Analyzing class discussion forums with author-sensitive topic modeling

Lauren Berk, Massachusetts Institute of Technology

ABSTRACT

We extended the generative Latent Dirichlet Allocation framework for topic modelling to include distinct author priors and allow for multiple authors to collaborate on a text. Revised inference methods were designed to handle the additional complexity and larger parameter space.  Extending LDA in this way not only gives it more modeling power, but leads to insights about author interests, allowing for more accurate inference and the personalization of recommendations in online systems.  We applied this analysis to forum data from two MOOCs at MIT to understand the evolution of conversation topics in the courses and evaluate student engagment and learning outcomes.  We compared this analysis to hand-coded topic analysis to understand the benefits and drawbacks of the automated approach.

A look ahead

The motivating problem - increasing collaboration in MOOC discussion forums

The method of attack - applied topic modeling

The theoretical context - latent Dirichlet allocation and an extension for author-specific priors

Theory to computation - performing LDA in Wolfram Language

Computation to application - Simulated and anticipated results

Extensions and areas for further work

A quick disclaimer - I am first and foremost a math graduate student, studying the theoretical properties and guarantees of machine learning algorithms.  This presentation will be more words and math than code.

The motivating problem

Growth of Massive Open Online Courses (MOOCs) has exploded in the last decade.  The largest providers of MOOCs - Coursera, Udacity, and edX - have a combined 50 million users.  And yet, MOOCs have failed to live up to their promise of providing inexpensive education to all of these members.  Participation in courses declides steadily week by week, completion rates between 5% and 15% are the norm, and students with wearker educational backgrounds are at even higher risk, showing completion rates as low as 3%.

Community forums make up a common and impactful element of modern MOOCs. The forums restore the collaborative element to dis- tance learning that is typically absent. On these forums, students can ask questions, discuss projects, and submit short assignments for peer viewing and revision.  Studies have shown that interpersonal interaction and peer encouragement can go a long way to improving course retention and completion rates.

While these interactions are important to student learning, these forums can be difficult for students to navigate, leaving them with the impression that the forums are not relevant to them, and discouraging them from contributing to the conversa- tion and learning from others.

A case study - two MIT online courses

Our research focused on two MOOCs run out of MIT.  

The first was Launching Innovation in Schools, a course for educators which provided us with data from its 2017 and 2018 runs.  

The second course us still being prepared, and will introduct students to community-centric engineering projects to provide personzlied aid to people with disabilities.  

In both courses, communication is a key skill and learning outcome, and so the forum components of these courses are especially important.

Isolating the problem of thread naming

One trend we identied in student behavior was that students create new “threads” or discussion topics when they need to post on the forum for a class assignment, and the titles they give to these threads give little insight into the posts’ contents.

Below are two responses to the assignment asking students to define what makes a powerful learning environment

Neither of these posts received responses from other students.  When a student logs in and sees hundreds of posts that are all titled “Assignment 1.1”, they might get overwhelmed, and even if they wanted to engage with other students, the prospect of clicking into and reading every post to decide what might interest them can be prohibitive.  

Question - how might you rewrite the titles for these posts to be more helpful?

The method of attack - applied topic modeling

Topic modeling is a branch of natural langauge processing that focuses on automated inference of the theme and context of individual documents as a part of a larger corpus.  

One example might be - if you were given a pile of articles cut out of newpapers, separated from their section headings, could you figure out what section each article belongs to?

Online news websites add an additional complexity to the problem, since articles can be listed under multiple topics.  An article about taking a knee for the pledge of allegiance at football games might fall under both sports and politics, or maybe culture.

A key aspect to the problem is that you don’t know the categories in advance - you need to infer what sections the newspaper has from the articles themselves.

Topic modeling and discussion forums

The goal of topic modeling can be put this way: Given a collection (corpus) of documents, discover which topics are being covered, the proportion of each document that comes from each topic, and what defines each topic (what words are markers of a topic)

How will that help us solve our original problem of MOOC forums?  If we can have the computer “read” each student’s post, understand what topics are being discussed across the form, and apply those topic labels back to the individual posts, we might get somethign close to what we imagined for new thread titles.  

More ambitiously, if we also infer student interests from what they post, we can begin to recommend discussions to students that might match their interest, or even connect peers together who share similar interests.

Theoretical context - generative models of text

Below we see a generative model of text using “plate notation”.  The rectangles or “plates” represent one of many objects at a given level of specificity.  I.e. there are many documents in the corpus, and many words in each document.  

The circles represent parameters and variables which are indexed at the level of the plate the object sits on.  Looking at this diagram, we see that the prior on topics and word distribution per topic are fixed at the corpus level, there is a document topic vector for each document, and there are topic and word variables for each document’s word.

An example of generative modeling of text

An example might go like this.  

Say you want to write a news article using this generative process.  You pick a topic vector according to the topic prior.  That topic vector might say your article should be 30% about sports and 70% about politics.  

Then to write your article, you step word by word through it.  Before you write each word, you flip a weighted coin that will give you sports 30% of the time and politics 70% of the time, which will tell you what topic that word should come from.  

Once you know that, say, the first word in your article should be about sports, you look up how words tend to be distributed in sports articles, and pick a word according to that distribution.  Your first word might be “Baseball”.

Continuing this way, you might get a document like:

Baseball senator the is win election election these did she ballot national...

Of course, this approach will result in grammatical nonsense, and no one actually generates text this way.  What is important is this this framework gives us, explicitly, what the probability of getting specific documents is given the topic and word distributions.  For example, we would be able to tell that the sentence

Tomorrow's mayoral election will coincide with the final game of the world series.

has high likelihood in the model.  And with these probabilities, we can compute the likelihood of the corpus as a function of the parameters, and tune the parameters to maximize this likelihood, which will tell us what topics exist in the corpus.

Getting specific - Latent Dirichlet Allocation

Now that we have an expression for the likelihood of a document as a function of parameters, we can ask - given a corpus of documents, what parameters (topics) would make the documents we see most likely to have been generated?

LDA with author-specific prior distributions

Think of three prominent researchers in your field or related fields.  If you found out the three of them were co-authoring a paper, you could probably guess a lot about the contents of the paper without seeing a single word of it.

One novel contribution of this work is allowing authors to have personalzied prior distributions over topics that reflect their individual interests.  This gives more degrees of freedom to the corpus topic prior while providing insight into individual author interests.

Theory to computation - inferring model parameters

The natural first approach to an inference problem like this is to try expectation maximization, but due to some nasty variable coupling, a direct approach like this is intractble.

A common way to deal with this problem is to use variational inference.  The main idea is to define some distributions over the latent variables in the problem (in this case, the hidden topic assignments for each individual word).  

In the case of LDA, we use “z” for the true word-topic assignments and “q” for the variational parameters.  Then we alternate between adjusting the distributions for “q” and adjusting the corpus-wide parameters for topic prior distribution and the word-given-a-topic distirbutions.

After a few rounds of approximations and bounding, we can get the alternating update equations:



Where gamma represents the estimate of each document’s topic vector.  To account for author specific priors, we adjust these using:

Performing LDA inference in Wolfram Language

The code is for the most part a straightforward translation from the math, but there were a few places where the uniue characteristics of Wolfram Langauge provided in advantage.

One key was in representing the data in key-value pairs rather than a matrix of individual counts.  In almost all natural language processing application, one of the greatest challenges is handling the large size of the dictionary, or number of possible words that might occur in a corpus.  Keeping counts in a matrix indexed by this dictionary takes up a ton of memory and computing power, but is still the way most statistical software likes to run.

Instead, in WL, I could transform a whole corpus of documents into the exact form I needed in one line:

niw = Lookup[(corpus[[#]] // Flatten // Counts ), dictionary, 0]& /@ Range[numDocuments];

The ability to use natural math expressions and mapping made it much more natural to translate mathematical expressions like “sum over” and “for all” into readable code.  This is how we go about computing an inverse Hessian:

partialdocTerm[d_, a_,t_]:=lambda[[d,a]]*
			(diGamma[Total[docAlphas[[d]]]]-
							diGamma[docAlphas[[d,t]]]+ ss[d,t]); 
partialTerm[a_, t_]:= Sum[partialdocTerm[d, a,t],{d,numDocuments}];
partials = Map[Function[a,Map[Function[t,partialTerm[a,t]],
			Range[numTopics]]],Range[numAuthors]];

c0 = Total[lambda^2]*triGamma[Total[Transpose[alpha]]] //N; 
ct = -Total[lambda^2]*triGamma[alpha]; 
b=((Sum[1/ct[[#,j]]*partials[[#,j]], {j,numTopics}])/(1/c0[[#]] + Total[1/ct[[#]]]))&/@ Range[numAuthors];
HinvDL = (partials - b) / (ct);

Computation to application - Simulated and anticipated results

Early results from simulations on author-specific prior generated docuemnts

We cannot measure model performance on real data using “accuracy”, since real data does not come with ground truth topic labels.  Instead, there are a number of methods in the literature for evaluating topic quality.  

One of these is “perplexity” which measures the distance of the word counts in the corpus from the expected words given the estimated parameters.  This is somewhat akin to the Chi-squared statistic used for measuring distributional fit for categorical variables.  



In our case, we are more interested in the ability for these models to improve student experiences and outcomes than we are interested in abstract properties.  This is why we are planning something new for our experiments this fall.  We will bring in human coders with experience in coding text for social science experiments to read over portions of the forum and use their intuition to decide what topics best describe the spectrum of conversations on the forum.  We will then qualitatively look at the similarities and differences between the expert-designed topics and LDA-generated topics.  We expect the LDA-generated topics might be a bit noisier and have some overlap, but might also identify emergent topics that escaped the experts unnoticed.

Extensions and further work

Incorporating entire forum discussions in topic modeling

Other related areas of work:

Sparse PCA for non-gaussian data as a topic modeling method

Compressed sensing for word embeddings

Hierarchical topic modeling

Robust topic modeling