NLP for Automated Grading of Mathematical Word Problems

Maik Meusel | University of Zurich | maik.meusel@business.uzh.ch
Maria Meier | University of Zurich | maria.meier4@uzh.ch

Join the Conversation #WolframTechConf

ABSTRACT

Educators spend a significant amount of time preparing and grading assignments and exams. With the introduction of online assessments in recent years, new possibilities for efficiency improvement in education have arisen. SYLVA uses Wolfram technologies to provide a fully-integrated platform allowing educators to create assessments that can be graded automatically. To increase the accessibility for teachers with little or no programming knowledge we use NLP and GrammarRules to derive computable conditions for mathematical text questions. We analyze a dataset with arithmetic word problems. We show how we built a foundation for creating GrammarRules automatically and discuss extension possibilities in order to create a sophisticated system for grading mathematical word problems.

Do you like manual grading?

Manually grading assessments is:

1.	Tedious and boring

2.	Labor-intense (expensive)

3.	Slow and prone to human error

How to translate any question or task into an auto-gradable problem:

Find an instance of entity type, such that conditions (Subscript[c, i]) hold.

Example (CREO for Mathematica)

What do we want to achieve?

1) Remove the coding barrier for educators

2) Increase the grading accuracy compared to purely statistical methods

Example Math Word Problem

Imagine an elementary school teacher gives her students this problem:

What the teacher wants: different names, numbers and objects for each student (to prevent cheating)

What the teacher doesn’t want: to calculate each solution and grade each student’s answer; write code

What needs to be automated

1.	Parse any given problem stated in plain English

2.	Derive conditions and model solution

3.	Evaluate student’s answer (SYLVA)

Approach

Dataset: Aggregate

1492 arithmetic word problems

Overview of Question-Types: 					({{Addition, 460}, {Subtraction, 355}, {Multiplication, 179}, {Division, 180}, {Combination, 318}})

713 different Text Structures based on comparison by Distance Matrices*

1318 different Text Structures based on POS-Tagging sequences

Example

datasetConf [[1]]

datasetConf[All,"FullText"][[1]]

1. Allocation to an Operator Class

a. By using Keywords

Addition: 		sum, total, altogether, combined etc.

Subtraction: 		difference, less, left, fewer etc.

Multiplication:		times, if each,  etc.

Division: 			average, per, divided, evenly etc.

b. By using Classify-Function

2. Creation of GrammarRules

A simple GrammarRule:

gr=GrammarRules[{FixedOrder[x:GrammarToken["GivenName"],y:GrammarToken["City"]]:>x->y}];
gr1=CloudDeploy[gr];
GrammarApply[gr1,"Michael Zurich"]

(Entity["Michael", Entity["GivenName",{"Michael","UnitedStates","male"}], "Entity[\"GivenName\", {\"Michael\", \"UnitedStates\", \"male\"}]", "given name"])->(Entity["Zurich", Entity["City",{"Zurich","Zurich","Switzerland"}], "Entity[\"City\", {\"Zurich\", \"Zurich\", \"Switzerland\"}]", "city"])

Steps:

Grid[{{Text[Style["Steps to create a GrammarRule",Bold]]},
{"1. POS-Tagging"},
{"2. Create GrammarTokens"},
{"3. Create left-hand part of GrammarRule"},
{"4. Create right-hand part of GrammarRule"},
{"5. Create full GrammarRule"},
{"6. Deploy to Cloud"}},Frame->All,Spacings->2,Alignment->Left]

a. Functions to Create GrammarRules

correctlyAllocatedAdditionsConf[All,"AllSentencesWithNumericalInfo"][[2]]

1. 	POS-Tagging:

TextStructure[correctlyAllocatedAdditionsConf[All,"AllSentencesWithNumericalInfo"][[2]],"PartsOfSpeech"]

2.&3.  Creation of left-hand part of GrammarRules

LeftHandSideOfGrammarRule[2,2,correctlyAllocatedAdditionsConf]

4.  Creation of right-hand part of GrammarRules

RightHandSideOfGrammarRule[LeftHandSideOfGrammarRule[2,2,correctlyAllocatedAdditionsConf]]

5.  Creation of full GrammarRules

CreateGrammarRule[2,2,correctlyAllocatedAdditionsConf]

6.  Deploy Grammar to the Cloud

grammar=CloudDeploy[CreateGrammarRule[2,2,correctlyAllocatedAdditionsConf][[1]]]

Compare derived solution with correct solution from dataset:

derivedSolution=GrammarApply[grammar,"Tom has 9.0 yellow balloons. Sara has 8.0 yellow balloons ."]

derivedSolution==correctlyAllocatedAdditionsConf[All,"Solution"][[2]][[1]]

GrammarApply[grammar,"Sam has 18.0 red apples. Tony has 8.0 red apples ."]

2. Creation of GrammarRules

b. Application to several examples

Creation of CloudObjects

Compare Grammar’s solution with correct solution

3. Analysis of False & $Failed

Reason why the derived solution is false (→ “4.0” not recognized as numeral):

TextStructure["Joan went to 4.0 football games this year . She went to 9.0 games last year .","PartsOfSpeech"]

GrammarApply[CloudObject[(HyperlinkURL["https://www.wolframcloud.com/obj/f7efc188-5bfa-4327-81e0-8f4e07265239", https://www.wolframcloud.com/obj/f7efc188-5bfa-4327-81e0-8f4e07265239])],
"Joan went to 4.0 football games this year . She went to 9.0 games last year ."]

TextStructure["Joan went to 4 football games this year . She went to 9.0 games last year .","PartsOfSpeech"]

GrammarApply[CloudObject[(HyperlinkURL["https://www.wolframcloud.com/obj/89096a04-e5d1-4105-bbad-13e026dba7df", https://www.wolframcloud.com/obj/89096a04-e5d1-4105-bbad-13e026dba7df])],
"Joan went to 4.0 football games this year . She went to 9.0 games last year ."]

Model Improvement:

Another reason why the derived solution is false (→ irrelevant number):

Reasons why GrammarApply failed:

Technical issues:

https://mathematica.stackexchange.com/questions/208060/error-message-for-grammarrules-and grammarapply

ProperNoun = GrammarToken[GivenName] at the moment!

Recap

1. Allocation to Operator Type
2. Automated Creation of GrammarRules
3. Analysis of False and $Failed
4. Improve Model

Source

Dataset:

Source: https://github.com/CogComp/arithmetic/blob/master/data/questions.json
Subhro Roy and Dan Roth.  
Mapping to Declarative Knowledge for Word Problem Solving.  
TACL 2018.

*https://www.wolfram.com/language/11/text-and-language-processing/compare-the-structure-of-sentences.html?product=language