The Question Framework: from Assessments to Applications

Bob Sandheinrich
2022 Wolfram Technology Conference

What

A collection of tools for creating and distributing questions with automated assessment typically for quizzes.

Why

Automated assessment of answers is a computational challenge. Wolfram’s technology is well suited for the challenge. 

We also now have the accompanying tools necessary for making this technology accessible to those who need it.

An example

Hyperlink["https://wolfr.am/wtcquiz"]

BarcodeImage["https://www.wolframcloud.com/obj/bobs/Quizzes/WTC2022-sample/quizapi","QR"]

Major Pieces

AssessmentFunction

QuestionObject

QuestionSelector and QuestionGenerator

Question Notebooks

AssessmentFunction

AssessmentFunction[key][answer]

Simple

The correct answer is 100, the student gives 100

AssessmentFunction[100][100]

The student gives 1

AssessmentFunction[100][1]

More Complicated

Use an “Interval” comparison method and include explanations and custom scores

pujolsHR=AssessmentFunction[{Interval[{0,500}]-><|"AnswerCorrect"->False,"Score"->-5,"Explanation"->"500 home runs is ordinary for a hall of fame slugger, but Pujols is extraordinary"|>,
Interval[{501,660}]-><|"AnswerCorrect"->False,"Score"->-3,"Explanation"->"Albert has more home runs than Willie Mays, who had 660"|>,
Interval[{661,696}]-><|"AnswerCorrect"->False,"Score"->-3,"Explanation"->"Albert has more home runs than Alex Rodriguez, who had 697"|>,
Interval[{703,703}]-><|"AnswerCorrect"->True,"Score"->10,"Explanation"->"Yes Albert hit 703 home runs"|>,
Interval[{700,705}]-><|"AnswerCorrect"->True,"Score"->1,"Explanation"->"Close enough."|>,
Interval[{706,Infinity}]-><|"AnswerCorrect"->False,"Score"->-5,"Explanation"->"Albert has not hit that many, yet"|>},"Interval"]

assessment=pujolsHR[703]

assessment[All]

Fully Customized

An assessment function for a question like “Name a city with a population similar to St. Louis, Missouri” that awards five points for a correct answer and deducts one point for an incorrect answer:

populationNearSTL=AssessmentFunction[{(Entity["Saint Louis", Entity["City",{"SaintLouis","Missouri","UnitedStates"}], "Entity[\"City\", {\"SaintLouis\", \"Missouri\", \"UnitedStates\"}]", "city"])->5,(Entity["Chicago", Entity["City",{"Chicago","Illinois","UnitedStates"}], "Entity[\"City\", {\"Chicago\", \"Illinois\", \"UnitedStates\"}]", "city"])->0,(Entity["Indianapolis", Entity["City",{"Indianapolis","Indiana","UnitedStates"}], "Entity[\"City\", {\"Indianapolis\", \"Indiana\", \"UnitedStates\"}]", "city"])->0,(Entity["Des Moines", Entity["City",{"DesMoines","Iowa","UnitedStates"}], "Entity[\"City\", {\"DesMoines\", \"Iowa\", \"UnitedStates\"}]", "city"])->0},<|"Selector"->(First@TakeSmallestBy[#1,Function[city,Abs[#2["Population"]-city["Population"]]],1]&)|>]

populationNearSTL/@{(Entity["Detroit", Entity["City",{"Detroit","Michigan","UnitedStates"}], "Entity[\"City\", {\"Detroit\", \"Michigan\", \"UnitedStates\"}]", "city"]),(Entity["Louisville", Entity["City",{"Louisville","Kentucky","UnitedStates"}], "Entity[\"City\", {\"Louisville\", \"Kentucky\", \"UnitedStates\"}]", "city"]),(Entity["Pittsburgh", Entity["City",{"Pittsburgh","Pennsylvania","UnitedStates"}], "Entity[\"City\", {\"Pittsburgh\", \"Pennsylvania\", \"UnitedStates\"}]", "city"])}

An assessment function that awards partial credit for each prime number given and considers the full answer to be correct if most values are prime.

customPartialCredit=AssessmentFunction[{_?PrimeQ->1},<|"ElementwiseAssessment"->True,"ScoreCombinerFunction"->(Total[#]/Length[#]&),"AnswerCorrectCombinerFunction"->First@*Commonest|>]

assessment=customPartialCredit[{5,11,15,20,40}]

assessment["Score"]

QuestionObject

QuestionObject[interface, assessmentfunction]

Simple

q1=QuestionObject["What is 50 + 50?",EqualTo[100]]

More Complicated

q2=QuestionObject[QuestionInterface["NumericRange",<|"Range"->{500,800,1},"Prompt"->"How many career home runs has Albert Pujols hit?"|>],
pujolsHR]

Fully Customized

q3=QuestionObject[QuestionInterface["ShortAnswer",<|"Interpreter"->"City","Prompt"->"Name a city with a population similar to St. Louis, Missouri"|>],populationNearSTL]

q4=QuestionObject[QuestionInterface["ListMultiple",<|"Prompt"->"Name up to 10 prime numbers","FieldType"->Number|>],
customPartialCredit]

AssessmentResultObject[][All]

QuestionSelector

Serves a random, pre-defined question from a list of questions.

$questionbank={q1,q2,q3,q4};
Column[%]

selector=QuestionSelector[$questionbank]

selector[]

QuestionGenerator

Generates and serves a random question from a function.

Simple

addints=QuestionGenerator[{"a":>RandomInteger[100],"b":>RandomInteger[100]},
QuestionObject[StringTemplate["What is `a` + `b`?"][#]
,EqualTo[#a+#b]]&
]

addints[]

Complicated

cityquiz=QuestionGenerator[{"city":>RandomEntity["City"]},
QuestionObject[QuestionInterface["ShortAnswer",<|"Interpreter"->"City","Prompt"->StringTemplate["Name a city with a population similar to `city`"][#]|>],AssessmentFunction[Prepend[Thread[RandomEntity["City",10]->0],#city->1],<|"Selector"->(First@TakeSmallestBy[#1,Function[city,Abs[#2["Population"]-city["Population"]]],1]&)|>]]&
]

cityquiz[]

Deployment

The Wolfram Cloud provides a mechanism to both distribute and secure question content.

Any piece of content can be deployed to the cloud

ResourceFunction["QuestionDeploy"]

Assessment

deployedAF=[AssessmentFunction[{_?PrimeQ}],"DemoQuestionDeployments/Assessment1",Permissions->"Public"]

deployedAF[23]

QuestionObject

deployedQO=[QuestionObject["What is 50 + 50?",EqualTo[100]],"DemoQuestionDeployments/Question1",Permissions->"Public"]

deployedQO["QuestionObject"]//InputForm

deployedQO["QuestionObject"]

QuestionSelector

deployedQS=[QuestionSelector[$questionbank],"DemoQuestionDeployments/Selector1",Permissions->"Public"]

deployedQS["QuestionSelector"]

deployedQS["QuestionSelector"]//InputForm

selected=deployedQS["QuestionSelector"][]

selected//InputForm

QuestionGenerator

deployedQG=[cityquiz,"DemoQuestionDeployments/Generator1",Permissions->"Public"]

deployedQG["QuestionGenerator"]

deployedQG["QuestionGenerator"]//InputForm

generated=deployedQG["QuestionGenerator"][]

generated//InputForm

Get the results

ResourceFunction["GetQuestionAssessments"]

[deployedQG["CloudObject"],All,All]

Question Notebook

All the pieces above combine within a Question Notebook.

CreateNotebook["QuestionNotebook"]

NotebookOpen["/Users/bobs/Documents/Bob/WTCPresentations/2022/SampleQuiz-WTC.nb"]

Thanks

Bob Sandheinrich

Credits

Stephen Schroeder

Christophe Pakleza

Avery Davis

Sylvia Haas (former)