Question Framework and Applications to Computational Thinking

Bob Sandheinrich, Xiaofan Zhang, Chistophe Pakleza

#WolframTechConf

ABSTRACT

This talk introduces new features in the Wolfram Language’s question framework, emphasizing improved question customization, including LLM evaluation in AssessmentFunction and support for EpilogFunction in QuestionObject. Through illustrative examples, we demonstrate how QuestionObject can significantly enhance computational thinking and facilitate interactive learning experiences on the CBM (Computer Based Math) Teaching Platform.

Agenda

Updates in Question Framework

Customizations

LLM Integrations

Application on the CBM Teaching Platform

Example use cases

Real-life implementation

QuestionObject Customizations

EpilogFunction

Give code to run when an assessment is completed.

QuestionObject[interface,assessment, EpilogFunction->efunc]

For example, Echo the results from within a QuestionObject:

QuestionObject[QuestionInterface["MultipleChoice",<|"Prompt"->"What is 4^3?"|>],
AssessmentFunction[{16, -64, 64->1}], EpilogFunction->(Echo[#Assessment,"assessmentresults"]&)]

Or make a playful celebration

celebrate[assessment_]:=If[assessment["AnswerCorrect"],
{Directive[Thick, Green],ResourceFunction[]["Celebration"]},
{Directive[Dashed, Red],ResourceFunction[]["Dissappointed"]},
{White,""}]

DynamicModule[{graphics="",framestyle=White},
Grid[{
{
QuestionObject[QuestionInterface["MultipleChoice",<|
"Prompt"->StringForm["What is ``?",HoldForm[Integrate[Cos[x],x]]],
"ShowAssessment"->True
|>],
AssessmentFunction[{HoldForm[Cos[x]^2]/2, HoldForm[Sin[x]]->1, HoldForm[1/x]}], 

EpilogFunction->Function[{framestyle,graphics}=celebrate[#Assessment]]],

Dynamic@graphics}
}, Frame->True, FrameStyle->framestyle]
]

Styles

We’ve added support for many more formatting and styling options. Now I can make ugly questions:

QuestionObject[QuestionInterface["TextCompletion",<|"Prompt"->Style["Complete the Mad Libs", "Section"],"Template"->"I opened a drawer and found `` ``.\nI called `` because I needed a `` expert on ``.",

BaseStyle->Directive[FontFamily->"Courier"],
FieldSize->{3,{10}},FieldHint->{"Number","Plural noun","Famous person","Adjective","Expertise"},FieldHintStyle->{Directive[Red, 8], Directive[Red, 28],{Underlined}},
Background->{{Blue},1->Cyan,3->Yellow,5->Red}|>
],ImageSize->{300,600}]

AssessmentFunction Customizations

ScoringFunction

Instead of assigning specific, fixed scores to each value in the answer key, define arbitrary code to determine to score.

AssessmentFunction[key,<|"ScoringFunction"->sfunc|>]

For example create score based on the distance from the correct answer:

ktest=AssessmentFunction[{QuantityMagnitude[(Quantity[100, "°C", degrees Celsius, "DegreesCelsius"]),"Kelvins"]},<|"ComparisonMethod"->"Number","ScoringFunction"->(5.-Abs[QuantityMagnitude[(Quantity[100, "°C", degrees Celsius, "DegreesCelsius"]),"Kelvins"]-#GivenAnswer]&)|>,Tolerance->1]

ktest[373][All]

Use it in a question object:

QuestionObject["What is the boiling point in Kelvin?",ktest]

ExplanationFunction

Use code to generate explanations.

AssessmentFunction[key,<|"ExplanationFunction"->sfunc|>]

For example, choose an explanation:

QuestionObject[QuestionInterface["ShortAnswer",<|"Prompt"->"Give an anagram of 'skate'"|>],
AssessmentFunction[ResourceFunction["Anagrams"]["skate"],

<|IgnoreCase->True,"ExplanationFunction"->(Apply[StringTemplate["The mismatched letters are `` and ``"]][UniqueElements[{Characters@#GivenAnswer,Characters@"skate"}]]&)|>]]

LLM Integration

LLM Assessment

This prompt is predefined to ask an LLM to determine if an answer matches the answer key.

LLMPrompt["QuestionAnswerAssess"]

Use it to determine the correctness:

heroassess=AssessmentFunction[{
"A real or fictional person widely considered to be a hero"
},LLMResourceFunction["QuestionAnswerAssess"]]

assessment=heroassess["Nelson Mandela"]

This (currently undocumented) comparison method name uses the same prompt internally:

heroassess=AssessmentFunction[{
"A real or fictional person widely considered to be a hero"
},"LLMQuestionAssess"]

assessment=heroassess["Sauron"]

This way, it also provides an LLM-generated explanation:

assessment[{"AnswerCorrect","Explanation"}]

See it in a QuestionObject:

QuestionObject[QuestionInterface["ShortAnswer",\[LeftAssociation]"Prompt"->"Name a hero"\[RightAssociation]],
heroassess]

LLM Explanation

Don’t trust an LLM to assess. 
Use AssessmentFunction’s sophisticated automatic comparisons to determine correctness, but use an LLM to explain why it is correct or not.

LLMPrompt["QuestionAssessmentExplanation"]

QuestionObject[QuestionInterface["ShortAnswer","Expand (x+1)(x-1)"],AssessmentFunction[{Hold[x^2-1]->1,Hold[x^2+1]->-1},<|"ComparisonMethod"->"PolynomialResult","ExplanationFunction"->LLMResourceFunction["QuestionAssessmentExplanation"]|>]]

Putting it all together

Make an API for recording grades (the cheapest LMS on the planet):

gradebookAPIfunction=APIFunction[{"Score"->"Number","AnswerCorrect"->"Boolean","GivenAnswer"->"String","Explanation"->"String"->None},
PutAppend[{Now,$RequesterCloudUserID}->#,CloudObject["StudentResults"]]&
]

gradebookAPI=CloudDeploy[gradebookAPIfunction,"gradebook",Permissions->{"Authenticated"->"Execute"}]

Create a AssessmentFunction that takes a date and awards points based on the distance to a secret date.

dateassess=AssessmentFunction[{},
<|"ComparisonMethod"->"Date","ScoringFunction"->(QuantityMagnitude[(Quantity[4, "yr", years, "Years"])-Abs@DateDifference[#GivenAnswer,],"Years"]&),

"ExplanationFunction"->LLMResourceFunction["QuestionAssessmentExplanation"]|>,Tolerance->(Quantity[2, "yr", year, "Years"])]

Create a QuestionObject with the assessment, using custom styles and an EpilogFunction that reports the result to the gradebook API:

wtcquestion=QuestionObject[QuestionInterface["ShortAnswer",<|"Prompt"->Style["When was the first Wolfram Technology Conference?","Subsection",Bold],"Interpreter"->"Date",BaseStyle->Directive[Purple,Italic]|>],
dateassess,
EpilogFunction->(URLExecute[CloudObject[(HyperlinkURL["https://www.wolframcloud.com/obj/bobs/gradebook", https://www.wolframcloud.com/obj/bobs/gradebook])],TextString/@#Assessment[{"Score","AnswerCorrect","GivenAnswer"}]]&)]

Deploy the question to the cloud, allowing any logged in user to answer:

cloudobj=CloudDeploy[wtcquestion,"WTCQuestion",Permissions->{"Authenticated"->Automatic}]

Create a dynamically updating tracker for the results:

$assessments={};
task=SessionSubmit[ScheduledTask[$assessments=ReadList[CloudObject[(HyperlinkURL["https://www.wolframcloud.com/obj/bobs/StudentResults", https://www.wolframcloud.com/obj/bobs/StudentResults])]],10]]

Dynamic[Column@$assessments]

Everyone answer the question:

BarcodeImage[First[cloudobj]]

TaskRemove[task]

Application of Question Framework on
the CBM (Computer Based Math) Teaching Platform

Example use cases

Constrain input field values for “TextCompletion” questions

Each field has constraints: it has to be a number, and it has to conform to a specific range.
The assessment function checks:
1. if all fields have an input.
2. if the inputs are numbers.
3. if the inputs fall into a their respective ranges.

Customized LLM prompt for assessment

LLMScoringPrompt = "Your task is to score the answer I have given to a question. 
You must return only valid JSON with the keys Score and Explanation 
where Score is from 0 to `MaxScore`. 
The Explanation contains up to two sentences of feedback about what I should think about to improve my answer. Do NOT mention there is an answer list. Do NOT provide any correct answer.
The question is '`Question`'.\nScore my answer according to the following answer rubrics:\n`Key`.\nMy answer is: '`Answer`'. You can give partial points if my answer is partially correct.
";

assocCheck[assoc_, max_]:=
If[AssociationQ[assoc],
<|"Score"->Lookup[assoc,"Score",0],"Explanation"->InsertLinebreaks[StringJoin[Lookup[assoc,"Explanation",""]," ","(",ToString[Lookup[assoc,"Score",0]],"/",ToString[max],")"],60],
"AnswerCorrect"->(Lookup[assoc,"Score",0]>0),
"MaxScore"->max|>,
$Failed];

postProcessResult[result_, max_]:=Module[{post},
If[Not[StringQ[result]],
post = <|"Score"->0,"Explanation"->"Please retry.","AnswerCorrect"->False|>,
post = Quiet[ImportString[StringReplace[result,___~~"{"~~Longest[x___]~~"}"~~___:>StringJoin["{",x,"}"]],"RawJSON"]]];
assocCheck[post, max]
];

makeLLMAssessment[question_,key_]:=
With[{max=Max[key[[All,2]]] },
AssessmentFunction[Automatic,
Block[
{result=Quiet@LLMSynthesize[
TemplateApply[
LLMScoringPrompt,
<|"MaxScore"->ToString[max],
"Question"->ToString[question],"Key"->StringReplace[ToString[key,InputForm],{"\""->"'","{"->"","}"->""}],
"Answer"->ToString[#]
|>
],
LLMEvaluator-><|"Model"->{"OpenAI","gpt-4"}|>]
},
postProcessResult[result, max]
]&]];

bear={"American black bear, Andean bear, Asiatic black bear, brown bear, giant panda, sloth bear, sun bear, polar bear"};makeLLMAssessment["Name three types of bear.",{"Any THREE correct types or synonyms from: "<>bear->3,"Any TWO correct types or synonyms from: "<>bear->2, "Any ONE correct type or synonyms from: "<>bear->1,"All given types are not in the list: "<>bear->0 }]

AssessmentFunction[]["panda and koala"]

AssessmentResultObject[][All]

Real-life implementation

Live demonstration of the application in a CBM module deployed in the Wolfram Cloud
answer question -> assess -> send answer and score to server -> retrieve data from server

Thank you