Question Object Updates

Math Assessment, LLMs and More

ImageSynthesize["Realistic photo of a robot teacher grading a math test"]

Bob Sandheinrich

Question Framework Recap

AssessmentFunction

Computational tool for assessing answers

AssessmentFunction contains the answer key, but not the question. Here is an assessment function where 42 is the correct answer:

assessfunc=AssessmentFunction[{42->True,41->False,40->False}]

Check an answer:

assessment=assessfunc[42]

The assessment contains several pieces of information:

assessment[All]

Here is a more interesting assessment function:

nearcity=AssessmentFunction[{(Entity["New York City", Entity["City",{"NewYork","NewYork","UnitedStates"}], "Entity[\"City\", {\"NewYork\", \"NewYork\", \"UnitedStates\"}]", "city"])->10,(Entity["Baltimore", Entity["City",{"Baltimore","Maryland","UnitedStates"}], "Entity[\"City\", {\"Baltimore\", \"Maryland\", \"UnitedStates\"}]", "city"])->-2,(Entity["Boston", Entity["City",{"Boston","Massachusetts","UnitedStates"}], "Entity[\"City\", {\"Boston\", \"Massachusetts\", \"UnitedStates\"}]", "city"])->-5},"GeoPosition",Tolerance->(Quantity[200, "mi", miles, "Miles"])]

Try some answers:

nearcity[(Entity["New Haven", Entity["City",{"NewHaven","Connecticut","UnitedStates"}], "Entity[\"City\", {\"NewHaven\", \"Connecticut\", \"UnitedStates\"}]", "city"])]

nearcity[(Entity["District of Columbia, United States", Entity["AdministrativeDivision",{"DistrictOfColumbia","UnitedStates"}], "Entity[\"AdministrativeDivision\", {\"DistrictOfColumbia\", \"UnitedStates\"}]", "administrative division"])]

QuestionObject

Symbolic object representing a whole question including the interface and assessment

QuestionObject["Who won the 2023 World Series?",
AssessmentFunction[{"Braves"->-1,"Rangers"->2,"Diamondbacks"->0,"Cardinals"->-1}]]

QuestionInterface

Tool for customizing interfaces of a QuestionObject

Here is a question where inputs are treated as locations (lat/long) and including an image and custom styles in the interface.

QuestionObject[
QuestionInterface["ShortAnswer",<|
"Interpreter"->"Location",
"Prompt"->Column[{Style["Where might a Yankees fan live?","Subsubtitle"],Image[,ImageSize->Large]}]|>
],
nearcity
]

The same assessment function can be used in a different question:

labels=GeoGraphics[GeoMarker[#],ImageSize->Small]&/@{(Entity["New York City", Entity["City",{"NewYork","NewYork","UnitedStates"}], "Entity[\"City\", {\"NewYork\", \"NewYork\", \"UnitedStates\"}]", "city"]),(Entity["Baltimore", Entity["City",{"Baltimore","Maryland","UnitedStates"}], "Entity[\"City\", {\"Baltimore\", \"Maryland\", \"UnitedStates\"}]", "city"]),(Entity["Boston", Entity["City",{"Boston","Massachusetts","UnitedStates"}], "Entity[\"City\", {\"Boston\", \"Massachusetts\", \"UnitedStates\"}]", "city"])}

QuestionObject[
QuestionInterface["MultipleChoice",<|
"Labels"->labels,
"Prompt"->Row[{," Where can I find great bagels? ",}]|>
],
nearcity
]

QuestionGenerator

Function for generating a randomized QuestionObject

A simple example

simpleggenerator=QuestionGenerator[{"a":>RandomInteger[10],"b":>RandomInteger[20]},
QuestionObject[
StringForm["What is `` * `` ?",#a,#b],
AssessmentFunction[#a*#b]
]&]

simpleggenerator[]

A more complicated example

Helper functions

makeAssessmentRules[stagea_,stageb_]:=Append[
(#-><|"Score"->1,
(* CORRECT: INCLUDE AN EXPLANATION FOR THE USER *)
"Explanation"->Column[{Style["CORRECT!",18],StringForm["`` evolves into ``",stagea["Name"],#["Name"]],#["Image"]},Alignment->Center]|>)&/@Echo[stageb,"This is for cheating during the demo"],
_->incorrectAnswer[stagea]
]

incorrectAnswer[stagea_]:=<|"Score"->0,
(* INCORRECT: INCLUDE AN EXPLANATION FOR THE USER *)"Explanation"->Column[Join[{Style["Incorrect",18],StringForm["`` evolves into ``",stagea["Name"],StringRiffle[stagea["EvolvesInto"],"\n"]]}],]|>

randomPokemon[]:=RandomEntity@FilteredEntityClass["Pokemon" , EntityFunction[x, Not@MissingQ[x["EvolvesInto"]]]]

pokemonPrompt[stagea_]:=Column[{
StringForm["Who does `` evolve into?",stagea["Name"]],
Row[{Image[stagea["Image"],],Style["→ ?",36]}]
},]

Question Generator

evolutiongen=QuestionGenerator[{"pokemon":>randomPokemon[]},
QuestionObject[
QuestionInterface["ShortAnswer",
<|"Prompt"->pokemonPrompt[#pokemon],
"Interpreter"->"Pokemon"|>
],
AssessmentFunction[
makeAssessmentRules[#pokemon,#pokemon["EvolvesInto"]]
]
]&
]

Demo

evolutiongen[]

QuestionDeploy Function Repository

Any of the above objects can be deployed to Wolfram Cloud to secure the answer key. This happens automatically when using a Question Notebook. See 2022 Presentation

Math Assessment

Comparing mathematical expressions during assessment is complicated!

These questions have mathematically equivalent answers, but should accept different sets of answers.

QuestionObject[StringForm["What is the derivative of ``?",3 (x^3)+2 (x^2)],
AssessmentFunction[Hold[4 x+9 (x^2)],"CalculusResult" ]]

QuestionObject[StringForm["Expand the polynomial: ``",x(4 +9 x)],
AssessmentFunction[Hold[4 x+9 (x^2)] ,"PolynomialResult"]]

Existing Math Assessment Methods

These three comparison methods are built in and available

CalculusResult

Checks if the answer is equal to the solution after allowing most code to evaluate.

QuestionObject[StringForm["What is the derivative of ``?",3 (x^3)+2 (x^2)],
AssessmentFunction[Hold[4 x+9 (x^2)],"CalculusResult" ]]

PolynomialResult

Checks that the answer has the same factorization as the solution.

QuestionObject[StringForm["Expand the polynomial: ``",x(4 +9 x)],
AssessmentFunction[Hold[4 x+9 (x^2)] ,"PolynomialResult"]]

ArithmeticResult

Allows no evaluation and only very basic equivalences.

QuestionObject[
StringForm["How much is ``? \n``",HoldForm[🍏+2*🍊-🍔+🍏-🍊+🍔],],
AssessmentFunction[🍊+2 🍏,"ArithmeticResult"]]

Coming soon: LLM Based Assessment

There are many ways LLMs are impacting our educational technology.
This is one simple case:

Let the LLM decide if the answer is correct and explain why:

QuestionObject[
"What is an elephant?",
AssessmentFunction["What is an elephant?","LLMQuestionAssess"]
]

Coming soon: New Question Notebook

Button[Style[ResourceFunction["RainbowText"]@"Demo New Question Notebook",32],QuestionFramework`NotebookIntegration`QuestionNotebookDefinitionNotebook`CreateQuestionNotebook[]]

More information

Hyperlink["Documentation","https://reference.wolfram.com/language/guide/QuestionsAndAssessment.html"]

Hyperlink["From Assessments to Applications (2022 WTC Video)","https://www.youtube.com/watch?v=1Tufjv-4OmI"]