Computational Thinking, LLMs, and the Future of Education

Questions Summarized by LLM

Q&A Link: https://wolfr.am/1fGUkGIld

Teaching with LLMs

Co-Design Classroom Activities with AI Written Code

https://www.wolframcloud.com/obj/jmcnally0/Published/ExperimentDesignDemo.nb

What were the human inputs that make this example work?

An Initial Idea

“Help design...”

Instructions to Refine

“After discussing with students...”

“...introduce and independent and dependent variable.”

“...help students analyze and present data...”

Introduction of Key Concepts

“linear model”, “inelastic bounce”, “coefficient of restitution”

Corrections

“This looks correct until...”

Relevant Extensions

“... help a student interpret their model.”

Follow up with alternate model

Note the AI mentioned this before the human!

“New” Ways to Assess

While not really new assessment advice in general, generative AI provides an entirely new tool for focusing on strategy and problem formulation instead of fine details.

Check out this variation on a “Prompt Problem” (Denny et al, 2023) in which the student’s role is to describe the method of solution in sufficient detail that the LLM can implement those details:

A pendulum is released from rest at an initial angle of . Suppose that the ratio .

Two differential equations to model this system are given:

Running the first result given:

What were the human inputs that make this example work?

An Initial Idea

The student was only given a physical situation, two quantitative models, and their background knowledge from other classroom activities.

Then, the student framed a relevant problem implied by the set-up.

In practice, the instructions to the student should include scaffolding so that they can direct available tools at relevant problems.

Instructions to Refine the Output

Use particular functions as part of the solution.

“Remember to...”

Include legends and labels.

Introduction of Key Concepts

small angle approximation

numerical solutions

Relevant Extensions

comparison of models

The Key Point for Educators and Students:

What were the common ingredients to these LLM interactions?

Initial Idea / Question

Lean into this for student-driven exploration

Refine Details

Iterate as the “solutions” become better defined

Introduce Key Concepts

Context is everything to an LLM!

Students need content understanding to do this well!

Critique / Corrections

Engage in critical thinking to find LLM mistakes

Relevant Extension

More student-driven critical thinking, exploration, and learning

Some may worry LLMs and computational tools are a reason for students to not think about their work. However, providing the key ingredients above will require students to critically and creatively think.

More Advice on Assessment

Remember: every assessment needs criteria for evaluation

This helps with feedback, evaluation, and subsequent learning.

“Is it right?” is not a specific enough criterion (except for very particular formative assessments).

Aligning assessments to learning goals is a much older unsolved “alignment problem” than human and AI alignment.

What should I assess given new technology?

Project based assessments require design decisions and communication that “go beyond” prompting an LLM.

Simple, atomic problems are fine as formative assessment. Summative assessments should go beyond to see if students can apply and extend simple tasks.

Revising and peer assessment are still powerful classroom tools. LLMs can be treated like an always-available “peer” for generating content to revise and assess.

None of the advice about assessment is new! Relevant tools have expanded, but educators already have frameworks for thinking about their use.

Teaching about LLMs

Artificial neural networks and machine learning are already basically absent from “general” curricula, in spite of the fact that ML has long been a large part of the “behind the scenes” technology in our daily lives.

Since appropriate background knowledge is an important part of new learning, there is a need and opportunity to tell the “story” of these technologies.

Neural Networks and Classification Tasks

The topic “as such”:

LeNet=NetModel["LeNet Trained on MNIST Data"];
Information[ LeNet,"FullSummaryGraphic" ]

LeNet/@{,,,,,,,,,}

LeNet[,"TopProbabilities"]

visualizeLeNet[  ]

visualizeLeNet=;

Applied to other domains, in this case social studies:

titanicClassifier=Classify[ExampleData[{"MachineLearning","Titanic"},"Data"],Method->"LogisticRegression"]

With[
{categories=SortBy[Tuples[{{"1st","2nd","3rd"},{age},{"female","male"}}],Last]},
Plot[ Evaluate@Quiet@Map[titanicClassifier[#,{"Probability","survived"}]&,categories],{age,0,100},PlotLegends->MapApply[#3<>", "<>#1<>" class"&,categories],Frame->True,FrameLabel->{"Age (years)","Survival probability"},PlotLabel->"Titanic Estimated Survival Probabilities"]
]

LLM Concepts

Sampling and Temperature

If your students are learning about statistics and distributions, all the better. If not, you

gpt2=NetModel[{"GPT2 Transformer Trained on WebText Data","Task"->"LanguageModeling"}];

gpt2["The next word in this sentence"]

gpt2["The next word in this sentence",{"TopProbabilities",5}]//Column

Nest[StringJoin[#,gpt2[#]]&,"The next word in this sentence",50 ]

This example turns the temperature parameter up to 0.6

Nest[ StringJoin[ #,gpt2[#,{"RandomSample","Temperature"->0.6}] ]&,"The next word in this sentence",25]

Run the exact same thing again with the same temperature setting, and get a new result:

Nest[ StringJoin[ #,gpt2[#,{"RandomSample","Temperature"->0.6}] ]&,"The next word in this sentence",25]

Temperature of 1.0 leads to complete nonsense for this task:

Nest[ StringJoin[ #,gpt2[#,{"RandomSample","Temperature"->1.0}] ]&,"The next word in this sentence",50 ]

Tokenization

ResourceFunction["GPTTokenizer"]["The cat is on the mat."]

ResourceFunction["GPTTokenizer"]["The dog is on the mat."]

AssociationMap[ResourceFunction["GPTTokenizer"],{"cat","dog","Cat","Dog","finance"}]

Embeddings

Needs["ChristopherWolfram`OpenAILink`"]

Module[{cosdist=Function[{x,y},CosineDistance[OpenAIEmbedding[x],OpenAIEmbedding[y]]],pairs={{"cat","dog"},{"cat","Cat"},{"cat","Mars"},{"cat","lion"},{"cat","wolf"},{"cat","finance"}}},
BarChart[ParallelMap[Apply[cosdist],pairs],ChartLabels->MapApply[StringJoin[#1,"\n",#2]&,pairs],PlotLabel->Style["Cosine Distance of OpenAI Embeddings",Red,Bold]]]

publicdata=CloudGet["https://www.wolframcloud.com/obj/jmcnally0/ExampleEmbeddings"]

ListPlot[publicdata[All,All,Tooltip[#ReducedEmbedding,#ParagraphText]&]]

With[
{reducer=DimensionReduction[Flatten[Normal[publicdata[Values,All,"Embedding"]],1],3]},
ListPointPlot3D[publicdata[All,All,Tooltip[reducer[#Embedding],#ParagraphText]&],PlotLegends->Normal[publicdata[Keys]]]
]

Make it interdisciplinary:

Teach about prompting, zero-shot classification with LLMs, and literature all at the same time.

bookpredictor=LLMFunction["Please answer only with the title of a book. Which famous book contains the following paragraph:\n`paragraph`"]

publicdata[All,All,bookpredictor[<|"paragraph"->#ParagraphText|>]&]

Some of the mistakes here are not so different from those a person might make.

EntityValue[{(Entity["Agnes Grey", Entity["Book","AgnesGrey"], "Entity[\"Book\", \"AgnesGrey\"]", "book"]),(Entity["Jane Eyre", Entity["Book","JaneEyre"], "Entity[\"Book\", \"JaneEyre\"]", "book"]),(Entity["War and Peace", Entity["Book","WarAndPeace"], "Entity[\"Book\", \"WarAndPeace\"]", "book"]),(Entity["Crime and Punishment", Entity["Book","CrimeAndPunishment"], "Entity[\"Book\", \"CrimeAndPunishment\"]", "book"])},
{"Author","FirstPublished"},"Dataset"]

As an exercise for the attendees, come up with a lesson on literature in which students find the part of Crime and Punishment that most reminds them of War and Peace. Then, look at embeddings and zero-shot classification to see if the LLM “agrees”.

TextCases[Entity["Book","CrimeAndPunishment"]["Plaintext"],"Paragraph",5]//Column[#,Frame->All]&

Prompting Techniques

One way to improve LLM outputs is to elicit what is known as a Chain-of Thought (Wei et al, 2022 preprint).

Let’s illustrate this concept using a small part of the gsm8k dataset of grade school math questions.

trainexamples=CloudGet["gsm8kExamples"];

testexamples=CloudGet["gsm8kTestExamples"];

testexamples[1,"question"]

testexamples[1,"answer"]

Note that on this common type of reasoning problem, newer OpenAI models can display chain of thought without the need for special prompting (this was not always the case).

This older model just provides the answer without “reasoning” its way through.

LLMSynthesize[testexamples[1,"question"],LLMEvaluator-><|"Model"->"text-davinci-003","Temperature"->0|>]

Using few-shot learning, you can often get newer models to reason their way through unfamiliar problems. In this case, let’s use few-shot learning to get gpt-4 to not reason its way through grade school math. The drop in accuracy is remarkable.

fewshot=StringTemplate["`generalInstruction`\n\nInput: `exemplarIn1`\nOutput: `exemplarOut1`\n\nInput: `exemplarIn2`\nOutput: `exemplarOut2`\n\nInput: `realQuestion`"]

(* prepare some exemplar in-out pairs *)
exemplarIn1=trainexamples[1,"question"];
exemplarOut1=Last@TextCases[trainexamples[1,"answer"],DigitCharacter..];
exemplarIn2=trainexamples[2,"question"];
exemplarOut2=Last@TextCases[trainexamples[2,"answer"],DigitCharacter..];

dontreason=fewshot[<|
"generalInstruction"->"Answer the math problem with ONLY a number:",
"exemplarIn1"->exemplarIn1,
"exemplarOut1"->exemplarOut1,
"exemplarIn2"->exemplarIn2,
"exemplarOut2"->exemplarOut2,
"realQuestion"->testexamples[1,"question"]|>]

LLMSynthesize[dontreason,LLMEvaluator-><|"Model"->"gpt-4","Temperature"->0|>]

smallexperiment=CloudGet@CloudObject[(HyperlinkURL["https://www.wolframcloud.com/obj/jmcnally0/PromptExamples/Summary", https://www.wolframcloud.com/obj/jmcnally0/PromptExamples/Summary])]

answerparser[str_String]:=Interpreter["Integer"][Last[StringCases[str,DigitCharacter..]]]

answerparser[x_List]:=First[Commonest[Map[Function[{str},Interpreter["Integer"][Last[StringCases[str,DigitCharacter..]]]],x],1]]

smallexperiment[All,All,Lookup[True]@*Counts,SameQ[answerparser[#rawresult],answerparser[testexamples[#index,"answer"]]]&]

Note that gpt-4 also displays chain of thought without any additional prompting on this particular domain

smallexperiment["gpt-4","in-out",1,"rawresult"]

“Sampling” in this case was done at a temperature of 0.7. Note that on question 8, the model is not so sure of the answer, but the most commonly arrived at answer is indeed the correct one:

smallexperiment["gpt-3.5-turbo","sampling",8,Map[answerparser][#rawresult]&]

testexamples[8,"answer"]

Other Techniques

Techniques for solving problems with language models is a large area with lots of new developments. You have seen how a chain of thought and self consistency among various reasoning paths impact performance in the small example above. These are good reading resources if you want to dive in.

Chain of Thought with Self Consistency (Wang et al, 2022 pre-print)

This involves sampling different reasoning paths, as you saw in the mini-example previously

Tree of Thoughts (Yao et al, 2023 pre-print)

Check out the Wolfram Summer School project on this topic: https://community.wolfram.com/groups/-/m/t/2959295

Reasoning via Planning (Hao et al, 2023 pre-print)

ReAct (Yao et al, 2022 pre-print)

Language Agent Tree Search (Zhou et al, 2023 pre-print)

Graph of Thoughts (Besta et al, 2023 pre-print)

Computational Thinking

As you’ve hopefully been convinced, learning with and learning about LLMs can engage students’ metacognitive skills. If they are thinking about the “reasoning” of an LLM, this is an opportunity to develop critical skills.

But now for something completely different -

But LLMs are just the latest technology that gives opportunity for bringing computational thinking into new domains. Since project-based learning and computational thinking go so well together, below is a starter activity you can use to bring computational thinking into the history classroom.

Question: Are there patterns in the construction of castles in England?

If you know it’s possible to get a list of all such castles...

englishcastles=EntityList[EntityClass["Castle",(EntityProperty["country", EntityProperty["Castle","Countries"], "EntityProperty[\"Castle\", \"Countries\"]"])->(Entity["United Kingdom", Entity["Country","UnitedKingdom"], "Entity[\"Country\", \"UnitedKingdom\"]", "country"])]];
Take[englishcastles,10]

If you know it’s possible to look up known construction dates...

castledates=EntityValue[englishcastles,"ConstructionStartDate","NonMissingEntityAssociation"];
castledates//Short

You can indeed spot patterns...

DateHistogram[castledates,"Century"]

...and investigate further!

DateHistogram[castledates,{{900},{1300},"Decade"}]

Refined Question: Are there geographic patterns in English castle construction?

If you know it’s possible to select and group by criteria...

castlesOfInterest=Select[englishcastles,(DateObject["Year: ""1060", DateObject[{1060},"Year"]])≤#["ConstructionStartDate"]≤(DateObject["Year: ""1110", DateObject[{1110},"Year"]])& ];
castlesByDecade=GroupBy[castlesOfInterest,DateValue[#["ConstructionStartDate"],"Decade"]& ]//KeySort

If you know data can be visualized for better communication...

GeoListPlot[castlesByDecade//Values,PlotLegends->Keys[castlesByDecade],PlotMarkers->Style[■,14],PlotStyle->{Red,Orange,Yellow,Green,Blue}]

You can create an interactive visual explorer:

With[
{labels=Keys[castlesByDecade],
places=Values[castlesByDecade]},
TabView[
Table[labels[[n]]->GeoListPlot[Take[places,n],PlotLegends->Take[labels,n],GeoRange->GeoBoundingBox[places,Scaled[0.05]],
PlotMarkers->Style[■,14],PlotStyle->{Red,Orange,Yellow,Green,Blue}],{n,Length[places]}],
ControlPlacement->Left]
]

Now you can even ask an LLM for input to this process!

It’s always nice to check an LLM with something verifiable:

Module[{labels,places,castles1060s},
castles1060s=KeySort[GroupBy[First[castlesByDecade],DateValue[#["ConstructionStartDate"],"Year"]& ]];
labels=Keys[castles1060s];
places=Values[castles1060s];
TabView[
Table[labels[[n]]->GeoListPlot[Take[places,n],PlotLegends->Take[labels,n],GeoRange->GeoBoundingBox[places,Scaled[0.05]],
PlotMarkers->Style[■,14],PlotStyle->{Red,Orange,Yellow,Green,Blue},PlotLabel->"Castles Started in 1060s"],{n,Length[places]}],
ControlPlacement->Left]
]

Further student exploration:

Who were the Normans and where did they live before the 1060s?

What human factors led to the Norman Conquest of England?

What primary sources are available from this time?

Was there an analogous event corresponding to the spurt of construction after 1100 as well?

Wrapping Up

Resources for Learning to Think Computationally

Wolfram Summer School

Wolfram U

Computational Notebooks

Wolfram Language

Data Science

LLMs 101

Back to School with AI Tools and Wolfram

Online Books

Elementary Introduction to the Wolfram Language

Introduction to Machine Learning

Wolfram Community

Wolfram Emerging Leaders Program

Wolfram Student Ambassador Program