LLM Integration and Agentic Workflows in Wolfram Language

Timothée Verdier, Wolfram Research

#WolframTechConf

ABSTRACT

Large language models (LLMs) are rapidly becoming essential tools for research and development. This session presents an overview of Wolfram Language’s LLM integration framework, showing how LLM functions, external model connections and structured tool-calling workflows can be used in practice. We’ll also highlight recent developments enabling complex, multistep reasoning chains that blend LLM output with symbolic computation.

What are Neural Networks

Family of tunable mathematical functions

Nature/bio inspired

Row[{,Style["⟹",14],(Subscript[y, k])=σ((∑_j)(Subscript[x, j]) (Subscript[w, k,j])-(Subscript[b, j]))},Spacer[10]]

Data encoded as numbers

Computation described as succession of layers of  interconnected “neurons”

Learning achieved by tuning neurons connection weights

powerful, versatile, scalable, generic, rich in by-products

What are Large Language Models (LLM)

Language Models are trained at reproducing the patterns and statistics of written languages (natural or formal).

Large Language Models (LLMs) have a large capacity and are trained on on a vast amount of text.

Training result after many examples updates...

GPT (as in Chat-GPT) stands for “Generative Pre-trained Transformer”

NetModel[{"GPT2 Transformer Trained on WebText Data","Task"->"LanguageModeling","Size"->"345M"}]["The cat sat on the",{"TopProbabilities", 5}]

Text generation achieved with incremental sampling

Connect to an External Provider...

LLM functionality requires external APIs to work. The Wolfram Language support many external services that you can use providing your own credentials.

If you have no saved connections to Anthropic, this code will prompt for an API key

LLMSynthesize["Hi",LLMEvaluator-><|"Model"-> <|"Service"-> "Anthropic", "Name"-> "claude-opus-4-20250514"|>|>]

...Or just use Wolfram

An LLMKit subscription will enable all the LLM-powered functions (and additionally give access to the Notebook Assistant).

LLMSynthesize["The cat sat on the "]

The simplest LLM interaction – LLMSynthesize

The most basic form of LLM interaction is provided by (TextRefLink[LLMSynthesize, paclet:ref/LLMSynthesize, https://reference.wolfram.com/language/ref/LLMSynthesize.html]).

Submit to the LLM for completion

LLMSynthesize["Give me three warm colors"]

Specify generation options

LLMSynthesize["Give me three warm colors", LLMEvaluator->LLMConfiguration@<|"MaxTokens"-> 40, "Temperature"->0, "Prompts"-> "Write only one RGB specification per line"|>]

Request other properties

LLMSynthesize["Give me three warm colors","Prompt", LLMEvaluator->LLMConfiguration@<|"MaxTokens"-> 40, "Temperature"->0, "Prompts"-> "Write only one RGB specification per line"|>]

Parametrize the prompts – LLMFunction

When a prompt that depends on some input is needed, you can use (TextRefLink[LLMFunction, paclet:ref/LLMFunction, https://reference.wolfram.com/language/ref/LLMFunction.html]).

Define a parametric prompt

llmfun = LLMFunction["Give me `number` `type` colors","Color"]

Evaluate the LLMFunction

llmfun[<|"number"->"three","type"-> "warm"|>]

[Update] LLMFunction now automatically uses (TextRefLink[Interpreter, paclet:ref/Interpreter, https://reference.wolfram.com/language/ref/Interpreter.html]) schema

Augment the model, delegate expertize – LLMTools

LLM can be made more powerful by giving them access to external tools. Those can be defined using (TextRefLink[LLMTool, paclet:ref/LLMTool, https://reference.wolfram.com/language/ref/LLMTool.html]).

Define a tool for the LLM

randomColor=Function[BlockRandom[RandomColor[#n],RandomSeeding->1234]];
colorSampler=LLMTool[{"ColorSample", "Random color generator"},<|"n"-> "Integer"|>,randomColor]

Test the LLMTool answer generation:

colorSampler[<|"n"-> 3|>]

Provide the LLM with the tool

LLMSynthesize["Give me a color per day in the week", LLMEvaluator->LLMConfiguration[<|"Tools"-> {colorSampler}|>]]

Check the generation steps

LLMSynthesize["Give me a color per day in the week", "CompletionToolsText",LLMEvaluator->LLMConfiguration[<|"Tools"-> {colorSampler}|>]]

Send contextualized prompts – LLMPromptGenerator

The choice of whether calling a tools belongs to the LLM. The prompt can be augmented programmatically using (TextRefLink[LLMPromptGenerator, paclet:ref/LLMPromptGenerator, https://reference.wolfram.com/language/ref/LLMPromptGenerator.html]).

LLMs diligently provide answers - correctness is optional. This is also known as hallucinating

LLMSynthesize["What day are we ? Give me a color per day in the week. Conclude the answer with today's color", LLMEvaluator->LLMConfiguration[<|"Tools"-> {colorSampler}|>]]

Define a dynamic call-time prompt including the answer elements

LLMPromptGenerator[{"Current date/time: ",DateString[]}&][<|"Input"-> "What day are we ? Give me a color per day in the week. Conclude the answer with today's color"|>]

Retry providing the LLMPromptGenerator:

LLMSynthesize["What day are we ? Give me a color per day in the week. Conclude the answer with today's color", LLMEvaluator->LLMConfiguration[<|"Tools"-> {colorSampler},"Prompts"-> LLMPromptGenerator[{"Current date/time: ",DateString[]}&]|>]]

Send contextualized prompts – LLMPromptGenerator

(TextRefLink[LLMPromptGenerator, paclet:ref/LLMPromptGenerator, https://reference.wolfram.com/language/ref/LLMPromptGenerator.html]) automatically augment the input prompt with similar (TextRefLink[SemanticSearchIndex, paclet:ref/SemanticSearchIndex, https://reference.wolfram.com/language/ref/SemanticSearchIndex.html]) entries.

Prompting based on context aware information retrieval is know as Retrieval-Augmented Generation (RAG)

index=CreateSemanticSearchIndex@ResourceData["Alice in Wonderland"]

Context is used to query the index

SemanticSearch[index, "a sport game",MaxItems->3]//Map[Framed[Style[#,8]]&]

Use a dynamic call-time prompt including the elements relevant to the question

LLMSynthesize["What sport game does Alice watch in the wonderland?", LLMEvaluator->LLMConfiguration[<|"Prompts"-> LLMPromptGenerator[index]|>]]

Build and run workflows — LLMGraph

(TextRefLink[LLMGraph, paclet:ref/LLMGraph, https://reference.wolfram.com/language/ref/LLMGraph.html]) combines the power of (TextRefLink[LLMFunction, paclet:ref/LLMFunction, https://reference.wolfram.com/language/ref/LLMFunction.html]) and (TextRefLink[Function, paclet:ref/Function, https://reference.wolfram.com/language/ref/Function.html]) altogether in a controlled flow

Sometimes we need a pure Function,

Function[{"Current date/time: ",DateString["Date"]}]

other times we need an LLMFunction,

LLMFunction["Give me a color per day in the week.",LLMEvaluator->LLMConfiguration[<|"Tools"-> {colorSampler}|>]]

but many times what works is a combination of both!

workflow=LLMGraph[<|
"Today" -> Function[{"Current date/time: ",DateString["Date"]}],
"DayColors"-> LLMFunction["Give me a color per day in the week.",LLMEvaluator->LLMConfiguration[<|"Tools"-> {colorSampler}|>]],
"CurrentColor"-> LLMFunction["Pick among `DayColors`, the one for `Today`", "Color"]
|>]

Inspect this first workflow

Information[workflow,"Graph"]

To evaluate, just run the LLMGraph

workflow[<||>]

Human in the loop - inject node value / force outputs anywhere

workflow[<|"DayColors"-> "Monday,Tuesday,Wednesday:Red\nThursday,Friday,Saturday:Green\nSunday:Blue"|>,All]

Request result from any node in the graph

workflow[<|"Today"-> "Monday"|>, All]

Build and run workflows — LLMGraph

(TextRefLink[LLMGraph, paclet:ref/LLMGraph, https://reference.wolfram.com/language/ref/LLMGraph.html]) is efficiency tailored both at build time and at run time

Automatic dependencies management

at build time:

(AplusBsquare = LLMGraph[<|"Sum" -> Function[#A+#B],"Square"-> Function[#Sum^2]|>])//Information[#, "Graph"]&

and at run time:

AplusBsquare[<|"Sum"-> 4|>, {"Square"}]
AplusBsquare[<|"A"-> 4, "B"-> 3|>, {"Sum"}]

Independent submissions of (TextRefLink[LLMFunction, paclet:ref/LLMFunction, https://reference.wolfram.com/language/ref/LLMFunction.html]) nodes

Distinct node (TextRefLink[LLMEvaluator, paclet:ref/LLMEvaluator, https://reference.wolfram.com/language/ref/LLMEvaluator.html])

Efficient parallel submissions under the hood

LLMGraph[<|"LLMCall1"->"Hello! Who are you?","LLMCall2"->LLMFunction["Hello! Who are you?",LLMEvaluator->LLMConfiguration[<|"Model"->{"Anthropic",Automatic}|>]]|>
][]

Multi-step evaluation

partial=AplusBsquare[<|"A"-> 2|>, "LLMGraph"]

partial[<|"B"-> 3|>]

Asynchronous evaluation

task = LLMGraphSubmit[workflow, HandlerFunctions-><|"TaskFinished"->Function[dayColor=#GraphResults]|>]

2+2
TaskWait@task; dayColor

And more: conditional node evaluation, listable LLM nodes...

LLM workflow application — Summarization

The LLMs are powerful tools, the key to success when using them is often to effectively constrain their scope of action to the right place.

$LLMAttentionWindow = 12000

(summarize =LLMGraph[
<|
"Chunk"-><|"EvaluationFunction"-> Function @StringPartition[#Text, UpTo[$LLMAttentionWindow]], "Input"-> {"Text"}|>,
"ChunkSummary"-><|"ListableLLMFunction"->"Summarize the following part in a few lines:\n\n`Chunk`"|>,
"FinalSummary"->"Summarize the following:\n\n`ChunkSummary`"
|>
])//Information[#, "Graph"]&

summarize[<|"Text"-> ExampleData[{"Text","AliceInWonderland"}]|>]

THANK YOU

QUESTIONS ?