Integrating LLMs into Wolfram 
Work Flows

Leveraging Mathematica Tools and AI Innovations to Process and Analyze Sustainable Development Data Efficiently

Ben Kickert and Maureen Baehr, SDGCounting

(RowDefault[, (Spacer1), , (Spacer1), ])  #WolframTechConf

ABSTRACT

This presentation outlines how Wolfram Tools can be utilized to create workflows that maximize the potential of emerging Large Language Models such as OpenAI’s GPT, Google’s Palm, Meta’s LLaMA and more.  

Special attention is paid to processing and analyzing large unstructured data such as those present in Sustainable Development Goal reports or conference video collections.

This demonstration builds on previous presentations by the authors to highlight new technologies while emphasizing familiar methodology.

A Few Introductory Notes

Work with StartingUpGood and SDGCounting as hobbyists

Support for startups around promoting societal good

Advocate for Sustainable Development data access, quality, and usage

Building on previous work and presentations

Working with structured and unstructured data

Machine Learning for data analysis

Incorporating non-traditional data sources such as YouTube

Connecting with external tools like GPT

WTC 2023: “Using the Wolfram Stack and LLM Tech to Combat Misinformation on the SDGs and ESG”

Start with concrete example and move to a survey of team tools

Additional information and examples will be included in the notebook for items we are unable to cover verbally in the presentation

Rapid Developments in the AI space as well as within Wolfram: everything is subject to change

Presentation Overview

Today we are going to demonstrate how to use Wolfram Tools as the backbone for utilizing external LLM resources with an emphasis on preprocessing, batching multiple calls, and structuring information.

Specifically we will look at content from the Voluntary National Reviews for SDG progress and video resources from the recent AI for Good Summit.

Key Learnings

Connecting with LLMs and the various Wolfram tools available

“Chunking” data for batch processing

Structuring calls and Prompt Engineering

Compiling Final Information in useful forms

Expanding workflows to include more complicated use cases

Using Large Language Models in Wolfram

Today we will focus primarily on OpenAI’s GPT model(s) although our team also works regularly with tools such as Palm, Claude 2, and Perplexity.

NOTE: Accessing these services requires an external account.  Details available here: https://support.wolfram.com/62525

There are currently three possible methods for accessing external LLM resources:

Built in Functions

Launched with 13.3

LLMSynthesize for basic calls

LLMTool or LLMResourceFunction for more complicated calls

Custom Defined Function for API calls

Most versatile

Allows for easier integration of emerging features

ServiceConnect

Blend of both

Wolfram Built In Functions

Starting with Mathematica 13.3, several LLM specific functions have been included.

Check out the WTC sessions from Wolfram Staff to get a better overview of these tools.

There are currently ~13 LLM related functions.  Many, such as LLMResourceFunction or LLMTool are structured to connect the user with a pre-designed processing/prompt stack.

We highly recommend using the simplest function LLMSynthesize so you can clearly see what is going in and what is coming out.

LLMSynthesize["What is the capital of North Carolina?"]

json=LLMSynthesize["Provide a list of 5 different programming languages most like Wolfram Language. Output should be formatted as JSON with the headings: 'name','year introduced','main url','similarities to WL'."];
Dataset[ImportString[json,"RawJSON"]]

LLMSynthesize can be customized using the LLMEvaluator function. So if you need a different model that accepts more tokens, or is generally quicker, you can make that adjustment to 3.5-Turbo-16K.

alice=ExampleData[{"Text","AliceInWonderland"}];

LLMSynthesize["Summarize the following text:\n"<>alice]

LLMSynthesize["Summarize the following text:\n"<>alice,LLMEvaluator-><|"Model"->"gpt-3.5-turbo-16k"|>]

Custom Functions for direct API Calls

While tools like LLMSynthesize can be customized and provide a direct view of what is going in and out, we often prefer using our our own custom functions.

Benefits:

Access to all arguments available through the API

Ability to adjust API endpoints that may be released

Even better control over how inputs are structured

Full output from API is delivered directly allowing for troubleshooting

Flexible for features/syntax that may change

Custom Function Definition

$openAISecretKey=

Clear[gpt];

Options[gpt]={"Model"->"gpt-4","SystemMessage"->"","TokensToGenerate"->1000,"Temperature"->0.6,"Metadata"->False};

gpt[prompt_String,OptionsPattern[]]:=Module[{systemmessage,messages,response,textonly},
(systemmessage=<|"role"->"system","content"->OptionValue["SystemMessage"]|>;

messages=If[OptionValue["SystemMessage"]==="",{<|"role"->"user","content"->prompt|>},{systemmessage,<|"role"->"user","content"->prompt|>}];

response=URLExecute[
HTTPRequest["https://api.openai.com/v1/chat/completions",
<|Method->"POST","Body"->ExportString[<|"model"->OptionValue["Model"],"messages"->messages|>,"JSON"],"Headers"->{"Authorization"->"Bearer "<>$openAISecretKey,"Content-Type"->"application/json"}|>
],"RAWJSON"];

textonly=If[MissingQ[response[["choices"]]],"Error: "<>response[["error","message"]],response[["choices",1,"message","content"]]];

If[OptionValue["Metadata"],response,textonly])]

Using the same examples as from the LLMSynthesize slide, we now see we have more flexibility

gpt["Summarize the following text:\n"<>alice]

gpt["Summarize the following text:\n"<>alice,"Model"->"gpt-3.5-turbo-16k","SystemMessage"->"All of your answers should come the form of an eight line rhyming poem"]

A Brief Note on System Messages

Introduced in March 2023 with GPT4, these are designed as a way to pass processing instructions on to the model.  According to OpenAI, they are given a higher priority than regular chat content and thus are more reliable.  This is the one “killer feature” that most aren’t regularly utilizing.

Full Chat Options

Although we are only demonstrating a single call, it is possible to simulate a full chat environment using this GPT code.

Service Connect

The OpenAI ServiceConnect provides a solution for those interested in connecting to the API and having more flexibility, but don’t want to worry about hand-coding all the specific interactions

Start by opening up a ServiceConnect

openAI=ServiceConnect["OpenAI"]

And ensure your prompt is formatted appropriately.  Here we are using the “Completion” endpoint.

ServiceExecute[openAI,"Completion",{"Prompt"->"What is the Capital of North Carolina?"}]

Most people are familiar with the “Chat” endpoint which requires slightly different structure, and allows for a back and forth interaction in addition to single calls.

question=ServiceExecute[openAI,"Chat",
{"Messages"->
{<|"role"->"user","content"->"What is the capital of North Carolina?"|>}
}]

question[[1,"message","content"]]

System Messages can be included as an additional message.

messages={
<|"role"->"system","content"->"Talk like a pirate when answering"|>,
<|"role"->"user","content"->"What is the best thing to do in Asheville?"|>
};

systemquestion=ServiceExecute[openAI,"Chat",{"Messages"->messages}]

And the “chat” can be continued.

messages2=Flatten[{
messages,
systemquestion[[1,"message"]],
\[LeftAssociation]"role"->"user","content"->"How much does it cost?"\[RightAssociation]
}];

ServiceExecute[openAI,"Chat",{"Messages"->messages2}][[1,"message","content"]]

Here we see the answer is based both on data from the original question and system message as well as the follow up question.

Which approach should you use?

While all three have their advantages, our team has worked almost exclusively with the custom function option as our tool development started prior to Wolfram Function integration and also because it allows for the most adaptation as things change without being overly complicated.

However, for this presentation, we will use the base built in function of LLMSynthesize to make it as easy as possible to copy and paste our work if you so desire

Example Exercise: Voluntary National Reviews

Every year, a subset of UN Member Nations submit their Voluntary National Reviews to the High Level Political Forum on Sustainable Development.  These documents provide a rich source of country-level information about what is working and the challenges faced.  Unfortunately, due to the dense nature of these reports, they are often overlooked.

Let’s take a look at the data available since 2015 from these nearly 1,300 reports and supporting documents.

vnrds=

Country / Entity Count: 187

Length[DeleteDuplicates[vnrds[[All,1]]]]

Most Recent Year: 175 Documents

vnrs2023=Dataset[Select[vnrds,#[[2]]=="2023"&],MaxItems->5]

Canada in 2023: 9 Documents

ca=Select[vnrs2023,#[[1]]=="Canada"&][All,{"File"->URLDecode}]

Summarizing a Single Large Document

Looking at Canada’s submission, we see one of the documents is the English version of their full report.  Let’s import that and see what we have.

cavnrpages=Import[File[(FileArgument["Z:\\vnrs\\canada\\2023\\VNR%202023%20Canada%20Report%20EN.pdf"])],"PagePlaintext"];
cavnrstring=StringRiffle[cavnrpages,"\n"];
Length[cavnrpages]
StringLength[cavnrstring]

It is 92 pages long and over 265,000 characters!

Documents like these are usually very dense and cover a wide variety of topics so to get the full picture, we want to ensure we can work with the entire text.  Unfortunately, even the most robust publically available models only work with a max of 64,000 characters (16K tokens) while many are capped at less than half that figure.

Tools for Working with Large Strings

tokenLength to approximate the number of tokens

Clear[tokenLength]
tokenLength[string_] := IntegerPart[N[StringLength[string]/4]];

tokenLength[cavnrstring]

SequenceSelect (adapted from Bob Sandheinrich’s Function Repository Entry) and tokenPartsSplit to create chunks up to a maximum token length.

Clear[SequenceSelect]
SequenceSelect[list_List,test_,rest___]:=SequenceCases[list,_?test,rest];

Clear[tokenPartsSplit]
tokenPartsSplit[list_,tokens_:3000]:=Module[{pagetokens,lengths},(
pagetokens=tokenLength/@list;
lengths=Length/@SequenceSelect[pagetokens,Total[#]<tokens&];
TakeList[list,lengths])]

Chunking the Data for Processing

7,000 tokens for GPT-4: Most powerful model, but slow and token-limited

cavnrchunks=StringRiffle[#,"\n"]&/@tokenPartsSplit[cavnrpages,7000];
Length[cavnrchunks]
tokenLength/@cavnrchunks

14,000 tokens for GPT-3.5-Turbo-16K: Faster, more tokens, but less creative

cavnrchunks16=StringRiffle[#,"\n"]&/@tokenPartsSplit[cavnrpages,14000];
Length[cavnrchunks16]
tokenLength/@cavnrchunks16

Batch Processing for Bullet Points

GPT-4: Ten Chunks and Nine Minutes

AbsoluteTiming[bullets4=LLMSynthesize[

"The following text is an excerpt from Canada's Voluntary National Review, submitted to the High Level Political Forum on Sunstainable Development in 2023.\n\n- - - - -\n\n"

<>#<>

"\n\n- - - -\n\n Provide a detailed bulleted list summarizing the content above.  Be sure to inclue significant accomplishments and challenges as well as any important statistics. Do not include any information not specifically in the text provided.",

LLMEvaluator-><|"Model"->"gpt-4"|>,MaxItems->1000]&/@cavnrchunks;]

3.5-Turbo-16K: Five Chunks and One Minute

AbsoluteTiming[bullets=LLMSynthesize[

"The following text is an excerpt from Canada's Voluntary National Review, submitted to the High Level Political Forum on Sunstainable Development in 2023.\n\n- - - - -\n\n"

<>#<>

"\n\n- - - -\n\nProvide a detailed bulletted list summarizing the content above.  Be sure to inclue significant accomplishments and challenges as well as any important statistics. Do not include any information no specifically in the text provided.",

LLMEvaluator-><|"Model"->"gpt-3.5-turbo-16k"|>,MaxItems->1500]&/@cavnrchunks16;]

In our experience, 3.5-Turbo does a good job of summarizing into bullets, but not as good of a job with writing long form summaries.  Given the larger token capacity of 3.5-Turbo-16K, as well as the overall speed per batch, it make sense to use it over GPT-4.

allbullets=StringRiffle[bullets,"\n"];
tokenLength[allbullets]
Pane[allbullets,{1300,500},Scrollbars->True]

From Bullets to Summary Using GPT-4

AbsoluteTiming[cavnrsummary=LLMSynthesize[allbullets<>"\n\n- - - -\n\nThe bullets above are a summary of Canada's Voluntary National Review submitted to the High Level Political Forum on Sustainable Development in 2023.  Please provide a detailed one page write up summarizing the content.  Be sure to include details on the accomplishments, challenges, and commitments made in the document as well as any significant statistics.",LLMEvaluator-><|"Model"->"gpt-4"|>,MaxItems->1500];]

The result is a roughly 2 page summary of a 92 page document that takes into account the entirety of the textual content.

Pane[cavnrsummary,{1300,500},Scrollbars->True]
StringLength[cavnrsummary]

Standardized All-In-One Function with Additional Processing

Now we can combine the techniques discussed into  single PDF summarizing tool while adding a few changes and adding a couple tricks:

Prompt is now more generic

Use filename to provide context

Requesting output as HTML

Algorithmically formatting the original bullets so no information is lost

Export as file

Clear[pdfSummarizer]
pdfSummarizer[file_]:=Module[{filename,pages,chunks,bullets,allbullets,summary,summaryhtml,htmlbullets,html},(filename=Last[StringSplit[file,{"/","\\"}]];

pages=Import[file,"PagePlaintext"];

chunks=StringRiffle[#,"\n"]&/@tokenPartsSplit[pages,14000];

bullets=LLMSynthesize[StringJoin["The following text is from the file \"",filename,"\":\n\n",#,"\n\n- - - -\n\nProvide a detailed bulleted list summarizing the content above."],LLMEvaluator-><|"Model"->"gpt-3.5-turbo-16k"|>,MaxItems->1500]&/@chunks;

allbullets=StringRiffle[bullets,"\n"];

summary=LLMSynthesize[StringJoin[allbullets,"\n\n- - - -\n\nThe bullets above are a summary of the file \"",file,"\".  Please provide a detailed one page write up summarizing the content. Use appropriate headers to present the information clearly. Output should be formatted as HTML."],LLMEvaluator-><|"Model"->"gpt-4"|>,MaxItems->1000];

summaryhtml=StringReplace[summary,"<html>"|"</html>"|"<body>"|"</body>"->"",IgnoreCase->True];

htmlbullets=StringJoin[Flatten[{"<ul>","<li>"<>#<>"</li>"&/@StringTrim[StringReplace[StringSplit[allbullets,"\n"],"-"->""]],"</ul>"}]];

html=StringJoin["<html><head><title>",filename," Summary</title></head><body>",summaryhtml,"<h1>BULLETS</h1>",htmlbullets,"</body></html>"];

Export[StringJoin["Summary of ",filename,".html"],html,"Text"]
)]

Testing on Belgium’s VNR:

AbsoluteTiming[besummary=pdfSummarizer["Z:\\vnrs\\belgium\\2023\\VNR%202023%20Belgium%20Report.pdf"]]

So in a bit over two minutes, we have a summary of a 130 page document!

Pushing the Limits with the EU’s VNR

The EU presented their first VNR this year and it came in at over 300 pages and 250K tokens requiring 19 14,000 token chunks!

AbsoluteTiming[eusummary=pdfSummarizer["Z:\\vnrs\\european-union\\2023\\VNR%202023%20EU%20Report.pdf"]]

Due to the size, this one takes longer at about 4.5 minutes, but you know you have processed the entirety of the text and you have all the summary bullets to dig deeper if you so desire.

A few notes on the process

This had no problem with the token limit of the bullets.  However, if you were working with a very large document, you may want to run an intermediary step to chunk and condense those.

Same basic process can be used for any large textual file.

Based on existing smaller chunks (PDF Pages).  For other documents you may need an alternative approach but this can also be done by StringSplitting on line breaks, or even simply doing a String Partition.

Wait... aren’t there already tools for this? 

Yes.  But, do you know what is going on in the “Black box”?  With methods like these, you know what preprocessing, condensing, and prompts are being used.

Survey of Team Tools

The following examples expand on the code demonstrated above, but have been tailored specifically to our team’s needs.  We are sharing them with the hope that others will recognize how Wolfram workflows can serve as the backbone of LLM related tools -- not just as a pass through to an API, but as a powerful resource for processing and structuring data.

Summarizing Multiple Documents

The chunking and summarizing process works well with single large documents.  But what if you want to look at several? Let’s go back to our VNR Example.

vnrds=

What if we want an overall summary of all the VNRs from 2023?  While we could do this for all the documents, VNRs are submitted in such a way that every country also submits a “Main Messages” summary.

mainmessages=;
Length@mainmessages
Take[mainmessages,4]

Using our Custom Function “fileListSummary” to Batch Process

AbsoluteTiming[mainmessagesummary=fileListSummary["2023 Voluntary National Reviews",mainmessages];]
mainmessagesummary

In 10 minutes we have a summary of all 39 documents:

A table of all the country specific notes:

Dataset[SemanticImport[mainmessagesummary[[2]]],MaxItems->3]

And a list of all intermediate notes used to create the overall summary:

Pane[Import[mainmessagesummary[[3]]],{1300,500},Scrollbars->True]

Medium Article on the Process

Global Goals, AI Solutions
Unlocking Insights from the Voluntary National Review Process Using Machine Learning

Working with YouTube Playlists

In our first example, we looked at a list of documents from a known source.  Realistically, this could have been a folder of files, a list of transcripts, or a series of downloaded PDFs on a particular topic

For our last example, we want to explore some non-traditional sources of textual data.

Theoretical Use Case

A conference you were supposed to attend has posted the contents of its sessions on YouTube.  You were unable to attend as many as you hoped, but still need to provide a summary to your colleagues.  What do you do?

2023 AI For Good Summit

First, you would need a tool that could import YouTube Playlist details.

ai4goodvids=ytPlaylist["https://www.youtube.com/playlist?list=PLpoIPNlF8P2N0VNgcbnQC3D_bT62r2xGw"]

Then you would need a tool that could import the details of those videos:

Dataset[ytvidexample=ytElements[ai4goodvids[[1,"URL"]]]]

This would include a full transcript of each video as well as important metadata for context

Pane[ytvidexample[["Transcript Text"]],{1300,500},Scrollbars->True]

Once that information is available, an all-in-one function could be developed that automatically summarizes each session and provides an overview of all the content.

AbsoluteTiming[ai4goodoutputs=playlistSummary["2023 ITU AI for Good Global Summit","https://www.youtube.com/playlist?list=PLpoIPNlF8P2N0VNgcbnQC3D_bT62r2xGw"];]
ai4goodoutputs

This took over 30 minutes, but we were also able to process and summarize 65 video into on cohesive output!

Dataset[SemanticImport[ai4goodoutputs[[2]]],MaxItems->3]

Pane[Import[ai4goodoutputs[[3]]],{1300,500},Scrollbars->True]

Web Base News and Web Topic Summarizer

In Maureen’s presentation yesterday she mentioned how we gathered news and web articles, filtered to only include trusted and unbiased sources, and created auto generated articles.  All of this is based on the processes outlined above.  We have taken in step further to provide our team with a web-based interface for accessing these tools.

Conclusion

LLMs and AI tools in general are changing the way people work by providing solutions never before possible.

The Wolfram Technology Stack offers much more than just pass through connectivity to these external resources.

The textual and procedural functionality available within Mathematica enables easy prototyping of AI based tools that allow users to know more about what is going on inside the “black box”.

While new LLM implementations are being released daily, the ability to customize tools based on your needs and constraints will ensure homegrown solutions are around for the foreseeable future.

But, we expected continued rapid change requiring developers stay up-to-date and continue to innovate.

Q&A

Integrating LLM into Wolfram Work Flows
Leveraging Mathematica Tools and AI Innovations to Process 
and Analyze Sustainable Development Data Efficiently

Ben Kickert and Maureen Baehr

www.startingupgood.org

https://startingupgood.medium.com
https://sdgcounting.medium.com