Natural Language Processing Template Engine

Anton Antonov, Accendo Data LLC

(RowDefault[, (Spacer1), , (Spacer1), ])  #WolframTechConf

Abstract

Code generation using semantic parsing of computational workflows specifications via Question Answering Systems

In this presentation we discuss the completion of computational templates with parameters that are extracted from text specifications using a question answering system (QAS) [Wk1]. We outline the general method and then demonstrate it with several types of computational workflows: classification, latent semantic analysis (LSA), quantile regression (QR), random data generation (RDG) and recommendations. We show how to leverage the engine by bringing your own templates (BYOT).

The demonstrated functionalities use the packages in the project "NLP Template Engine" [AAr1], which is based on FindTextualAnswer [WRI1, JL1].

We concentrate on data science (DS) and machine learning (ML) computations, but the described method and overall algorithm are applicable to computation workflows from other fields.

Presentation plan

Part 1

Problem formulation

Big picture

Demo examples

Interactive interface

Design

Walkthrough

Part 2

Bring your templates

Introspection

Workflow classifier training

Introduction: problem formulation

Problem formulation

We want to have a system -- Template Engine (TE) -- that:

Generates relevant, correct, executable programming code based natural language specifications of computational workflows

Can automatically recognize the workflow types

Can generate code for different programming languages and related software packages

The points above are given in order of importance; the most important are placed first.

Introduction: examples

For example, consider the following -- intentionally short and non-specific -- computational workflow specifications:

lsCommands={
"Make a classifier over dsTitanic.", 
"Use the document collection aAbstracts; extract 32 topics",
"Do quantile regression over dfFinData."
};

Here we generate the code -- note the only the list of commands is given to the function Concretize, [AAp1]:

aRes=Concretize[lsCommands];

Here we tabulate the code generation results (i.e. computational templates that are filled-in):

Grid[List/@Flatten[List@@@Normal[KeyMap[Style["\n"<>#,Blue,FontFamily->"Times",FontSize->16]&,aRes]]],Dividers->All,FrameStyle->Directive[Gray],Alignment->Left]

Pareto within Pareto

We can go even further and say that within each ML workflow 75% of the time we use the same workflow.

Question Answering System (QAS)

In this presentation we outline the functionalities and software design of a system that generates executable code based on natural language specifications

The system belongs to the type of software generally known as “template processor” or “template engine”, [Wk2].

The system uses a specialized Question Answering System (QAS), [Wk2], that is based on a generic, open QAS built-in in Wolfram Language (WL) --  see FindTextualAnswer, [WRI1, JL1].

FindTextualAnswer[
"Today it is 45 degrees hot, I think.",
"What is the temperature?"]

The “big picture”

QAS

General/open and specialized/closed QAS can be made in a variety ways.

Here we present a general QAS that is specialized into a “computational workflows” one.

It uses one or several neural networks.

Motivation

Rapid specification of prediction (computational) workflows using natural language commands.

Data science, Machine learning, Scientific computing

Too many packages and languages for doing prediction computations.

Same workflows, but different syntax and “small” details.

Target audience

Data science, data analysis, and scientific computing practitioners.

(Professionals, wannabes, full time, part time, etc.)

Disclaimers

I do not “believe“ that much in using “stochastic” QAS for producing executable code. I think that kind of approach is second or third best.

Nevertheless, I tried to do an “intellectually honest” and “complete enough” effort to make the described approach work and be useful.

At some point, though, I started considering further development to produces too little Return Of Investment (ROI).

The best approach is to use a combination of grammar-based and stochastic-QAS-based methods.

In my view, in such combined system, the precision of responses is handled by the grammar-based part, and the flexibility and generality by the stochastic based part.

Such system would also adhere to the UNIX philosophy principle: “build tools for experienced users.” (While also being useful to new-comers.)

Comparison mind-map

plMindMap=Import["https://github.com/antononcube/NLP-Template-Engine/raw/main/Documents/Diagrams/General/Conversational-agents-with-QAS-demo-mind-map.pdf","PageGraphics"][[1]]

Pareto principle in ML

We can say that 65-85% ML workflows are covered by the four types of workflows:

Classification

Quantile Regression (QR)

Latent Semantic Analysis (LSA)

Recommendations

Within each workflow type 60-75% are “just” one standard workflow.

Pareto principle in ML: LSA

80% LSA workflows are like this:

Concretize["Templates"]["LatentSemanticAnalysis"]["WL"]

Let us concretize it:

Concretize["Make latent semantic analysis workflow for aAbstracts, using 30 topics.","TargetLanguage"->"WL"]

aAbstracts//Length

lsaObj=
LSAMonUnit[aAbstracts]⟹
LSAMonMakeDocumentTermMatrix["StemmingRules"->Automatic,"StopWords"->Automatic]⟹
LSAMonEchoDocumentTermMatrixStatistics["LogBase"->10]⟹
LSAMonApplyTermWeightFunctions["GlobalWeightFunction"->"IDF","LocalWeightFunction"->"None","NormalizerFunction"->"Cosine"]⟹LSAMonExtractTopics["NumberOfTopics"->20,Method->"ICA","MaxSteps"->16,"MinNumberOfDocumentsPerTerm"->20]⟹
LSAMonEchoTopicsTable["NumberOfTerms"->10]⟹
LSAMonEchoStatisticalThesaurus["Words"->Map[WordData[#,"PorterStem"]&,{"notebook","computational","function","neural","talk","programming"}]];

Interactive interface

Here is an interactive interface that gives “online” access to the functionalities discussed: https://antononcube.shinyapps.io/DSL-evaluations/ .

Interactive interface: components diagram

plDSLWebServiceWithWE=Import["https://github.com/antononcube/RakuForPrediction-book/raw/main/Part-0-Introduction/Diagrams/DSL-Web-Service-via-Cro-with-WE-QAS.jpg"];
plDSLWebServiceWithWE=ImageCrop[plDSLWebServiceWithWE]

The interactive interface utilizes:

A server at RStudio’s shinyapps.io

Launched on demand

A server / droplet at DigitalOcean

Permanent

Raku and the Raku package Cro

Wolfram Engine (WE) connected to Raku through ZeroMQ

WE is free for developers

NLP Template Engine Design

plSpecCompl=Import["https://raw.githubusercontent.com/antononcube/RakuForPrediction-book/main/Part-0-Introduction/Diagrams/Computation-spec-completion-by-QAS.jpg"];
plSpecCompl=ImageTake[plSpecCompl,{50,-150},{50,-20}]

Example: Classification

Here is a classification workflow specification and the corresponding template filled-in:

clCommand="Make a classifier for the data dsTitanic and using the method RandomForest. Show TruePositve rate and FalsePositiveRate.";

Here we generate code for the a typical ML classification workflow, [AAp3]:

clRes=Concretize["ClCon",clCommand]

Remark: Note that the correct dataset name and classifier method are filled-in in the ClCon template.

Remark: Due to some wrong answers for the classifier measures different measures are given.

Walkthrough

clRes=Concretize["ClCon",clCommand]

GetRawAnswers["ClCon",clCommand]

GetAnswers["ClCon",clCommand]

Example: Classification (built-in)

Here is a classification workflow specification and the corresponding template filled-in:

clCommand="Make a classifier for the data dsTitanic and using the method RandomForest. Show TruePositve rate and FalsePositiveRate.";

Here we generate code for the software monad ClMon, [AAp3]:

clRes=Concretize["Classification",clCommand]

Here the code generated above is evaluated:

clRes//ReleaseHold

Example: Quantile Regression

Here is QR workflow specification:

qrCommand="Do quantile regression with 22 knots and probabilities 0.1, 0.5 and 0.9 over the dataset dfTemperatureData. Use a datelist plot.";

qrCommand2="Do quantile regression with 22 knots and probabilities 0.1, 0.5 and 0.9 over the dataset dfTemperatureData. Use a datelist plot. With interpolation order 2.";

qrCommand2a="Do quantile regression with 22 knots and probabilities 0.1, 0.5 and 0.9 over the dataset dfTemperatureData. Use a datelist plot. Using the interpolation order 2.";

Here we generate code for the software monad QRMon, [AAp5]:

qrRes=Concretize["QRMon",qrCommand]

qrRes=Concretize["QRMon",qrCommand2]

qrRes=Concretize["QRMon",qrCommand2a]

Here we evaluate the generated code:

qrRes//ReleaseHold

Remark: Note that the correct dataset name, number of knots, probabilities, and data list plot Boolean indicator are filled-in in the QRMon template.

Details

Concretize["Questions"]["QRMon"]

GetRawAnswers["QRMon",qrCommand]

GetAnswers["QRMon",qrCommand]

Bring your own

Templates (you are familiar with)

Questions

Shortcuts

Examples

dsSendMailTemplateEngineData=ResourceFunction["ImportCSVToDataset"]["https://raw.githubusercontent.com/antononcube/NLP-Template-Engine/main/TemplateData/dsQASParameters-SendMail.csv"];
Dimensions[dsSendMailTemplateEngineData]

dsSendMailTemplateEngineData

NLPTemplateEngineAddData[dsSendMailTemplateEngineData]//Keys

Concretize["SendMail","Send email to joedoe@gmail.com with content RandomReal[343], and the subject this is a random real call.",PerformanceGoal->"Speed"]

Workflow classifier

We want to make a classifier for computational specifications texts into workflow types.

In order to be able to give only one argument to Concretize:

I.e. to automatically guess the workflow type of the given specification.

Workflow classifier -- examples

Here are examples:

Concretize["Show me classifier workflow for textual data"]

Concretize["Use the programming language Mathematica and show me classifier workflow for textual data.","TargetLanguage"->Automatic]

Concretize["Make a classifier over some data"]

Concretize["Make a recommender over some data"]

Here is how we obtain the classifier object used by Concretize:

Information[GetComputationalWorkflowTypeClassifier[],"Classes"]

Workflow classifier -- wrong results

The classifier has some deficiencies:

ReverseSort@GetComputationalWorkflowTypeClassifier[]["Get top 23 recommendations for 'project manager' and 'software engineer'.","Probabilities"]

Ideally those deficiencies are traced and evaluated.

Making of a workflow classifier: flow chart

Here is a flow chart that corresponds to the above steps for creation of a text-to-workflow-type classifier:

ImageCrop@Import["https://raw.githubusercontent.com/antononcube/RakuForPrediction-book/main/Part-0-Introduction/Diagrams/Computation-workflow-type-classifier-making.jpg"]

Making of a workflow classifier: training data

The training data can be randomly generated using:

Interlacing manually selected “key” or “signature” words withing sequences of random words (simple)

Simple random sentences of (EBNF) grammars (harder)

Generation of random sentences with grammars

GrammarRandomSentences[LSAMonCommandsGrammar[Normalize->True],12]//

GrammarRandomSentences[QRMonCommandsGrammar[Normalize->True],12]//

Generate interlaced random texts

Clear[RandomTrainingText];
RandomTrainingText[keyWord_String]:=RandomTrainingText[{keyWord},12];
RandomTrainingText[keyWords:{_String..},n_Integer:12]:=
Table[StringRiffle[RandomSample[Join[RandomSample[keyWords,RandomChoice[Range[Length[keyWords]]]],RandomWord["CommonWords",Max[3,RandomVariate[PoissonDistribution[6]]]]]]],n];

RandomTrainingText[{"recommend","recommendations","suggest"}]//

Making of a workflow classifier

Parameters

Number of random sentences per grammar:

nRandomSentences=200;

Resampling for larger texts:

meanNumberOfSamples=3;
numberOfTexts=70;
recursionLimit=12;

Grammar to labels correspondence:

(*ListManagementCommandsGrammar->"DataTransformation"*)

aGrammarToLabelRules=<|LSAMonCommandsGrammar->"LatentSemanticAnalysis",ClConCommandsGrammar->"Classification",QRMonCommandsGrammar->"QuantileRegression",RandomDataGenerationGrammar->"RandomTabularDataset",SMRMonCommandsGrammar->"Recommendations",NetMonCommandsGrammar->"NeuralNetworkCreation",ListManagementCommandsGrammar->"DataTransformation"|>
aLabelToGrammarRules=Association[Reverse/@Normal[aGrammarToLabelRules]];

Generate training and testing data

Here we generate random sentences for each grammar:

SeedRandom[7872];
AbsoluteTiming[
aSentences=Association@KeyValueMap[#1->GrammarRandomSentences[#2[Normalize->True],nRandomSentences,"RecursionLimit"->recursionLimit]&,aLabelToGrammarRules];
]

Here we combine samples for random sentences into “composite” specifications:

aSentences2=Association@KeyValueMap[#1->Table[StringRiffle[RandomSample[#2,Max[1,Round@RandomVariate[PoissonDistribution[meanNumberOfSamples]]]]," "],numberOfTexts]&,aSentences];
aSentences3=Merge[{aSentences,aSentences2},Flatten[{##}]&];

Here are the number of training texts (specifications) for each grammar:

Length/@aSentences3

Here we combine the training data into one association:

aSentences4=Join@@KeyValueMap[Thread[#2->#1]&,aSentences3];
Length[aSentences4]

Train a classifier

Here we train a classifier using a pipeline specification for the software monad ClCon, [AA2, AAp2]:

SeedRandom[7878];
clObj=
ClConUnit[aSentences4]⟹
ClConSplitData[0.75]⟹
ClConMakeClassifier[{"RandomForest","TreeNumber"->30}]⟹
ClConClassifierMeasurements[]⟹ClConEchoFunctionValue["mesaurements:",#&]⟹
ClConClassifierMeasurements["ConfusionMatrixPlot"]⟹ClConEchoFunctionValue["confusion matrix:",Show[#[[1]],ImageSize->Medium]&]⟹
ClConROCPlot[];

Here we take the classifier from the monad object:

specClassifier=clObj⟹ClConTakeClassifier

Alternative workflows classifier

Using Tries with Frequencies

"Fast and compact classifier of DSL commands"

"Trie based classifiers evaluation"

Using recommenders

Eat your own dog food

✓ How the templates are discovered?

✓ How we get examples?

"Show me example of specs"

□ Automatic recognition of introspection queries

Examples

Concretize["Introspection","What are the templates?"]

Concretize["Introspection","Show me an example of QRMon spec"]

Concretize["Introspection","Show me an example of recommendations spec"]

Concretize["Introspection","Show latent semantic analysis workflow template"]

Keys@Concretize["Questions"]
%//Length

Concretize`Private`txtSQASDescription

FindTextualAnswer[Concretize`Private`txtSQASDescription,"What is the template for recommendations"]

Concretize["Introspection","What is the template for recommendations"]

Using context free grammars instead

plMindMap

command="use aAbstracts;
make document term matrix;
show document term matrix statistics;
extract 23 topics using the method SVD;
show topics table;"

lsCommandLines=StringTrim/@StringSplit[command,";"]

ToDSLCode[#]&/@lsCommandLines

SetOptions[ToDSLCode,Method->"Evaluate"]

ToDSLCode["use aAbstracts"]⟹
ToDSLCode["make document term matrix"]⟹
ToDSLCode["show document term matrix statistics"]⟹
ToDSLCode["extract 23 topics using the method SVD"]⟹
ToDSLCode["show topics table"];

ToQuantileRegressionWorkflowCode["use dfTemperatureData"]⟹
ToQuantileRegressionWorkflowCode["echo data summary"]⟹
ToQuantileRegressionWorkflowCode["compute quantile regression with 20 knots and probabilities 0.5 and 0.7"]⟹
ToQuantileRegressionWorkflowCode["show date list plots"]⟹
ToQuantileRegressionWorkflowCode["show error plots"];

Observations, comparisons, conclusions

Why using grammars, not GPT-3, BERT, etc?

Heuristics breed special and corner cases.

Are concrete algorithmic steps and data that address special and corner cases applicable for the “next” versions of those statistical methods/models?

When are we going to get reliable results?

What training and re-training data is going to be used?

How it is obtained? Is it generated?

References

Articles

[JL1] Jérôme Louradour, "New in the Wolfram Language: FindTextualAnswer", (2018), blog.wolfram.com.

[Wk1] Wikipedia entry, Question answering.

[Wk2] Wikipedia entry, Template processor.

Books

[AA1] Anton Antonov, Simplified Machine Learning Workflows, (2018), GitHub/antononcube. 
           (Unfinished.)

Functions

[AAf1] Anton Antonov, GridTableForm, (2020), Wolfram Function Repository.

[AAf2] Anton Antonov, RandomTabularDataset, (2021), Wolfram Function Repository.

[WRI1] Wolfram Research, FindTextualAnswer, (2018), Wolfram Language function, https://reference.wolfram.com/language/ref/FindTextualAnswer.html (updated 2020).

Packages

[AAp1] Anton Antonov, Computational Spec Completion Mathematica package, (2021), MathematicaForPrediction at GitHub/antononcube.

[AAp2] Anton Antonov, Computational Spec Completion Mathematica package, (2021), MathematicaForPrediction at GitHub/antononcube.

[AAp3] Anton Antonov, ClCon, (2021), MathematicaForPrediction at GitHub/antononcube.

[AAp4] Anton Antonov, LSACon, (2019), MathematicaForPrediction at GitHub/antononcube.

[AAp5] Anton Antonov, QRMon, (2019), MathematicaForPrediction at GitHub/antononcube.

[AAp6] Anton Antonov, SMRMon, (2019), MathematicaForPrediction at GitHub/antononcube.

Load packages and data

Packages

Import["https://raw.githubusercontent.com/antononcube/NLP-Template-Engine/main/Packages/WL/NLPTemplateEngine.m"]

Data

Quantile Regression

dfFinData=FinancialData["NYSE:GE",{{2016,1,1},Now,"Day"}];

dfTemperatureData=WeatherData[{"Orlando","Florida"},"Temperature",{{2016,1,1},Now,"Day"}];

Latent Semantic Analysis

dsAbstracts=ResourceFunction["ImportCSVToDataset"]["https://raw.githubusercontent.com/antononcube/SimplifiedMachineLearningWorkflows-book/master/Data/Wolfram-Technology-Conference-2016-to-2019-abstracts.csv"];
aAbstracts=Normal@dsAbstracts[Association,#ID->#Abstract&];

Length[aAbstracts]

RandomSample[dsAbstracts,4]

Classification

dsTitanic=ResourceFunction["ImportCSVToDataset"]["https://raw.githubusercontent.com/antononcube/MathematicaVsR/master/Data/MathematicaVsR-Data-Titanic.csv"];

RandomSample[dsTitanic,6]

Random data generation

Clear[RandomString,RandomPetName];
RandomString[n_Integer,args___]:=Block[{},Table[ResourceFunction["RandomString"][Max[2,RandomVariate[PoissonDistribution[6]]]],n]];
RandomPetName[n_Integer,args___]:=ResourceFunction["RandomPetName"][n];

Grammars

Import["https://raw.githubusercontent.com/antononcube/ConversationalAgents/master/EBNF/English/Mathematica/LatentSemanticAnalysisWorkflowsGrammar.m"];
Import["https://raw.githubusercontent.com/antononcube/ConversationalAgents/master/EBNF/English/Mathematica/ClassifierWorkflowsGrammar.m"];
Import["https://raw.githubusercontent.com/antononcube/ConversationalAgents/master/EBNF/English/Mathematica/TimeSeriesWorkflowsGrammar.m"];
Import["https://raw.githubusercontent.com/antononcube/ConversationalAgents/master/EBNF/English/Mathematica/RandomDataGenerationGrammar.m"];
Import["https://raw.githubusercontent.com/antononcube/ConversationalAgents/master/EBNF/English/Mathematica/RecommenderWorkflowsGrammar.m"];
Import["https://raw.githubusercontent.com/antononcube/ConversationalAgents/master/EBNF/English/Mathematica/NeuralNetworkSpecificationsGrammar.m"];
Import["https://raw.githubusercontent.com/antononcube/ConversationalAgents/master/EBNF/English/Mathematica/ListManagementGrammar.m"];

DSL Mode

Import["https://raw.githubusercontent.com/antononcube/ConversationalAgents/master/Packages/WL/RakuMode.m"]
Import["https://raw.githubusercontent.com/antononcube/ConversationalAgents/master/Packages/WL/DSLMode.m"]

DSLMode[]

Raku process

(*KillRakuSockets[]*)

KillRakuProcess[]
StartRakuProcess["Raku"->$HomeDirectory<>"/.rakubrew/shims/raku"]

Load Raku packages