Multi-language Data Acquisition Conversational Agent

Anton Antonov
Accendo Data LLC

(RowDefault[, (Spacer1), , (Spacer1), ])  #WolframTechConf

ABSTRACT

Last year, at WTC-2020, I gave a presentation titled "Multi-language Data Wrangling Conversational Agent". A natural extension of that work is making a Conversational Agent (CA) for Data Acquisition Workflows (DAWs). A closely related problem domain is the acquisition of Machine Learning (ML) models. (For example neural nets models.)

In this presentation we discuss the design of a DAWs CA and exemplify its various facets and subparts:

Gathering and utilizing metadata taxonomies

The making of datasets recommender systems and search engines

In/for both R and WL

Making (ingredient) variables queries

Recommendations by data profiles

Introspection queries

Random data generation specifications

Data obfuscation specifications

Data qualities verification specifications

Extensions to ML models acquisition workflows

We also outline the considered trade-offs and architectural design.

Who am I?

MSc in Mathematics, General Algebra.

University of Ruse, Bulgaria.

MSc in Computer Science, Data Bases.

University of Ruse, Bulgaria.

PhD in Applied Mathematics, Large Scale Air-Pollution Simulations.

The Danish Technical University, Denmark

Former kernel developer of Mathematica, 2001-2008.

Wolfram Research, Inc.

Currently working as a “Senior data scientist.”

Accendo Data LLC

Problem formulation

(These too dry and general... better look into concrete examples ideal end results.)

Data Acquisition System

A system software that manages data storage, data resources, related software resources and provides common services for computer programs for the finding, retrieval, and preparation of datasets.

Data Acquisition Conversational Agent

A conversational agent that prompts, supports, and guides users of a Data Acquisition System.

Simplifications

We are interested in tabular data, or lists or associations of tabular data.

Single line interpreter is an acceptable Minimal Viable Product (MVP).

Target audience

Data science, data analysis, and scientific computing practitioners.

(Professionals, wannabes, full time, part time, etc.)

Ideal end result

Let us consider a Command Line Interface (CLI) perspective.

Imagine we can make the following specification from the command line:

dsl translate -c "get a sample of 3000 JSON commodity files from AWS; 
                  parse them into long form data frame; 
                  make a data package for them; 
                  open a notebook with that data package loaded" | 
dsl data-acquire | 
dsl data-wrangle |
dsl make-notebook

The chain of commands above:

Parses and interprets the natural language commands

And produces, say, JSON or XML records that have executable code.

Gets the commodities files from some default location

Might ask for credentials.

Uses some JSON parsing package in some programming language to make the long form data frame

Makes a data package with the long form data frame

Python and R data packages are regular packages.

WL data package can be a resource function or resource object.

Uploads the data package in some repository

Local packages installation folder or a private cloud.

Creates a notebook and populates with command(s) loading the data package

Opens the notebook in some notebook interpreter.

Jupyter notebooks can be opened in a Web browser or VSCode.

R notebooks, in IntelliJ or RStudio

WL/Mathematica notebooks, in Mathematica

Ideal end result (cont.)

An alternative representation of the workflow is:

dsl interpret -c "get a sample of 3000 JSON commodity files from AWS" |
dsl interpret -c "parse JSON files collection" | 
dsl interpret -c "transform into long form data frame" | 
dsl interpret -c "make a data package" |
dsl interpret -c "open a notebook with that data package loaded"

Here is a diagram that shows the interaction between workflow steps and DAW components:

plDAWCLI=Import["https://github.com/antononcube/ConversationalAgents/raw/master/Projects/DataAcquirer/Diagrams/DAW-CLI-workflow-execution-example.png"]

(The NLP Template Engine has its own repository, [AAr4].)

The chain of commands above:

Parses and interprets the natural language commands

And produces, say, JSON or XML records that have executable code.

Gets the commodities files from some default location

Might ask for credentials.

Uses some JSON parsing package in some programming language to make the long form data frame

Makes a data package with the long form data frame

Python and R data packages are regular packages.

WL data package can be a resource function or resource object.

Uploads the data package in some repository

Local packages installation folder or a private cloud.

Creates a notebook and populates with command(s) loading the data package

Opens the notebook in some notebook interpreter.

Jupyter notebooks can be opened in a Web browser or VSCode.

R notebooks, in IntelliJ or RStudio

WL/Mathematica notebooks, in Mathematica

Example commands and dialogs

We have to gather or make commands and dialogs for such a system.

Here are examples:

lsExampleCommands=
{"recommend customer service data that has product descriptions",
"what data can I get for time series investigations?;why did you recommend those",
"I want to investigate data that cross references good purchases with customer demographics;
keep only datasets that can be transformed to star schema","verify the quality of the database dbGJ99;
what fraction of records have missing data;
what are the distributions of the numerical columns",
"how many people used customer service data last month;
what is the breakdown of data sources over data types;
where textual data is utilized the most;
plot the results;",
"what data people like me acquired last month;
which of those I can use for classfier investigations;
show me the data sizes and metadata;"};
lsExampleCommands2=Map[StringRiffle[StringTrim/@StringSplit[#,";"],"\n"]&,lsExampleCommands];
Multicolumn[lsExampleCommands2,2,Dividers->All]

General design

Similar conversational agents

Food preparation workflows (Sous Chef Susana)

ResourceFunction["RecordsSummary"][dsSCSMeals[Select[AbsoluteTime[DateObject["2020-08-01"]] <= AbsoluteTime[#Timestamp] <= AbsoluteTime[DateObject["2021-08-08"]] &]]]

Recruiting workflows (Head Huntress Gemma)

smrHHGJobs ⟹ SMRMonRecommendByProfile[ {"Skill:java"}] ⟹ SMRMonJoinAcross["Warning"->False] ⟹ SMRMonTakeValue[]

Comparison

aCAComponents=<|
"Data Acquisition Engine"->Import["https://raw.githubusercontent.com/antononcube/Data-Acquisition-Engine-project/main/Diagrams/Data-Acquisition-Workflows-design.jpg"],
"Sous Chef Susana"->Import["https://raw.githubusercontent.com/antononcube/ConversationalAgents/master/Projects/SousChefSusana/Diagrams/Sous-Chef-Susana-design.jpg"],"Head Huntress Gemma"->Import["https://github.com/antononcube/ConversationalAgents/raw/master/Projects/HeadHuntressGemma/Diagrams/Head-Huntress-Gemma-design.jpg"]
|>;
TabView[Show[ImageCrop@#,ImageSize->1200]&/@aCAComponents]

General design

plDAEDesign=Import["https://raw.githubusercontent.com/antononcube/Data-Acquisition-Engine-project/main/Diagrams/Data-Acquisition-Workflows-design.jpg"];

Multi-language in both senses

The Data Acquisition Workflows Conversational Agent (CA) is part of a bigger effort: CAs for computational workflows.

The system is designed to translate multiple natural DSLs into multiple programming DSLs:

Note that the interpreter can be made with WL, Raku, and/or other relevant systems.

Data wrangling demo

... from last year.

Here are data wrangling examples that support the statements above:

commands="
use dfStarwars; 
rename homeworld as HW and age as HOWOLD;
group by species; 
counts;";

aRes=Association@Map[#->ToDataQueryWorkflowCode[commands,"Target"->#,"Execute"->False,"StringResult"->True]&,{"Julia-DataFrames","Python-pandas","R-base","R-tidyverse","Spanish","WL-System"}];

ResourceFunction["GridTableForm"][List@@@Normal[aRes]]

Remark: Shorter code is produced for dedicated data wrangling packages. (As expected.)

See this interactive interface : https://antononcube.shinyapps.io/DSL-evaluations/

Data wrangling demo (cont.)

From last year, WTC-2020, "Multi-language Data-Wrangling Conversational Agent"

Tabular data transformation workflows

Here is a flow chart that shows the targeted workflows:

plWorkflows=ImageCrop@Import["https://github.com/antononcube/ConversationalAgents/raw/master/ConceptualDiagrams/Tabular-data-transformation-workflows.jpg"]

Only the data loading and summary analysis are not optional. (The left-most diagram elements.)

All other steps are optional.

Remark: The Split-Transform-Combine pattern (orange blocks) is implemented in ParallelCombine.

Also, see the article "The Split-Apply-Combine Strategy for Data Analysis" by Hadley Wickham, [HW1].

Conversational agents breakdown

First let us discuss Conversational Agents (CAs) in general.

plCABreakdown=ImageCrop@Import["https://github.com/antononcube/ConversationalAgents/raw/master/ConceptualDiagrams/Conversational-agents-breakdown.jpg"]

Computational Conversational Agents Design (generic)

Here is a generic CA architecture:

plCAFrameBased=Import["https://github.com/antononcube/ConversationalAgents/raw/master/ConceptualDiagrams/Frame-based-conversational-agent-flowchart.jpg"]

Computational Conversational Agents Design

This diagram shows the framework of my current efforts:

plCAMonadic=Import["https://github.com/antononcube/ConversationalAgents/raw/master/ConceptualDiagrams/Monadic-making-of-ML-conversational-agents.jpg"]

Using Finite State Machines

Basically, we move within states of a Finite State Machine (FSM) using different natural language commands interpreters.

Here is an example (from the project OOPDataAcquisitionDialogsAgent):

Import["https://raw.githubusercontent.com/antononcube/ConversationalAgents/master/Projects/OOPDataAcquisitionDialogsAgent/Mathematica/OOPDataAcquisitionDialogsAgent.m"]

daObj["Run"["WaitForRequest",{"get a dataset from Statistics","","the one with 31 rows","","the second","","","quit"}]]

Data acquisition system components

The following mind-maps shows the data acquisition system components:

plGeneralDAWMindMap=Import["https://github.com/antononcube/Data-Acquisition-Engine-project/raw/main/Diagrams/Data-Acquisition-Workflows-mind-map.png"]

Data acquisition conversational agent components

Here is a mind-map Data Acquisition Conversational Agent (CA) components:

plDAWCAMindMap=Import["https://github.com/antononcube/Data-Acquisition-Engine-project/raw/main/Diagrams/Data-Acquisition-Workflows-Conversational-Agent-Modules-mind-map.png"]

Clothes have no emperor

We use the principle “Clothes have no emperor”, we want Data Acquisition Conversational Agents to be have the minimum required awareness of the data:

plDAWComponents=ImageCrop@Import["https://github.com/antononcube/Data-Acquisition-Engine-project/raw/main/Diagrams/Data-Acquisition-Workflows-components-interaction.png"]

Random data generation

There is a dedicated WFR function RandomTabularDataset.

DSL mode

ResourceFunction["RandomTabularDataset"][{10, 3}, "MaxNumberOfValues" -> 20]

NLP Template Engine

Concretize["Make some random table with 10 rows, with 3 columns, and with maximum of 20 values",PerformanceGoal->"Speed"]

ReleaseHold[%]

Through Raku external evaluator

Through a web service

command="generate a random dataset with 10 rows and 3 columns and with 20 max number of values";

ColumnForm@DSLWebServiceInterpretation[command]

Metadata 1

Consider the metadata: https://schema.org/ .

(The data is taken from the GitHub repository: https://github.com/schemaorg/schemaorg .)

dsAllHTTPSTypes=ResourceFunction["ImportCSVToDataset"]["https://raw.githubusercontent.com/schemaorg/schemaorg/main/data/releases/12.0/schemaorg-all-https-types.csv"]

Metadata 2

Here is the corresponding graph:

lsEdges=Map[DirectedEdge[FileNameSplit[#id][[-1]],If[StringQ[#subTypeOf]&&Length[FileNameSplit[#subTypeOf]]>0,FileNameSplit[#subTypeOf][[-1]],#subTypeOf]]&,Normal@dsAllHTTPSTypes];
lsSplitEntities=SplitWordsByCapitalLetters/@Union[Flatten[List@@@lsEdges]];
grAllTypes=Graph[lsEdges,VertexLabels->"Name",ImageSize->2000,EdgeShapeFunction->"CurvedEdge",EdgeStyle->Arrowheads[Medium]]

Metadata Name Entity Recognition

Consider the retrieval and recognition of metadata entities using the following Raku code.

Dataset recommender making 1

“Common” recommender

The recommender’s items are two types: (1) datasets, and (2) columns of datasets.

That principle allows to find dataset similarities and “opportunistic joins.”

Recommendation by profile

Recommendation by profile (i.e. search engine):

smrWLExampleData⟹
SMRMonRecommendByProfile[{"DataType:TimeSeries","ApplicationArea:Finance"},6]⟹
SMRMonJoinAcross[dsWLExampleData,"ID"]⟹
SMRMonTakeValue[]

Retrieve by query elements

Assume that we want to find time series datasets that have “financial data” in their descriptions, but are not with application areas “actuarial” and “aviation”, and do not have “categorical” data:

dsRes=
smrWLExampleData⟹
SMRMonRetrieveByQueryElements[
"Should"->{"Word:financial","Word:data"},
"Must"->{"DataType:TimeSeries"},
"MustNot"->{"ApplicationArea:Actuarial","ApplicationArea:Aviation","ColumnType:Categorical"}]⟹
SMRMonJoinAcross[dsWLExampleData,"ID"]⟹
SMRMonTakeValue[];

ResourceFunction["RecordsSummary"][dsRes[All,ToString/@#&]]

Recommendation by history

Recommend datasets based on previous “dataset consumption”:

smrWLExampleData⟹
SMRMonRecommend[{"Statistics-USLifeTable2003","Statistics-UKLungDiseaseDeaths"},8,"RemoveHistory"->False]⟹
SMRMonJoinAcross[dsWLExampleData,"ID"]⟹
SMRMonTakeValue[]

Note that the history is not removed.

Dataset recommender making 2

Dataset data gathering

In Wolfram Language

In R

Metadata unification

Data identifiers

Column types

Including dataset descriptions

LSA application

Annexing

Recommender representation

Map[MatrixPlot[#,ImageSize->{Automatic,250}]&,smrWLExampleData⟹SMRMonTakeMatrices]

Example datasets interactive interface

Try out this interactive interface that uses a recommender objects over example datasets.

Introspection commands

Scaling over size and complexity

There are two principle ways to scale the MVP version of the data acquisition workflow functionalities:

Large data sizes

Automatic discovery of inter-relationship between datasets

Large data

Consider the following data warehouses or storage services:

Amazon Web Services S3

Google Cloud Storage

BigQuery

A few questions to answer implementation-wise:

What are the ingredients of data packages for accessing the large data?

Using the data itself is not option.

How representative samples of the data are taken and used?

Where the “data scientist environment” is created?

Opportunistic joins

Assume we have a fairly large set of datasets and we want to recommend possible (inner, left, or right) joins between small subsets of them.

Find types and distributions of all variables (dataset columns)

Say, with FindDistrbution

Find variables pairs for which:

The distributions are similar (according to some test)

All values of  one subset of the other

Try to confirm correlations using other “signals”

E.g. dataset descriptions and column descriptions.

Propose "opportunistic joins” to the user

Opportunistic joins (cont.)

Let us illustrate with a set of random tabular datasets.

Generate a few random datasets

SeedRandom[222];

GetDists[]:={PoissonDistribution[RandomReal[{2,19}]],NormalDistribution[RandomReal[{-3,3}],RandomReal[{0.2,12}]],SkewNormalDistribution[RandomReal[{-3,10}],RandomReal[{0,10}],RandomReal[{-3,-3}]]}

aRandomTables=Association@Table[
StringReplace[ResourceFunction["RandomPetName"][],{" "->"-","\""->""}]->
ResourceFunction["RandomTabularDataset"][
{RandomInteger[{30,190}],3},
"Generators"->RandomChoice[GetDists[],3]
],3]

Convert to long format

aLongFormTables=ResourceFunction["LongFormDataset"]/@aRandomTables;

lsLongFormTables2=KeyValueMap[Function[{k,v},v[All,Join[<|"Dataset"->k,"AutomaticKey"->k<>"-"<>ToString[#AutomaticKey]|>,#]&]],aLongFormTables];
lsLongFormTables2

All tables together

dsAllTables3=Join@@lsLongFormTables2

Find distributions for each dataset-variable pair

aColumnDistributions=GroupBy[Normal@dsAllTables3,{#Dataset,#Variable}&,FindDistribution[N[#Value&/@#]]&]

Nearest neighbors

k=50;
nnFunc=Nearest[Keys[aColumnDistributions]->All,DistanceFunction->(1-DistributionFitTest[RandomVariate[aColumnDistributions[#1],k],aColumnDistributions[#2]]&)]

Keys[aColumnDistributions][[1]]
nnFunc[Keys[aColumnDistributions][[1]],3]

Nearest neighbors graph

NearestNeighborGraph[Keys[aColumnDistributions],DistanceFunction->(1-DistributionFitTest[RandomVariate[aColumnDistributions[#1],k],aColumnDistributions[#2]]&),VertexLabels->"Name"]

How the parsing and interpreting are done?

Three kinds of parsing and interpreting are used.

Native WL grammar

Import["https://raw.githubusercontent.com/antononcube/ConversationalAgents/master/Projects/OOPDataAcquisitionDialogsAgent/Mathematica/DataAcquisitionDialogsGrammar.m"]

DADCommandsGrammar[]

Multicolumn[Union@GrammarRandomSentences[GrammarNormalize[DADCommandsGrammar[]],12],3]

Raku grammars

NLP Template Completion

Import["https://raw.githubusercontent.com/antononcube/NLP-Template-Engine/main/Packages/WL/NLPTemplateEngine.m"]

Concretize["Make a random tabular dataset with 20 rows, 5 columns and no more than 40 values."]

%//ReleaseHold

Additional topics

Prefix trees (Tries) with frequencies

Using Prefix Trees with Frequencies we can make frequency-based auto-completions and detect/correct misspellings.

Numeric word forms

SeedRandom[7732];
MapThread[#1->IntegerName[#1,{#2,"Words"}]&,{RandomInteger[{0,1200},5],{"Japanese","Bulgarian","Russian","Spanish","English"}}]

Using FoxySheep and BearGoat (instead of WolfRam)

There is an impressive project named "FoxySheep" for translating WL to Python.

I want(ed) to have something similar that translates WL to Raku.

I want to automate the translation of data acquisition and wrangling unit tests written in WL.

Initially, I thought naming the project BearGoat, but then opted for something more easily understood, "Mathematica::Grammar".

Also, with this kind of grammar it would be easier to translate WL code to English; and from English to translate to other programming languages.

UNIX design similarities

Am I re-inventing UNIX (badly)?

In many ways the presented philosophy and design resembles that of UNIX.

That statement can be seen as “appeal to authority”, but probably is going to introduce and clarify the “message” faster.

Almost of all of Eric Raymond's 17 Unix rules are adhered to:

Magnify[ResourceFunction["ImportCSVToDataset"]["https://raw.githubusercontent.com/antononcube/RakuForPrediction-book/main/Part-0-Introduction/R4P-vs-UnixRules.csv"],1.4]

Connecting Raku to notebooks

Raku cells in DSLMode or RakuMode use an OS process of sandboxed Raku through the library ZeroMQ.

Here is an infographic that summarizes my “journey” of implementing Raku connections to Mathematica and RStudio notebooks:

plMakingRakuConnections=Import["https://github.com/antononcube/RakuForPrediction-book/raw/main/Part-0-Introduction/Diagrams/Raku-hook-up-to-notebooks-journey.jpg"]

The translation execution loop

In this notebook we use the following translation (parser-interpreter) execution loop:

plRakuExecutionLoop=Import["https://github.com/antononcube/RakuForPrediction-book/raw/main/Part-0-Introduction/Diagrams/Raku-execution-in-Mathematica-notebook.jpg"]

Conclusion

Import["https://github.com/antononcube/Data-Acquisition-Engine-project/raw/main/Diagrams/Data-acquisition-conversational-agent-breakdown-mind-map.png"]

Acknowledgements

Kuba Podkalicki

WL notebooks to Markdown export (M2MD). Notebook creation and manipulation. Notebook stylesheets and interactive features.

Diego Zviovich

General discussions on data acquisition workflows and ZeroMQ.

References

Articles, movies

[AA1] Anton Antonov, "How to simplify Machine learning workflows specifications? (useR! 2020)", (2020), MathematicaForPrediction at WordPress.

[HW1] Hadley Wickham, "The Split-Apply-Combine Strategy for Data Analysis", (2011), Journal of Statistical Software.

Repositories

[AAr1] Anton Antonov, Data Acquisition Engine project, (2021), GitHub/antononcube.

[AAr2] Anton Antonov, DSL::English::DataQueryWorkflows Raku package, (2020), GitHub/antononcube.

[AAr3] Anton Antonov, DSL::English::DataAcquisitionWorkflows Raku package, (2021), GitHub/antononcube.

[AAr4] Anton Antonov, DSL::English::FoodPreparationWorkflows Raku package, (2021), GitHub/antononcube.

[AAr5] Anton Antonov, "Raku for Prediction" book, (2021), GitHub/antononcube.

[AAr6] Anton Antonov, NLP Template Engine, (2021), GitHub/antononcube.

[RS1] RStudio, https://www.tidyverse.org.

[RS2] RStudio, https://github.com/tidyverse.

Interactive interfaces

Hosted at  www.shinyapps.io.

[AAii1] DSL evaluations

[AAii2] FindTextualAnswer evaluations interface

[AAii3] Example datasets recommender interface

Videos

[AAv1] Anton Antonov, “Multi-language Data-Wrangling Conversational Agent”, WTC-2020 presentation, (2020), Wolfram at YouTube.

[AAv2] Anton Antonov, "useR! 2020: How to simplify Machine Learning workflows specifications (A. Antonov), lightning", (2020), R Consortium at YouTube.

Code

NLP Template Engine

Import["https://raw.githubusercontent.com/antononcube/NLP-Template-Engine/main/Packages/WL/NLPTemplateEngine.m"]

DSLMode

Import["https://raw.githubusercontent.com/antononcube/ConversationalAgents/master/Packages/WL/DSLMode.m"]

DSLMode[]
KillRakuProcess[]
StartRakuProcess[]

SplitWordsByCapitalLetters

Clear[SplitWordsByCapitalLetters];
SplitWordsByCapitalLetters[word_String]:=StringCases[word,CharacterRange["A","Z"]~~(Except[CharacterRange["A","Z"]]...)];
SplitWordsByCapitalLetters[words:{_String..}]:=Map[SplitWordsByCapitalLetters,words];

SplitWordAndNumber

Clear[SplitWordAndNumber];
SplitWordAndNumber[word_String]:=If[StringMatchQ[word,(LetterCharacter..)~~(DigitCharacter..)],StringCases[word,x:(LetterCharacter..)~~n:(DigitCharacter..):>x<>" " <>n],word];