Patterns in Functional Data Analysis

Alan Calvitti PhD
Head of Research, Statigrafix
alancalvitti@gmail.com

Join the Conversation #WolframTechConf

Outline

Why “functional” data analysis?

Hierarchical and temporal data: commercial example

Tour of Functional Data Analysis API

Not covered in this talk

• Outline, not a how-to // wait for my book

• Important WL technicalities // ascending vs descending operators

• Important aspects of data  // cloud, distributed processing, streaming

Introduction

Background

• Release V10 2014 accelerated our science work at VA San Diego

• Naturally led to continuous dev of API of mostly 1-liners // FDA book

• Now consultant in healthcare data, fintech - application to commercial space

• See Mitch Stonehocker’s related, application-oriented talk: “Healthcare and Finance...”

Why “functional” data analysis?

• “The real hero of programming is the one who writes negative code” - Douglas Mcllroy

• “By relieving the brain of all unnecessary work, a good notation sets it free to concentrate on more advanced problems” -- Alfred N Whitehead [1861-1947]

• “We should have some ways of coupling programs like garden hose - screw in another segment when it becomes necessary to massage data in another way....” 
   -- Doug McIlroy “Functional at scale: applying functional programming principles to distributed computing projects” Comm ACM Dec 2016

• Software engineering // patterns, abstractions to contain, manage complexity

• Aim for simplicity // readable, reusable, modular, negative code = time savings

• Data “pipes and filters” // is a predominant pattern in data; uses composition

WL is uniquely suited to functional paradigm

• Unique symbolic expression rewriting // dynamic typing, optional schema multi-media ready

• Associations // highly optimized key-value stores can contain (almost) arbitrary expressions

• Orthogonal // with rest of language; combine functional and pattern-matching features

Hierarchical Data //  Transaction Log → Report Example

Input data

arafDemo[1][{1},{"Anthetum"},{"unbilled"},{1},{1}]//Normal

Levels 1;;5

arafDemo[1][All,All,Identity,All,Keys]

Levels 5;;8

arafDemo[1][1,"Anthetum", 1,1,{"xsxs"},{"accountSummary"}]

arafDemo[1][1,"Anthetum", 1,1,{"xsxs"},{"actual"}]

“Level Keys” view

{arafDemoGraph[1],arafDemoGrid[1]}// Row

keyCounts view

keyCounts[k_]:=Table[Query[Sequence@@Table[Values,{i}],Keys],{i,0,k}]/*Query[All,Flatten/*sortedCounts];

arafDemo[1][keyCounts[7]]//nmd

Notable features of nested Associations

• Hierarhical DBs predate NoSQL // MUMPS, InterSystems Cache

http://www.datasciencecentral.com/profiles/blogs/mumps-the-most-important-database-you-probably-never-heard-of

Healthcare // Care Centric, Allscripts, Epic, Coventry Healthcare, EMIS, Partners HealthCare (including Massachusetts GH), MEDITECH, GE Healthcare (formerly IDX Systems and Centricity), Sunquest Information Systems,  laboratories such as DASA, Quest Diagnostics,, Dynacare

Finance // Ameritrade, largest online trading service in US (~12bn trans/day), Bank of England, Barclays

Science // MUMPS/Cache to support ESA Gaia billion-star map of Milky Way.

Strengths // Elastic horizontal scaling across multiple low-cost commodity servers; supports big data, analytics, BI (and SQL) beyond RDBMS, ACID/OLTP; no DBAs; low cost resulting from open source code; Flexible data modeling able to easily duplicate the features of RDBMS, key value, columnar, document, and graph architectures.

• Keys need not be strings // Lists, Associations, and combinations in any Association

• Perform logic on keys // key-oriented functions eg keyGroupBy etc - see API below

• Values general WL expressions // Graphics, TimeSeries, Entities, etc

• Nested Associations are malleable // GroupBy, Transpose etc, change levels; lack of schema double-edged sword

Functional Query Pipeline //  Transaction Log → Report

Goal: summary business report table

Pulti-stage data and pipe schematic

pipelineGraph

<|stylize[18]@"(input) arafDemo[1]"-> arafDemoGraph[1],stylize[18]@"(output) arafDemo[3]"->arafDemoGraph[3]|>// nc

Code: ~50% of actual workflow

arafDemo[2 ] =arafDemo[1][All,All,associationKeyValueMap[Function[{bucket,data},Qu[All,All,Query[Append[#,
<|"model"-> <|"Payment"->PAMBucketModel[bucket][#["accountSummary","report_group"],#["accountSummary","account_basis"],#["accountSummary","plan_code"],#["actual","history","Charge"]]|>|>]&]][data]]]];

arafDemo[2.1] = arafDemo[2][All,All,All,All,GroupBy[Query["accountSummary","account_basis"]],GroupBy[Query["accountSummary",{"report_group","account_basis","plan_code"}/* Values/* Replace[{rg_,ab_,pc_}:> capDS[planKeys,1][rg,ab,Lookup[#,pc,"low_support_plan"]&]]]]];

arafDemo[2.2] = arafDemo[2.1][All, All, All, All, All, All, Query[{Query[Total,KeyDrop["accountSummary"]],
Query[First,"accountSummary",{"report_group","account_basis"}/*Values/*Apply[Normal[ePPP[M7, 2]]]]/*associateToKey["PTM"],Query[Keys/*associateToKey["account_pool"]]}/*Apply[Join]]];

arafDemo[2.3 ] = arafDemo[2.2][All,All,All,All,All,All,{"PTM",KeyDrop["PTM"]}/* Replace[{ptm_,data_}:> queryAt["model","Payment",#*ptm&][data]]];

arafDemo[2.35] = arafDemo[2.3][All,All,All,All,Values/* Values/* Merge[Identity]/* Query[{"actual"-> Merge[Total],"model"-> Nest[Merge,Total,2],"account_pool"-> Apply[Join]}]];

arafDemo[2.37] = arafDemo[2.35][All, All, All,  associationKeyValueMap[Function[{dasof,data},Query[queryAt["model","Payment",KeyMap[#+dasof&]]][data]]]];

arafDemo[2.38]= arafDemo[2.37][All,All,All,Values/* Merge[Identity]/* keySplitQuery["account_pool",Query[All,Apply[Join]],Query[All,Nest[Merge,Total,2]]]/* Apply[Join]];

arafDemo[2.4]= arafDemo[2.38][All,All,All,{"model"-> Query[{"Payment"},keyGroupBy[Query[Sign/*Replace[{0|-1->"history",1->"future"}]]]]/*(Transpose[#]&)}];

arafDemo[3] = arafDemo[2.4][All,All,All,{Identity,{Query["actual","history",{"Payment"}],Query["model","history",{"Payment"},Total]}/* Apply[Subtract]/* associateToKey["trueup"]}/* Apply[Join]];

Notable features of Query pipes

• Is the sequence ds[k+1] = ds[k][Q] a pipe? need to export intermediates

• Functional composition used extensively //  RightComposition: Q1 /* Q2 /* Q3

• Composition, processing at multiple levels coexist as subqueries // ds[Q1, Q2 /* Q3]

• High degree of abstraction and efficient indexing // Sequence of All in ds[All,All,All,... Q]

• Uses reusable 1-liner helper functions // keyGroupBy, queryAt, associationKeyValueMap...

• Operators streamline syntax // postfix; minimizes f[#]&; pseudo-op ‘bullet’ function in API

• Orthogonal with WL // combine w/ pattern matching functions, eg Replace to merge branches

• Raises innovation opportunities // how to store “LevelKeys”, metadata, other DB structures, eg efficient column, graph, triple-stores

Functional Data Analysis API // helper functions

Selected functions in API

associationAccumulate, associationFold, associatationIndex, associationKeyUnion, associationPartition, associationReverse, associationSplitBy, deleteEmpty, filterComplementKeys, filterKeys, keyCounts, keyFlatten, keyGroupBy, keyLookup, keyOperate, keyPath, keyPop, keyPush, keyRegroupBy, keyReplace, keySpan, keySplit, keySplitQuery, keySubKeyMap, keyTrie, keyValueOuter, keyValueReverse, nestedMerge, queryAt, queryMerge, transposeCommute...

Pseudo-operator form

• /: h_[pre___,•,post___]:= Function[expr,h[pre,expr,post]];

keyGroupBy

keyGroupBy[f_][expr_]:=Association/@GroupBy[Normal@expr,Keys/*f];

[note also use of bullet operator above]

{as,"            ",as[keyGroupBy[StringTake[•,-1]]]}

Toy dataset

as = <|"a1"-> <|"x"-> 1,"y"-> 2|>,"b1"-> <|"x"-> 3|>,"b2"-> <|"y"->4|>|>//Dataset

keySubKeyMap

jointKeyValueApply[f_][k_-> v_]:= k-> f[k,v];

associationKeyValueMap[f_]:= AssociationMap[jointKeyValueApply[f]];

{as,"            ",as[All,KeyValueMap[Subscript]],"            ",as[All,associationKeyValueMap[Subscript]]}

keySubKeyMap[f_][as_Association]:=associationKeyValueMap[KeyMap[#1][#2]& /* KeyMap[Replace[k_[v_]:> f[k,v]]]][as];

{as,"            ",as[keySubKeyMap[List/* StringRiffle[•,"-"]]]}

keyValueOuter

keyValueOuter[f_,g_][as_List]:= as // Query[{Map[Keys]/*(Outer[f,Sequence@@#]&),Map[Values]/*(Outer[g,Sequence@@#]&)}] // Transpose // Map[Query[{First,Last/* Transpose}]/*Transpose/*Map[Apply[Rule]]]//Flatten//Association;

keyValueOuter[{f_,fOpts__},{g_,gOpts__}][as_List]:= Module[{keyout,valout},
keyout =   as // Map[Keys] /*(Outer[f,Sequence@@#,fOpts]&) ;
valout = as // Map[Values]/* (Outer[g,Sequence@@#,gOpts]&);
{keyout,valout}// Transpose // Map[Transpose/* Map[Apply[Rule]]]//Flatten//Association
];

keyValueOuter[f_,{g_,gOpts__}][as_List]:= keyValueOuter[{f,1},{g,gOpts}][as];

keyValueOuter[{List/*StringRiffle[•,"-"],1},{List/*Merge[Total],1}][{Normal[as],Normal[as]}]// Map[Dataset]

Functional Data Analysis API // Recursive Queries

Tries, filesystem map

"http://mathematica.stackexchange.com/questions/83507/handy-flexible-directory-tree-operations/84952#84952"

trie =Query[GroupBy[Query[First]],If[First@#=={},{},trie@#]&,Rest];

keyTrie = Query[keyGroupBy[First]]/* Query[All,KeyMap[Rest]]/* Query[All,If[Keys[#]=={{}},First@Values[#],keyTrie[#]]&];

fileNamesMap[form_,cleanerf_,path_]:= FileNames[form,path,Infinity] // AssociationMap[ Drop[StringSplit[#,"/"],Length@StringSplit[path,"/"]]&]   // AssociationMap[Reverse]// KeyMap[MapAt[Query[cleanerf],•,-1]]// keyTrie;

fileNamesMap[form_,cleanerf_][path_]:= fileNamesMap[form,cleanerf,path];

Dataset["~/Desktop"][fileNamesMap["*.png",StringSplit[•,"."]/* Most/* StringRiffle[•,"."]]]

nestedMerge // “How to organically merge nested associations?”  stackexchange → goo.gl/o38DYV

https://mathematica.stackexchange.com/questions/102466/how-to-organically-merge-nested-associations/154855#154855

nestedMerge

nestedMerge[f_] := Query[Merge[Identity]/* Query[All,{MatchQ[{__Association}],Identity}/* Replace[{{True,data_}:>nestedMerge[f][data],{False,data_}:> Query[f][data]} ]]];

keyRegroupBy

keyRegroupBy[f_]:= keyGroupBy[f]/* Query[All,Values/* nestedMerge[First]];

M.SE example:

peopleFacts=<|alice-><|age->29,shoeSize->7|>,bob-><|age->27,sex->male|>|>

{peopleFacts,<|steve-><|hairColor->red|>|>,<|bob-><|age->22|>|>,<|steve-><|major->physics|>|>}//nestedMerge[Last]//Dataset

Quad tree index  // “How to use classical data structure in Mathematica” stackexchange → goo.gl/MEbwKj

groupByQuad[bucketSize_][box_Rectangle,data_List]:=If[Length[data]>bucketSize,data//GroupBy[Thread[#pos>boxMean[box]]&/*boxClassify[box]]//associationKeyValueMap[groupByQuad[bucketSize]],data];

Graphic // Monte Carlo simulations suggest n*log[n] time complexity // see M.SE post for details and results

output // nested association

<|Rectangle[{-1.44554,-1.22553},{1.85318,0.992471}]-><|Rectangle[{0.203821,-0.116532},{1.85318,0.992471}]-><|Rectangle[{0.203821,0.43797},{1.0285,0.992471}]->{<|pos->{0.315051,0.992471},foo->0|>},Rectangle[{0.203821,-0.116532},{1.0285,0.43797}]->{<|pos->{0.496316,-0.0548369},foo->0|>}|>,Rectangle[{-1.44554,-0.116532},{0.203821,0.992471}]-><|Rectangle[{-0.62086,-0.116532},{0.203821,0.43797}]-><|Rectangle[{-0.20852,-0.116532},{0.203821,0.160719}]->{<|pos->{0.0416775,0.0817036},foo->0|>},Rectangle[{-0.62086,-0.116532},{-0.20852,0.160719}]->{<|pos->{-0.559806,-0.106714},foo->0|>}|>,Rectangle[{-1.44554,-0.116532},{-0.62086,0.43797}]->{<|pos->{-1.44554,-0.112904},foo->0|>}|>,Rectangle[{-1.44554,-1.22553},{0.203821,-0.116532}]-><|Rectangle[{-1.44554,-1.22553},{-0.62086,-0.671033}]-><|Rectangle[{-1.0332,-1.22553},{-0.62086,-0.948284}]-><|Rectangle[{-0.82703,-1.22553},{-0.62086,-1.08691}]-><|Rectangle[{-0.723945,-1.22553},{-0.62086,-1.15622}]->{<|pos->{-0.646667,-1.22553},foo->0|>},Rectangle[{-0.723945,-1.15622},{-0.62086,-1.08691}]->{<|pos->{-0.654308,-1.09372},foo->0|>}|>|>|>,Rectangle[{-0.62086,-1.22553},{0.203821,-0.671033}]->{<|pos->{-0.431171,-1.08481},foo->0|>},Rectangle[{-1.44554,-0.671033},{-0.62086,-0.116532}]->{<|pos->{-1.24569,-0.514815},foo->0|>}|>,Rectangle[{0.203821,-1.22553},{1.85318,-0.116532}]->{<|pos->{1.85318,-1.01392},foo->0|>}|>|>

Conclusions

• WL “Hyper High Level Language” // Alan Kay; abstraction

• Data features orthogonal with WL // modular, compact, reusable codes

• Fertile ground for innovation // eg, “LevelKeys”

• Functional Data Analysis // wait for book & video lectures for tutorials

[[Abstract and Bio]]

Abstract

Mathematica 10 introduced innovative features for data, including Associations for relational and hierarchical data, with keys that can be almost arbitrary expressions, along with a range of new methods and query mechanisms.In this talk, based on material from the forthcoming book : "Functional Data Analysis: Practical Methods for Science Workflows" I describe some common programming and query patterns encountered in NoSQL data workflows using a functional approach.I focus on hierarchical and temporal logic on multi - GB scale data.The modular, orthogonal nature of the Wolfram Language is illustrated via a range of highly reusable one - liner helper functions developed during the course of real world scientific and commercial work, primarily in healthcare and fintech domains but of general applicability - The resulting API will be freely available with the book.Topics include recursive queries, navigating and processing deeply nested and ragged datasets, utilizing composite keys, combining and mapping data from different branches, handling missing values, leveraging pattern matching, and advanced operators.The presentation is rounded out with a list Key counterparts of existing functions and desirable extensions to the Association model to help automate redundant tasks and reduce programmer errors.

Bio

I have a computer engineering PhD from CWRU, and have used Mathematica since version 2. Focus areas are temporal data analysis and visualization. Currently I am consulting in the healthcare and fintech data space. Previously I was Statistician at Veterans Affairs San Diego and USCD where I helped developed a cloud-based collaborative workflow centered around Mathematica and participated in VA national initiatives to redesign health IT user interfaces.