A Tour of Functional Data Workflow: 
The Book, The Videos, The Camp

Alan Calvitti PhD
Data Scientist, Statigrafix
alancalvitti@gmail.com

Join the Conversation #WolframTechConf

Functional Data Workflow // Talk Overview

Book Title:

Functional Data Workflow: 
    Practical Methods with a Time-Motion UX Case Study

Motivation

Content & Organization

Focus on Case Study [Healthcare]

Functional Programming Patterns

Planned Tutorial Videos, Coding Camp

Functional Data Workflow // Motivation

Significant Case Study / End-to-End Analysis

Many books on data use toy datasets and episodic chapters

Near total coverage on project data processing

Significance: results used in VA national Health IT redesign initiatives and IT prototyping

Showcase Wolfram Language

Mathematica V10 [2014] introduced powerful features for data

Compact, reusable codes; operator pipelines

Make the case for WL for data science

Intended Audience

Data Scientists and Statisticians

Grads and Advanced Undergrads

Prerequisite: exposure to WL, Functional Programming

Functional Data Workflow // Content Organization

Time-Motion UX Case Study

Topics

Data & metadata management

I/O, indexing, prep, wrangling

Statistical analysis, visualization

Results management

Not Emphasized

Not focused on math [eg Machine Learning]

Not the place for Big Data issues [eg Map-Reduce]

Exception: FICO 2018 xML Data Challenge [Excursion]

Excursion Chapters

Self-Contained utility chapters

Additional small data analysis projects [FICO 2018 xML Challenge]

Functional Code Library

In development since  2014 /  tested and stable

Applied in other projects [eg fintech startup]

Mostly 1-3 liner operators

SystemOpen["~/Google Drive/STATIGRAFIX/DataTools/StatigrafixDataTools.nb"];

Wolfram Data Repo

https://datarepository.wolframcloud.com

Raw data + Intermediate datasets ready for analysis chapters

Can query metadata prior to import

Functional Data Workflow // Chapters

[[Code to Open Chapters]]

NotebookEvaluate["~/Google Drive/STATIGRAFIX/DataTools/StatigrafixDataTools.nb"];

$chaptersDir = "~/Google Drive/STATIGRAFIX/FunctionalDataWorkflow/Chapters/";

bookChapters = FileNames["*nb",$chaptersDir] // AssociationMap[Identity]//KeyMap[StringSplit[•,"/"]/* Last/* StringSplit[•,"."]/*First];

openChapter[chapterNum_String]:= SystemOpen[bookChapters[chapterNum]];

How to Query Associations vs Dataset

Query composition is a combination of // and RightComposition

FileNames["*nb",$chaptersDir]//Query[1;;3] // AssociationMap[Identity]// KeyMap[StringSplit[•,"/"]/* Last/* StringSplit[•,"."]/*First]

Query is more compact  and uses only RightComposition (/*)

Dataset[ FileNames["*nb",$chaptersDir]][(1;;3)/* AssociationMap[Identity]/* 
KeyMap[StringSplit[•,"/"]/* Last/* StringSplit[•,"."]/*First]]

Introductory Chapters

00. Introduction //  01. The QUICK Time-Motion Study // 02. The Functional Paradigm // 03. Hierarchies and Recursive Queries [Excursion]

openChapter["03"]

Data Prep Chapters

04. Project Index and Code Library // 05. Project Master Table and Dictionaries // 06. Small Data Prep and Census // 07. TimeMotion Data Census // 08. TimeMotion Data Prep 1: Timestamps // 09. TimeMotion Data Prep 2: Synchronization // 10. TimeMotion Data Prep 3: Linking Dictionaries and Reshaping // 11. Keyscaping [Excursion]

openChapter["07"]

Analysis Chapters

12. Population Groups // 13. Simpson’s Paradox [Excursion]  14. Summary Statistics // 15. Correlation and Outer Products // 16. Number Crystals [Excursion] // 17. Distinguishable Marks and Color Cliques [Excursion] // 18. Summary Visualization // 19. Sequential Pattern Analysis // 20. Time-Resolved Pattern Analysis // 21. Simulating Intercoder Reliability [Excursion] // 22. Time-Resolved Correlations // 23. Semi-Structured Data // 24. Exploratory Log Analysis // 25. Wrangling a Document [Excursion] // 26. FICO xML 2018 Data Challenge [Excursion]

openChapter["17"]

Concluding Chapters & Appendix

27. Missing Data Recap // 28. Conclusions // A. Functional Data Tools Library // B. QUICK Project Code Library

openChapter["A"]

Functional Data Workflow // Case Study [Background]

“Quantifying EHR Usability to Improve Clinical Workflow” [QUICK]

Series of similar studies, VA San Diego 2010-2016

Interdisciplinary: Operations Research, Clinical Informatics, UX Usability, Human Factors

Measure doctors’ workflow [time-at-task] and computer use [eg functions used, navigation patterns, efficiency]

A Calvitti “Physician activity during outpatient visits and subjective workflow” J Biomed Informatics 2017

Significance

Real-world data collected during patient visits [not lab setting] at large clinics: VA San Diego, UCSD Health

Results shared with  VA national Health IT redesign initiatives

Results led to prototyping novel UIs

Clinical Data Collection

Almost 3 years, 200+ clinical encounters, 120+ hours of video & EHR screen capture

Functional Data Workflow // Case Study [Data Layers]

Small Data Tables

Project “Master” / data dictionaries / “small” data layers
Multiple visits in Gsheets 
During project enabled co-editing dictionaries in remote meetings

Example: data dictionary [Charleson Comorbidity]

Time-Motion Data Layout

Each visit in own CSV file(s) // Using cloud sync no need for a DBA

Room Video → Human review/coding [60+hr] → Physicians’ discrete behavior events

Room Audio → Human coding → Patient & physician vocalization events

EHR: Screen capture, Mouse, Keyboard, Eyetracking → Human coding → EHR Activity

Example Coded Clickstream Data File

Timestamped / semi-structured / ragged / hierarchical data

Project Data Index

Main hierarchy

quick[figures,"Aux","quick[index] Graph","Step 2"]

Dictionaries Branch

quick[figures,"Aux","quick[index] Graph","Dictionaries Step 2"]

Functional Data Workflow // Case Study [Time-Motion Viz]

Visualizing Workflow in an Example Visit

Functional Data Workflow // Functional Paradigm

Motivational Quotes

The alternative to functional programming is unreasonable and incoherent code. Functional programming is a huge win, even with all these annoyances.
-- Runar Bjarnason “Functional programming is terrible”, 2014 // https://youtu.be/hzf3hTUKk8U?t=1231

“This world of statements is a disorderly one, with few useful mathematical properties. Structured programming can be seen as a modest effort to introduce some order into this chaotic world, but it accomplishes little in attacking the fundamental problems created by the word-at-a-time von Neumann style of programming, with its primitive use of loops, subscripts and branching flow of control.” --  John Backus, 1977 Turing Award

• "It feels like playing with Legos. Constructing from simple pieces that snap cleanly together." -- Bradford

Features of FP

Composition & Factorization

Functions as Objects

Higher Order Functions [Operators]

Lambdas, Curry and Closures...

Compositional

Plug-n-play: the whole is constructed from the meaning of its modular parts

Highly Reusable, Testable, Parallelizable...

Scalable

Interface looks similar at large and small scales

Usable

Less friction and frustration, more predictable, less cognitive load

Advantages of Wolfram Language

Abstraction enabled by Large Scale Symbolic Processing

Code Compactness [Compressed Quad Tree Indexing [Chapter 03] in ~10 lines of code]

Dataflow, Operators and Query Pipes

• Dataflow

Directed Acyclic Graph: Nodes = Data states [analogous to git commits]  / Links = Queries

• Data Intermediates

data[raw]  →  data[stage1] → data[stage2] → ... → data[final]

Operator Forms

• Make it easy to pipeline

Use Patterns not libraries, minimize ad-hoc function definitions

Functional Data Workflow // Example Queries

Time-Motion Data Storage Census

The quick[index] is a key-value dictionary of data layer name pointing to a stored data file

quick[index]["TimeMotion",preLeafMap[quick[query,prep,keySelectMasterVisits] /* KeyValueMap[List]]][associationKeyFlatten][All,GroupBy[Query[First/* quick[master][All,"Site"]]],Query[Total/* UnitConvert[•,"Megabytes"]/* Round[•,0.01],Last/* FileSize]][Transpose][Transpose][1;;12] (* first few rows shown *)

Scaling Statistical Queries to Population Groups

quick[results,"Aim-3","TLX Subscales"][keyGroupBy[quick[groups][All,"Site"]]/*Query[All,Transpose/* Query[All,quantileStats[quartiles]]]]

quick[data]["Demographics","Patient"][keyGroupBy[quick[groups][All,{"Site","SpecialtyGroup"}]/*Normal]][All,GroupBy["AgeDecade"],GroupBy["Gender"],Length][keyGroupBy[Query["Site"]-> Query["SpecialtyGroup"]]/* Query[All,All,KeySort,KeySort]][All,All,Transpose]

Outer Products for Key Value Pairs

quick[results,"Aim-3","TLX outer"]= quick[data]["Surveys","TLX", Transpose][doubler /* Apply[outer[List-> List/* Merge[Identity]]]];

quick[results,"Aim-3","TLX outer"][keyTrie,{associationCorrelate[SpearmanRho/*N],Values}/* Replace[{corr_,pts_}:> ListPlot[pts,AspectRatio-> 1,Epilog-> Inset[Round[corr,0.01],{3,18}]]]]

Summary Visualizations

• Distributions

(quick[results,figures,"Aim-4","Visit Measures 1"][quick[query,group,"JBI"]][keyAssociationTrie
]["PhysicianID",SortBy[Query[Median,"VisitLength"]]][
All,Values/* medianIQRPlot[AxesStyle->Directive[{Gray,Opacity[0.5]}]]/* Show[•,PlotRange-> {{0,45},All}]/*Dataset,"VisitLength"/* UnitConvert[•,"Minutes"]])[1;;-1;;5] (* select rows shown *)

• Time-At-Task

quick[figures,"Aim-1","MouseClick Chromograms"]= quick[data]["TimeMotion","EHR","MouseClicks","SiteCombinedCoder1"][quick[query,group,"JBI"]][keyAssociationTrie]["PhysicianID",All,All,quick[query,visualize,"Aim-1","MouseClick Chromograms"]/* chromogramGraphics[2] ];

quick[figures,"Aim-1","MouseClick Chromograms"][1;;3,Normal/*Column/*Dataset]

• Aggregated Time-at-Task

quick[figures,"Aim-2","JBI Figure 4A"]= {quick[results,"Aim-2","NonVerbal Summary JBI Figure 4A"][keyAssociationTrie]["Site",All,"SpecialtyGroup",All,ReverseSort/* KeyMap[quick[dictionaries]["TimeMotion","NonVerbal","Gazes",associateValuePairs,{"JBIGroup","JBIGroupHTMLColor"}]]/* associationHorizontalRectangles/*Graphics[•,PlotRange-> {{0,25},All},Frame->{True,False,False,False},ImageSize->300]/* Dataset,"medianTimeAtTask"/* QuantityMagnitude],
quick[figures,"Aim-2","JBI Figure 4A Legend"]}// Column

Sequential Pattern Analysis // Clickstream Analysis

quick[results,"Aim-1","Clickstream Transition Graph","Step 3"][All,Key[True]/* Keys/* Map[Apply[Rule]]/* Graph[•,ImageSize-> 400,GraphLayout->"CircularEmbedding",VertexLabels->"Name"]]

Epic Log Analysis

quick[data]["TimeMotion","EHR","EpicLogs","Longitudinal",{"Q178","Q234","Q157","Q129","Q201"},"Encounter",GroupBy["Clinician_Specialty"] ,GroupBy[Query["LogDateTime"/* Quotient[•,Quantity[7,"Days"]]]],All,{"Provider_Type"/*Replace[Normal@Normal@providerTypeMarks],"Encounter_type"/* Replace[Normal@Normal@encounterTypeHTMLColor]}/* Reverse][All,All,KeyValueMap[List/*Replace[{day_,data_}:> Inset[Graphics[{Thickness[0.25],Normal@data}],{day,0},{0,0},1]]]][All,All,Replace[insets_:> Graphics[{White,Line[{{-52,0},{0,0}}],insets},GridLines->{Range[-52,0],None},Background->LightGray,AspectRatio->1/40,ImageSize->800]]/*Dataset][KeyMap[quick[data]["Demographics","Patient",All,{"PatientID"/* Identity,"AgeDecade"/* Identity,"Gender"/*StringTake[•,1]}/*Column]],keyDeleteMissing][SortBy[Length]/*Reverse]//Normal//Map[Dataset]

Functional Data Workflow // Videos, Coding Camps

Tutorial Videos

• Code Stepping Videos

It’s difficult to convey what parts of a Query do - need video/animation

Currently in planning stage

Coding Camps

• Learn by Doing

• Real World Projects

Q? Suggestions?