The Wolfram Data Science Platform: Data Science in the Cloud

MouseOver

MouseOver

Dillon Tracy

Senior Kernel Developer, Wolfram Research

Wolfram Cloud Platforms

Wolfram Programming Cloud: launched summer 2014

Optimized for programming projects, startups, prototyping

Other Platforms: Mathematica Online

Mathematica Online: launching this month

Extends the desktop Mathematica experience into the cloud

The Wolfram Data Science Platform

Emphasis on import-analyze-publish workflows

Why do data science in the cloud?  Collaboration; accessibility and reach.

Cloudifying Computation: API Functions

Create an API function that returns a country map:

f=APIFunction["country"->"Country",GeoGraphics[Polygon[#country]]&, "PNG"];

f[<|"country"->"Chad"|>]

Deploy this to the cloud:

o=CloudDeploy[f,Permissions->"Public"]

Anyone who can make HTTP requests can invoke this function:

Hyperlink[First[o]<>"?country=Italy"]

Computable Documents

“Cloudifying” other concepts, like documents, benefits data science.

Author computable documents in cloud or desktop, view in cloud or desktop

Cloudifying CDF lowers the barrier to getting results in front of decision makers

GraphicsGrid[
Partition[
Tooltip[Button[cdfThumb[#],NotebookOpen[FileNameJoin[{Directory[],"../WDS 2013/CDF-Examples",#}]],Appearance->Automatic,FrameMargins->10],#]&/@cdfSamples,
3,3,{1,1},{}
],ImageSize->600,Dividers->LightGray
]

Data Science Platform Workflows

The Wolfram Data Science Platform supports in-cloud, uploaded, url-based, and external API-based data sources.

A data source:

$url="https://docs.google.com/spreadsheet/pub?key=0An-lvWEXFqTvdElnLUNEOG5QTUJJaUh2aEpabUdIbEE&single=true&gid=0&output=csv";

SemanticImport attempts to recognize entities in structured data:

SemanticImport[$url]

Trees

An inventory of tree species and locations in Champaign, Illinois:

$src="ChampaignTrees-short.csv";

Take[Import[$src],4]//Column

d=Quiet@SemanticImport[FileNameJoin[{Directory[],$src}],{"Plant",Automatic,Automatic,Automatic,Automatic,Automatic},
"Rows"
];

Take[d,3]//Grid[#,Dividers->LightGray]&

LinguisticAssistant["Properties"]//Length

LinguisticAssistant["FoliageColor"]

Document Generators

Computable documents, notebook templating, and scheduled tasks converge in document generators.

A document generator consists of a template, and may also have a schedule and logic for computing template parameters.

Create a basic document generator:

co=CloudDeploy[DocumentGenerator[File["ExampleData/BasicTemplate.nb"],None],"WDS/r1"]

Examine the template:

NotebookPut@Import["ExampleData/BasicTemplate.nb"]

Trigger the generator to run synchronously using EvaluateScheduledTask.  Evaluation is taking place in the cloud.

With[{o=co},EvaluateScheduledTask[o]]

Trigger the generator to run asynchronously using RunScheduledTask.  Evaluation is taking place in the cloud.

With[{o=co},RunScheduledTask[o]]

Document Generators: Templates and Drivers

There are multiple ways to parameterize a generator. One way is by directly supplying a mapping of slot names to values.

The notion of “template” is general, and we can use other kinds of templates besides notebooks:

co=With[{
template=StringTemplate["Time: `time`\nCurrent temperature: `temp`"],
driver=<|"time":>Now,"temp":>AirTemperatureData[]|>
},
CloudDeploy[DocumentGenerator[template,driver,"Hourly"],"WDS/r2"]
]

With[{o=co},EvaluateScheduledTask[o]]

Document Generators: Repeating Elements

Here is a deeper example that builds a report of the quiz scores of a variable number of students. This is the template:

NotebookPut@Import["ExampleData/RepeatingTemplate.nb"]

Simulate scores by sampling them from a NormalDistribution:

quizScore[μ_,σ_]:=Min[Round@RandomVariate[NormalDistribution[μ,σ]],100];
quizScore["science"]:=quizScore[75,8];quizScore["english"]:=quizScore[82,6];quizScore["maths"]:=quizScore[70,13];

Deploy a generator that evaluates three times a week at 17:00 and compiles a report on the day's scores. This one will also email us a PDF of the results:

driver=Block[
{names={"John Smith","Anu Reddy","Sarah Cook","Sam Jones"},subjects={"science","english","maths"},raw},
raw=Table[Prepend[(quizScore[#1]&)/@subjects,n],{n,names}];Association["people"->(AssociationThread[Prepend[subjects,"name"]->#1]&/@raw)]
]&;

co=CloudDeploy[DocumentGenerator[
File["ExampleData/RepeatingTemplate.nb"],
driver,
DateObject[{_,_,Monday|Wednesday|Friday,17}],
"MailOptions"->{
<|"To"->$WolframID,"MessageInsert"->"WDS example","AttachmentFormat"->"PDF"|>
},
"OutputFormat"->"CloudCDF"
],"WDS/r3"]

Peek at what the driver is doing:

driver[]

Manually trigger the generator:

With[{o=co},RunScheduledTask[o]]

Document Generators: Stock Report

Create a document generator that produces a daily report on the performance over the past m days of the n trucking industry stocks with the highest trading volumes on the previous day. Here is the template:

NotebookPut@Import["ExampleData/StocksTemplate.nb"]

Deploy the generator with a suitable driver:

Clear[driver];
driver[m_Integer, n_Integer]:=Block[{members,reportOn},
members=FinancialData["Trucking","Members"];
reportOn=Take[Reverse[SortBy[
({#,Quiet[FinancialData[#,"Volume",{Yesterday}]]}&/@members)/.{{_,_Missing|_FinancialData}->Sequence[],{t_String,{{_List,i_Integer}}}:>{t,i}},
Last
]],n][[All,1]];
<|
"Start":>DateString[DatePlus[Today,-m],"DateShort"],
"Stocks"->(
<|"Name"->FinancialData[#,"Company"],"Ticker"->#|>&/@reportOn)
|>
]&;

co=CloudDeploy[DocumentGenerator[
File["ExampleData/StocksTemplate.nb"],
driver[30,5],
DateObject[{_,_,Tuesday|Wednesday|Thursday|Friday|Saturday,1}],
"MailOptions"->{
<|"To"->$WolframID,"MessageInsert"->"WDS example"|>
},
"OutputFormat"->"CloudCDF"
],"WDS/r4-1"]

The generation time for this report is potentially long, so this is an example of a case where asynchronous evaluation is preferable:

With[{o=co},RunScheduledTask[o]]

Notebook Templating

A weather report template:

o=NotebookOpen[FileNameJoin[{Directory[],"weather report.nb"}]]

Generate a document from the template, supplying our own slot values:

GenerateDocument[o,<|"theCity"->"Washington"|>]

Create a fresh template:

o=CreateNotebook["Template"]

Generate a document from the template:

GenerateDocument[o,<|"x"->"world"|>]

Used extensively in business reporting scenarios

Distribution is only as good as consumers’ ability to view CDF

Cloudifying templating opens possibilities for automated document generation

Scheduled Tasks

Scheduled tasks will run code for you in the background of your session:

n=1;
boxes=Invisible@"x";
Dynamic[boxes]
RunScheduledTask[boxes=Nest[Framed,Invisible["x"],n++],{1,12}];

When this functionality is cloudified, cron-like tasks become possible:

o=CloudDeploy[ScheduledTask[
SendMail["To":>$WolframID,"Subject"->"Current temp","Body"->AirTemperatureData[]],
"Hourly"
],"current temp"]

With[{o=o},RunScheduledTask[o]]

Here is a task that checks the moon phase:

LunarReport[]:=Module[
{phase,phaseText,subject,leader,body},
phase=MoonPhase[Tomorrow];
phaseText=CommonName@MoonPhase[Tomorrow,"Name"];
{subject,leader}=If[phase>0.99,
{"Full moon alert","There is a full moon tomorrow ...\n\n"},
{"Tomorrow's phase: "<>phaseText,phaseText<>"\n\n"}
];
body=StringJoin[leader,"http://www.wolframalpha.com/input/?i=moon+phase\n\n"];
SendMail["To"->$WolframID,"Subject"->subject,"Body"->{body,MoonPhase[Tomorrow,"Icon"]}]
]

Save our functions to the cloud so they are accessible to the task:

obj=CloudSave[LunarReport]

Make a scheduled task that evaluates at 18:00 every day and checks tomorrow’s moon phase:

o=With[{def=obj},CloudDeploy[
ScheduledTask[
CloudGet[def];LunarReport[],
DateObject[{_,_,_,18}]
],
"moon phase check"
]]

With[{o=o},RunScheduledTask[o]]

Questions?

dillont@wolfram.com

Team members also at the conference:
Clayton Voyles
Jeremy Lee
Deepak Balachandar

Init

Init

Needs["JLink`"];

Needs["DatabaseLink`"];

Needs["HadoopLink`"];

Needs["CorporateAnalysisUtilities`"];

SetDirectory[NotebookDirectory[]]

PrependTo[$Path,"../WDS 2013"];

PrependTo[$Path,"../WDS 2013/CDF-Examples"];

PrependTo[$Path,"./cdf thumbs"];

cdfSamples=Complement[
Last[FileNameSplit[#]]&/@FileNames["*.cdf","../WDS 2013/CDF-Examples"],
{"CDF and ReportGeneration.cdf"}
]

Scan[
(cdfThumb[#]=Import[FileBaseName[#]<>".jpg"];&),
cdfSamples
]

Off[FrontEndObject::notavail]

Cloud connection

PacletDirectoryAdd["/Users/dillont/eclipse/workspace/CloudPlatformComponents/apache-tomcat-6.0.41/data/PID-617d2f9e-84de-43c0-b38d-2ac266e8574f/"]

FindFile["CloudObject`"]

<<CloudObject`;

$CloudDebug=True;
$CloudDebugLevel=3;

$CloudDebug=False;
$CloudDebugLevel=3;

$CloudBase="http://localhost/";

$CloudBase="http://www.wolframcloud.com/";

$CloudBase="http://www.devel.wolframcloud.com/";

$CloudBase="http://www.test.wolframcloud.com/";

CloudConnect["dillont@wolfram.com","Marflow17"]

CDF Thumbs

ClearAll[nb];
Scan[
(nb[#]=Import[#];&),
cdfSamples
]

Clear[makeNbThumb];
makeNbThumb[nb_Notebook,width_:175]:=Module[
{nbObj=NotebookPut[nb],res},
res=makeNbThumb[nbObj,width];
NotebookClose[nbObj];
res
];
makeNbThumb[nbObj_NotebookObject,width_:175]:=Module[
{rs,top},
rs=Rasterize[nbObj,"RasterSize"];
top=With[{res=Rasterize[nbObj]},
If[res===$Failed,
Return[$Failed],
ImageTake[res,1.294*rs[[1]]]
]
];
ImageResize[top,width]
]

ClearAll[cdfThumb]

Scan[
(If[!MatchQ[cdfThumb[#],_Image|$Failed],
cdfThumb[#]=makeNbThumb[nb[#]];
]&),
Take[cdfSamples,6]
]

no=NotebookPut[nb[#]]&@cdfSamples[[12]]

t=(cdfThumb[#]=makeNbThumb[no])&@cdfSamples[[12]]

CreateDirectory["cdf thumbs"]

Export["cdf thumbs/"<>FileBaseName[#]<>".jpg",cdfThumb[#]]&/@cdfSamples

Special Cells

Picture Cell

PicturePrint[expr_,opts:OptionsPattern[Cell]]:=CellPrint[TextCell[expr,"Picture",opts,ShowStringCharacters->False]]

Overview Mouseover

OverviewMouseover[header_,subs_]:=CellPrint@Cell[BoxData[ToBoxes@Mouseover[Framed[Style[header,"OverviewSection"],FrameMargins->{{15,15},{5,5}},BoxFrame->2,FrameStyle->None,RoundingRadius->5],

Evaluate@If[subs=="",Framed[Style[header,"OverviewSection",White],FrameMargins->{{15,15},{5,5}},BoxFrame->2,FrameStyle->None,RoundingRadius->5,Background->m8red[1]],Row[{Framed[Style[header,"OverviewSection",White],FrameMargins->{{15,15},{5,5}},FrameStyle->None,RoundingRadius->5,Background->m8red[1]]," » ",Framed[Style[subs,"OverviewSection"],FrameMargins->{{15,15},{5,5}},FrameStyle->m8red[1],BoxFrame->2,RoundingRadius->5,Background->White]}]]]],"OverviewSection"]

(*MapThread[OverviewMouseover,Transpose[{{"Chapter One","Chapter One topics, …"},{"Chapter Two","Chapter Two topics, …"},{"Chapter Three","Chapter Three topics, …"},{"Chapter Four","Chapter Four topics, …"}}]]*)

Large Overview Mouseover

LargeOverviewMouseover[header_,subs_]:=CellPrint@Cell[BoxData[ToBoxes@Mouseover[Framed[Style[header,"LargeOverviewSection",FontFamily->"Helvetica"],FrameMargins->{{15,15},{5,5}},FrameStyle->None,RoundingRadius->5],

Evaluate@If[subs=="",Framed[Style[header,"LargeOverviewSection",White],FrameMargins->{{15,15},{5,5}},FrameStyle->None,RoundingRadius->5,Background->Orange],Row[{Framed[Style[header,"LargeOverviewSection",{FontFamily->"Helvetica",White}],FrameMargins->{{15,15},{5,5}},FrameStyle->None,RoundingRadius->5,Background->Orange]," » ",Framed[Style[subs,"LargeOverviewSection",FontFamily->"Helvetica"],FrameMargins->{{15,15},{5,5}},FrameStyle->Orange,RoundingRadius->5,Background->White]}]]]],"LargeOverviewSection"]

MapThread[LargeOverviewMouseover,Transpose[{
{"Deployment","Scheduling Tasks, Computable Documents, Document Generators"}
}]
];

DG removal

RemoveScheduledTask[CloudObject["WDS/r1"]]

RemoveScheduledTask[CloudObject["WDS/r2"]]

RemoveScheduledTask[CloudObject["WDS/r3"]]

RemoveScheduledTask[CloudObject["WDS/r4"]]

Scratch

t=Import["ChampaignTrees.csv"];

t=Import["ChampaignTrees-short.csv"];

t[[1]]

Export["ChampaignTrees-short.csv",Prepend[RandomSample[t[[2;;]],500],t[[1]]]
]

Directory[]

t=DocumentGenerators[];

DocumentGeneratorInformation/@t

MapThread[LargeOverviewMouseover,Transpose[{
{"The Cloud","Platforms, Motivation"},
{"Components","Computable Documents, Scheduled Tasks, Document Templating"},
{"Workflow","Utilities, Trees, Stocks ..."}
}]
];