End-to-End Machine Learning Project

Timothee Verdier & Etienne Bernard, Wolfram Research

Join the Conversation #WolframTechConf

End-to-End Machine Learning Project

Data wrangling

Data understanding

Modeling

Use & Deployment

Data wrangling

Importing from a spreadsheet

sales1=Import["ExampleData/RetailSales.tsv","Data"];

sales2=SemanticImport["ExampleData/RetailSales.tsv"]

Scraping from the web

Hacky, often long

Typically:

Store the URLs locally with URLDownload or URLDownloadSubmit

Grab URLs using Import[..., "Hyperlinks"]

Import the URLs with Import or SemanticImport

Usually: Import[..., "XMLObject"]

Parse the data using Position + Part or Cases

The data is always structured

Simple Problem

From http://www.wolfram.com/resources/?source=nav

1. Obtain all the hyperlinks

2. Extract all the titles in the grid (“Wolfram Language Home Page”, “Wolfram Language Reference”, etc.)

a. Using Position + Part

b. Using Cases

url=URL["http://www.wolfram.com/resources/?source=nav"]

links=Import[url,"Hyperlinks"];

file=URLDownload[url];

xml=Import[file,"XMLObject"];

Solution 1.

Position[xml,"Wolfram Language Reference"]

Position[xml,"Wolfram Language Home Page"]

Position[xml,"Wolfram Tweet-a-Program"]

xml[[2,3,2,3,2,3,8,3,16,3,Range[2,18,2],3,{2,4,6},3,3,3,1]] // Grid

Solution 2.

xml[[2,3,2,3,2,3,8,3,16,3,2,3,4,3]]

Cases[xml, HoldPattern[{"\n                ",XMLElement["a",{"shape"->"rect","href"->_},{"\n                    ",XMLElement["img",{"src"->_,"alt"->"","width"->_,"height"->_},{}],"\n                    "}],XMLElement["p",{},{str_String}],"\n                    ",XMLElement["p",{},{_}],"\n                \n            "}]:>str,Infinity]

Data cleaning, interpretation, feature augmentation

Removing bad examples (importation errors, encoding missing data, ...)

Select, Replace

Interpreting variables

Interpreter

Augmenting the dataset with new variables

EntityValue

Example

From the spreadsheet ExampleData/RetailSales.tsv

1. Add the population of each city as a new variable

data=Import["ExampleData/RetailSales.tsv"][[2;;]];

cities=DeleteDuplicates[data[[All,2]]]

interpretedcities=Interpreter["City"][cities]

populations=#["Population"] & /@ interpretedcities

populationsassoc=AssociationThread[cities->populations]

allpopulations=QuantityMagnitude[Lookup[populationsassoc,data[[All, 2]]]];

data2=MapThread[Append,{data,allpopulations}];
Dataset[data2]

Data storage

Organize all your Metadata on a searchable database (path to large files, properties...)

Access you database programmatically to retrieve / generate new dataset: a cloud example

Needs["MongoLink`"]
client=OpenMongoConnection[MongoURIConstruct["ml-data-shard-00-00-hgqzh.mongodb.net:27017,ml-data-shard-00-01-hgqzh.mongodb.net:27017,ml-data-shard-00-02-hgqzh.mongodb.net",27017,"Username"->"WTCtestaccount","Password"->"wolframtest"],"CAFile"->$MongoDefaultCAFile]

db=client["wtctest"]

MongoDatabaseGetCollectionNames[db]

collection=db["titanic"]

Aggregate data. Group by sex and class and show total passenger count and average age:

agg=Dataset[MongoCollectionAggregate[coll, {<|"$group"-><|"_id"-><|"sex"->"$sex", "class"->"$class"|>,"passengers"-><|"$sum"->1|>,"average age"-><|"$avg"->"$age"|>|>|>}]]

Add new elements to your database

newdocuments={\[LeftAssociation]"class"->"1st","age"->23,"sex"->"female","survived"->True\[RightAssociation],\[LeftAssociation]"class"->"3rd","age"->21,"sex"->"male","survived"->False\[RightAssociation]}

MongoCollectionInsert[collection,newdocuments]

Data understanding

Features

Histogram

PieChart

Min val / max val

Fraction of missing features

Trivial baselines

players=Import["https://www.dropbox.com/s/45pgt5fpfhrer7g/players.m?dl=1"];

matches=Import["https://www.dropbox.com/s/d2lz75r7aibvu3w/matches.m?dl=1"];

ByteCount[d]

nplayers=Length[players]
keysplayers=DeleteDuplicates[Flatten[Keys[Values[players]]]]
playersd=Dataset[Values[players]];
Dataset[RandomSample[players,3]]

nmatches=Length[matches]
keysmatches = DeleteDuplicates[Flatten[Keys[Values[matches]]]]
Dataset[RandomSample[matches,3]]

Missing Data

Sort[Transpose[Dataset[Values[players]]][All,Count[#,_Missing]/nplayers &]]

Sort[Transpose[Dataset[Values[matches]]][All,Count[#,_Missing]/nmatches &]]

Distribution of Features

Transpose[Dataset[Values[players]]][4;;, Histogram]

Transpose[Dataset[Values[matches]]][{"Attendance","Score_1","Score_2"}, Histogram]

counts=Normal@Transpose[Dataset[Values[matches]]]["Continent_1", Counts];
PieChart[Values[counts], ChartLabels -> Keys[counts]]

Time-dependency of some important values

s=Lookup[#, {"Score_1", "Score_2","Date"}] & /@ Values[matches];
s={#1+#2, #3} & @@@ s;
s=GatherBy[s, #[[2,1]]&];
s = {#[[1,2,1]], Mean[#[[All,1]]]} & /@ s;
ListLinePlot[s, PlotRange-> {0,5},PlotTheme->"Business",  Filling->Bottom, FrameLabel->{"Year", Automatic}, PlotLabel-> "Mean number of goals per match for each year"]

s=Lookup[#, {"Score_1", "Score_2","TeamRef_1","TeamRef_2", "HomeTeam", "Date"}] & /@ Values[matches];
s={#1-#2, #3===#5, #4===#5, #6} & @@@ s;
s1 = Select[s, !And[#[[2]],#[[3]]]&];
s1 = {Boole[#1!=0]/2,#4}& @@@ s1;
s2 = Select[s, Or[#[[2]],#[[3]]]&];
s2 = {If[#2,Boole[#1<0], Boole[#1>0]],#4}& @@@ s2;
s3 = Select[s, Or[#[[2]],#[[3]]]&];
s3 = {If[#2,Boole[#1>0], Boole[#1<0]],#4}& @@@ s3;
{s1,s2,s3}=Map[
({#[[1,2,1]], N@Mean[#[[All,1]]]} & /@ GatherBy[#, #[[2,1]]&]) &,
{s1, s2, s3}
];
ListLinePlot[{s3,s1, s2}, PlotRange-> {0,0.7},PlotTheme->"Business",  Filling->Bottom, FrameLabel->{"Year", "probability"}, PlotLegends->{"Home", "Neutral", "Away"},PlotLabel-> "Estimated victory probability per year"]

Modeling

Classify / Predict

dataset=Import["https://www.dropbox.com/s/09yzpvt2qixkwfv/dataset.m?dl=1"];

Length[dataset]

training=dataset[[;;25000]];
test=dataset[[25001;;]];

c=Classify[training]

c=ClassifierMeasurements[c,test]

c["Accuracy"]

c["ConfusionMatrixPlot"]

NetTrain

Local

dataset  = (#-> N@(Sin[#]+RandomReal[]*0.2))& /@(RandomReal[1,1000]*2*Pi);

trainSet=dataset[[;;600]];
validSet=dataset[[601;;800]];
testSet=dataset[[801;;]];

RulesToList[dataset_]:=Map[{First[#],Last[#]}&,dataset]

ListPlot[RulesToList/@{trainSet,validSet,testSet}]

net=NetChain[{32,Tanh,1}];

trainedNet=NetTrain[net,trainSet,ValidationSet-> validSet,MaxTrainingRounds->Quantity[5,"Seconds"]]

Show[Plot[trainedNet[x],{x,0,2 Pi},PlotStyle-> Red],ListPlot[RulesToList[testSet]]]

RelPrecision[x_,y_]:=(x-y)/y;

Histogram[RelPrecision[trainedNet[testSet[[;;,1]]],testSet[[;;,2]]]]

On a server

prepare the data for transfer:

CreateDirectory[FileNameJoin[{NotebookDirectory[],"training"}]];
Export[FileNameJoin[{NotebookDirectory[], "training", "trainSet.m"}], trainSet];
Export[FileNameJoin[{NotebookDirectory[], "training", "validationSet.m"}], validSet];
Export[FileNameJoin[{NotebookDirectory[],"training","net.wlnet"}], net];

Prepare a script to launch training

script ="
(*--------TEMPLATE FOR NEURAL NETWORK TRAINING--------*)
(*--------    USER SHOULD SET THE FOLLOWING   --------*)

SetDirectory[FileNameJoin[{Directory[]}]]

netPath = FileNameJoin[{Directory[],\"net.wlnet\"}]	(*remote/path/to/your/net.wlnet*);
trainSetPath = FileNameJoin[{Directory[],\"trainSet.m\"}];	(*\"path/to/your/training_set.m/on/remote/machine\"*);
valSetPath = FileNameJoin[{Directory[],\"validationSet.m\"}];	(*\"path/to/your/ValidationSet.m/on/remote/machine\"*);
checkpointsPath = FileNameJoin[{Directory[]}];	(*\"path/to/your/checkpointing/directory/on/remote/machine\"*);
(*-----------------------------------------------*)

net = Import@netPath;
trainingSet = Import@trainSetPath;
validationSet = Import@valSetPath;

(*EDIT THE NetTrain CALL WITH ANY CUSTOM OPTIONS YOU NEED*)
trainedNet =
	NetTrain[
		net,
		trainingSet,
		ValidationSet -> validationSet,
		TrainingProgressCheckpointing-> {\"Directory\", checkpointsPath, \"Interval\"->Quantity[1,\"Hours\"]},
		TargetDevice->\"GPU\"
	]

Export[FileNameJoin@{checkpointsPath,\"final.wlnet\"}, trainedNet];
";

WriteString[CreateFile[FileNameJoin[{NotebookDirectory[],"training","script.wl"}]], script]

<< "!tar -cvzf training.tar.gz training"

Create an EC2 instance (we will use a machine image AMI with Mathematica already installed)

URL[(URLArgument["https://arg-group.signin.aws.amazon.com/console"])]

Connect to the instance to transfer the script and data to the instance

(we will need a terminal at least once this time to add the ec2 machine into the known host list...)

<<"! scp -i mykey.pem training.tar.gz ubuntu@ec2-52-207-246-94.compute-1.amazonaws.com:~/."

<< "! ssh -i mykey.pem ubuntu@ec2-52-207-246-94.compute-1.amazonaws.com 'tar xvzf training.tar.gz'";

<< "! ssh -i mykey.pem ubuntu@ec2-52-207-246-94.compute-1.amazonaws.com 'math -script ~/script.wl'"

Transfer back the trained net:

<<"! scp -i mykey.pem ubuntu@ec2-52-207-246-94.compute-1.amazonaws.com:~/training/final.wlnet ./trainedNet.wlnet"

remotlyTrainedNet= Import["trainedNet.wlnet"]

Show[Plot[remotlyTrainedNet[x],{x,0,2 Pi},PlotStyle-> Red],ListPlot[RulesToList[testSet]]]

AWSLink is coming!

PacletDirectoryAdd[{"/Users/timotheev/git/awslink/awslink"}]
<< AWSLink`

client = EC2ClientCreate[]

EC2InstancesList[client]

Create an EC2 instance with this image (using - default profile), tag it, terminates all

EC2InstancesRun[client]

EC2TagSet[..., <|"Owner"-> "timotheev"|>]

EC2InstanceTerminate/@...

Spot instances are cheaper

spotPricehistory = EC2SpotInstancePriceHistory[client,"InstanceTypes"-> {"p2.xlarge"},"From"-> DatePlus[Now,-1], "ProductDescriptions"->"Linux/UNIX"]

DateListPlot[spotPricehistory[[1, 1, All]],PlotRange->All,PlotLabel->"2 x K80 GPU spot prices"]

Usage & Deployment

On your machine

Export the model locally

Export["~/classifier.wmlf",c]

Import the model for a new use

c2=Import["~/classifier.wmlf"]

On the cloud

FormFunction / APIFunction

CloudDeploy

types=Normal@ClassifierInformation[c,FeatureTypes]

types2=types /.{"Numerical" -> "Number","Nominal"->"String"}

form=FormFunction[types2,c]

CloudDeploy[form]

api=APIFunction[types2,c]

CloudDeploy[api]

netform=FormFunction[{"image"->"Image"},net[#image] &]

CloudDeploy[netform]

netapi=APIFunction[{"image"->"Image"},net[#image] &]

CloudDeploy[netapi]

On a specific device

Export["mytrainedNet.json",trainedNet, "MXNet"]

Tools to compile MXNet -> device