Machine Learning Automation: Looking for the Next Generation

Timothee VERDIER, Wolfram Research

(RowDefault[, (Spacer1), , (Spacer1), ])  #WolframTechConf

What is Automatic Machine Learning (AutoML)?

The Machine Learning workflow

Supervised:

Classify[{->,->,->,->,->,->,}]

Predict[{->25.3`,->11.`,->21.7,}]

Self-Supervised:

SequencePredict[{"the cat is grey","my cat is fast",}]

Unsupervised:

LearnDistribution[{,,,,,,,,,}]

ClusterClassify[{,,,,,,,,,,,,,}]

The Machine Learning workflow

ClusterClassify[{,,,,,,,,,,,,,}, Method->"Agglomerate"]
Classify[{->,->,->,->,->,->,},Method->"LogisticRegression", PerformanceGoal->"DirectTraining"]

The Machine Learning workflow

ClassifierFunction[][]

ClassifierFunction[][,"Probabilities"]

The Machine Learning workflow

ClusteringMeasurements[{,,,,,,,,,,,,,}-> ClassifierFunction[], "Silhouette"]

ClassifierMeasurements[ClassifierFunction[],{ -> }, "Accuracy"]

What Is Automatic Machine Learning (AutoML)?

The Automatic Machine Learning workflow

What Is Automatic Machine Learning (AutoML)?

Classify[{->,->,->,->,->,->,}]

Predict[{->25.3`,->11.`,->21.7,}]

LearnDistribution[{,,,,,,,,,}]

SequencePredict[{"the cat is grey","my cat is fast",}]

ClusterClassify[{,,,,,,,,,,,,,}]

Arbitrary mixture of input types (to figure out and conform)

{"Audio","Boolean","Color","Complex","Date","Entity","Expression","Graph","Image","Image3D","Integer","Location","Molecule","Nominal","NominalBag","Numerical","NumericalBag","PositiveInteger","Sound","Text","Time","TimeSeries"}\[Cross]{"Vector", "Tensor", "Sequence"}

Automatic features (to engineer)

Model training (parameters to optimize)

Models search and validation (configurations space to sample)

Diversity of tasks means diversity of utility function to accommodate

Basic: focused on model results they do not correspond to any real usage scenario
mean squared error, mean cross entropy, silhouette criterion, ...

Composite: aggregating resource usage (tied to PerformanceGoal)
training and evaluation time and memory, model performances (same as above)

Type inference and feature extraction

c = Classify[{{1.5,,}->"A",{3.2,,}->"A",{4.1,,}->"B",{5.3,Missing[],}->"B",{10.,,}->"C",{12.4,,}->"C"}]

Mostly fixed,  data driven (lot of heuristics) processing pipe-line to extract a numerical or/and a nominal vector.

Blackbox-ish

Information[c, "FeatureExtractor"]

c[[1, "Input", "Processor"]]

Configuration selection

Search strategies

The search space is large and complex, highly varying parameters influence

Prefer RandomSearch over GridSearch (outperformed more and more with growing dimension)

Trainings are usually expensive (in time or computation resources)

Exploit the information of know configuration results to improve on vanilla RandomSearch

Use “any-time” and or parallelizable training schemes

Validation results are noisy (dependant on train-test split, initializations...), which sets worse case scenario for any strategies.

Estimate variance and avoid extreme samples

Configuration selection

Exploit the information of know configuration results to improve on vanilla GridSearch

Prioritize sensible configurations portfolio [warm start optimizer] -> sampler prior

Use early observations to prune search candidates [Multi-resolution optimisation] (SuccessiveHalving, Hyperband, ...)

Use gathered final results to sample new candidates [Bayesian optimisation] (SMAC, Hyperhopt, ...)

Configuration selection

Estimate variance and avoid extreme samples

k-fold validation: Compensate for train-test split choice in the validation measures (at least take their variance into account)

randomization & stratified sampling: train and test data must both be representative of the full data set (data homogeneity, target balance) - even within cross validation runs - this reduce extreme cases wasted computation

Training in operation

SetOptions[InputNotebook[], Magnification-> 1.25]

classifier =Classify[RandomSample[ResourceData["MNIST", "TrainingData"], 20000], TimeGoal->Quantity[3, "Minutes"]]

Training in operation

SetOptions[InputNotebook[], Magnification-> .75]

Information[classifier]

Information[classifier, "LearningCurve"]

MachineLearning`PackageScope`UnrollTrainingInformation[classifier]//Dataset

Future directions

In progress

Reporting

Extra automation for clustering and unsupervised functions

Challenges

Processing pipeline makeover:

high performance underlying data-structures

user exposed

Out of core training

On-line training

Getting to best out of during the training

use Bayesian optimisation in the configuration sampling (heavy resources usage use case)

References

Selected Material

Building the Automated Data Scientist [blog]

An Overview of Machine Learning [course]

Automated Structure Discovery [course]

Automated Data Science [course]

WL Guides

Machine Learning

Neural Networks

Cluster Analysis

Statistical Model Analysis

Books & Blogs

Introduction to Machine Learning

Wolfram Blog | Machine Learning

Courses

Wolfram U | Machine Learning

Communities

Wolfram Community | Machine Learning

Mathematica Stack Exchange | machine-learning

Repositories

Wolfram Data Repository

Wolfram Neural Net Repository

Wolfram Function Repository | Machine Learning