Building the Automated Data Scientist:
The New Classify & Predict

Etienne Bernard, Wolfram Research

Join the Conversation #WolframTechConf

Automated Data Scientist

Human: “How many croissants are we going to sell next Sunday?”

Computer: “According to your recorded data and other factors such as the predicted weather, there is a 90% chance that between 62 and 67 croissants will be sold.”

Human understanding

Data understanding

Accessing external data

Automatic modeling

Communicating results back

Classify & Predict

First version in 10.0 (3 years ago)

Automatic modeling

Supervised Learning

c=Classify[{1->"A",2->"A",3.5->"B",4->"B"}]

Training Progress Panel

Progress bar

Current values

Learning curves

Stop button

Controlled by TrainingProgressReporting

"Panel"

None

"Print"

TimeGoal

mnist=RandomSample[ResourceData["MNIST"],10000];

c=Classify[mnist]

ClassifierInformation / PredictorInformation

Input type, number of classes, number of training examples, etc.

Quality of the model:

Loss, Accuracy

Evaluation speed

Model memory

Method and hyperparameters

Learning curves

Methods

All methods are documented

New methods:

DecisionTree

GradientBoostedTrees

Improved methods:

NeuralNetwork

LinearRegresion & LogisticRegression

Under the Hood...

New automation procedure

Goal: find best configuration (method + hyperparameters)

Procedure:

Start with a set of configurations

Train configurations on a small dataset (“experiments”)

Predict how well the configurations would perform on the full dataset

Give more data to promising configurations

Training time at least linear -> experiments are almost free

Side effects:

Progress bar

Abortability

TimeGoal

Works even when method & hyperparameters are specified by the user

Model performance estimation

Learning curves

Future of Automation?

Include pre-processing in the procedure

Method combinations

Meta-learning a.k.a. “Learning to learn”