Learning from Input and Output

Supervised Machine Learning

© Wolfram Research, Inc.

Outline

A simple example of learning from labeled input

Supervised learning with the Wolfram Language™

Functions that automatically select and build models, using labeled input, for a wide variety of learning tasks

Classify

Predict

Model validation—understand parameters of the model and evaluate its performance

ClassifierInformation

ClassifierMeasurements

Learning a formula to fit the data

FindFormula

Recognize Handwritten Digits

The Digit Classifier

digitClassifier=Classify[
{->2,->5,->8,->0,->2,->7,->5,->1,->3,->0,->3,->9,->6,->2,->8,->2,->0,->6,->6,->1,->1,->7,->8,->5,->0,->4,->7,->6,->0,->2,->5,->3,->1,->5,->6,->7,->5,->4,->1,->9,->3,->6,->8,->0,->9,->3,->0,->3,->7,->4,->4,->3,->8,->0,->4,->1,->3,->7,->6,->4,->7,->2,->7,->2,->5,->2,->0,->9,->8,->9,->8,->1,->6,->4,->8,->5,->8,->0,->6,->7,->4,->5,->8,->4,->3,->1,->5,->1,->9,->9,->9,->2,->4,->7,->3,->1,->9,->2,->9,->6}]

digitClassifier[{,,,,,,,,,}]

Supervised Learning

The Task of Classification

Predict a label for the sample.

Labeled[(Underscript[{"Age", "Income"}, ⏟]),Style["Features",Bold,Darker@Green]]-> Labeled["Yes/No(Credit Risk)",Style["Label",Bold,Darker@Blue]]

Training data is usually a list of labeled samples.

Grid[Join[{Join[Style[#,Bold,Darker[Green]]&/@{"Feature1","Feature2"},{Style["Label",Bold,Darker@Blue]}],Style[#,Bold]&/@{"Age","Income","Credit Risk"}},Table[{RandomInteger[{18,80}],RandomInteger[{30000,90000}], RandomChoice[{"Yes","No"}]},{i,1,5}]],Frame->All]

Infer a function from the data, mapping from feature values to label

Given a new data point, use this function to return a label based on the feature values

Labeled[{"35", "65000"},Style["Sample Features",Bold,Darker@Green]]-> Labeled["No",Style["Predicted Label",Bold,Brown]]

Supervised Learning with Classify

The main Wolfram Language function for classification tasks is Classify.

Example of a Classifier to Recognize a Dog or a Cat

inputData = {->"cat",->"cat",->"cat",
->"dog",->"dog",->"dog"};
dogCatClassifier =Classify[inputData]

dogCatClassifier[]

Decision Making with Classify

Classify can provide either a decision or probabilities for most likely classes.

c=Classify[{
{1.5,Blue}->"A",
{3.2,Blue}->"A",
{4.1,Red}->"B",
{5.3,Red}->"B",
{10.,Green}->"C",
{12.4,Red}->"C"}]

Decision

c[{10.1,Blue}]

Most likely classes

c[{10.1,Blue},"TopProbabilities"]

c[{10.1,Blue},"Probabilities"]

c[{10.1,Blue},"Probability"-> "A"]

Probability of a class as function of a feature

Plot[c[{x,Blue},"Probability"->"A"],{x,0,15},Exclusions->None]

Customizing Classify

Method

Classify can be configured to use one of many well-known classification algorithms.

Logistic regression

Nearest neighbors

Naive Bayes

Support vector machines

Neural networks

Random forests etc.

trainingset = {1,2,3,4,5,6,7}-> {"a","a", "b", "a", "b","b","b"};

Use logistic regression

logistic=Classify[trainingset, Method->"LogisticRegression"]

Use naive Bayes

naiveb=Classify[trainingset, Method->"NaiveBayes"]

Compare the class probabilities for the two classifiers

Plot[{
logistic[x,"Probability"->"a"],
naiveb[x,"Probability"->"a"]
}, {x,0,8}, Exclusions->None,PlotLegends->{"Logistic Regression","Naive Bayes"}]

Customizing Classify

UtilityFunction

A utility function may be specified for predicting each class correctly or incorrectly.

trainingset = {1->"healthy",2->"diseased",3->"healthy",4->"diseased",5->"diseased"};

Train with default settings

c=Classify[trainingset]

c[2.5]

Train with a specific utility function

cWithUtility=Classify[trainingset,UtilityFunction-> 
<|"healthy"-> <|"healthy"-> 1, "diseased"-> 0|>,"diseased"-> <|"healthy"->-1,"diseased"->1|>|>]

cWithUtility[2.5]

The probabilities for individual classes remain the same for both classifiers.

c[2.5,"Probabilities"]

cWithUtility[2.5,"Probabilities"]

Customizing Classify

PerformanceGoal

The performance of a classifier can be optimized for training speed, actual runtime speed, memory usage or accuracy of predictions.

RandomSample[irisData,5]

Optimize training speed

c1 = Classify[irisData,PerformanceGoal->"TrainingSpeed"]

Optimize accuracy of predictions

c2 = Classify[irisData,PerformanceGoal->"Quality"]

Compare the training time

ClassifierInformation[#, "TrainingTime"]&/@ {c1,c2}

More details can be found in the documentation for Classify.

Supervised Learning

The Task of Regression

Compute a target value for a sample.

Labeled[(Underscript[{"Age", "Income"}, ⏟]),Style["Features",Bold,Darker@Green]]-> Labeled["Credit Score",Style["Target",Bold,Darker@Blue]]

The training data contains samples with recorded values.

Grid[Join[{Join[Style[#,Bold,Darker@Green]&/@{"Feature1","Feature2"},{Style["Target",Bold,Darker@Blue]}],Style[#,Bold]&/@{"Age","Income","Credit Score"}},Table[{RandomInteger[{18,80}],RandomInteger[{30000,90000}], RandomInteger[{1,100}]},{i,1,5}]],Frame->All]

Infer a function from the data—mapping from the feature values to the numeric target

Given a new data point, use the regression function to compute a target value optimal for the given features

Labeled[{"35", "65000"},Style["Sample Features",Bold,Darker@Green]]-> Labeled["60",Style["Predicted Target",Bold,Brown]]

Supervised Learning with Predict

The main Wolfram Language function for regression tasks is Predict.

p=Predict[{
{1.3,"P"}->1,
{1.8,"Q"}->2.5,
{1.9,"Q"}->3,
{0.2,"P"}->1,
{-3.2,"P"}->-4.2,
{0.3,"Q"}->2
}]

p[{1.8,"Q"}]

Predicting Output

Example of Predicting Gauge-Reading from Image of the Gauge

trainingGauges =gaugeData[[;;95]];
RandomSample[trainingGauges,3]

predictor=Predict[trainingGauges]

The PredictorFunction can either return the predicted value or a distribution for the predicted value conditioned on the input.

Predicted value

testGauges = gaugeData[[96;;]];
test = testGauges[[1,1]]

predictor[test]

Distribution of the predicted value

predictor[test,"Distribution"]
Plot[PDF[%,x],{x,0.2,0.35}]

Customizing Predict

Method

While linear regression is perhaps the most commonly used regression algorithm, Predict can internally use other algorithms as well.

Linear regression

Nearest neighbors

Neural networks

Random forest

Gaussian process

trainingset = {1->2,2->3,3->5,4->8,5->9,6->7};

Linear regression

linearPredictor=Predict[trainingset, Method->"LinearRegression"]

Nearest neighbors

nnPredictor=Predict[trainingset, Method->"NearestNeighbors"]

Compare results for changing feature value

Plot[{linearPredictor[x], nnPredictor[x]}, {x,0,7},Exclusions->None, PlotLegends->{"Linear Regression","Nearest Neighbors"}]

Customizing Predict

UtilityFunction

The default utility function for Predict is the Dirac delta utility function, which corresponds with the value for the highest probability density being predicted as the target. This can be changed at training or when predicting by setting the UtilityFunction option.

trainingset = {1->1.1,2->4.4,3->6.1,4->7.1,5->9.2};

Default

p1=Predict[trainingset]
p1[2.4]

Custom

A custom utility function that penalizes the predicted value's being smaller than the actual value can be defined as follows.

utility[a_,p_]:=-Piecewise[{{Exp[p-a],a<p},{Exp[3*(a-p)],a≥p}}]

Plot[utility[0,p], {p,-1,2}]

p2=Predict[trainingset,UtilityFunction->utility]
p2[2.4]

Customizing Predict

PerformanceGoal

The performance of the Predict function can be optimized for training speed, actual runtime speed, memory usage or accuracy of predictions.

data = bodyBrainWeights;
RandomSample[data,5]

Optimize for training speed

p1=Predict[data[[All,2]]->data[[All,3]],PerformanceGoal->"TrainingSpeed"]

Optimize for accuracy of predictions

p2=Predict[data[[All,2]]->data[[All,3]],PerformanceGoal->"Quality"]

Compare the time taken for training

PredictorInformation[#, "TrainingTime"]& /@{p1,p2}

By default, a compromise between prediction speed and performance is sought.

p3=Predict[data[[All,2]]->data[[All,3]]]
PredictorInformation[p3, "TrainingTime"]

More details can be found in the documentation for Predict.

More Information about the Classifier

ClassifierInformation or PredictorInformation can be used to extract information about the parameters of the final model.

Train a classifier

Train an SVM classifier.

trainingset={1->"A",2->"A",3->"B",4->"A",5->"B",6->"B"};

c=Classify[trainingset,Method->"SupportVectorMachine"]

Understand the Parameters

ClassifierInformation[c,"Properties"]

Find values of specific parameters

For the SVM classifier, look specifically at the parameters "KernelType", "MulticlassMethod", "SoftMarginParameter" and "GammaScalingParameter".

options=Normal[AssociationMap[ClassifierInformation[c,#] &, {"KernelType", "MulticlassMethod", "SoftMarginParameter", "GammaScalingParameter"} ]]

Retrain the Classifier

By default, Classify chooses the best parameter values that optimize the quality in terms of performance. You can also specify values of your own choice.

cSpecific=Classify[trainingset, Method->{"SupportVectorMachine", Sequence@@options}]

When parameter values are specified, less time is spent on searching the parameter space.

ClassifierInformation[#,"TrainingTime"]& /@ {c,cSpecific}

Evaluating the Model

ClassifierMeasurements or PredictorMeasurements can be used to assess how well the model is performing.

Train a Model

Train a classifier on the Fisher’s Iris dataset to predict the species of the Iris flower.

c=Classify[irisData]

The classifier can now predict the class for a single new sample.

c[{4.3,3.1,1.2,0.3}]

Evaluate the Model

Evaluate the performance using a test dataset.

cm=ClassifierMeasurements[c,irisDataTest]

Accuracy

cm["Accuracy"]

Confusion Matrix

cm["ConfusionMatrixPlot"]

Other metrics of evaluation

“ROCCurve”

“AreaUnderROCCurve”

“FalsePositiveRate”

“Precision”

“Recall”

Details about these and other evaluation metrics can be found in the documentation pages for ClassifierMeasurements and PredictorMeasurements.

A Formula to Fit the Data

FindFormula

A function/formula that approximates the data—a really simple interpretable model.

Example: Find a formula from the first 100 prime numbers

primes = Table[Prime[n],{n,100}];

primes[[{1,100}]]

Find a formula to fit the data.

fit = FindFormula[primes,x]

Compute estimates approximating the next five prime numbers.

{Prime[#],fit /.x->#}& /@ {101,102,103,104,105}//Column

Compare the fit with the data for the 300 primes.

primesNext = Table[Prime[n],{n,100, 300}];

Show[ListPlot[ {primes,Transpose[{Range[100, 300], primesNext}]}, PlotMarkers->Automatic,PlotLegends-> {"First 100 Primes","Next 200 Primes"}], Plot[fit, {x, 0, 300}, PlotStyle->Black], PlotRange->All,PlotLabel-> fit]

FindDistribution

FindFormula has a sister function FindDistribution for the unsupervised learning task of learning a distribution from which the data may have been generated.

Thank You

Initialization

(*gaugeData = Image[AngularGauge[#]]->#&/@RandomReal[1,100];*)
gaugeData={->0.24327106051315028`,->0.07992082096045294`,->0.024432598018887752`,->0.6562541396670079`,->0.7099351911861005`,->0.64117248260226`,->0.011838985868150687`,->0.504163317804045`,->0.724911503950445`,->0.33770415742276283`,->0.7799335914202916`,->0.2537134132340233`,->0.5703364590918116`,->0.6337618837167223`,->0.8243257085564326`,->0.6135041922452988`,->0.4922844865448226`,->0.7625012205781787`,->0.9520685486935256`,->0.1502626882906346`,->0.6567244714114513`,->0.6152272176306046`,->0.2288045949563542`,->0.8853933697831227`,->0.5277378424111197`,->0.014310726546641117`,->0.40137357317009004`,->0.8320534273373859`,->0.1404815137443418`,->0.2919112985024006`,->0.7661245152861218`,->0.5946193343176833`,->0.3728566947594947`,->0.09474360903667467`,->0.004861464717130559`,->0.4017593327660369`,->0.4669455910268956`,->0.8422510251764099`,->0.49275995618991497`,->0.06490946969605527`,->0.015294980746579867`,->0.19184747241005673`,->0.7587054403588323`,->0.5658660847843786`,->0.43518918608667767`,->0.16481440343385767`,->0.5192114124264571`,->0.963323628547138`,->0.038305369895976105`,->0.5706082224862543`,->0.8727481490266167`,->0.39578448991477644`,->0.20098381226688766`,->0.885709176374277`,->0.5366726331248621`,->0.3502778096581536`,->0.02301943565415443`,->0.2590317595946141`,->0.5799441589055025`,->0.26061213414258133`,->0.8609711312269088`,->0.48462392185861924`,->0.6711572577479739`,->0.26502010214135785`,->0.734989996468715`,->0.190325622354935`,->0.5651059538693097`,->0.29662148083601503`,->0.44383017862061713`,->0.3240933033322113`,->0.5828313336367827`,->0.9138162080845744`,->0.38816660358669086`,->0.8368633783049009`,->0.345556161036207`,->0.2010534322760642`,->0.9521247972835922`,->0.3091900430820582`,->0.8103982186857204`,->0.4112928834113583`,->0.27674758722226334`,->0.4803161889486727`,->0.7907865298521513`,->0.8708088485952898`,->0.6316657170129587`,->0.2474133732202599`,->0.38496346959302774`,->0.007657204026337627`,->0.4034369541444003`,->0.47359654514035276`,->0.2552432172348498`,->0.5376568396088597`,->0.29412427651059514`,->0.6371982442479236`,->0.6974237858579782`,->0.21683976165345586`,->0.9519117347507458`,->0.07028713318677648`,->0.10209075707447246`,->0.9341307328323436`};

dayNightData = 
{->"Night",->"Day",->"Night",->"Night",->"Day",->"Night",->"Day",->"Day",->"Night",->"Night",->"Day",->"Night",->"Night",->"Day",->"Night",->"Night",->"Day",->"Day",->"Day",->"Day",->"Night",->"Night",->"Day",->"Night",->"Night",->"Day",->"Day",->"Day",->"Night",->"Day"};

bodyBrainWeights = ExampleData[{"Statistics","AnimalWeights"},"DataElements"];

irisData = ExampleData[{"MachineLearning","FisherIris"},"TrainingData"];
irisDataTest = ExampleData[{"MachineLearning","FisherIris"},"TestData"];