Machine Learning Workshop

Jesse Galef, Giulio Alessandrini WRI

(RowDefault[, (Spacer1), , (Spacer1), ])  #WolframTechConf

ML Pipeline

data import

cleaning

feature visualization

feature selection

training

analysis

Dataset

Student Performance Data Set

https://archive.ics.uci.edu/ml/datasets/Student+Performance

P. Cortez and A. Silva. Using Data Mining to Predict Secondary School Student Performance. In A. Brito and J. Teixeira Eds., Proceedings of 5th FUture BUsiness TEChnology Conference (FUBUTEC 2008) pp. 5-12, Porto, Portugal, April, 2008, EUROSIS, ISBN 978-9077381-39-7. [PDF]

Data Import

Download the dataset from the URL https://archive.ics.uci.edu/ml/machine-learning-databases/00320/student.zip

Extract the archive

Import the student-mat.csv file as a Dataset

Import the student.txt and extract the column numbers, variable names and descriptions

Solution

Set the working directory for convenience

SetDirectory[NotebookDirectory[]]

Download the archive

URLDownload["https://archive.ics.uci.edu/ml/machine-learning-databases/00320/student.zip","student.zip"]

Extract the ZIP archive and import the document within

{mathFile,portugueseFile,merge,notes}=ExtractArchive["student.zip",OverwriteTarget->True]

Import the student-mat.csv file as a Dataset

This CVS file has semicolon separator

Short[Import[mathFile,"CSV"],5]

Convert them to commas to import it as CSV

StringReplace[Import[mathFile,"Text"],{","->".",";"->","}]//Short

CSV import supports automatic conversion to the Dataset format

rawMathData=ImportString[%,{"CSV","Dataset"},"HeaderLines"->1]

Import the student.txt and extract the column numbers, variable names and descriptions

metadata=StringCases[Import[notes,"String"],Shortest[StringExpression[
StartOfLine,
number:DigitCharacter..,
" ",
name:WordCharacter..,
" - ",
description:__,
EndOfLine]]:>{number,name,description}
];
metadata//TableForm

Data Cleaning

Convert the “yes”/”no” values to boolean variables (True/False)

Create a version of the dataset where the G3 column have only the binary values “Pass” (G3 < 10) and  ”Fail” (G3 ≥ 10)

Solution

Convert the “yes”/”no” values to boolean variables (True/False)

dataset=rawMathData;

dataset//=ReplaceAll[{"yes"->True,"no"->False}];

dataset[[1]]//Normal

Create a version of the dataset where the G3 column have only the binary values “Pass” (G3 < 10) and  ”Fail” (G3 ≥ 10)

datasetBinary=dataset[All,{
"G3"->Replace[{x_/;x>=10->"Pass",_->"Fail"}]
}];

datasetBinary[[1]]//Normal

Preliminary Analysis

Feature Types

Find the type of each column based on their content

Find the types automatically using FeatureExtraction and Information

Compute a tally of the found data types and compare the results

Solution

Find the type of each column based on their content

typesManual=dataset//Transpose//Normal//Map[ReplaceAll[{_String->"Nominal",_?NumericQ->"Numerical",_?BooleanQ->"Boolean"}]/*
Counts/*
Sort/*
Keys/*
First]

Find the types automatically using FeatureExtraction and Information

types=Information[FeatureExtraction[dataset],"FeatureTypes"]

Compute a tally of the found data types and compare the results

Counts[typesManual]

Counts[types]

Merge[{typesManual,types},Identity]

Position[%,{a_,b_}/;b!=a]

Preliminary Analysis

Feature Relationship

Plot the column G3 as a function of G1 and of G2

Take only the score columns G1, G2 and G3 and make them all binary with the previous method,

Compute the counts of all the possible triplets

Compute the counts of all the possible couples (G2, G3) when G1 was a passing score

Solution

Plot the column G3 as a function of G1 and of G2

ListPlot[dataset[All,{"G1","G3"}],AxesLabel->Automatic,PlotStyle->Directive[PointSize[Large],Opacity[.5]]]

ListPlot[dataset[All,{"G2","G3"}],AxesLabel->Automatic,PlotStyle->Directive[PointSize[Large],Opacity[.5]]]

Take only the score columns G1, G2 and G3 and make them all binary with the previous method,

scoreBinary=dataset[All,{"G1","G2","G3"}]//ReplaceAll[{x_/;x>=10->"Pass",_Integer->"Fail"}]

Compute the counts of all the possible triplets

Counts@scoreBinary

Compute the counts of all the possible couples (G2, G3) when G1 was a passing score

Counts@scoreBinary[Select[#G1=="Pass"&],{"G2","G3"}]

Feature Visualization

Basic Visualization

Create a dataset with only the column corresponding to numerical features

Create a histogram of each column

Create a pie chart of the nominal features

Create a bar chart of the nominal features

Solution

Create a dataset with only the column corresponding to numerical features

dataset[All,Keys@Select[types,EqualTo["Numerical"]]]

Create a histogram of each column

Histogram/@Transpose[%]

Create a pie chart of the nominal features

SetOptions[PieChart,ChartLegends->Automatic];

PieChart@*Sort@*Counts/@Normal@Transpose[dataset[All,Keys@Select[types,EqualTo["Boolean"]]]]

Create a bar chart of the nominal features

BarChart@*Counts/@Normal@Transpose@dataset[All,Keys@Select[types,EqualTo["Nominal"]]]

Feature Visualization

Dimensionality Reduction

Drop the G3 column and compress each row to 2 dimensions using the UMAP method

Group the 2D feature using the binarized G3 column

Remove  the target and reduce the dataset to a 2-dimensional space to plot it

Plot the two list of points

Compare different dimensionality reduction methods

Solution

Drop the G3 column and compress each row to 2 dimensions using the UMAP method

featuresUMAP=DimensionReduce[KeyDrop[dataset,"G3"],2,Method->"UMAP"];

Short[featuresUMAP]

Group the 2D feature using the binarized G3 column

GroupBy[Thread[{featuresUMAP,target}],Last->First]

Remove  the target and reduce the dataset to a 2-dimensional space to plot it

target=Normal@datasetBinary[All,"G3"];

GroupBy[Thread[{featuresUMAP,target}],Last->First]//Short

Plot the two list of points

ListPlot[%]

Compare different dimensionality reduction methods

features=AssociationMap[DimensionReduce[KeyDrop[dataset,"G3"],2,Method->#]&,{"PrincipalComponentsAnalysis","Autoencoder","UMAP","TSNE"}];

KeyValueMap[
ListPlot[GroupBy[Thread[{#2,target}],Last->First],
Frame->True,Axes->False,ImageSize->Scaled[0.3`],PlotLabel->Style[#1,20],PlotStyle->Directive[PointSize[0.01`],Opacity[0.9`]]]&,
features
]//Multicolumn

Feature Selection

Build a feature extractor directly  from the  data

Test the feature extractor on the first dataset row

Solution

Build a feature extractor directly  from the  data

fe=FeatureExtraction[KeyDrop[dataset,"G3"]]

Test the feature extractor on a dataset row

fe[dataset[1]]

Model Training

Regression

Randomize the rows and split the data in a  training and test set (3/4 and 1/4 respectively)

Train a linear regression model on G3 using the computed feature extractor and an L2 regularization of 0.1

Measure the Root Mean Squared Error on the test set

Solution

Randomize the rows and split the data in a  training and test set (3/4 and 1/4 respectively)

SeedRandom[1234];
{test,train}=TakeDrop[RandomSample@dataset,Ceiling[1/4Length[dataset]]];

Train a linear regression model using the computed feature extractor

predictor=Predict[
train->"G3",
Method->{"LinearRegression","L2Regularization"->0.1},
FeatureExtractor->fe
]

Measure the Root Mean Squared Error on the test set

Sqrt@PredictorMeasurements[predictor,test,"MeanSquare"]

Model Training

Classification

Train a logistic regression classifier on the binarized G3 using the computed feature extractor and an L2 regularization of 0.1

Measure the accuracy on the test set

Train a classifier automatically (no method specification) for 20 seconds using the “Quality” goal and providing the test set for validation

Compare the accuracy of the two classifiers

Solution

Train a logistic regression classifier on the binarized G3 using the computed feature extractor

SeedRandom[1234];
{testBinary,trainBinary}=TakeDrop[RandomSample@datasetBinary,Ceiling[1/4Length[dataset]]];

classifier=Classify[
trainBinary->"G3",
Method->{"LogisticRegression","L2Regularization"->0.1},
FeatureExtractor->fe
]

Measure the Accuracy on the test set

Unprotect[KnownUnitQ];
KnownUnitQ[1]=False;

ClassifierMeasurements[classifier,testBinary,"Accuracy"]

Train a classifier automatically (no method specification) for 10 seconds using the “Quality” goal and providing the test set for validation

classifierAuto=Classify[trainBinary->"G3",ValidationSet->testBinary,PerformanceGoal->"Quality",TimeGoal->Quantity[10,"Seconds"]]

Compare the accuracy of the two classifiers

ClassifierMeasurements[classifier,testBinary,"Accuracy"]

ClassifierMeasurements[classifierAuto,testBinary,"Accuracy"]

Analysis

Classification metrics

Visualize the information panel of the classifier

Visualize the learning curve for all the testes algorithms in the automatic classifier

Create a measurement object using the test set and query the F1 scores

Plot the receiver operating characteristic curve for the “Pass” class

Plot the calibration curve

Solution

Visualize the information panel of the classifier

Information[classifier]

Visualize the learning curve for all the testes algorithms in the automatic classifier

Information[classifierAuto,"LearningCurve"]

Create a measurement object using the test set and query the F1 scores

cm=ClassifierMeasurements[classifier,testBinary]

cm["F1Score"]

Plot the receiver operating characteristic curve for the “Pass” class

cm["ROCCurve"->"Pass"]

Plot the calibration curve

cm["CalibrationCurve"]

Analysis

Regression Interpretability

Plot the distribution of the impact of each feature on the predictor using the test set (FeatureImpactPlot)

Plot the the impact of the “G1”,”G2”,”age”,”sex”,”school” and ”absences”  feature values (FeatureValueImpactPlot)

Plot the cumulative feature impact in on the test rows 1, 5 and 10 (CumulativeFeatureImpactPlot)

Solution

Plot the distribution of the impact of each feature on the predictor using the test set

FeatureImpactPlot[predictor,test,ImageSize->Large]

Plot the the impact of the “G1”,”G2”,”age”,”sex”,”school” and “absences”  feature values

FeatureValueImpactPlot[predictor->{"G1","G2","age","sex","school","absences"},test]

Plot the cumulative feature impact in on the test rows 1, 5 and 10

CumulativeFeatureImpactPlot[predictor,test[{1,5,10}],ImageSize->Large]

References

WL Guides

Machine Learning

Neural Networks

Cluster Analysis

Statistical Model Analysis

Books & Blogs

Introduction to Machine Learning

Wolfram Blog | Machine Learning

Courses

Wolfram U | Machine Learning

Communities

Wolfram Community | Machine Learning

Mathematica Stack Exchange | machine-learning

Repositories

Wolfram Data Repository

Wolfram Neural Net Repository

Wolfram Function Repository | Machine Learning

Thank you!