Human in the loop: 
Interpretable Machine Learning

- Jesse Galef  jesseg@wolfram.com

~ comic by Christoph Molnar, Interpretable Machine Learning

Machine Learning is Brilliant, Except When It' s Not

“On two occasions I have been asked by members of Parliament:
'Pray, Mr . Babbage, if you put into the machine wrong figures, will the right answers come out?'

Machine Learning is Brilliant, Except When It' s Not

“On two occasions I have been asked by members of Parliament:
'Pray, Mr . Babbage, if you put into the machine wrong figures, will the right answers come out?' 

...I am not able rightly to apprehend the kind of confusion of ideas that could provoke such a question .”

Machine Learning is Brilliant, Except When It’s Not

ML: Good at accomplishing goals
ML: Bad at knowing which goals we want

Specification gaming : the flip side of AI ingenuity (DeepMind Safety Research team, April 2020)

Goal: Stack Red Lego on Blue

Import["/Users/jesseg/Documents/1*hAslNmSRAmDtJDWM3RZVfw.gif", "AnimatedImage"]

ML: Really, really good at spotting patterns in data
ML: Really, really bad at knowing which pattern we want

Help your AI help you help it help you

(HyperlinkTemplate[Humans in the Loop : The Design of Interactive AI Systems - Ge Wang, Stanford HAI Blog, https://hai.stanford.edu/blog/humans-loop-design-interactive-ai-systemsNone, https://hai.stanford.edu/blog/humans-loop-design-interactive-ai-systems, HyperlinkActionRecycled, HyperlinkActive, , ])

1) How well does it work in different situations and 
2) How is it working?

Why We Might Want Decision Explanations:

Identify Specification Gaming

Improve data gathering and feature engineering

Build public trust and buy-in for the model

Identify discriminatory behavior

Suggest ways we can intervene to change the outcome

Develop an understanding of the patterns our algorithm found

Wolfram Mathematica Tools for Interpreting Machine Learning

Interpretable Models

Linear Regression
Logistic Regression

Information
Predictor/Classifier Measurements

ICEPlots

Individual Conditional Expectation plots

SHAPValues & SHAPPlot

Use case on single example
Use case to explore a model
Feature Impact plots
Different types of SHAPValues

Model Measurements

Information & PredictorMeasurements

SeedRandom[42];
wine=RandomSample[ResourceData["Sample Data: Wine Quality"]];
wine[[;;5]]

{winetest, winetrain}=TakeDrop[wine,100];
predictor = Predict[winetrain->"WineQuality", Method->"LinearRegression"]

Information - About the Model

info=Information[predictor]

info["Function"]

info["Properties"]

PredictorMeasurements - Explore with a Testset

wineMeasurements=PredictorMeasurements[predictor,winetest]

wineMeasurements[{"RSquared", "MeanSquare"}]

wineMeasurements["Properties"]

wineMeasurements["ResidualPlot"]

residuals = wineMeasurements["Residuals"];
sulfur = Normal@winetest[[All,"TotalSulfurDioxide"]];

ListPlot[Thread[{sulfur,residuals}],
AxesLabel->{"TotalSulfurDioxide","Residual"}]

worst=wineMeasurements["WorstPredictedExamples"]

ClassifierMeasurements

RandomSeeding[42];
(*titanic=RandomSample[ResourceData[ResourceSearch["Titanic"][[1,1]]]];*)
titanic = RandomSample@ExampleData[{"MachineLearning","Titanic"},"TrainingData"];

titanic[[;;2]]

{titanictest, titanictrain}=TakeDrop[titanic,250];

classifier=Classify[titanictrain]

cm=ClassifierMeasurements[classifier,titanictest]

cm["Properties"]

Individual Conditional Expectation Plot

Keeping all else constant, change one feature and predict the result

treemodel=Predict[wine->"WineQuality", Method->"GradientBoostedTrees"];
treemeasurements = PredictorMeasurements[treemodel, winetest];

treeiceplots = treemeasurements["ICEPlots"];

treeiceplots["FreeSulfurDioxide"]

ICEPlots for three types of model: Linear, Nearest Neighbors, and Boosted Trees:

linearmodel=Predict[wine->"WineQuality", Method->"LinearRegression"];
lineariceplots = PredictorMeasurements[linearmodel, winetest, "ICEPlots"];
neighborsmodel=Predict[wine->"WineQuality", Method->"NearestNeighbors"];
neighborsiceplots = PredictorMeasurements[neighborsmodel, winetest, "ICEPlots"];

ICEPlots={lineariceplots, neighborsiceplots, treeiceplots};

GraphicsGrid[Transpose[{ICEPlots[[All,"Chlorides"]]}]]

ClassifierMeasurements[classifier, titanic, "ICEPlots"]

Use “ICEData” property to explore on your own

ICEData=treemeasurements["ICEData"];

feature="VolatileAcidity";
featurevalues=Normal@ICEData["FeatureValues"][feature];
lines=Normal@ICEData["ConditionalExpectation"][feature];
threaded=Thread[{featurevalues, #}]&/@lines;

alcohols=Rescale[Normal[winetest[[All,"Alcohol"]]]];
alcohols=Rescale@Clip[alcohols,{.15,.85}];
colors=ColorData["TemperatureMap"]/@alcohols;

ICE Plot of Volatile Acidity, Colored by Alcohol Content

ListLinePlot[threaded, PlotStyle->Thread[{Opacity[.65],colors}], PlotRange->{Automatic, {3,8}}]

SHAPValues - SHapley Additive exPlanations

Shapley: Game Theoretic Attribution of Credit in a Group

Named after Lloyd Shapley who won the Nobel Prize in Economics in 2012 for this

Model-Agnostic way to explain each feature’s impact on a decision

Can take any model, even black-box
“Explains” one example at a time
Can find multi-feature dependencies

Idea behind SHAPValues

For every combination of “knowing” each feature or not,
compare the impact of adding a feature

Example: When predicting income from Age, Gender & Job,
What impact did Age have?

Example of SHAPValues Property: 
Explain a complex decision

myBottle=Normal@Rest@wine[[42]]

Print["Mean Quality of all bottles: ",winemean= Mean[winetrain[[All,"WineQuality"]]]//N];
Print["Predicted Quality for my bottle: ", treemodel[myBottle]];

Why does the model say my bottle of wine is worse than average?

shaps = treemodel[myBottle,"SHAPValues"]

BarChart[KeySort@shaps,ChartLabels->Placed[Automatic,Above]]

Total[shaps]

winemean+Total[shaps]

forest[myBottle]

Classifier SHAPValues give the odds multiplier for each feature

(*classifier=Classify[titanictrain->"SurvivalStatus"]*)

classifier

titanic[[1]]

classifier[titanic[[1,1]], "Probabilities"]

shaps=classifier[titanic[[1,1]], "SHAPValues"->50]["survived"]

prior = Information[classifier, "TrainingClassPriors"]

priorodds = prior / (1-prior)

p=classifier[titanic[[1,1]], "Probabilities"]["survived"];
estimatedodds =p/(1-p)

priorodds["survived"]*shaps[[1]]*shaps[[2]]*shaps[[3]]

%==estimatedodds

Used in Concert, SHAPValues can Give Insight Into Models

Length[winetest]

allShaps = treemodel[winetest,"SHAPValues"->3];//AbsoluteTiming

importance = Mean[Abs[allShaps]];
importance = ReverseSort[importance];

BarChart[importance, ChartLabels->Placed[Automatic,Above], PlotLabel->Style["Feature Importance:\nMean Impact on Prediction",24]]

keys = Rest[Keys[winetest[[1]]]];

predictions=treemodel[winetest];

threaded=Transpose/@Thread[{Transpose[Values@Normal@KeyDrop[winetest,"WineQuality"]], Transpose[Values[allShaps]]}];
predthread =Thread[{#,predictor[winetest]}]&/@Transpose[Values@Normal@KeyDrop[winetest,"WineQuality"]];

shapscatter = MapIndexed[ListPlot[#1,PlotLabel->keys[[First@#2]], AxesLabel->{"Feature Value", "Impact of Feature"}, ImageSize->400, PlotMarkers->{Automatic,Small}]&,threaded];
predscatter = MapIndexed[ListPlot[#1,PlotLabel->keys[[First@#2]], AxesLabel->{None,"Predicted Quality"},ImageSize->400, PlotMarkers->{Automatic,Small}]&,predthread];

GraphicsGrid[Thread[{predscatter,shapscatter}]]

PredictorMeasurements[treemodel, wine[[;;350]], "SHAPPlots"]

What do we mean by “If we didn’t have” a Feature?

Philosophically Ambiguous!
Observation vs Intervention, Correlation vs Causation, Conditional vs Marginal

Predicting Heart Attacks from: Age, Blood Pressure, Salt Intake

72 Year-old man, BP: 140/90, High Salt Diet
Model says: Risk of Heart Attack: High

What was the impact of Blood Pressure on our prediction?

If we ask the model to INFER blood pressure: Observational

Fill in the feature(s) given the information we do have

“Doctor, what do you predict for a 72-year old man, BP: 140/90, with a High Salt Diet?”
	“His blood pressure is high, so he’s high risk.”
	
“What if I hadn’t told you his blood pressure?”
	“With that high salt diet I’d figure he has high blood pressure, and still say high risk.”

So... is blood pressure an unimportant feature?

If features are correlated, the predictions will be similar, and impact of Blood Pressure is considered comparatively low and shared with diet.

This is true EVEN IF the model inner workings use blood pressure exclusively!

Can discover hidden proxy variables:
E.g. If a model doesn't use Race but relies on ZipCode

True to the data; not to the model.

If we ask the model to CHANGE blood pressure: Interventional

Fill in the feature(s) with values independent of known values

“Doctor, what do you predict for a 72-year old man, BP: 140/90, with a High Salt Diet?”
	“His blood pressure is high, so he’s high risk.”

“What if I hadn’t told you his blood pressure?”
	“That’s all I go by, so it’s hard to say. I’d have to guess he’s average risk.”
	
“What if we could get the blood pressure down to 110/75?”
	“That’s excellent; he would be low risk.”

If the model only takes blood pressure into account, all “interventional” impact is attributed to it.

This is true to the model, but may involve testing on unlikely or nonsensical data.
(e.g. Assessing an 18 year old’s risk of cancer given 22 years of taking Hormonal Birth Control)

Getting Interventional SHAPValues:

One way: Tell the model your features are independent

distribution = LearnDistribution[KeyDrop[winetrain, "WineQuality"], Method->{"Multinormal","CovarianceType"->"Diagonal"}]

treemodel[winetest[[1]],"SHAPValues", MissingValueSynthesis->distribution]

treemodel[winetest[[1]],"SHAPValues"]

conditionalshaps=treemodel[winetest,"SHAPValues"->3];
interventionalshaps=treemodel[winetest,"SHAPValues"->3, MissingValueSynthesis->distribution];