Automatic Density Estimation with LearnDistribution

Etienne Bernard, Wolfram Research

Join the Conversation #WolframTechConf

GIT["Dataset`"]

GIT[ML]

Learning a probability distribution of...

Numeric variable

{1.3,2.7,1.5...}->

Nominal variable

{"A","A","B", ...} -> <|"A"->0.66, "B"->0.33|>

Numeric vectors

->

Numeric and nominal variables

Images

{,,,,...,,,,,}

Text

{"the cat is grey",...,"this dog is scary"}

Audio

{, ...}

...

Holly grail of unsupervised learning

Generating new examples (RandomVariate)

Anomaly detection (AnomalyDetection)

Predict any variable as function of others

Fill-in missing values (SynthesizeMissingValues)

Denoising / Error correction

Expectation or probability queries

Find variable sharing information

Semi-supervised learning

...

LearnDistribution

Nominal space

ld=LearnDistribution[{"A","A","B","B","B"}]

Information[ld]

RandomVariate[ld,100]

PDF[ld,"A"]

PDF[ld,"B"]

Numeric space

iris=ExampleData[{"MachineLearning","FisherIris"},"Data"][[All,1,{1,3}]];

ListPlot[iris]

ld=LearnDistribution[iris]

Information[ld]

Show[ContourPlot[PDF[ld,{x,y}],{x,4,8},{y,0,7},PlotRange->All,Contours->10],ListPlot[iris, PlotStyle->Red]]

SynthesizeMissingValues[ld,{5.5,Missing[]}]

Histogram[Table[Last@SynthesizeMissingValues[ld,{5.5,Missing[]}],1000]]

Dataset

titanic=ExampleData[{"Dataset","Titanic"}]

ld=LearnDistribution[titanic]

samples=RandomVariate[ld,10]

PDF[ld,samples]

toimpute=Dataset[{\[LeftAssociation]"class"->"1st","age"->5,"sex"->"female","survived"->Missing[]\[RightAssociation],\[LeftAssociation]"class"->"2nd","age"->40`,"sex"->Missing[],"survived"->False\[RightAssociation]}]

SynthesizeMissingValues[ld, toimpute]

Colors, Date, ...

Quit

GIT[ML]

ld=LearnDistribution[{,,,,,,,,,}]

RandomVariate[ld,100]

PDF[ld,]

PDF[ld,]

PDF[ld,]

ld=LearnDistribution[{(DateObject["Hour: ""Sat 28 Jul 2018 12 pm""GMT"-7.`, DateObject[{2018,7,28,12},"Hour","Gregorian",-7.`]]),(DateObject["Thu 16 Aug 2018 18:19:13""GMT"-7.`, DateObject[{2018,8,16,18,19,13.906976`7.895807690783149},"Instant","Gregorian",-7.`]]),(DateObject["Mon 30 Jul 2018 20:15:22""GMT"-7.`, DateObject[{2018,7,30,20,15,22.188818`8.098709151107931},"Instant","Gregorian",-7.`]]),(DateObject["Mon 13 Aug 2018 11:22:17""GMT"-7.`, DateObject[{2018,8,13,11,22,17.61177`7.998377989701336},"Instant","Gregorian",-7.`]]),(DateObject["Fri 10 Aug 2018 08:37:27""GMT"-7.`, DateObject[{2018,8,10,8,37,27.063134`8.18495306970004},"Instant","Gregorian",-7.`]]),(DateObject["Sun 5 Aug 2018 15:17:54""GMT"-7.`, DateObject[{2018,8,5,15,17,54.472131`8.48874934003741},"Instant","Gregorian",-7.`]])}]

RandomVariate[ld,10]

Image

cifar=ResourceData["CIFAR-100"][[All,1]];

RandomSample[cifar,10]

ld=LearnDistribution[cifar]

ld=LearnedDistribution[];

RandomVariate[ld,100]

testimages={,,,,};

Grid[{#,PDF[ld,#]} & /@ testimages]

Text, Sound, Sequence

To come...

Methods

Multinormal

ld=LearnDistribution[iris,Method->"Multinormal"]

Show[ContourPlot[PDF[ld,{x,y}],{x,4,8},{y,0,7},PlotRange->All,Contours->10],ListPlot[iris, PlotStyle->Red]]

GaussianMixture

ld=LearnDistribution[iris,Method->"GaussianMixture"]

Show[ContourPlot[PDF[ld,{x,y}],{x,4,8},{y,0,7},PlotRange->All,Contours->10],ListPlot[iris,PlotStyle->Red]]

ContingencyTable

ld=LearnDistribution[iris,Method->"ContingencyTable"]

Show[DensityPlot[PDF[ld,{x,y}],{x,4,8},{y,0,7},PlotRange->All],ListPlot[iris, PlotStyle->Red]]

KernelDensityEstimation

ld=LearnDistribution[iris,Method->"KernelDensityEstimation"]

Show[ContourPlot[PDF[ld,{x,y}],{x,4,8},{y,0,7},PlotRange->All,Contours->10],ListPlot[iris,PlotStyle->Red]]

DecisionTree

ld=LearnDistribution[iris,Method->"DecisionTree"]

Show[DensityPlot[PDF[ld,{x,y}],{x,4,8},{y,0,7},PlotRange->All],ListPlot[iris,PlotStyle->Red]]

RealNVP

Reversible neural network

Very experimental

ld=LearnDistribution[iris,Method->"RealNVP"]

Show[ContourPlot[PDF[ld,{x,y}],{x,4,8},{y,0,7},PlotRange->All,Contours->10],ListPlot[iris,PlotStyle->Red]]

Future methods...

RandomForest

RNN based methods (LSTM, ...)

VariationalAutoEncoder

GenerativeAdversarialNetworks

RestrictedBoltzmannMachine, DeepBoltzmannMachine

Other reversible network: GLOW, FFJORD

MADE / MAF / IAF

GraphicalModel

ProbabilisticProgram

Automation

Loss function: cross-entropy (log likelihood) on test sets

Also takes into account model size, pdf query speed, sampling time, ...

Procedure:

Start with a set of configurations

Train configurations on a small dataset (“experiments”)

Predict how well the configurations would perform on the full dataset

Give more data to promising configurations

Problems:

More brittle than supervised learning

repeated elements

mix of discrete and continuous values

...

Difficult to do feature extraction (need to be reversible)

Some models cannot compute PDF (GAN, energy based model, ...)

Future

Other methods

Good reversible feature extractor for image & sound (& text ?)

Various distribution transformation

Marginalizing

Conditioning

MutualInformation

Denoising

Multi-output Predict