Neural Networks 
in the Wolfram Language

Piotr Wendykier, Wolfram Research

Agenda

Why Deep Neural Networks?

Neural Networks in Mathematica 11

Applications in Image Processing

Applications in Audio Processing

Wolfram Neural Net Repository

Why Deep Neural Networks?

Deep neural nets achieve better results!

Large Scale Visual Recognition Challenge (ILSVRC)

Classify millions of images into thousands of categories.

BarChart[
{28.2,25.8,16.4,11.7,6.7,3.57,2.99},
ChartLabels->Map[ToString,Range[2010,2016]],
Frame->  True,FrameLabel->"ImageNet classification error rate",
Epilog-> {Dashing[{0.01,0.01}],Line[{{0.3,5},{7.5,5}}]}
]

2011  Classical image recognition techniques by Nec exhibited an error rate of 28.2 %.
2011  Classical image recognition techniques by Xerox exhibited an error rate of 25.8 %.
2012  “SuperVision” deep neural net by Hinton et al achieved an error rate of 16.4%
2013  “Clarifai” by Matthew Zeiler achieved an error rate of 11.7%
2014  “GoogLeNet” by Google achieved an error rate of 6.7%
2015  Deep residual neural nets by Microsoft achieved an error rate of 3.57%
2016  An ensemble of nets by Trimps-Soushen, China, won with an error rate of  2.997%

NetModel["Inception V3 Trained on ImageNet Competition Data","SourceMetadata"]

NetModel["Inception V3 Trained on ImageNet Competition Data","TrainingSetInformation"]

NetModel["Inception V3 Trained on ImageNet Competition Data","ByteCount"]

inceptionV3Net=NetModel["Inception V3 Trained on ImageNet Competition Data"]

inceptionV3Net[]

Deep neural nets are now doable!

Big Data

Much more data, thanks to cheap sensors and bandwidth

More user-generated data on the internet

Broader range of applications

Better Hardware

Key players, like NVIDIA and Intel provide:

Faster training on GPUs

Optimized network primitives both on GPU and CPU

Optimized Software

Optimized frameworks for deep neural network implementations:

TensorFlow, Torch, Caffe, CNTK, theano, MXNet, Keras, .... , and the Wolfram Language

Why Are Neural Networks Interesting?

Flexibility

Neural networks can be built and trained to solve a variety of problems

High Tolerance

of faults: corruption of individual weights has a limited effect on a network response

of noise: neural networks handle noisy data well

Scalability

Parallelizable computation scale with extra CPU and GPU cores

Better Performance

In many tasks, neural networks already perform better than humans (e.g. lip reading)

Neural  Networks in the Wolfram Language

The Wolfram Language has state-of-the-art capabilities for the construction, training and deployment of neural networks.

Many standard layer types are available and are assembled symbolically into a network:

Network layers (LinearLayer, ElementwiseLayer, CrossEntropyLossLayer, ...)

Network structures (NetChain, NetGraph)

End-to-end training (NetTrain) on available CPUs and GPUs.

Model format to save and load trained networks (WLNet, NetModel).

Integration with the rest of the language (NetEncoder, NetDecoder).

Example datasets (ResourceObject of MNIST, CIFAR-10, CIRFAR-100, ...).

Image Classification

Concept

Train a simple network to perform image classification among 10 categories.

Multicolumn[Row[{"classifier"[#],"->",trained[#]},Spacer[5]]&/@({,,,}),2,Spacings->5]

Image Classification

Data

Import an external resource—in this case the CIFAR-10 dataset.

resource=ResourceObject["CIFAR-10"];
resource["Description"]

Extract the training data.

Short[trainingData=ResourceData[resource,"TrainingData"]]

Length[trainingData]

Short[testData=ResourceData[resource,"TestData"]]

Length[testData]

Extract the class names.

classes=Union@Values[testData]

Image Classification

Neural Network

Define the structure of a convolutional network.

Graphics[{
FaceForm[Orange],EdgeForm[Black],
Reverse@Table[Disk[{i,j+.4i},.6],{i,6},{j,6}],
Black,Opacity[.5],AbsoluteThickness[2],
Table[Line[{{i,j+.4i},{7.5,5}}],{i,1,3},{j,4,6}],
Opacity[1],FaceForm[Green],EdgeForm[Black],
Reverse@Table[Disk[{i,j+.4i}+{6,0},.6],{i,1.5,5},{j,1.5,5}],
Black,Opacity[.5],AbsoluteThickness[2],
Table[Line[{{i,j+.4i}+{6,0},{13,4}}],{i,1.5,4},{j,2.5,5}],
Opacity[1],FaceForm[Pink],EdgeForm[Black],
Reverse@Table[Disk[{i,j}+{12,0},.6],{i,1},{j,2,6}],

Black,Opacity[.5],AbsoluteThickness[2],
Table[Line[{{i1,j1}+{12,0},{i2,j2}+{15,0}}],{i1,1},{j1,2,6},{i2,1},{j2,2.5,6}],

Opacity[1],FaceForm[Lighter@Blue],EdgeForm[Black],
Reverse@Table[Disk[{i,j}+{15,0},.6],{i,1},{j,2.5,6}]
},ImageSize->600
]

lenet = NetChain[
	{
	ConvolutionLayer[20,3],  (* first convolution \[Implies] 20 feature images *)
	ElementwiseLayer[Ramp],  (* activation function (ReLU) \[Implies] non-linearity, sparsity *)
	PoolingLayer[2,2],       (* max pooling \[Implies] downsampling *)

	ConvolutionLayer[50,3],  (* second convolution \[Implies] 50 feature images *)
	ElementwiseLayer[Ramp],  (* activation function (ReLU) \[Implies] non-linearity, sparsity *)
	PoolingLayer[2,2],       (* max pooling \[Implies] downsampling *)

	FlattenLayer[],          (* flattening \[Implies] images to vector *)

	LinearLayer[500],        (* first fully connected layer \[Implies] feature vector from image features *)
	ElementwiseLayer[Ramp],  (* activation function (ReLU) \[Implies] non-linearity, sparsity *)

	LinearLayer[10],         (* second fully connected layer \[Implies] class prediction *)
	SoftmaxLayer[]           (* normalization *)
	},
	"Input" -> NetEncoder[{"Image",{32,32}}],  (* encoder \[Implies] image to tensor *)
	"Output" -> NetDecoder[{"Class",classes}]  (* decoder \[Implies] tensor to class *)
];

Image Classification

Training

Initialize and train the weights in the layers.

classifier=NetTrain[lenet,trainingData, CrossEntropyLossLayer["Index"]]

Speed up the training process by using a GPU.

classifier=NetTrain[lenet,trainingData,CrossEntropyLossLayer["Index"],TargetDevice->"GPU"]

Import a previously trained network.

classifier=Import[FileNameJoin[{NotebookDirectory[],"CIFAR-10-classifier.wlnet"}]];

Image Classification

Application

Predict the most likely classes for an image.

classifier[]

Use the network to classify a collection of images.

classifier[{,,,,}]

For a specific example, display the probabilities for each label assignment.

ClearAll[probabilities]
probabilities[i_Image]:=Dataset[MapAt[Style[#,Bold,Red]&,#,Position[#,Max[#]]]&[classifier[i,"Probabilities"]]]

probabilities[]

Image Classification

Testing

Use test data to initialize a ClassifierMeasurementsObject.

cm=ClassifierMeasurements[classifier,testData]

Query for different classification properties.

cm["Accuracy"]

Dataset[
Transpose@AssociationThread[
{"Precision","Recall","FScore"},
cm[{"Precision","Recall","FScore"}]/.{
x_/;x>.75:>Style[x,Darker@Green],
x_/;x<.5:>Style[x,Darker@Red]
}
]
]

Explore the classification results graphically.

cm["ConfusionMatrixPlot"]

ImageIdentify

A large classifier network:

NetModel["Wolfram ImageIdentify Net for WL 11.1","SourceMetadata"]

NetModel["Wolfram ImageIdentify Net for WL 11.1","TrainingSetInformation"]

imgIdNet=NetModel["Wolfram ImageIdentify Net for WL 11.1","EvaluationNet"]

imgIdNet[]

Built-in classification command.

ImageIdentify[,(Entity["dog", Entity["Species","Infraspecies:CanisLupusFamiliaris"], "Entity[\"Species\", \"Infraspecies:CanisLupusFamiliaris\"]", "species specification"])]

Four most probable results.

ImageIdentify[,(Entity["dog", Entity["Species","Infraspecies:CanisLupusFamiliaris"], "Entity[\"Species\", \"Infraspecies:CanisLupusFamiliaris\"]", "species specification"]),4,"Probability"]

Searching for people.

ImageIdentify[]

Use the "NotablePerson" built-in classifier to identify who is depicted in a given image.

Classify["NotablePerson",]

Determine a dog breed that would fit Donald Trump.

ImageIdentify[,(Entity["dog", Entity["Species","Infraspecies:CanisLupusFamiliaris"], "Entity[\"Species\", \"Infraspecies:CanisLupusFamiliaris\"]", "species specification"]),6,"Probability"]

Prediction vs Reality

bull terrier			                                                  Lhasa

Audio Classification

This is not available in Mathematica 11.2

Data

dir=NotebookDirectory[];
Get[FileNameJoin[{dir,"ProtoNetEncoder.m"}]];

Get file names and classes.

fileNames=FileNames["*.wav",FileNameJoin[{dir,"audio-cats-and-dogs","cats_dogs"}]];
dataNames=AssociationMap[StringSplit[StringSplit[FileNameSplit[#][[-1]]][[-1]],"_"][[1]]&,fileNames];
classes=dataNames//Values//DeleteDuplicates

Length[fileNames]

Divide samples to test and training data.

SeedRandom[23487];
trainingData=RandomSample[dataNames,Round[.85Length@dataNames]];
testData=RandomSample[Complement[dataNames,trainingData],Round[.15Length@dataNames]];
{trainingData,testData}=Normal/@{trainingData,testData};

Length[trainingData]

Length[testData]

Audio Classification

Neural Network and Encoder

enc=AudioNetEncoder`Private`createNetEncoder[{"Audio","MelSpectrogram",SampleRate->16000,"ExtractPart"->{Automatic,Automatic,True},PartitionGranularity->{1024,512},"NumberOfThreads"->4,"NumberOfFrames"->31,"NumberOfFilters"->64,"NoiseLevel"->0.005}];

encoder[data_,assoc_]:=Module[{temp},temp=RandomSample[data,assoc["BatchSize"]];Thread[List/@Log[Normal@First@enc@temp[[All,1]]+.001]->temp[[All,2]]]]

uninitializedNet=
NetChain[{
ConvolutionLayer[16,3],
PoolingLayer[2],
BatchNormalizationLayer[],
Ramp,
PoolingLayer[2],
ConvolutionLayer[32,3],
PoolingLayer[2,"Stride"->2],
BatchNormalizationLayer[],
Ramp,
ConvolutionLayer[64,3],
PoolingLayer[2,"Stride"->2],
BatchNormalizationLayer[],
Ramp,
PoolingLayer[2,"Stride"->2],
FlattenLayer[],
LinearLayer[128],
Ramp,
LinearLayer[Length@classes],
SoftmaxLayer[]},
"Input"->{1,31,64},
"Output"->NetDecoder[{"Class",classes}]
];

Audio Classification

Training

Initialize and train the weights in the layers.

trainedNet=NetTrain[uninitializedNet,encoder[trainingData,#]&,BatchSize->128,
MaxTrainingRounds->200]

Import a previously trained network.

trainedNet=Import[FileNameJoin[{NotebookDirectory[],"trainedNet.wlnet"}]];

Audio Classification

Testing

Query for different classification properties.

cm=ClassifierMeasurements[trainedNet,Flatten[Table[encoder[testData,<|"BatchSize"->All|>],10]]];
cm["Accuracy"]
cm["ConfusionMatrixPlot"]

Visualization.

chopped=Take[trainedNet,{1,18}]

visualizeCatsAndDogs[data_,net_]:=Module[
{d,dd,red,labels},
red=NetExtract[chopped[[-1]],"OutputDimensions"]=!={2};
d=Function[class,Select[Association@data,#===class&]]/@classes;
dd=chopped[
encoder[#,<|"BatchSize"->All|>]//Keys
]&/@Normal[d];
If[red,dd=TakeList[Quiet@DimensionReduce[Join[Sequence@@dd],2],Length/@dd]];
labels=Keys@KeyMap[Audio[#,Appearance->"Minimal"]&,#]&/@d;
ListPlot[
MapThread[Thread[#1->#2]&,{dd,labels}]
,
ImageSize->Large,PlotRange->All
]
]

visualizeCatsAndDogs[RandomSample[trainingData,30],chopped]

Wolfram Neural Net Repository

Hyperlink[,"http://resources.wolframcloud.com/NeuralNetRepository/"]

http://resources.wolframcloud.com/NeuralNetRepository/

Comprehensive

Large assortment of high quality net models

Updated regularly with new models

Pre-trained

Saves training time

Some training sets are not publicly available

Within the Wolfram Language

No need to import from other frameworks

Net models can be used as-is

Layers can be added or removed to construct new models

Untrained net can be re-trained with different data

Submit your own

Examples: Image Processing

Colorization

Raw model: doesn’t give you the final result straight away

Outputs A and B channel of the LAB color space

net=NetModel["ColorNet Image Colorization Trained on ImageNet Competition Data (Raw Model)"];

Get the A and B channels:

output=net[]

Dimensions[output]

Write an evaluation function to merge the A and B channels with the luminance of the grayscale input

evaluationFunction[img_Image]:=Image[
Prepend[net[img],ImageData[ImageResize[ColorSeparate[img,"L"],{224,224}]]],
Interleaving->False,ColorSpace->"LAB"
]

evaluationFunction[]

Style Transfer

ImageRestyle

net=NetModel["AdaIN-Style Trained on MS-COCO and Painter by Numbers Data"];

Restyle the photograph:

net[<|"Content"->,"Style"->|>]

Compare with ImageRestyle:

ImageRestyle[,]

Examples: Facial Analysis

faces=ImageTrim[,#]&/@FindFaces[]

Gender Prediction

net=NetModel["Gender Prediction VGG-16 Trained on IMDB-WIKI Data"];

Guess the gender:

net[faces]

Obtain the probabilities:

net[faces,"Probabilities"]

Facial Keypoints

net=NetModel["Vanilla CNN for Facial Landmark Regression"];

Get the facial keypoints (eyes, nose, mouth):

keypoints=net[faces]

Visualize the predictions:

colorCodes=<|"LeftEye"->,"RightEye"->,"Nose"->,"LeftMouth"->,"RightMouth"->|>;

MapThread[
HighlightImage[#1,{PointSize[0.04],Riffle[Values@colorCodes,#2]},DataRange->{{0,1},{0,1}}]&,
{faces,keypoints}
]

3D Facial Model

net=NetModel["Unguided Volumetric Regression Network for 3D Face Reconstruction"];

Get the 3D volume:

volume=net[];

Dimensions[volume]

Visualize the model:

image3D=Image3D[volume,ViewPoint->Below,BoxRatios->{1,1,0.5},ImageSize->Medium]

region=ImageMesh[image3D,Method->"DualMarchingCubes",ViewPoint->Below,BoxRatios->{1,1,0.5}]

Questions