Deep Learning 
in the Wolfram Language

Giulio Alessandrini, WRI

Join the Conversation #WolframTechConf

ABSTRACT

The Wolfram Language contains a scalable, industrial-strength and easy-to-use neural net framework. In this talk, we are going to demonstrate its capabilities and its integration with the rest of the language. We are going to show how to build and train a net model from scratch, as well as work with a ready-to-use model from the Neural Net Repository. Last, we would like to cover some of our short- to mid-term-future plans.

Outline

Why are Neural Nets interesting?

Application Survey

What are Neural Nets?

Deep Learning in the Wolfram Language

A full example: training a handwritten digit classifier

The future of the Wolfram Language Neural Net Framework

Why Are Neural Networks Interesting?

Data availability

Big Data it’s not (just) a buzzword!

Much more data, thanks to cheap sensors and bandwidth

More user-generated data on the internet

Broader range of applications

Much better Hardware and Software

Key players, like NVIDIA and Intel, are very committed

Dedicated hardware

Optimized network primitives both on GPU and CPU

Faster training on GPUs

Faster execution time

Huge community effort to improve deep learning frameworks

Why Are Neural Networks Interesting?

Desirable Properties

Flexibility

Neural networks can be built and trained to solve a variety of problems

End-to-end training or...

Integration with classical processing pipelines (e.g. feature extraction)

High tolerance

Of faults: corruption of individual weights has a limited effect on the network response

Of noise: neural networks handle noisy data well

Scalability

Parallelizable computation scale with extra CPU or GPU cores

Learning algorithms are trained on few examples at a time

Better performance

In many tasks, neural networks already perform better than humans (e.g. lip reading)

Why Are Neural Networks Interesting?

They Work

Neural nets provide state-of-the art results for:

Image classification:

ImageNet Challenge

Object and face detection

Facial landmarks detection

Scene parsing

Image segmentation (at pixel level)

Image captioning

Optical character recognition

Self-driving cars

Applications Survey: Image Classification

How would you write a program that can take arbitrary images of objects and correctly recognize them?

One such dataset: ImageNet Large Scale Visual Recognition Challenge, O. Russakovsky et al

ImageNet Challenge: 1000 categories of object, ~1.5 million training images, unseen test images

Applications Survey: Image Classification

First deep learning approach in 2012:

ImageNet Classification with Deep Convolutional Neural Networks by A. Krizhevsky et al

A few key innovations, primarily using GPUs to train the neural net

The Data that Transformed AI Research—and Possibly the World by Dave Gershgorn

Applications Survey: Image Classification

Non-deep learning approaches

Careful feature engineering (HOG, SIFT) to be used in a shallow classifier (eg SVM)

Rapidly diminishing improvements (best method: 25% error) at the cost of massive engineering effort

Deep learning

Use pixels directly, zero feature engineering!

Let the net learn appropriate features as part of the learning problem

Rapid improvements (best method: <5% error)

Applications Survey: Image Classification

In the WL

Many models trained on ImageNet are available via NetModel and can be viewed on the soon-to-be-released Wolfram Neural Net Repository:

net=NetModel["Inception V3 Trained on ImageNet Competition Data"]

net[]

ImageIdentify uses this technology, and the underlying net is available to users:

NetModel["Wolfram ImageIdentify Net V1"]

Applications Survey: Translation

Google Translate switched to using neural nets in 2016:

“As dawn broke over Tokyo, Google Translate was the No. 1 trend on Japanese Twitter, just above some cult anime series and the long-awaited new single from a girl-idol supergroup. Everybody wondered: How had Google Translate become so uncannily artful?” ~ The Great A.I. Awakening from the New York Times

Major performance improvements

Google’s Neural Machine Translation System: Bridging the Gap between Human and Machine Translation by Wu et al

This neural net-based translation service can be used via:

TextTranslation["There is no such thing as a free lunch",(Entity["German", Entity["Language","German"], "Entity[\"Language\", \"German\"]", "language"]),Method-> "Google"]

Applications Survey: Autonomous Vehicles

A key component in most self-driving car systems

Training AI for Self-Driving Vehicles: the Challenge of Scale
by Adam Grzywaczewski

An example is the Dilated ResNet-105 Trained on Cityscapes Data model:

NetModel["Dilated ResNet-105 Trained on Cityscapes Data"];

in=;

out=

Applications Survey: NLP

Speech Recognition

Almost every modern speech recognition system users neural nets

Deep Speech 2: End-to-End Speech Recognition in English and Mandarin by D. Amodei et al

An example is the Deep Speech 2 Trained on Baidu English Data (Raw Model) in the Wolfram Neural Net Repository.

Question Answering

Almost every modern speech recognition system users neural nets

FindTextualAnswer was trained with the Wolfram Neural Net Framework. See this blog post by Jerome Louradour from our group

Its the only game in town. Every method on the SQuAD Question Answering Leaderboard is a neural net

questions={"Where was the dodo?","How big was the dodo?","What's the dodo's closest relative?"};
text="The dodo (Raphus cucullatus) is an extinct flightless bird that was endemic to the island of Mauritius, east of Madagascar in the Indian Ocean. The dodo's closest genetic relative was the also extinct Rodrigues solitaire, the two forming the subfamily Raphinae of the family of pigeons and doves. The closest living relative of the dodo is the Nicobar pigeon. A white dodo was once thought to have existed on the nearby island of Réunion, but this is now thought to have been confusion based on the Réunion ibis and paintings of white dodos.\n
Subfossil remains show the dodo was about 1 metre (3 ft 3 in) tall and may have weighed 10.6–17.5 kg (23–39 lb) in the wild. The dodo's appearance in life is evidenced only by drawings, paintings, and written accounts from the 17th century.";
{questions,FindTextualAnswer[text,questions]}//Transpose//TextGrid

Applications Survey: ImageRestyle

ImageRestyle uses a neural net underneath

There are a great number of methods available now

Deep Painterly Harmonization by F. Luan et al

An example of a restyling net:

NetModel["AdaIN-Style Trained on MS-COCO and Painter by Numbers Data"][<|
"Content"->,"Style"->|>
]

Applications Survey: Colorization

Colorize a black and white image:

netevaluation[img_Image]:=With[{lum=ColorSeparate[img,"L"]},
Image[Prepend[ArrayResample[NetModel["Colorful Image Colorization Trained on ImageNet Competition Data"][lum],Prepend[Reverse@ImageDimensions@img,2]],ImageData[lum]],Interleaving->False,ColorSpace->"LAB"]
]

netevaluation[]

Applications Survey: The Rest

Go to https://resources.wolframcloud.com/NeuralNetRepository to see more!

Will be officially released soon

There are vast amounts of extra applications

medical imaging

sentiment analysis

etc etc

What Are Neural Nets?

“...a computing system made up of a number of simple, highly interconnected processing elements, which process information by their dynamic state response to external inputs.”

In "Neural Network Primer: Part I" by Maureen Caudill, AI Expert, Feb. 1989

Originally inspired by biological neural networks

Computing systems that learn by examples

Modern term: differentiable programming

“It’s really very much like a regular program, except it’s parameterized, automatically differentiated, and trainable/optimizable.”

Yann LeCun, Director of FAIR

What Are Neural Nets?

Layers

Computational units that perform the same operation on the same input are aggregated into layers

The output of a layer is the input of the successive one

A three layers network with two fully connected layers:

A three layers network with two convolution layers:

Images from Convolution Networks by Stanford CS231

What Are Neural Nets?

Gradient Descent

Training is very similar to NMinimize or FindMinimium

Working in a very high dimensional space

Source: http://sebastianraschka.com/faq/docs/closed-form-vs-gd.html

What Are Neural Nets?

Deep Learning

Learn multiple representations at multiple levels

Deeper levels correspond to higher levels of abstraction

Hierarchy of concepts

What Are Neural Nets?

Deep Learning

models={"LeNet Trained on MNIST Data","Inception V1 Trained on ImageNet Competition Data","ResNet-101 Trained on ImageNet Competition Data"};
ListLinePlot[AssociationMap[NetInformation[NetModel[#],"LayersCount"]&,models],PlotLegends->Automatic,Ticks->{None,Automatic}]

Deep Learning in the Wolfram Language

1. Layers

A layer is the simplest component of a network.

It represents an operation and by default it’s completely symbolic

elem=ElementwiseLayer[Tanh]

Layers only act on numeric tensors:

elem@{1,2,3}
N@Tanh@{1,2,3}

Layers are differentiable. Differentiability is a key property that allows for the efficient training of nets, which we will see later:

elem[{1,2,3},NetPortGradient["Input"]]

D[Tanh[x],x]/.x->{1.,2.,3.}

Deep Learning in the Wolfram Language

1. Layers

They can run on both NVIDIA GPUs and CPUs:

elem[{1,2,3},TargetDevice->"GPU"]

They do shape inference:

ElementwiseLayer[Tanh]

ElementwiseLayer[Tanh,"Input"->{4,32}]

Certain layers have learnable parameters

without this, no learning would be possible!

dot=LinearLayer[3,"Input"-> 2]

Initialize the parameters in the layer:

dot2=NetInitialize@dot

NetExtract[dot2,"Weights"]

Deep Learning in the Wolfram Language

1. Layers

So far, we have seen layers that have exactly one input. Some layers have more than one input.

For example, (RefLink[MeanSquaredLossLayer, paclet:ref/MeanSquaredLossLayer]) compares two arrays, called the input and the target, and produces a single number that represents (RefLink[Mean, paclet:ref/Mean])[(input-target)^2].

msloss=MeanSquaredLossLayer[]

The inputs of the layer are named and must be supplied in an association when the net is applied:

msloss[<|"Input"->{1,2,3},"Target"->{4,0,4}|>]

The full list of available layers is:

?*Layer

Deep Learning in the Wolfram Language

1. Layers

The simplest learnable layer is the LinearLayer

This is just:

linear[data_, weight_,bias_]:=Dot[weight,data]+bias

Comparing this to a LinearLayer:

layer=NetInitialize@LinearLayer[2,"Input"->3]
layer[{2,10,3}]

linear[{2,10,3}, NetExtract[layer,"Weights"],NetExtract[layer,"Biases"]]

Deep Learning in the Wolfram Language

2. Containers

Single neural net layers are generally not useful by themselves. We usually need to combine multiple layers together to do something interesting.

The simplest container is a chain

Chain together two operations:

net=NetChain[{ElementwiseLayer[Tanh],ElementwiseLayer[LogisticSigmoid]}]

Equivalent to:

f[x_]:=LogisticSigmoid@Tanh@x

Equivalent on data:

data={1.,2.,3.};
net@data
f@data

Deep Learning in the Wolfram Language

3. Containers

(RefLink[NetChain, paclet:ref/NetChain]) does not allow a net to take more than one input, so we need to use (RefLink[NetGraph, paclet:ref/NetGraph]) to build the training network

Create a (RefLink[NetGraph, paclet:ref/NetGraph]):

net=NetGraph[{ElementwiseLayer[Tanh],ElementwiseLayer[LogisticSigmoid],TotalLayer[]},{NetPort["Input1"]->1,NetPort["Input2"]->2,{1,2}->3}]

Equivalent to:

func=(Tanh@#Input1 + LogisticSigmoid@#Input2)&;

Evaluate on data:

data=<|"Input1"->{0.1,-2.4},"Input2"->{-1.2,3.4}|>;
net@data
func@data

As all of the layers are differentiable, so is the container

net[data,NetPortGradient["Input1"]]

Deep Learning in the Wolfram Language

4.  Containers

Containers behave exactly like normal layers!

differentiable, run on GPUs, etc

Containers can be nested, as they are just like normal layers:

NetChain[{NetChain[{LinearLayer[]}]}]

Models in the Repository are almost all some form of container:

NetModel["AdaIN-Style Trained on MS-COCO and Painter by Numbers Data"]

Deep Learning in the Wolfram Language

4.  Containers

There is a rich shape inference between all the layers in a container

NetChain[{LinearLayer[],LinearLayer[]}]

NetChain[{LinearLayer[3],LinearLayer[]}]

NetChain[{LinearLayer[3],LinearLayer[4]}]

NetChain[{LinearLayer[3],LinearLayer[4]},"Input"->5]

Helpful error messages point to the exact mismatches

NetChain[{LinearLayer[3,"Input"->6],LinearLayer[4]},"Input"->5]

Deep Learning in the Wolfram Language

5.  NetEncoders

Fundamentally, because they must be differentiable, neural net layers operate on numeric tensors. However, we often want to train and use nets on other data, such as images, audio, text, etc.

We can use a (RefLink[NetEncoder, paclet:ref/NetEncoder]) to translate this data to numeric tensors.

Create an image (RefLink[NetEncoder, paclet:ref/NetEncoder]) that produces a 1×12×12 tensor:

imageenc=NetEncoder[{"Image",{12,12},"ColorSpace"->"Grayscale"}]

Apply the encoder to an image:

imageenc[]

Can also be applied to files

f=File@FindFile["ExampleData/coneflower.jpg"]

imageenc[f]

Allows for out-of-core learning on image and audio files!

See the tutorial Training on Large Datasets for more

A large collection of encoders are available for different datatypes

“Audio”

“Characters”

Deep Learning in the Wolfram Language

5.  NetEncoders

Encoders can be attached to the input ports of layers or containers. Attach an image (RefLink[NetEncoder, paclet:ref/NetEncoder]) to a (RefLink[PoolingLayer, paclet:ref/PoolingLayer]) via the "Input" option:

pool=PoolingLayer[10,"Input"->NetEncoder[{"Image",64}]]

Apply the (RefLink[PoolingLayer, paclet:ref/PoolingLayer]) directly to an image, which will use the image (RefLink[NetEncoder, paclet:ref/NetEncoder]) to translate the image to a tensor for (RefLink[PoolingLayer, paclet:ref/PoolingLayer]) to operate on:

pool[]//Shallow

Deep Learning in the Wolfram Language

5.  NetEncoders

Encoders are what allows trained models to be used directly on the type of interest:

net=NetModel["Inception V3 Trained on ImageNet Competition Data"]

net[]

NetExtract[net,"Input"]

Deep Learning in the Wolfram Language

6.  NetDecoders

A net will always output a numeric tensor. But for a task like classification, one wants class-labels as output. A NetDecoder is a mechanism for returning non-numeric tensors from nets.

dec=NetDecoder[{"Class",{"dog","cat"}}]

This decoder will interpret a vector of probabilities over classes as a class label:

dec[{0.1,0.9}]

The probabilities can also be obtained:

dec[{0.1,0.9},"Probabilities"]

Deep Learning in the Wolfram Language

6.  NetDecoders

A decoder can be attached to the output of a layer or container:

soft=SoftmaxLayer["Output"->NetDecoder[{"Class",{"dog","cat"}}]]

soft[{44,41},"Probabilities"]

This mechanism allows pre-trained nets to output class-labels:

net=NetModel["Inception V3 Trained on ImageNet Competition Data"];

net[]

NetExtract[net,"Output"]

Deep Learning in the Wolfram Language

7.  Training

To train a net, it must have one output, the loss

Training involves finding parameters to minimize the loss

A very simple example:

data={{1}->{1.9},{2}->{4.1},{3}->{6.0},{4}->{8.1}}

net=NetInitialize@NetGraph[{LinearLayer[1],MeanSquaredLossLayer[]},{1->2},"Input"->{1}]

Evaluating this net:

net[<|"Input"->{2},"Target"->{4.1}|>]

Deep Learning in the Wolfram Language

7.  Training

Train the net:

trainednet=NetTrain[net,data]

The output is now much smaller:

trainednet[<|"Input"->{2},"Target"->{4.1}|>]

Training has changed the parameters:

NetExtract[net,{1,"Weights"}]
NetExtract[trainednet,{1,"Weights"}]

Deep Learning in the Wolfram Language

8. Surgery

Very easy to look into nets and modify them

Remove the weights from the trainable layers:

model=NetModel["LeNet Trained on MNIST Data"]

NetInitialize[model,None]

Drop the liner layers and make the resulting fully convolutional net size-independent:

NetReplacePart[NetDrop[model,-4],"Input"->Automatic]

Replace the activation function:

NetReplace[model,ElementwiseLayer[Ramp]->ElementwiseLayer[Tanh]]

Training a Digit Classifier

Obtain the MNIST dataset, which contains 60,000 training and 10,000 test images:

trainingData=ResourceData["MNIST","TrainingData"];
testData=ResourceData["MNIST","TestData"];

Display a few random examples from the training set:

RandomSample[trainingData,5]

Training a Digit Classifier

We could use a pre-trained version of LeNet from the Wolfram Neural Net Repository:

lenetModel=NetModel["LeNet Trained on MNIST Data"]

Classify a list of images using the pretrained net:

lenetModel[{,,,}]

But lets do this from scratch, making use of all of the components.

We are also going to do this in the most general way

For a faster way, see the Tutorial MNIST Digit Classification

Training a Digit Classifier

Define encoders + decoders:

dec=NetDecoder[{"Class",Range[0,9]}]

enc=NetEncoder[{"Image",{28,28},"Grayscale"}]

Training a Digit Classifier

Define a convolutional neural network that takes in 28x28 grayscale images as input:

uninitializedLenet = NetChain[
{
	ConvolutionLayer[20,5],     (*first convolution => 20 feature images*)
	ElementwiseLayer[Ramp],     (*activation function (ReLU) => non-linearity, sparsity*)
	PoolingLayer[2,2],          (*max pooling => downsampling*)

	ConvolutionLayer[50,5],     (*second convolution => 50 feature images*)
	ElementwiseLayer[Ramp],     (*activation function (ReLU) => non-linearity, sparsity*)
	PoolingLayer[2,2],          (*max pooling => downsampling*)

	FlattenLayer[],             (*flattening => images to vector*)
	LinearLayer[500],           (*first fully connected layer => feature vector from image features*)
	ElementwiseLayer[Ramp],     (*activation function (ReLU) => non-linearity, sparsity*)
	LinearLayer[10],            (*second fully connected layer => class prediction*)
	SoftmaxLayer[]              (*normalization*)
},
	"Input"  -> enc, (*encoder => image to tensor*)
	"Output" -> dec  (*decoder => tensor to class*)
]

Training a Digit Classifier

Construct a (RefLink[NetGraph, paclet:ref/NetGraph]) by supplying a list of layers and connections:

trainingNet=NetInitialize@NetGraph[<|"lenet"->uninitializedLenet,"loss"->CrossEntropyLossLayer["Index"]|>,
{NetPort["Input"]->"lenet"->NetPort["loss","Input"],NetPort["Target"]->NetPort["loss","Target"]}]

Evaluate the training net on some examples:

trainingNet[\[LeftAssociation]"Input"->{,,,,},"Target"->{2,4,7,9,8}\[RightAssociation]]

These losses summarize how well LeNet did at predicting the targets when given the images. Because LeNet was randomly initialized, we expect it to be no better than chance (on average). During training, the learnable parameters in LeNet are gradually adjusted to bring the average loss down.

Training a Digit Classifier

Convert the training and test data into association form, using (RefLink[Keys, paclet:ref/Keys]) and (RefLink[Values, paclet:ref/Values]) to obtain the images and the labels from the lists of rules in the training and test data:

trainAssoc=<|"Input"->Keys[trainingData],"Target"->Values[trainingData]+1|>;
testAssoc=<|"Input"->Keys[testData],"Target"->Values[testData]+1|>;

Show a small sample of the training association:

Part[trainAssoc,All,{1,10^4,4 10^4,5 10^4}]

Training a Digit Classifier

Let us now perform the training using (RefLink[NetTrain, paclet:ref/NetTrain]). Notice several things about this net example:

We have specified (RefLink[MaxTrainingRounds, paclet:ref/MaxTrainingRounds])->5, which will scan the entire training set five times before finishing.

We have given (RefLink[All, paclet:ref/All]) as the third argument, to obtain a (RefLink[NetTrainResultsObject, paclet:ref/NetTrainResultsObject]) that summarizes various information about the training session.

We have given a (RefLink[ValidationSet, paclet:ref/ValidationSet]), which allows us to measure how well the trained classifier generalizes to new examples that it has not trained on. This helps avoid a common pitfall known as "overfitting".

See the Tutorial Training Neural Networks with Regularization for extra understanding of overfitting and preventing it

We have specified (RefLink[TargetDevice, paclet:ref/TargetDevice])->"CPU" (which is already the default). If you have an NVIDIA graphics card, however, you can change this to "GPU" to achieve a big speedup in training.

Train LeNet:

results=NetTrain[trainingNet,trainAssoc,All,ValidationSet->testAssoc,MaxTrainingRounds->5,TargetDevice->"CPU"]

Training a Digit Classifier

Obtain the loss evolution plot from the (RefLink[NetTrainResultsObject, paclet:ref/NetTrainResultsObject]):

results["LossEvolutionPlot"]

Obtain the trained net from the (RefLink[NetTrainResultsObject, paclet:ref/NetTrainResultsObject]) and from it extract the prediction network:

trainedLenet=NetExtract[results["TrainedNet"],"lenet"]

Reattach the (RefLink[NetEncoder, paclet:ref/NetEncoder]) and (RefLink[NetDecoder, paclet:ref/NetDecoder]) that were removed when the classification net was embedded inside the training network:

trainedLenet=NetReplacePart[trainedLenet,{"Input"->enc,"Output"->dec}]

Training a Digit Classifier

Make a classification on an image:

trainedLenet[]

Obtain the top probabilities for a difficult image:

trainedLenet[,"TopProbabilities"]

Compare with the fully trained model:

NetModel["LeNet Trained on MNIST Data"][,"TopProbabilities"]

Evaluate the Digit Classifier

measurements=ClassifierMeasurements[trainedLenet,testData]

Obtain the overall accuracy:

measurements["Accuracy"]

This could have been done much more simply with Classify:

classifier=Classify[trainingData,ValidationSet->testData]
ClassifierMeasurements[classifier,testData,"Accuracy"]

Summary of the Framework

The WL Neural Net framework is an industrial-strength, scalable framework

uses latest NVIDIA libraries

backed by MXNet

used extensively internally by Wolfram Research

It has a great Neural Net Repository

curation from frameworks

access to a huge collection of nets trained by WL

ensures our framework can represent most modern nets

No lock-in

import/export to MXNet

hopefully support for the ONNX format

Easier to use than other frameworks

makes developers much more productive

true variable-length sequence support

Thank you