Neural Networks

Sebastian Bodenstein, Wolfram Research

Join the Conversation #WolframTechConf

Introduction

Introduction to Neural Networks in WL

Upcoming features in 11.1

Introduction

Neural Networks currently achieve state-of-the-art performance on many tasks:

Speech recognition

Machine translation

Question answering and fact extraction

All sorts of image tasks (classification, segmentation, colorization)

Version 11.0 focused on image related networks with convolutional networks

GPU training support on NVIDIA GPU’s

Out-of-core image training

Version 11.1 is adding recurrent support, necessary for all sequential data

What are neural networks?

Modern term: differentiable programming

Lots of differentiable operations composed to form complex function

1. Layers

elem=ElementwiseLayer[Tanh]

Apply to Data

elem@{1,2,3}
N@Tanh@{1,2,3}

What are neural networks?

Some layers come with weights:

dot=DotPlusLayer[3,"Input"-> 2]

NetExtract[dot,"Weights"]

Initialize:

dot2=NetInitialize@dot
NetExtract[dot2,"Biases"]

What are neural networks?

2. Encoders

Networks can only take in tensors. Encoders encode images etc to tensors:

enc=NetEncoder[{"Image",28}]

enc@

What are neural networks?

Or encode classes as vectors:

encClass=NetEncoder[{"Class",{"dog","cat","racoon"},"UnitVector"}];
encClass@{"racoon","dog"}

Attach to layer input:

elem=ElementwiseLayer[Tanh,"Input"->encClass]

Can take in class input now:

elem@"cat"

What are neural networks?

3. Containers

Build computational graphs from layers.

NetChain

Chain together two operations:

net=NetChain[{ElementwiseLayer[Tanh],ElementwiseLayer[LogisticSigmoid]}]

Equivalent to:

f[x_]:=LogisticSigmoid@Tanh@x

Equivalent on data:

data={1.,2.,3.};
net@data
f@data

What are neural networks?

NetGraph

Create arbitrarily shaped computational graphs:

net=NetGraph[{ElementwiseLayer[Tanh],ElementwiseLayer[LogisticSigmoid],TotalLayer[]},{NetPort["Input1"]->1,NetPort["Input2"]->2,{1,2}->3}]

Equivalent to:

func=(Tanh@#Input1 + LogisticSigmoid@#Input2)&;

Evaluate on data:

data=<|"Input1"->{0.1,-2.4},"Input2"->{-1.2,3.4}|>;
net@data
func@data

What are neural networks?

Example of a modern Net

Googlenet v3:

https://arxiv.org/pdf/1512.00567.pdf

Get model:

https://wolfr.am/dHDv9Kut

inception=Import@FileNameJoin[{NotebookDirectory[],"inceptionv3.wlnet"}]

What are neural networks?

Evaluate:

inception@

Get subgraph:

NetExtract[inception,25]

Take[inception,{1,25}]

What are neural networks?

A basic example

Define a simple network:

net=NetGraph[<|"dotLayer"->DotPlusLayer[1],"loss"->MeanSquaredLossLayer[]|>,{"dotLayer"-> "loss"},"Input"-> 1]

Initialize:

net=NetInitialize@net;

Define trainingset:

trainingset =<|"Input"->{{1},{2},{3},{4}},"Target"->{{2.2},{3.8},{6.4},{9.1}}|>;
net@trainingset

What are neural networks?

Gradient Descent

Neural networks are almost always trained via gradient descent:

Source: http://sebastianraschka.com/faq/docs/closed-form-vs-gd.html

Use this to update the weights

What are neural networks?

NetTrain

Automates the training process:

Uses clever ways to choose step sizes etc:

trainedNet=NetTrain[net,trainingset]

Get losses:

net@trainingset
trainedNet@trainingset

The losses have decreased!

Remove loss layer with take:

trainedNet=Take[trainedNet,{"dotLayer","dotLayer"}]

Make a prediction:

trainedNet@{3.4}

What are neural networks?

Example: digit recognizer

Get data:

resource = ResourceObject["MNIST"];

trainingData=ResourceData[resource,"TrainingData"];
testData=ResourceData[resource,"TestData"];

RandomSample[trainingData,5]

Define encoders + decoders:

dec=NetDecoder["Class",Range[0,9]]
enc=NetEncoder["Image",{28,28},"Grayscale"]

What are neural networks?

Define a convolutional neural network that takes in 28x28 grayscale images as input:

lenet=NetChain[{
ConvolutionLayer[20,{5,5}],
ElementwiseLayer[Ramp],
PoolingLayer[{2,2},{2,2}],
ConvolutionLayer[50,{5,5}],
ElementwiseLayer[Ramp],
PoolingLayer[{2,2},{2,2}],
FlattenLayer[],
DotPlusLayer[500],
ElementwiseLayer[Ramp],
DotPlusLayer[10],
SoftmaxLayer[]},
"Output"->dec,
"Input"->enc
]

What are neural networks?

Define a loss layer:

loss=CrossEntropyLossLayer["Target"->NetEncoder["Class",Range[0,9]]]

Train the network for 4 training rounds (this can take several minutes):

lenet=NetTrain[lenet,trainingData ,loss,ValidationSet->testData,MaxTrainingRounds-> 4]

What are neural networks?

Evaluate the trained network directly on images from the validation set:

imgs=Keys @ RandomSample[testData,5]
lenet[imgs]

Obtain probability estimates of the predicted classes:

lenet[imgs,"TopProbabilities"]

Use ClassifierMeasurements:

cm=ClassifierMeasurements[lenet,testData]

Get accuracy:

cm["Accuracy"]

Export the model:

Export["mnist.wlnet",lenet]

Why are neural networks such successful ML models?

0. Powerful Learner

Theorem (Universal Approximation Theorem): NetChain[{DotPlusLayer[n], ElementwiseLayer[Tanh]}] can approximate any continuous function (on a compact subset of (R^n)) arbitrary well for some finite number of neurons n.

Theorem: Recurrent neural networks are Turing Complete (Siegelmann, H. T. and Sontag, E. D. (1995))

1. Existence of Online Training Algorithms

Stochastic gradient descent: only needs to see a few training examples at a time

Not the case for many other ML algorithms (Random Forest, Support Vector Machine, etc)

Allows training on massive datasets

Why are neural networks such successful ML models?

2. Massively Parallelizable Training

Sequential execution speed improvements on processors has stalled

Only parallelizable computation can scaled

Neural networks allow for multiple parallelization modes: parallelize across examples (across many devices), single example execution built out of parallelizable bottlenecks (matrix multiplication)

Why are neural networks such successful ML models?

3. Composable, flexible architecture

Data often has symmetries/structure (eg sequential, spatial invariance, etc). Can compose necessary components to exploit structure/symmetry:

PoolingLayer -> spatial invariance

RNN -> Sequential data

Convolution -> spatial sequential data (like images, sound)

Can adapt architecture of neural network to specific ML task (compare with Predict/Classify):

object detection (returns bounding boxes of objects + classes of objects)

speech recognition

machine translation

question answering

image description generation

4. Deep representations

Efficiency: exponentially (in the depth of the network) fewer parameters are required for deep networks for same representational power

A very successful prior on the space of functions: functions that can be composed of lots of simple operations

Coming in 11.1

Note: the exact design of these layers is subject to change!

1. RNN + variable length sequence support

Many ML tasks deal with sequential data:

Translating English to {German, French, etc}

Speech recognition

Image -> description

Language modelling

Sequence Layers

A few layers to manipulate variable length sequences (SequenceLastLayer, SequenceReverseLayer):

layer=SequenceLastLayer["Input"->{Automatic,{2}}]

layer@{{1,2},{3,4},{5,6},{7,8}}

Coming in 11.1

Recurrent Layers

Recurrent layers operate on sequences (BasicRecurrentLayer, LongShortTermMemoryLayer, GatedRecurrentLayer):

net=NetInitialize@BasicRecurrentLayer[5,"Input"->{Automatic,2}]

net[<|"Input"->{{1,2},{3,4},{2,3}},"InitialState"->{0,0,0,0,0}|>]

Can be part of NetGraph, NetChain etc

NetChain[{BasicRecurrentLayer[5]}]

Coming in 11.1

Functional Constructs

NetFoldOperator, NetMapOperator, NetNestOperator:

Eg. NetMapOperator maps any layer across a sequence:

NetMapOperator[SummationLayer[]]@{{2,3},{4,5},{5,6}}

Higher-level operators (not certain for 11.1)

Make make sequence-to-sequence training (which has dynamic stopping conditions) easy to train and evaluate:

net=EncoderDecoderNet[encoderNetwork,decoderNetwork]
net@input

Coming in 11.1

Speed

targeting new cuDNN RNN implementation from NVIDIA as backend for GPU

A lot of work to make general networks with sequence inputs work as fast as possible.

Coming in 11.1

2. Model Gallery

State-of-the-art networks are hard to define and take a very long to train:

Eg image classification model Inception v4 (https://arxiv.org/pdf/1602.07261v2.pdf) takes ~ weeks to train on good GPU

Takes 300+ lines of code define (easy to make mistakes)

There exist many interesting pre-trained TensorFlow, Caffe and Torch models

Make these accessible via model gallery! (and do the hard work of converting other models into WL)

Version 11.1 will focus on image related models (image colorization, neural style transfer, object detection, semantic segmentation, various state-of-the-art image classifiers, and some NLP models like Glove word vectors).

Will continuously add more models

ro=ResourceObject["NeuralNet"-> "ResNet 101-Layer trained on Wolfram ImageIdentify V2 Data"];
ResourceData[ro]

Lost of useful metadata, like “ConstructionNotebook”

Coming in 11.1

2. Model Gallery

Ease of use

net=Import@FileNameJoin[{NotebookDirectory[],"googlenet_v1.wlnet"}]

im=;

net[im,"TopProbabilities"]

Coming in 11.1

Ease of use

Get[FileNameJoin[{NotebookDirectory[],"segmentation.wl"}]];
net=Import@FileNameJoin[{NotebookDirectory[],"dilated_conv.wlnet"}];

data=;
NetEvaluationFunction[net,t]

Coming in 11.1

3. Other improvements

Training Checkpointing + detailed training introspection with TrainingProgressFunction:

data=Table[x->Exp[-x^2],{x,-3,3,.1}];
plot=ListPlot[data[[All,2]],PlotStyle->Red,FrameTicks->False,Frame->True]

net=NetChain[{150,Ramp,150,Tanh,1},"Input"->"Scalar","Output"->"Scalar"];
solutions={};
plotSolution[net_]:=ListLinePlot[net@Range[-3,3,.1]];
NetTrain[net,data,MaxTrainingRounds->200,TrainingProgressFunction->{AppendTo[solutions,plotSolution[#Net]]&,"Interval"->5}];

Animate[Show[plot,solution],{solution,solutions}]

Coming in 11.1

3. Other improvements

Many new and improved layers, eg:

ElementwiseLayer[(Tanh[#]/2.3 + 4.5*Sin[2.3#])&]

Major CPU speedup (roughly 3-4x faster for convolutional networks)

Using new Intel MKL network primitives

Questions