Neural Networks

A construction toolkit for neural networks

Taliesin Beynon
Wolfram Research
training@wolfram.com

Introduction

A framework for representing and training neural networks

Why are neural networks popular again?

lots more data, thanks to cheap sensors and bandwidth, wider applications, user-generated data on the internet

fast training on GPUs

deep networks with larger representational power

specialized networks (convolutions, LSTMs)

Some applications

computer vision

machine translation

speech-to-text

sequence prediction

Mathematica 11

“construction kit” for building and training your own networks

symbolic representation of various layers

Model format to save and load trained ntworks

NetTrain, with flexible specs for training data and losses

Prepackaged MNIST, CIFAR-10, CIFAR-100 datasets, more to come

Basic Concepts

Tensors: data operated on by net layers

Layers: computational elements of a net

Chains, Graphs: ways of composing layers

Training Data: examples used to train a net

Regression, Classification: supervised learning applications of nets

Loss Functions: quantity used to perform training

Training: using NetTrain

Tensors

Tensors (in the context of neural nets) are rectangular arrays of numbers

rank 1 (vectors): {1,2,3}

rank 2 (matrices): {{1,2,3},{4,5,6}}

rank n (n-tensors): {⋯{⋯{1,2,3}⋯}⋯}

rank 0 (scalars): 1

(in practice, scalars are represented as length-1 vectors)

Rank corresponds to output of ArrayDepth function

“Tensor” has other meanings in Mathematica, such as representing symbolic tensors

Tensors

Examples of tensors

vectors (1-tensors) representing geometric points:

data={{1,4},{3,2},{2,1}};
plot=ListPlot[Labeled[#,Framed[#,FrameStyle->None]]&/@data,Frame->True,LabelStyle->Medium,AspectRatio->1,PlotRangePadding->1,PlotStyle->PointSize[0.04],ImageSize->Small];
shiftr[60,arrow[plot,Column[vec/@data,Spacings->0.5]]]

vectors (1-tensors) representing elements of classes (categorical variables):

shiftr[120,Column[MapThread[smallarrow[str[#1],MatrixForm@List@#2]&,{{"a","b","c"},{{1,0,0},{0,1,0},{0,0,1}}}]]]

(often called one-hot encoding, we use the term unit vector)

matrix (2-tensor) representing a grayscale image:

arrow[a_,b_]:=Row[{a,Style["  ≈ ",32],b}];
shiftr[50,arrow[ArrayPlot[DiskMatrix[3],Mesh->All,ImageSize->100], MatrixForm@DiskMatrix[3]]]

3-tensor representing a color image:

data={{re,bl,bl},{re,gr,bl},{re,re,bl}};
img=Image[data];
shiftr[50,arrow[ArrayPlot[ImageData@img,ColorFunction->RGBColor,Mesh->True,ImageSize->60], MatrixForm[{MatrixForm/@Round[Transpose[data,{2,3,1}]]}]]]

(outermost dimension is the RGB ‘color channel’ dimension)

Tensors

Encoding and Decoding

If data input to a network is not already a numeric tensor, we can use a NetEncoder to handle the job of encoding the data as a tensor

A NetEncoder acts like a function: it takes another kind of input and produces a tensor:

enc=NetEncoder[{"Image",{8,8},ColorSpace->"Grayscale"}]

enc[]

enc=NetEncoder[{"Class",{"cat","dog","frog"},"UnitVector"}]

enc["frog"]

AssociationMap[enc,{"cat","dog","frog"}]

A NetDecoder does the opposite: converts a tensor back into an image, class, etc.

Layers

Building blocks of a net

Layers are simple operations on numeric tensors that produce other tensors

Layers can take one or more inputs and return one or more outputs

Some layers contain learned parameters that are ‘tuned’ during training

Most layers take one input and return one output

Layers can be combined into a linear chain (NetChain) or directed acyclic graph (NetGraph)

Layers

Basic example: DotPlusLayer

DotPlusLayer[n] represents a layer with a vector output of size n

Takes an input vector x and computes the result w.x+b, where w (matrix) and b (vector) are fixed

Represent an affine transformation of the original input vector – though people often say ‘linear’

Input size is inferred from previous layer, or specified via "Input"->n

Example of how a randomly initialized DotPlusLayer[2,"Input"->2] transforms a grid:

inputs=Join@@Table[{x,y},{x,0,10},{y,0,10}];
lplot[inputs_]:=ListPlot[inputs,AspectRatio->1,PlotRangePadding->0.25,Axes->None,Frame->True,FrameTicks->False,ImageSize->Small];
n=NetInitialize@DotPlusLayer[2,"Input"->2];
lplot[inputs]->lplot[n@inputs]

Layers

Basic example: ElementwiseLayer

Applies a simple non-linearity to each individual element of a tensor

Without non-linearities, stacking linear (or affine) layers doesn’t do anything “new”

Known as the “transfer function” in old neural network literature

Non-linearity can be one of Tanh, LogisticSigmoid, or Ramp (also known as Rectified Linear or ReLu):

Row[Table[
{min,max}={-1,1}*If[f===Ramp,1,3];
Plot[f[x],{x,min,max},PlotLabel->f,ImageSize->200,Frame->True,PlotRange->{{min,max},{-1,1}},AspectRatio->0.5,FrameTicks->{{{-1,0,1},{}},{{min,0,max},{}}}],{f,{Ramp,LogisticSigmoid,Tanh}}], Spacer[15]]

Be careful using this as the final layer in regression!

Layers

Basic example: ConvolutionLayer

ConvolutionLayer[n,k] represents a convolution of an (image) 3-tensor against a “kernel bank” of n kernels, each of size k.

Combined with PoolingLayer, this is the key idea that allows nets to effectively address computer vision tasks

Each of the n kernels in the bank looks does something like this:

Each kernel in the bank produces one output channel in the output tensor.

For certain convolutional networks, early kernel banks resemble Gabor filters that also loosely mirror the response of neurons in the V1 stage of the visual cortex:

Chains

Simple composition of layers

NetChain represents a sequence of operations, with the operation (Subscript[op, i+1]) being applied to the result of operation (Subscript[op, i])

NetChain[{DotPlusLayer[5],ElementwiseLayer[Ramp],SummationLayer[]}]

The input (a vector of unspecified length)
...is fed into a DotPlusLayer whose output (a size-5 vector)
...is fed into an ElementwiseLayer whose output (a size-5 vector)
...is fed to a SummationLayer whose output (a scalar)
is returned.

Each layer in a NetChain must take one input and return one output

Chains

Simple examples

Classic MLP (Multi-Layer Perceptron) with scalar input and output:

net=NetChain[{DotPlusLayer[5],ElementwiseLayer[Tanh],DotPlusLayer[1]},"Input"->"Scalar","Output"->"Scalar"]

inet=NetInitialize[net,Method->{"Random","Biases"->2}];
Plot[inet[x],{x,-5,5}]

DotPlusLayer and ElementwiseLayer are common, so we can abbreviate them in NetChain:

NetChain[{5,Tanh,1},"Input"->"Scalar","Output"->"Scalar"]

Graphs

Complex composition of layers

NetGraph represents a set of layers and their connectivity

Allow for more complex dataflow than NetChain:

NetGraph[layers,connectivity]

Connectivity is in the form of rules (Subscript[layer, 1])->(Subscript[layer, 2])

Only way to use layers that have more than one input or output, by naming them with NetPort

Can click on a vertex to see more details

Graphs

Simple example: multiple outputs

Take a scalar input (modelled as a vector of length 1), apply two separate affine + non-linear transformations, and output both results:

graph=NetGraph[
{2,Tanh,1,1},
{1->2->3->NetPort["y"],2-> 4->NetPort["z"]},
"Input"->"Scalar","y"->"Scalar","z"->"Scalar"]

Randomly initialize the weights and biases:

net=NetInitialize[graph,Method->{"Random","Biases"->1}];

Apply to a single input, output is an association:

net[1]

Plot the outputs over a range of inputs:

ListLinePlot[net@Range[-4,4,.1]]

Training Data

Neural networks learn on training data

Neural networks are useful because they can be trained on examples (NetTrain)

We can think of training data as implicitly specifying a function, where training ‘tunes’ a net to approximate this function

When the net is performing classification, it is actually predicting posterior probabilities of the classes given the input, e.g. p(c|x)

For simple networks with one input and output, training data can be just a list of input->output pairs

For more complex applications with multiple inputs and outputs, an association or Dataset can be used to distinguish the different ‘ports’

Training Data

Examples

Three main image-related datasets exposed via ResourceData

More datasets will be exposed in 11.1

Use "TrainingDataset" property to get a browseable view, using "TrainingData" for NetTrain

MNIST (handwritten digits):

ResourceObject["MNIST"]

RandomSample[ResourceData["MNIST","TrainingDataset"],4]

CIFAR-10 (small images, one of 8 labels):

ResourceObject["CIFAR-10"]

RandomSample[ResourceData["CIFAR-10","TrainingDataset"],4]

CIFAR-100 (small images, one of 20 labels, one of 100 sub-labels):

ResourceObject["CIFAR-100"]

RandomSample[ResourceData["CIFAR-100","TrainingDataset"],4]

Prediction/Regression

Task: predicting a numeric value or tensor

Have a real number, or tensor of real numbers: y

Net computes a prediction y given input data x

Training data consists of  x->y pairs

The predicted y is not a distribution, but a maximum likelihood estimate of y

(in other words, the network has been trained to calculate (max_y) p(y|x) under certain noise assumptions)

Predicting arbitrary data from itself through a low-dimensional code (autoencoding)

(a kind of generative model that learns the underlying distribution of the data and can sample from it)

Method used by Predict are often superior for small-to-medium sized data, and are completely automatic

Using nets for regression can be adapted to specific purposes, model prior knowledge about target function

Suitable for high-dimensional inputs and outputs

Classification

Task: predicting one of several discrete classes

Have n distinct classes {(Subscript[c, 1]),(Subscript[c, 2]),…,(Subscript[c, n])}.

Net computes a distribution over these classes, in other words the probabilities p={(Subscript[p, 1]),(Subscript[p, 2]),…,(Subscript[p, n])} summing to 1.

(Subscript[p, i]) represents the net’s belief that the true class is (Subscript[c, i]).

Statistical view is that the net is computing the posterior p(c|x), where x is the input and c is the class.

Uses SoftmaxLayer to turn an arbitrary vector into a vector of probabilities that sum to 1.

Classification

Decoding the probability vector

Useful to add an ‘interpretation’ of a class probability vector

decoder=NetDecoder[{"Class",{"dog","cat","frog"}}]

Allow the net to return the most likely class by default

decoder[{0.8,0.1,0.1}]

decoder[{0.1,0.8,0.1}]

Second argument gives access to properties:

probs={0.1,0.8,0.1};

decoder[probs,"Probabilities"]

decoder[probs,"Entropy"]

decoder[probs,{"TopProbabilities",2}]

decoder[probs,{"Probability","frog"}]

Like an encoder, a decoder can be attached to the output of a net directly, shown later

Loss Functions

Measuring error on training data

Training is a minimization procedure, where we are minimizing some loss ϵ

The loss ϵ quantifies the error of the network’s predictions when compared to actual outputs

How to calculate loss depends on the task

NetTrain will display the loss on a log plot during training:

(training can be stopped manually if the loss starts to plateau)

Loss Functions

Choosing a loss function for a task

Loss functions are also represented by layers

Loss layers are attached to a net's output or outputs, and also take an input from the training data (the ‘ground truth’ the net is trying to predict):

MeanAbsoluteLossLayer – regression (Poisson noise)

MeanSquaredLossLayer – regression (Gaussian noise)

CrossEntropyLossLayer – classification

NetTrain will automatically choose loss functions for the net

Loss functions can be given explicitly to NetTrain, or embedded in your net.

Loss Functions

Regression loss

MeanAbsoluteLossLayer gives the sum of the (absolute) differences between prediction and data:

n=10;
data1=Range[n]/6;
data2=data1+0.3*RandomVariate[NormalDistribution[],n];
ListPlot[{data1,data2,{{-5,1}}},Joined->{True,False},PlotRange->{{1,n},All},PlotRangePadding->0.5,Filling->{2->{{1},Directive[Thick,Orange,Dotted]}},
PlotStyle->{Automatic,Automatic,Directive[Thick,Orange,Dotted]},Frame->True,FrameTicks->False,Axes->None,PlotLegends->{"prediction","data","error"}]

MeanSquaredLossLayer is similar, giving sum of the squared differences.

Minimizing this loss over all examples obviously requires the net to predict the data

Example:

loss=MeanAbsoluteLossLayer[]

loss[<|"Input"->{1,2,3},"Target"->{2,2,2}|>]

Mean[Abs[{1,2,3} - {2,2,2}]]

Loss Functions

Classification loss

Recall that when classifying inputs, the net computes a class distribution: a probability for each of the classes

CrossEntropyLossLayer computes an information-theoretic ‘distance’ between the class distribution q computed by the net and the ‘real’ distribution p

TraditionalForm[Style[Inactive[Set][H[p,q],-Sum[p[x]*Log[q[x]],x]],20]]

The true distribution comes from the training data, is usually a point-mass, with all probability lying in one class

... but need not be, if the training data is itself uncertain about the correct class, for example

Minimizing the CE loss requires the net assign a probability of 1.0 to the correct class and 0.0 to the other classes

Example:

loss=CrossEntropyLossLayer["Index"]

loss[<|"Input"->{0.8,0.1,0.1},"Target"->1|>]

loss[<|"Input"->{0.8,0.1,0.1},"Target"->2|>]

Training

Basic Theory

Training a network finds a (local) optimum for learned parameters, such as the w and b in DotPlusLayer

To train a parameter ϕ, the error or loss of the network, ϵ, is computed on subsets of the training data

The chain rule is then used to back-propagate the error ϵ through the network to produce a gradient ∂ϕ/∂ϵ at each parameter ϕ

Updating the parameter ϕ -> ϕ - r ∂ϕ/∂ϵ is known as gradient descent with learning rate r

The parameters of the network gradually converge on a local optimum, i.e. the error is minimized

More complicated update schemes like ADAM are used in practice

Training

Putting it all together with NetTrain

Training data is the CIFAR-10 object image database

trainingData=ResourceData["CIFAR-10","TrainingData"];
RandomSample[trainingData,5]

Create a decoder to interpret the net’s output as a class probability vector:

classes=Union@Values[trainingData]

decoder=NetDecoder[{"Class",classes}]

Create an encoder to take 32×32 images:

encoder=NetEncoder[{"Image",{32,32}}]

Create a simple convolutional network using stacked convolution and pooling layers, followed by a classification stage that produces a vector of 10 probabilities:

lenet=NetChain[
{ConvolutionLayer[20,5],Ramp,PoolingLayer[2,2],ConvolutionLayer[50,5],Ramp,PoolingLayer[2,2],FlattenLayer[],500,Ramp,10,SoftmaxLayer[]},
"Output"->decoder,"Input"->encoder
]

Train the network:

trained=NetTrain[lenet,trainingData,MaxTrainingRounds->4]

Predict the most likely classes for a set of images.

trained[{,,,,}]

For a specific (difficult) example, give the probabilities of the most likely label assignments.

trained[,{"TopProbabilities",3}]

Training

Measuring classifier performance

Obtain the test set

testData=ResourceData["CIFAR-10","TestData"];

Build a measurements object:

measurements=ClassifierMeasurements[trained,testData]

Obtain the accuracy on the test set:

measurements["Accuracy"]

Produce a confusion matrix plot:

measurements["ConfusionMatrixPlot"]

Find examples of trucks being misclassified as dogs:

measurements[{"Examples","truck"->"dog"}]

Future Directions

Roadmap

Recurrent nets: operate on arbitrary-length sequences, like text, audio, and timeseries

Higher-order constructs for constructing recurrent nets

Wide gallery of prebuilt nets

Multi-GPU training support

Checkpointing during training

Packing training data into a single file + efficient out-of-core streaming