What’s new in the Neural Network Framework

Jérôme Louradour and Matteo Salvarezza
Advanced Research Group, Wolfram Research

Join the Conversation #WolframTechConf

Outline

Speed-ups

New training facilities

Parametrized NetModel

Higher Layer Capabilities

Capsule Networks

Transformers

Byte Pair Encoding (BPE)

Speed-ups

Much faster Recurrent Neural Networks (RNN) on CPU

gru = NetInitialize @ GatedRecurrentLayer[20,"Input"->{"Varying",10}];
$Version->First@AbsoluteTiming[Table[gru[RandomReal[1,{i,10}]],{i,50,150}]]

"11.3.0 for Linux x86 (64-bit) (March 7, 2018)"->0.523372

Multi-GPU: TargetDevice -> {"GPU", All}

TensorCore support: TargetDevice -> "GPU", WorkingPrecision -> "Mixed"

https://devblogs.nvidia.com/tensor-ops-made-easier-in-cudnn/

Support of NumericArray

NumericArray[RandomReal[1,{5,5}], "Real64"]

Choice of working precision: WorkingPrecision -> "Real32" / "Real64"

New training facilities

Monitoring

Option TrainingMeasurements

out=NetTrain["LeNet","FashionMNIST",All,ValidationSet->Scaled[0.2],MaxTrainingRounds->1,TrainingMeasurements->{"Recall","Precision","F1Score"}]

out["ValidationMeasurements"]

out["RoundMeasurements"]

Early-Stopping

Option TrainingStoppingCriterion

SeedRandom[2018];
x = RandomReal[{0.5,2.5},{1000}];
trainingData= Map[# -> Round[#]& , x];
validationData= Map[# ->Round[#  + RandomReal[0.4]]& , x];

RandomSample[trainingData,10]

net=NetChain[{8, Ramp, 4,SoftmaxLayer[]}];
out=NetTrain[net,trainingData, All, ValidationSet -> validationData,
TrainingStoppingCriterion-><|"Criterion"->"ErrorRate","Patience"->50|>]

out["ErrorRateEvolutionPlot"]

net=NetChain[{8, Ramp, 1}];
out=NetTrain[net,trainingData, All, ValidationSet -> validationData,
LossFunction->MeanSquaredLossLayer[],
 TrainingStoppingCriterion-><|"Criterion"->"Loss","Patience"->50,"RelativeChange"->0.01|>]

out["LossEvolutionPlot"]

Debugging

Associations can be passed in the list of output properties

NetTrain[net,trainingData,{
<|"Property"->"BatchLoss","Form"->"Plot","Interval"->"Batch"|>,
<|"Property"->"GradientsRMS","Form"->"Plot","PlotOptions"->{ScalingFunctions->"Linear"},"Interval"->"Round"|>,
<|"Property"->"GradientsRMS","Form"->"Plot","PlotOptions"->{ScalingFunctions->"Linear"},"Interval"->"Batch"|>
},MaxTrainingRounds->100,BatchSize->64]

New training facilities

Unlocking Reinforcement Learning and Active Learning

Interface with 	OpenAI Gym Reinforcement Learning environments: DeviceOpen

(RowDefault[,    , ])

Soon a tutorial on Reinforcement Learning

Access to several properties in the data generator, including the network (#Net)

{train,valid}=TakeList[RandomSample@ExampleData[{"MachineLearning","MovieReview"},"Data"], {500,100}];

dataGenerator[batchSize_]:=(
	$batches = Partition[train, 32];
	$numBatches = Length[$batches];
	$weights = Table[0.5, $numBatches];
	$lastSamplesIndices = None;
	$batchLosses = {};
	
	Function[
		If[NumericQ[#BatchLoss],
			$weights[[$lastSamplesIndices]]= #BatchLoss;
			AppendTo[$batchLosses, #BatchLoss]
		];
		$lastSamplesIndices = First @ RandomSample[
			$weights -> Range[$numBatches],
		1];
		$batches[[$lastSamplesIndices]]
	]);

netSentimentAnalysis=NetChain[{EmbeddingLayer[40],DropoutLayer[.3],GatedRecurrentLayer[32],SequenceLastLayer[],LinearLayer[],SoftmaxLayer[]},"Output"->NetDecoder[{"Class",{"positive","negative"}}],"Input"->NetEncoder[{"Tokens"}]]

NetTrain[netSentimentAnalysis, {dataGenerator[32], "RoundLength" -> $numBatches*32}, "BatchLossList", ValidationSet -> valid,BatchSize->32, MaxTrainingRounds->6]

ListLinePlot[{%,$batchLosses}]

Parametrized NetModels

Initial motivation: gather different NetModels from the Neural Net Repository into a single entity by using parameters

NetModel["ResNet-50 Trained on ImageNet Competition Data"]

NetModel["ResNet-101 Trained on ImageNet Competition Data"]

NetModel["ResNet-152 Trained on ImageNet Competition Data"]

NetModel[{"ResNet Trained on ImageNet Competition Data","Type"->"ResNet-152"}]

Many parametrized NetModels coming for 12

SketchRNN (Generative model for vector drawings)

SeNet (Image classification model, winner of ImageNet 2017)

MobileNet (Fast, lightweight convolutional neural networks)

BPEMB (BPE word embeddings for 275 languages)

Self-Normalizing Nets (Successful architecture to process generic numerical data)

Example 1: SketchRNN

Get a specific SketchRNN by specifying a parameter value

netCat=NetModel[{"SketchRNN Demo Trained on QuickDraw Data","Object"->"Cat"}]

Use the network to make sketches

netDraw[net_] := Block[
	{stateObject, lastPos, pos, state,maxLen, segments, time, action, lastAction, offset},
	stateObject = NetStateObject[net];
	lastPos = pos = {0, 0}; 
	state = {{0, 0, 0, 0, 0}};
  maxLen=300;
	segments = Table[{}, maxLen];
	time = 0; action = 1;
	While[time++ < maxLen, 
		lastAction = action;
		{offset, action} = stateObject[state, {"Sample", "Temperature" -> 0.01}];
		state = List @ Join[offset, UnitVector[3, action]];
		lastPos = pos; pos -= offset * {-1, 1};
		If[lastAction == 1, segments[[time]] = {lastPos, pos}];
		If[action == 3, Break[]];
	];
	Graphics[Line[segments]]
];

netDraw[netCat]

Try out a different one

netDog=NetModel[{"SketchRNN Demo Trained on QuickDraw Data","Object"->"Dog"}]

netDraw[netDog]

Parametrized NetModels have specific metadata about their own parameters

NetModel["SketchRNN Demo Trained on QuickDraw Data","ParameterNames"]

NetModel["SketchRNN Demo Trained on QuickDraw Data","ParametersAllowedValues"]

NetModel["SketchRNN Demo Trained on QuickDraw Data","Variants"]

NetModel["SketchRNN Demo Trained on QuickDraw Data","ParametersInformation"]

A default choice is also present

NetModel["SketchRNN Demo Trained on QuickDraw Data","DefaultVariant"]

NetModel["SketchRNN Demo Trained on QuickDraw Data"]

Example 2: Self-Normalizing Nets

The Parametrized NetModel paradigm also allow for on the fly generation of uninitialized architectures

NetModel["SELUNet","ParametersInformation"]

NetModel[{"SELUNet","Depth"->5,"Width"->32,"DropoutProbability"->0.3}]

NetModel["SELUNet","DefaultVariant"]

NetModel["SELUNet"]

Extension of layers

ReshapeLayer

Now supports Automatic and Inherited specs, as well as variable-length input:

Automatic infers one dimension from the others

Inherited copies the corresponding dimension from the output

ReshapeLayer[{Inherited,Automatic,2},"Input"->{"Varying",1,8}]

Automatic and Inherited adapt to the input specification, which can now be changed without having to redefine the layer entirely:

l=ReshapeLayer[{Automatic,3,Inherited},"Input"->{3,2,2,4}]

NetReplacePart[l,"Input"->{3,1,4,4}]

Convolution / Deconvolution / Pooling

Added support for “grouped” convolutions / deconvolutions

Legacy functionality (2012) originally appearing in Caffe in order to overcome GPU memory limitations

Now risen back to popularity for their speed (at the cost of some expressiveness), e.g. MobileNet

ConvolutionLayer[10,3,"ChannelGroups"->2,"Input"->{4,10,10}]

DeconvolutionLayer[10,3,"ChannelGroups"->2,"Input"->{4,10,10}]

One-dimensional deconvolutions

DeconvolutionLayer[10,3,"Input"->{4,10}]

Asymmetric padding specification for convolution / pooling layers

Previously available with PaddingLayer, now built-in

Also works for variable-length inputs, unlike PaddingLayer

Specified as a {before, after} pair for each axis: {{(Subscript[before, 1]), (Subscript[after, 1])}, {(Subscript[before, 1]), (Subscript[after, 1])}, ...} just like ArrayPad

Old, symmetric API still available

ConvolutionLayer[10,3,"PaddingSize"->{2,1},"Input"->{3,10,10}]

ConvolutionLayer[10,3,"PaddingSize"->{{3,1},{0,2}},"Input"->{3,10,10}]

New layers

Capsule Networks

OrderingLayer and ExtractLayer enable implementing Capsule Net without resorting to the internal MXLayer functionality

CapsNet was published in the NN repo for 11.3 using MXLayer

In 12, the MXLayer is now replaced by OrderingLayer and ExtractLayer

They are used to identify the capsule vector with highest norm (OrderingLayer) and select it for reconstruction (ExtractLayer)

NetModel["CapsNet Trained on MNIST Data"]

OrderingLayer

Works just like Ordering

Works on variable-length sequences

data=RandomReal[1,{100}];

Ordering[data,-7]

OrderingLayer[-7][data]

OrderingLayer[-2,"Input"->"Varying"][{{10.5,11.7},{5.1,1.7,21.3}}]

ExtractLayer

Extracts parts of an input array at runtime

Parts are specified by an additional port

ExtractLayer[]

ExtractLayer[]@<|"Input"->{3,9,4},"Position"->{2}|>

Can construct generic arrays from the parts of the input

ExtractLayer[]@<|"Input"->{3,9,4},"Position"->{{2},{2},{2},{3},{1}}|>

ExtractLayer[]@<|"Input"->{3,9,4},"Position"->{{{1},{3}},{{2},{1}}}|>

Supports variable length inputs in both ports:

l=ExtractLayer["Input"->{"Varying",6},"Position"->{"Varying",2}]

l@<|"Input"->RandomReal[2,{3,6}],"Position"->{{3,1},{2,1},{1,5}}|>

New layers

Transformers

(see animation)

New in 12.0:

AttentionLayer (previously SequenceAttentionLayer)

Key-Value attention

Scaling by the sequence length

Causal masking (for self-attention)

NormalizationLayer (previously InstanceNormalizationLayer)

Layer Norm

Instance Norm

NetMapThreadOperator

Key-Value Attention

keyValueAttention[numHiddens_, causal_:False] := NetGraph[<|
	"key" -> NetMapOperator[numHiddens],
	"value" -> NetMapOperator[numHiddens],
	"query" -> NetMapOperator[numHiddens],
	"attention" -> AttentionLayer["Dot", "ScoreRescaling" -> "LengthSqrt", "Mask"->If[causal,"Causal",None]]
	|>,
	{NetPort["Input"] -> "key" -> NetPort["attention","Key"],
	NetPort["Input"] -> "value" -> NetPort["attention","Value"],
	NetPort["Query"] -> "query" -> NetPort["attention","Query"]}
]

keyValueAttention[64]

Multi-Head Attention

multiHeadAttention[numHeads_, numHiddens_, args___] := NetGraph[
	Join[
		Table[keyValueAttention[numHiddens, args], numHeads],
		{CatenateLayer[2], NetMapOperator[numHiddens * numHeads]}
	],
	{Range[numHeads] -> numHeads+1 -> numHeads+2}
];

multiHeadAttention[8, 64]

LayerNorm

"The layer normalized recurrent layer re-centers and re-scales its activations"  (2016, Hinton)

NormalizationLayer[2,"Same","Input"->{"n",64}]

Shared weights

sharedEmbeddings[numTokens_,embeddingSize_] := With[{sharedWeights=NetSharedArray["EmbeddingWeights"]},
{
	EmbeddingLayer[embeddingSize, "Weights" -> sharedWeights, "Input" -> {"Varying", Restricted[Integer, numTokens]}],
	NetGraph[
  	{ConstantArrayLayer["Array" -> sharedWeights], TransposeLayer[], NetMapThreadOperator[DotLayer["Input"->{{Automatic},{Automatic,Automatic}}],<|"1"->1|>], NetMapOperator[ConstantPlusLayer["Input" -> numTokens ]]},
  	{1 -> 2, {NetPort["Input"], 2} -> 3 -> 4},
  	"Input" -> {"Varying", embeddingSize}
  ]
}]

sharedEmbeddings[255,512]

Blocks

multiHeadAttentionBlock[args__] := NetGraph[
	<|
		"attention" -> multiHeadAttention[args],
		"dropout" -> DropoutLayer[],
		"add" -> ThreadingLayer[Plus],
		"norm" -> NormalizationLayer[2,"Same"]
	|>,
	{"attention" -> "dropout", {NetPort["Query"], "dropout"} -> "add" -> "norm"}
];

selfAttentionBlock[args__] := With[{multihead = multiHeadAttentionBlock[args]},
	NetGraph[
		Normal[multihead],
		ReplaceAll[EdgeList[multihead], NetPort["Query"] -> NetPort["Input"]]
	]
];

feedForwardBlock[inputDim_, numHiddens_, normalization_:True] := NetGraph[
	<|
		"linear1" -> NetMapOperator[numHiddens],
		"relu" -> Ramp,
		"linear2" -> NetMapOperator[inputDim],
		"dropout" -> DropoutLayer[],
		"add" -> ThreadingLayer[Plus], (* Residual connection *)
		"norm" -> NormalizationLayer[2,"Same"] (* LayerNorm *)
	|>,
	{"linear1" -> "relu" -> "linear2" -> "dropout", {NetPort["Input"], "dropout"} -> "add" -> "norm"},
	"Input" -> {"Varying", Automatic}
];

multiHeadAttentionBlock[8,64]

selfAttentionBlock[8,64]

feedForwardBlock[512,512]

encoderBlock[attentionHeads_, attentionHiddens_, feedForwardHiddens_] := NetChain[{
	selfAttentionBlock[attentionHeads, attentionHiddens],
	feedForwardBlock[attentionHeads * attentionHiddens, feedForwardHiddens]
}]

encoderBlock[8,64,2048]

decoderBlock[attentionHeads_, attentionHiddens_, feedForwardHiddens_] := NetGraph[
	<|
	"self-attention"-> selfAttentionBlock[attentionHeads, attentionHiddens, True],
	"attention" -> multiHeadAttentionBlock[attentionHeads, attentionHiddens],
	"feed-forward" -> feedForwardBlock[attentionHeads * attentionHiddens, feedForwardHiddens]
	|>, {"self-attention" -> "attention" -> "feed-forward"}
]

decoderBlock[8,64,2048]

Full architecture

numStackEncoder = 6;
numStackDecoder = 6;

embeddingSize = 512;
characterEncoder = NetEncoder["Characters"];
numTokens = characterEncoder[["Count"]];
feedForwardHiddens = 2048;
attentionHiddens = 64;
attentionHeads = 8;

encoder=NetChain[
Table[encoderBlock[attentionHeads, attentionHiddens, feedForwardHiddens],numStackEncoder]
]

decoder=NetGraph[
Table[decoderBlock[attentionHeads, attentionHiddens, feedForwardHiddens],numStackDecoder],
Table[i->i+1,{i,numStackDecoder-1}]
]

{inputEmbedding, outputEmbedding}=sharedEmbeddings[numTokens,embeddingSize]

transformer=NetGraph[<|
"inputEmbedding"->inputEmbedding,
"encoder"->encoder,
"outputEmbedding"->inputEmbedding,
"decoder"->decoder,
"linear" -> outputEmbedding,
"softmax"-> SoftmaxLayer[]
|>,
{"inputEmbedding"->"encoder"->NetPort["decoder","Input"],
NetPort["PreviousOutput"]->"outputEmbedding"->NetPort["decoder","Query"],
"decoder"->"linear"->"softmax"
}
]

Positional Encoding

numEmbeddings = 30;

coeffs1[numEmbeddings_] := coeffs1[numEmbeddings] = Table[10000.^(2*i/numEmbeddings),{i,(numEmbeddings+1)/2}];
coeffs2[numEmbeddings_] := If[Mod[numEmbeddings,2]==0, coeffs1[numEmbeddings], Most[coeffs1[numEmbeddings]]];

positionalEmbedding1[numEmbeddings_][x_List] := N @ Sin[Outer[#1/#2&, Range[Length[x]],coeffs1[numEmbeddings]]];
positionalEmbedding2[numEmbeddings_][x_List] := N @ Cos[Outer[#1/#2&, Range[Length[x]],coeffs2[numEmbeddings]]];

positionalEmbedding[numEmbeddings][x_List] := CatenateLayer[2][{positionalEmbedding1[numEmbeddings][x],positionalEmbedding2[numEmbeddings][x]}];

ListLinePlot[Transpose @ positionalEmbedding[numEmbeddings][Range[100]]]

positionalEmbedding[30][Range[100]]//Transpose//ArrayPlot

positionalEncoder[numEmbeddings_] := NetEncoder[{"Function", positionalEmbedding[numEmbeddings], {"Varying", numEmbeddings}}]

positionalEncoder[30]

positionalEncoder[30][Range[100]]//Transpose//ArrayPlot

transformer2=NetGraph[<|
"inputEmbedding"->inputEmbedding,
"inputCombine"-> ThreadingLayer[#1+#2&],
"encoder"->encoder,
"outputEmbedding"->inputEmbedding,
"outputCombine"->  ThreadingLayer[#1+#2&],
"decoder"->decoder,
"linear" -> outputEmbedding,
"softmax"-> SoftmaxLayer[]
|>,
{{"inputEmbedding",NetPort["Input2"]}->"inputCombine"->"encoder"->NetPort["decoder","Input"],
NetPort["PreviousOutput"]->"outputEmbedding",
{"outputEmbedding",NetPort["PreviousOutput2"]}->"outputCombine"->NetPort["decoder","Query"],
"decoder"->"linear"->"softmax"
},
"Input2"->positionalEncoder[embeddingSize],
"PreviousOutput2"->positionalEncoder[embeddingSize]]

Byte Pair Encoding (BPE)

Subword tokenization

Character-level encoding

Leave a lot of additional work to the network, e.g. figure out the spelling

Very simple to set up, just segment the input into its characters and assign a unique integer code to each

Low-level approach

enc=NetEncoder[{"Characters",Append[Alphabet[]," "]}]

codes=enc["can you imagine an imaginary menagerie manager imagining managing an imaginary menagerie"]

Normal[enc]["Encoding"][[codes]]

Word-level encoding

Simplify the task for the network

Dealing with out-of-vocabulary tokens is inevitable

Can’t possibly assign unique integer codes to acronyms, neologisms, foreign expressions, compound words, misspelled words, etc

High-level approach

enc=NetEncoder[{"Tokens"}]

codes=enc["can you imagine an imaginary menagerie manager imagining managing an imaginary menagerie"]

Append[Normal[enc]["Tokens"],"??"][[codes]]

Subword encoding allows word subparts as individual tokens

Mid-level approach

Aims at the best of both worlds

Keep separate tokens for common words, e.g. “hello”

Use subpieces to represent rare words, e.g. “electro” + “encephalo” + “gram”

By allowing segmentation down to character-level in extreme cases, no out-of-vocabulary tokens can be generated

Particularly well suited for machine translation

If the two languages share the same alphabet, BPE can be trained on the concatenation of both corpuses

bpeEnc=NetEncoder[{"BPETokens",pieces,"AdditionalWhitespace"->Left}]

codes=bpeEnc["can you imagine an imaginary menagerie manager imagining managing an imaginary menagerie"]

Normal[bpeEnc]["Tokens"][[codes]]

BPE is a trained tokenization algorithm

Tokens are generated based on the structure of training data

Only evaluation of pre-trained BPE models will be supported in 12

WL 12 will support importing of BPE models trained with sentencepiece

Current reference implementation for BPE

Provides very simple command line scripts

% spm_train --input=<input> --model_prefix=<model_name> --vocab_size=8000 --character_coverage=1.0 --model_type=<type>

Import a sentencepiece model with

NetEncoder[{"BPETokens",File["path/to/pretrained_sentencepiece.model"]}]