Building Applications with the Wolfram Neural Net Repository

Tuseeta Banerjee and Matteo Salvarezza, Advanced Research Group, Wolfram Research

Join the Conversation #WolframTechConf

Outline

In this talk we will see what models have been added in the Neural Net Repository in the last one year and how to tailor the models in the Neural Net Repository for your applications

What’s new in the repository?

New models added in each section: Classification, Feature Extraction, Image Processing

New sections added: Semantic Segmentation, Speech and Audio models, Object Detection, Language Modeling

Building Applications: Use Cases from Summer Programs

Image Classification: Plug and  Connector Classifier (Rishab Nayak, Boston University)

Image Segmentation: Rooftop Recognition for Solar Energy Potential (Enrico Castro, IER-UNAM)

Building Applications: Live Training

Language Modeling

Sentiment Analysis

New Models in the Neural Net Repository

Classification

New image classification model

img=;

net=NetModel["Wide ResNet-50-2 Trained on ImageNet Competition Data"]

pred=net[img]

Obtain the top 5 guesses:

net[img,{"TopProbabilities",5}]

pred["Definition"]

Feature Extraction

Contextual Word Representations

Not just a lookup table for tokens

Produces context-dependent word embeddings

Fairly complicated architecture involving both 1-D convolutions and bidirectional recurrent layers

net=NetModel["ELMo Contextual Word Representations Trained on 1B Word Benchmark"]

Produces 3 distinct 1024-embeddings

2 are contextual

Suggested usage is to take the average of the embeddings

netevaluateWithContext[sentence_String]:=With[
{tokenizedSentence=TextElement[StringSplit[sentence]]},
AssociationThread[
Thread[{First[tokenizedSentence],sentence}],Mean@Values@NetModel["ELMo Contextual Word Representations Trained on 1B Word Benchmark"]@tokenizedSentence
]
]

embeddings=netevaluateWithContext["ELMo is amazing"];

Map[Dimensions,embeddings]

Find a semantically similar word given a context:

findSemanticNearestWord[{word_,context_},targetSentences_]:=First@Nearest[Association[Join@@Map[netevaluateWithContext,targetSentences]],netevaluateWithContext[context][{word,context}]]

findSemanticNearestWord[
{
"play","Harry receives a Nimbus 2000 in Philosopher's Stone so that he can play for Gryffindor house."
},
{
"Harry Potter and the Cursed Child is a two-part stage play written by Jack Thorne","Quidditch is a fictional sport played at Hogwarts"
}
]

For more information on NLP: Attend “What’s new in Neural Net Frameworks” and “Hands-on NN for NLP”

Image Processing

Super Resolution

Simple, linear architecture inspired by VGG

Takes an interpolated low-resolution image and refines the details to create a sharp upsampling

net=NetModel["Very Deep Net for Super-Resolution"]

img=;

zoom=3;

Downsample the image and re-upsample it with both the NN and naive interpolation

{nnUpscaled,naivelyUpscaled}=With[
{downScaled=ImageResize[img,Scaled[1/zoom]]},
{netevaluatevdsr[downScaled,zoom],
ImageResize[downScaled,Scaled@zoom,Resampling->{"OMOMS",3}]}
];

Compare the results

Manipulate[
image,
{{image,naivelyUpscaled},{naivelyUpscaled->"Naive",nnUpscaled->"NN"}},ControlType->SetterBar
]

Image Processing

Cycle GAN: Harry to Van Gogh

Trained using a novel technique for image translation with adversarial training

Doesn’t need matching image pairs

net=NetModel["CycleGAN Photo-to-Van Gogh Translation"]

With[{enc=NetEncoder[{"Image",ImageDimensions[img]}]},
NetReplacePart[net,{"Input"->enc,"Output"->NetDecoder[{"Image"}]}][img]
]

Other CycleGAN-s

New Sections in Repository: Semantic Segmentation

A tour of Diagon Alley

Pixelwise classification of street scenarios (Cityscapes Dataset)

Self-driving cars

NetModel["Ademxapp Model A1 Trained on Cityscapes Data"]

img2=ImageResize[,1500];

result[img2]

Currently 9 models available

For more information: “Neural Network for Images”.

New Sections in Repository: Language Modeling

Original English language model trained by Wolfram Research

Character-level model

Language models are often used to improve the performance of other systems, such as speech-to-text applications

NetModel["Wolfram English Character-Level Language Model V1"]

Generate text for 50 characters with temperature sampling (T = 0.4)

generateSample["Harry P",50,0.4]

Build a simple autocomplete system

autocomplete[word_]:=Block[
{obj=NetStateObject[NetModel["Wolfram English Character-Level Language Model V1"]]},StringJoin@Most@NestWhileList[obj,word,StringMatchQ[#,LetterCharacter|word]&]
]

autocomplete/@{"Harry P","Severus S","Albus D", "Hermione G", "Ron W"}

For more information on NLP: Attend “What’s new in Neural Net Frameworks” and “Hands-on NN for NLP”

New Sections in Repository: Speech to Text

Features convolutions on both time and frequency, as well as recurrent layers (GRU)

Trained with CTC loss

Uses beam search decoding in the space of output sequences

Backend for SpeechRecognize (new in 12)

NetModel["Deep Speech 2 Trained on Baidu English Data"]

audio=

output=StringJoin@NetModel["Deep Speech 2 Trained on Baidu English Data"][audio]

For other sections on Audio: Attend “ Neural Networks for Audio”/ “Understanding Speech”

Custom Classifier: Use Cases from Summer School Programs

Plug and Connector Classifier (Rishab Nayak, Boston University)

Image classification task: identify plugs and connectors

Transfer learning application using an image classification model trained on ImageNet

NetModel["Ademxapp Model A Trained on ImageNet Competition Data"]

Adapt the net to the new task

Swap the last classification layer, add image augmentation and new class list

ademNet=NetTake[NetModel["Ademxapp Model A Trained on ImageNet Competition Data"],{1,-3}]

connList={"MIDImale","MIDIfemale","RJ45male","RJ45female","TOSLINKmale","TOSLINKfemale","compositevideomalecable","compositevideoport","componentvideocable","componentvideoport","VGAcable","VGAport","dvicable","dviport","minidisplayportcable","minidisplayportport","HDMIcable","HDMIport","DisplayPortcable","DisplayPortport","usbamale","usbafemale","usbcmale","usbcfemale","microusbfemale","microusbmalecable","firewire800cable","firewire800port","firewire400port","firewire400cable","coaxcable","coaxport"};

Length@connList

myademnet=NetJoin[
NetPrepend[
ademNet,"augLayer"->ImageAugmentationLayer[{320,320},"ReflectionProbabilities"->{0.5,0.5}]
],
NetChain@<|
"linear"->LinearLayer[],
"softmax"->SoftmaxLayer[]
|>,
"Input"->NetExtract[ademNet,"Input"],
"Output"->NetDecoder[{"Class",connList}]
]

Let the last layers learn while keeping the initial layers frozen using LearningRateMultipliers

Training won’t run (we don’t have the training set!)

NetTrain[
myademnet,
trainingset,
All,
BatchSize->32,
LearningRateMultipliers->{
"6a"->1,
"7a"->1,
"bn7"->1,
"linear"->1,
_->0
},
TargetDevice->"GPU"
]

Import the pre-trained model and run it

plugID=Import[NotebookDirectory[]<>"trainedademNetv2.wlnet"]

plugID[]

Custom Image Segmentation Network: Use Cases from Summer School Programs

Rooftop Recognition for Solar Energy Potential (Enrico Castro, IER-UNAM)

Pixelwise classification task: segment rooftops from satellite images

Transfer learning application using a semantic segmentation model

netModel=NetModel["Ademxapp Model A1 Trained on PASCAL VOC2012 and MS-COCO Data"];

Adapt the net to the new task

Swap the last classification layer, modify input size

netModel500=NetReplacePart[netModel,"Input"->NetEncoder[{"Image",{500,500},"MeanImage"-> {0.485,0.456,0.406}}] ];
firstPartNet=NetDrop[netModel500,-3]

NetJoin[
firstPartNet,
NetChain@<|
"linear2"-> ConvolutionLayer[2,{3,3},"Stride"-> {1,1},"PaddingSize"-> {12,12},"Dilation"-> {12,12}],
"resize"-> ResizeLayer[{500,500}],
"softmax/1"->TransposeLayer[{1<-> 3,1<-> 2}],
"softmax/2"-> SoftmaxLayer[]
|>,
"Output"->NetDecoder[{"Class",{0,1},"InputDepth"->3}]
]

Import the pre-trained net and run it

roofdetector=Import[StringJoin[NotebookDirectory[],"RooftopRecognitionTrainedNet.wlnet"]]

geoImageResize[latitude_,longitude_]:=ImageResize[
GeoImage[GeoPosition[{latitude,longitude}],GeoRange->(Quantity[75, "m", meters, "Meters"]),GeoProjection->"Mercator",ImageSize->Small],
{500,500}
]

testImage=geoImageResize[40.11,-88.24]

mask=Image[roofdetector[testImage]]

Manipulate[image,{{image,testImage},{testImage->"GeoImage",mask->"Mask",ImageCompose[testImage,{mask,0.3}]->"Overlap"
}}]

Live Training: Pokemon Name Generator

Retrain the Wolfram Language Model from scratch using Pokemon names. Get the “teacher forcing” training architecture:

net=NetModel["Wolfram English Character-Level Language Model V1","TrainingNet"]

Get the data:

data=Activate@Thread@Inactive[StringJoin][EntityValue[EntityList["Pokemon"],"Name"],"."];

RandomSample[data,10]

Train the net for 30 rounds:

trained=NetTrain[net,<|"Input"->data|>,LossFunction->"Loss",MaxTrainingRounds->30,TargetDevice->"GPU"]

Creating the evaluation net requires some surgery to remove the “teacher forcing” architecture:

chars=Normal[NetExtract[trained,"Input"]]["Encoding"];
enc=NetEncoder[{"Characters",chars}];

evalNet=NetJoin[
NetExtract[trained,{"predict",1}],
NetChain@{SequenceLastLayer[],NetExtract[trained,{"predict",2,"Net"}],SoftmaxLayer[]},
"Input"->enc,
"Output"->NetDecoder[{"Class",Characters@StringJoin[Most[chars]]}]
]

Generate a name character by character until a full stop is produced:

generateSample[start_,temp_:1]:=With[{obj=NetStateObject[evalNet]},
StringJoin@NestWhileList[obj[#,{"RandomSample","Temperature"->temp}]&,start,#=!="."&,1,100]
]

Table[generateSample[RandomChoice@ToUpperCase@Alphabet[]],10]//Column

Live Training: Sentiment analysis with ELMo

Take text-processing dataset:

trainingData=ExampleData[{"MachineLearning","MovieReview"},"TrainingData"];
validationData=ExampleData[{"MachineLearning","MovieReview"},"TestData"];

trainingData[[;;3]]

Pre-compute the ELMo vectors on the training and the validation dataset (if available, GPU is recommended):

trainingDataELMo=Rule[
Total@Values[
NetModel["ELMo Contextual Word Representations Trained on 1B Word Benchmark"][Keys[trainingData],TargetDevice->"GPU"]/3.
],
Values[trainingData]
];

trainingData[[1,1]]

trainingDataELMo[[1,1]]//Dimensions

validationDataELMo=Total[Values[NetModel["ELMo Contextual Word Representations Trained on 1B Word Benchmark"][Keys[validationData],TargetDevice->"GPU"]]/3.]->Values[validationData];//AbsoluteTiming

Define a network that takes word vectors instead of strings for the text-processing task:

netArchitecture=NetChain[{DropoutLayer[],NetMapOperator[2],AggregationLayer[Max,1],SoftmaxLayer[]},"Output"->NetDecoder[{"Class",{"negative","positive"}}]]

Train the network on the pre-computed ELMo vectors:

trainResultsELMo=NetTrain[netArchitecture,trainingDataELMo,All,ValidationSet->validationDataELMo,MaxTrainingRounds->20,TargetDevice->"GPU"]

Check the classification error rate on the validation data:

trainResultsELMo["LowestValidationErrorRate"]

Compare the results with the performance of the same model trained on context-independent embeddings:

trainingDataGlove=Thread@Rule[
NetModel["GloVe 300-Dimensional Word Vectors Trained on Wikipedia and Gigaword 5 Data"][Keys[trainingData]],
Values[trainingData]
];

trainingDataGlove[[1,1]]//Dimensions

validationDataGlove=Thread@Rule[
NetModel["GloVe 300-Dimensional Word Vectors Trained on Wikipedia and Gigaword 5 Data"][Keys[validationData]],
Values[validationData]
];

trainResultsGlove=NetTrain[netArchitecture,trainingDataGlove,All,ValidationSet->validationDataGlove,MaxTrainingRounds->20,TargetDevice->"GPU"]

trainResultsGlove["LowestValidationErrorRate"]

Initialization Cell

netevaluatevdsr[img_,imgScale_,device_:"CPU"]:=Block[{net,interpolated,ycbcr,channels,resizedNet,diff,rgb},
net=NetModel["Very Deep Net for Super-Resolution"];
interpolated=ImageResize[img,Scaled[imgScale],Resampling->"Cubic"];(* upscale to the final size *)
ycbcr=ImageApply[{{0.257,0.504,0.098},{-0.148,-0.291,0.439},{0.439,-0.368,-0.071}}.#+{0.063,0.502,0.502}&,interpolated];
channels=ColorSeparate[ycbcr];
resizedNet=NetReplacePart[net,"Input"->NetEncoder[{"Image",ImageDimensions@interpolated,ColorSpace->"Grayscale"}]];
diff=Image@resizedNet[channels[[1]],TargetDevice->device];
ycbcr=ColorCombine[{channels[[1]]+diff,channels[[2]],channels[[3]]}];
rgb=ImageApply[{{1.164,0.,1.596},{1.164,-0.392,-0.813},{1.164,2.017,0.}}.#+{-0.874,0.532,-1.086}&,ycbcr];
rgb
]

netevaluatesemantic[img_,device_: "CPU"]:=Block[{net,resized,encData,dec,mean,var,prob},net=NetModel["Ademxapp Model A1 Trained on Cityscapes Data"];
resized=ImageResize[img,{1024}];
encData=Normal@NetExtract[net,"Input"];
dec=NetExtract[net,"Output"];
{mean,var}=Lookup[encData,{"MeanImage","VarianceImage"}];
prob=NetReplacePart[net,{"Input"->NetEncoder[{"Image",ImageDimensions@resized,"MeanImage"->mean,"VarianceImage"->var}],"Output"->Automatic}][resized,TargetDevice->device];
prob=ArrayResample[prob,Append[Reverse@ImageDimensions@img,19]];
dec[prob]]

net=NetTake[NetModel["Wolfram English Character-Level Language Model V1"],7];
sampler=NetTake[NetModel["Wolfram English Character-Level Language Model V1"],-1]
generateSample[start_,len_,temp_: 1]:=With[{obj=NetStateObject[net]},
StringJoin@NestList[sampler[obj[#]/temp,"RandomSample"]&,start,len]]

colors=Apply[RGBColor,{{128,64,128},{244,35,232},{70,70,70},{102,102,156},{190,153,153},{153,153,153},{250,170,30},{220,220,0},{107,142,35},{152,251,152},{70,130,180},{220,20,60},{255,0,0},{0,0,142},{0,0,70},{0,60,100},{0,80,100},{0,0,230},{119,11,32}}/255.,{1}];
labels={"wall","building","sky","floor","tree","ceiling","road","bed","windowpane","grass","cabinet","sidewalk","person","earth","door","table","mountain","plant","curtain","chair","car","water","painting","sofa","shelf","house","sea","mirror","rug","field","armchair","seat","fence","desk","rock","wardrobe","lamp","bathtub","railing","cushion","base","box","column","signboard","chest","counter","sand","sink","skyscraper","fireplace","refrigerator","grandstand","path","stairs","runway","case","pool","pillow","screen","stairway","river","bridge","bookcase","blind","coffee","toilet","flower","book","hill","bench","countertop","stove","palm","kitchen","computer","swivel","boat","bar","arcade","hovel","bus","towel","light","truck","tower","chandelier","awning","streetlight","booth","television","airplane","dirt","apparel","pole","land","bannister","escalator","ottoman","bottle","buffet","poster","stage","van","ship","fountain","conveyer","canopy","washer","plaything","swimming","stool","barrel","basket","waterfall","tent","bag","minibike","cradle","oven","ball","food","step","tank","trade","microwave","pot","animal","bicycle","lake","dishwasher","screen","blanket","sculpture","hood","sconce","vase","traffic","tray","ashcan","fan","pier","crt","plate","monitor","bulletin","shower","radiator","glass","clock","flag"};
indexToColor=Thread[Range[19]->colors];
result[img_,device_: "CPU"]:=Block[{mask,classes,maskPlot,composition},mask=netevaluatesemantic[img,device];
classes=DeleteDuplicates[Flatten@mask];
maskPlot=Colorize[mask,ColorRules->indexToColor];
composition=ImageCompose[img,{maskPlot,0.5}];
Legended[Row[Image[#,ImageSize->Large]&/@{maskPlot,composition}],SwatchLegend[indexToColor[[classes,2]],labels[[classes]]]]]