Neural Networks

Sebastian Bodenstein, Wolfram Research

Join the Conversation #WolframTechConf

Introduction

State of Neural Networks in Wider Industry

Neural Networks in the WL

Upcoming features (where are things going?)

State of Neural Nets in Industry: Highlights

Machine Translation

Google switched to neural nets in late 2016 for Machine Translation

“As dawn broke over Tokyo, Google Translate was the No. 1 trend on Japanese Twitter, just above some cult anime series and the long-awaited new single from a girl-idol supergroup. Everybody wondered: How had Google Translate become so uncannily artful?”

https://www.nytimes.com/2016/12/14/magazine/the-great-ai-awakening.html?_r=0

Lots of new translation startups (much easier to build translation pipelines with neural nets!)

https://www.deepl.com/

State of Neural Nets in Industry: Highlights

Reinforcement Learning

AlphaGo

Go is impossible to solve via brute force

Solution was generally considered at least a decade away

AlphaGo beat human champion Lee Sedol in 2016

Key component: neural net

Few days ago: AlphaGo Zero (https://deepmind.com/blog/alphago-zero-learning-scratch/)

AlphaGo ELO 3700, AlphaGo Zero ELO >5000

State of Neural Nets in Industry: Highlights

Reinforcement Learning

Robotics

eg “Robots that Learn”

https://blog.openai.com/robots-that-learn/

list=Import["https://blog.openai.com/content/images/2017/05/stacking_demo.gif"];
ListAnimate[list]

State of Neural Nets in Industry: Highlights

Reinforcement Learning

Robotics

eg “Robots that Learn”

https://blog.openai.com/robots-that-learn/

list=Import["https://blog.openai.com/content/images/2017/05/stacking_demo.gif"];
ListAnimate[list]

State of Neural Nets in Industry: Highlights

Medical Imaging

Massive amounts of medical image data, very expensive to analyse with trained humans

eg “Dermatologist-level classification of skin cancer”, https://cs.stanford.edu/people/esteva/nature/

MRI, X-Rays, Diabetic Retinopathy, etc etc

For review, “A Survey on Deep Learning in Medical Image Analysis”, https://arxiv.org/abs/1702.05747

State of Neural Nets in Industry: Highlights

Other

Self-Driving Cars

Speech Recognition

Every good system uses neural nets now

NLP

Question Answering

Neural net only game in town on competition benchmarks (eg https://rajpurkar.github.io/SQuAD-explorer/)

Parsing

Sentiment Analysis

Physics

Huge amounts of detector data

“Deep Transfer Learning: A new deep learning glitch classification method for advanced LIGO”, Daniel George, https://arxiv.org/pdf/1706.07446.pdf

Finance

State of Neural Nets in Industry

Consequences of Widespread Applications

Specialized Hardware

Nvidia: Volta V100 (https://devblogs.nvidia.com/parallelforall/inside-volta/)

Specialized TensorCores gives peak 120 TFLOPs for mixed precision (appropriate for deep learning)

Wolfram Language 11.3 will have support upcoming V100, and 12.0 should be fully optimized for TensorCores

Google: Tensor Processing Unit (TPU)

Intel: Nervana NNP

https://www.intelnervana.com/intel-nervana-neural-network-processors-nnp-redefine-ai-silicon/

State of Neural Nets in Industry

Consequences of Widespread Applications

Massive Deep Learning Framework Investment

Highly optimized libraries

NVIDIA: cuDNN (https://developer.nvidia.com/cudnn)

Wolfram Language has been using this for the last few versions

Intel MKL DNN (https://github.com/01org/mkl-dnn)

Will give around 8x speedup on CPU to our current CPU implementation!

Still in Beta, Wolfram Language will use this when its production ready. Hopefully 11.3, but more likely 12.0

The big corporations have their own low-level frameworks

Microsoft (CNTK), Google (TensorFlow), Facebook (PyTorch + Caffe 2)

Apache MXNet (http://mxnet.incubator.apache.org/) is the only major framework not directly governed by a single coroporation (under Apache)

Higher-level frameworks, like Keras or Wolfram Langauge Neural Net Framework, build on top of these lower level frameworks.

Abstraction of things like TensorFlow very low

State of Neural Nets in Industry

Consequences of Widespread Applications

MXNet

Wolfram Neural Net Framework uses MXNet as a backend currently (and contribute to MXNet development)

All work that NVIDIA, Intel, Amazon, and others (~400 contributors) do on improving MXNet automatically benefits Wolfram Language users.

Full support for importing and exporting to MXNet (no lockin) since 11.2:

net=NetChain[{2,3,4},"Input"->1];
vpath=Export["temp.json",net,"MXNet"]

We will also benefit from the work on a compiler stack the MXNet people are working on (TVM: http://tvmlang.org/)

Useful for deploying nets on any device

Official framework of Amazon Web Services

In the process of making it easier to train nets in Wolfram Language on EC2

Amazon has spent a great deal of time optimizing MXNet for optimal performance on EC2

Neural Networks in the Wolfram Language

To properly understand the Neural Net Framework, it helps to understand what we want to achieve:

Vision 1: Powerful enough for Industrial-Grade Applications

Working on 50+ functions that need neural nets:

ImageRestyle, ImageIdentify:

ImageIdentify[]

ImageContents

ImageContents[]

Upcoming: Speech Recognition, Question Answering, Translation, etc

We are using Wolfram Language Framework to produce all of these!

Major driver of development adding features to support internal projects

Complexity of framework needs to be at a level appropriate for research-level work

Neural Networks in the Wolfram Language

Vision 2: Make pretrained nets first-class citizen of language

Training high-quality nets require very large amounts of data + computation

Generally not available to most users

Neural Net Repository is now available (since a few days ago!)

http://resources.wolframcloud.com/NeuralNetRepository

Add all internally trained nets + nets from other frameworks carefully converted

net=NetModel["Vanilla CNN for Facial Landmark Regression"]

img=;
landmarks=net[img];
HighlightImage[img,{PointSize[0.04],landmarks},DataRange->{{0,1},{0,1}}]

Neural Networks in the Wolfram Language

Vision 2: Make pretrained nets first-class citizen of language (continued)

Also: best models are hard to define yourself:

NetModel["ResNet-152 Trained on ImageNet Competition Data","UninitializedEvaluationNet"]

All nets underlying 50+ internal functions will be made available!

As these are symbolic nets, can easily be manipulated

Take[net,{1,5}]

High-level net paradigm:

Construct nets from pretrained pieces (often trained on vast datasets)

Finetune net on your smaller dataset

Neural Networks in the Wolfram Language

To properly understand the Neural Net Framework, it helps to understand what we want to achieve:

Vision 3: Make the WL framework as simple and pleasant to use as possible

1. Type System

net=NetChain[{EmbeddingLayer[3,4],LinearLayer[2]},"Input"->{2}]

Great, informative error handling:

net2=NetInitialize[net]

net2[{1,6}]

Neural Networks in the Wolfram Language

Vision 3: Make the WL framework as simple and pleasant to use as possible

2. Encoders/Decoders

Hide all feature extraction details in NetEncoder, and make it “ready to use” with NetDecoder”

model=NetModel["SqueezeNet V1.1 Trained on ImageNet Competition Data"]

model[]
model[,"TopProbabilities"]

Or from file:

file=File[Export["temp.jpg",]]

model[file]

This is painful and messy in lots of other frameworks (set up your own pre- and post-processing pipelines)

Eg. Caffe: https://github.com/BVLC/caffe/blob/master/examples/00-classification.ipynb

Neural Networks in the Wolfram Language

Vision 3: Make the WL framework as simple and pleasant to use as possible

3. Variable Length Support (new 11.2)

Hard to support variable-length sequences and train efficiently

Most frameworks force users to deal with this

Either via padding, bucketing, etc

Simple example:

enc=NetEncoder[{"Characters",{DigitCharacter,"+"}}];
makeRule[a_,b_]:=IntegerString[a]<>"+"<>IntegerString[b]->a+b;
trainingData=Flatten@Table[makeRule[i,j],{i,0,99},{j,0,99}];
RandomSample[trainingData,4]//InputForm

Define net:

net=NetChain[{UnitVectorLayer[],LongShortTermMemoryLayer[40],LongShortTermMemoryLayer[20],SequenceLastLayer[],LinearLayer[]},"Input"->enc,"Output"->"Real"]

trained=NetTrain[net,trainingData,BatchSize->64,MaxTrainingRounds->200,ValidationSet->Scaled[0.1]]

Evaluate:

Round@trained[{"5+2","10+25","9+11","44+44"}]

Neural Networks in the Wolfram Language

Vision 3: Make the WL framework as simple and pleasant to use as possible

3. Variable Length Support (new 11.2)

Full support for defining custom recurrent layers:

core=NetGraph[{ThreadingLayer[Plus]},{{NetPort["Input"],NetPort["State"]}->1}]
fold=NetFoldOperator[core,"Input"->{"Varying",2}]

fold[{{1,2},{3,4},{5,0},{6,1},{0,0}}]

Neural Networks in the Wolfram Language

Roadmap

11.0

First version with Neural Net framework

Supported GPU training

No RNN’s, but decent support for convnets

11.1 and 11.2

Full recurrent net support with variable length sequences

LSTM, GRU, Attention Layer, etc

Higher-level ops like NetFoldOperator, NetMapOperator

30+ new layers

Massive numbers of small improvements

Multithreaded Image File NetEncoder

Generator syntax for NetTrain

MXNet Importer/Exporter

Neural Networks in the Wolfram Language

Roadmap

11.2+ (Upcoming)

Useability

Many new nets don’t fit into a ‘static graph’: computational graph depends on input

PyTorch and Chainer are main examples of frameworks supporting ‘Dynamic Graphs’: MXNet also supports!

Weight sharing is also hard at the moment (eg using the same layer in different places in the graph with the same parameters)

Many nets are annoying to write as NetGraph (easier to write imperative program)

Solution: NetFunction (like Module)

NetFunction[{in},
{temp,conv=ConvolutionLayer[3,4]},
temp=If[Max[in]>2,in^2,conv[in]];
conv[temp]
]

Could become a very general autograd engine

Can differentiate very general programs

Similar functionality to https://github.com/HIPS/autograd

Custom Layers

Allow users to write custom layers as compiled functions via new Compiler

Eventually: support WL -> LLVM -> CUDA.

This will be a very powerful advanced user feature (we really want it internally)

Import/Export

Support for ONNX format (https://github.com/onnx/onnx)

Microsoft CNTK, PyTorch and Caffe 2 support it. Probably most other frameworks will support it.

Better support for deploying nets on other devices

Performance Enhancements

multi-GPU support

Use faster RNN implementations from NVIDIA cuDNN + Intel MKL DNN

Audio

Scalable Audio NetEncoder (11.3 probably)

CTC Loss Layer (11.3 probably)

Convolutions with variable length support

Neural Networks in the Wolfram Language

Roadmap

11.2+ (Upcoming)

Training

Our group has been working hard on setting up data + training infrastructure for our internal training.

Planning on exposing to users

Automate training on EC2

EC2 is getting V100 GPU’s

AWSLink

Storing our ML metadata in MongoDB

We wrote fast LibraryLink interface to Mongo C driver

MongoLink will be available in 11.3

MongoDB running locally: a decent solution for streaming out-of-core data for NN training

Mongo also supports random lookup

NetTrain will probably support direct training from Mongo collection

Neural Networks in the Wolfram Language

Conclusion

The WL Neural Net framework is industrial strength

uses latest NVIDIA libraries

backed by MXNet

It has a fantastic neural net repository

curation other frameworks

access to a huge collection of nets trained by WL

Things like pre-trained speech recognition nets are harder to find in other frameworks

Friendlier to use than most other frameworks

makes developers much more productive

No lock-in

easy to move to another framework if needed for any reason

Questions