Multi-framework Neural Networks

Matteo Salvarezza, Wolfram Research

(RowDefault[, (Spacer1), , (Spacer1), ])  #WolframTechConf

ABSTRACT

In this talk I will present NetExternalObject, a new symbol in Wolfram Language 13.2. It exposes new functionality to interface with external deep learning frameworks and run neural networks in their native format without having to manually install any external software.

Setup Step

In the Wolfram Language 13.2 Beta released for the WTC, NetExternalObject might not work properly on all machines

Please run the following code to download a 20MB patch and load it into the current session

download=URLDownload["https://amoeba.wolfram.com/index.php/s/8CJErWMj7LmxwBr/download"];
ExtractArchive[download,$TemporaryDirectory,OverwriteTarget->True];
tempPacletDir=FileNameJoin@{$TemporaryDirectory,"ONNXRuntimeLink"};
PacletDirectoryLoad[tempPacletDir];
Get["ONNXRuntimeLink`"];

This code doesn’t make any permanent change to the Wolfram Language installation, which means that the download must be performed again in future sessions

In order to avoid having to repeat the download, backup tempPacletDir in a safe location pacletDir where it can be loaded from in the future:

PacletDirectoryLoad[pacletDir];
Get["ONNXRuntimeLink`"];

Introduction to NetExternalObject

NetExternalObject instances are created via Import by specifying the “NetExternalObject” property:

obj=Import["ExampleData/onnx_example.onnx","NetExternalObject"]

It is designed as a generic interface to wrap several distinct evaluation engines, each being able to run nets stored in their respective file format

Currently only the ONNX (Open Neural Network eXchange) format is supported

An open, cross-platform format to represent machine learning models

Evaluation engine provided by ONNXRuntime

NetExternalObject has the look and feel of a neural network in the Wolfram Language, however it can only be evaluated on inputs:

obj[{{1,2,3},{4,5,6}}]

It is not part of the neural network framework by any other means

It cannot be combined with other layers:

NetChain[{obj,ElementwiseLayer[Ramp]}]

It cannot be modified:

NetReplacePart[obj,"X"->{4,5}]

It cannot be trained:

NetTrain[obj,{RandomReal[1,{2,3}]-><|"Y1"->RandomReal[1,{2,5}],"Y2"->RandomReal[1,{2,5}]|>}]

Application example

NetExternalObject can run anything that its supported evaluation engines can, thus allowing generic deep learning applications

As example, let’s run a face detection pipeline using an external ONNX model:

url="https://github.com/onnx/models/raw/main/vision/body_analysis/ultraface/models/version-RFB-640.onnx";

net=Import[url,"NetExternalObject"]

Get a test image and preprocess it

NetEncoder cannot be directly attached to a NetExternalObject (at least yet), but it can be applied to the image separately and it’s still convenient to use:

img=;

encoder=List@*NetEncoder[{"Image",{640,480}}]

imgData=encoder[img]
Dimensions[imgData]

Run the net on the encoded image:

netOutput=net[imgData]

The net produces the coordinates 17,640 bounding boxes together with the probability that they contain a face:

Dimensions/@netOutput

rectangles=Rectangle[{#1,1-#4},{#3,1-#2}]&@@@netOutput[["boxes",1]];
probs=netOutput[["scores",1,All,2]];

The bounding boxes and probabilities should be aggregated via the non-maximum suppression algorithm, available via ResourceFunction:

nonMaxSup=ResourceFunction["NonMaximumSuppression"]

finalRectangles=nonMaxSup[rectangles->probs,AcceptanceThreshold->.9]

Show the face detection result:

HighlightImage[img,finalRectangles,DataRange->{{0,1},{0,1}}]

A wide array of other applications are possible

Comparison with model conversion: introduction

NetExternalObject can be seen as an alternative way to import ONNX models into the Wolfram Language

Since version 12.1, ONNX models could be imported as proper nets in the neural network framework (NetChain, NetGraph, layers, ...):

Import["ExampleData/onnx_example.onnx"]

The above code is equivalent to selecting the default “Net” property, while NetExternalObject is selected via the property with the same name:

net=Import["ExampleData/onnx_example.onnx","Net"]

obj=Import["ExampleData/onnx_example.onnx","NetExternalObject"]

Although equivalent at first sight, these two options are deeply different:

The “Net” property analyzes the model file and converts every single layer of the model to its Wolfram Language equivalent, returning a net just like any other net that can be created with the NN framework

The resulting net is integrated with all the functionality of the NN framework (NetReplacePart, NetExtract, NetJoin, NetTrain, ...)

It is ran by the evaluation engine used by the NN framework (MXNet)

The “NetExternalObject” property imports the model file but treats it as a black box

Not integrated with the NN framework

It is ran by its native engine (currently only ONNXRuntime for ONNX models)

Comparison with model conversion: batch dimension

One evident difference lies in the treatment of input dimensions

Compare the input shape of the NetGraph and the NetExternalObject:

{net,obj}

In ONNX and most other external frameworks, some common layers like LinearLayer, ConvolutionLayer and PoolingLayer are commonly defined as acting on a batch of examples as opposed to a single one.

In the Wolfram Language these layers can freely act on both a single example and a batch (just like Listable functions):

l=NetInitialize@LinearLayer[2,"Input"->{3}]

l[{1,2,3}]
l[{{1,2,3},{4,5,6}}]
l[{{1,2,3},{4,5,6},{7,8,9}}]

But in external frameworks the batch dimension is explicit: layers can only act on batches of a given size. As a consequence of this, their inputs have an extra dimension w.r.t their WL counterparts, called the batch dimension

When converting these operators to/from ONNX the batch dimension is added/removed

For example, export a PoolingLayer to ONNX. The batch dimension is added in the file:

l=PoolingLayer[2,"Input"->{2,3}]
file=Export[FileNameJoin@{$TemporaryDirectory,"model.onnx"},l];

Import[file,"RawData"][["graph","input",1,"type","tensor_type","shape","dim",All,"dim_value"]]

When the file is converted back into the Wolfram Language, the batch dimension is removed and the original layer is recovered:

net=Import[file]

When the file is imported as NetExternalObject, the batch dimension appears explicitly:

obj=Import[file,"NetExternalObject"]

The converted layer and the NetExternalObject perform the same computation:

input=RandomReal[1,{2,3}];

net[input]
obj[{input}]

Another example with a batch size of 10:

file=Export["model.onnx",ByteArray[],"Binary"];

Import[file]

Import[file,"NetExternalObject"]

The extra batch dimension doesn’t always appear. It is only added if layers which ONNX defines as acting on batches (“batched” layers) are involved:

l=AggregationLayer[Mean,"Input"->{2,3}]
file=Export[FileNameJoin@{$TemporaryDirectory,"model.onnx"},l];

Import[file,"NetExternalObject"]

The extra batch dimension “propagates” to non-batched layers if batched layers are involved:

chain=NetInitialize@NetChain@{AggregationLayer[Mean,"Input"->{2,3}],LinearLayer[3]}
file=Export[FileNameJoin@{$TemporaryDirectory,"model.onnx"},chain];

Import[file,"NetExternalObject"]

Import[file]

Comparison with model conversion: weights as inputs

Another notable difference is that uninitialized layers have extra inputs in NetExternalObject

Roundtripping an initialized ConvolutionLayer creates a NetExternalObject with a single input (having an extra batch dimension as discussed):

l=NetInitialize@ConvolutionLayer[2,3,"Input"->{3,10,10}]
Export["file.onnx",l];
Import["file.onnx","NetExternalObject"]

If the layer is uninitialized, the roundtripped NetExternalObject exposed extra inputs for the missing weights:

Notice that the extra weight inputs do not have the batch dimension

l=ConvolutionLayer[2,3,"Input"->{3,4,4}]
Export["file.onnx",l];
obj=Import["file.onnx","NetExternalObject"]

Plugging weights into the ConvolutionLayer and feeding the same weights into the NetExternalObject gives the same result:

weights=RandomReal[1,{2,3,3,3}];
biases={2,3};
input=RandomReal[1,{3,4,4}];

NetReplacePart[l,{"Weights"->weights,"Biases"->biases}]@input
obj@<|"Input"->{input},"1_Weights"->weights,"1_Biases"->biases|>

This happens because ONNX (as most of the external frameworks) effectively treats weights as additional inputs

They don’t show up the initialized case because the value of the weights is provided as a default value for those inputs (“initializer”)

Final discussion & future development

Why adding another way to import ONNX models as black boxes when it’s already possible to convert them as proper nets?

Model conversion to WL is a complicated and error-prone process, mostly because of the differences discussed previously (batch dimensions and weights an inputs)

Import into NetExternalObject is way easier to develop/maintain and much more reliable, as solid as using ONNXRuntime directly

What’s to come

NetExternalObject will be extended to incorporate other formats (for which no dedicated importer exists in the language):

MXNet (dedicated importer was deprecated)

PyTorch

TensorFlow

Other?

NetExternalObject is the natural entry point to incorporate other NN evaluation engines into the Wolfram Language, a first step towards their adoption as additional backends of the neural nnetwork framework alongside MXNet