New Additions to Wolfram Neural Net Repository

Maria Sargsyan,  Wolfram Research

(RowDefault[, (Spacer1), , (Spacer1), ])  #WolframTechConf

ABSTRACT

The Wolfram Neural Net Repository consistently evolves, offering a variety of pre-trained models to address diverse computational needs. This presentation will introduce the latest models added to the repository, covering areas like natural language processing, computer vision, and audio analysis. Attendees will learn about the functionalities and potential applications of these models.

OUTLINE

NetModel Overview

Key Features of NetModel

NetModel Retrieval

Net Information

Statistics on the Models in the WNNR

Distribution of NetModels by Task and Input Type

NetModels Parameters Count Over the Years

Evolution of Layer Usability in WNNR Over the Years

Recent Additions Summary By TaskType

Highlights on the Latest Models

Classification

Application Specific Classification

Vision Transformer

Action Recognition Nets

Speech Recognition

Semantic Segmentation

Multimodal Feature Extractors

How to Contribute?

Related Resources

NetModel Overview

Key Features of NetModel

https://resources.wolframcloud.com/NeuralNetRepository

Extensive Collection: NetModel provides access to a comprehensive and continuously growing repository of neural networks. These models have been trained on various datasets and optimized for different tasks, such as image recognition, natural language processing, generative tasks, and more.

Obtain available  pre-trained neural net models:

NetModel[]

Detailed Metadata: Each model in the repository comes with metadata that provides insights into its architecture, the dataset it was trained on, its performance metrics, as well as the original "SourceMetaData".

Get all the possible properties:

NetModel["EfficientNet Trained on ImageNet","Properties"]

NetModel["EfficientNet Trained on ImageNet","InputDomains"]

NetModel["EfficientNet Trained on ImageNet","TaskType"]

NetModel["EfficientNet Trained on ImageNet","SourceMetadata"]

NetModel Overview

NetModel Retrieval

Easy Access: With a simple command or function call, users can immediately access a pre-trained model from the repository. This eliminates the need to manually search, download, or train models from scratch.

Get the default evaluation net:

NetModel["EfficientNet Trained on ImageNet"]

NetModel["EfficientNet Trained on ImageNet","UninitializedEvaluationNet"]

Inspect  the  available  parameters:

NetModel["EfficientNet Trained on ImageNet","ParametersInformation"]

Pick a non-default net by specifying the parameters:

NetModel[{"EfficientNet Trained on ImageNet","Architecture"->"B1"},"UninitializedEvaluationNet"]

NetModel Overview

Net Information

Information[ElementwiseLayer[Ramp],"Properties"]

Obtain the total number of parameters:

Information[NetModel["LeNet Trained on MNIST Data"],"ArraysTotalElementCount"]

Extract "InputPorts" and "OutputPorts":

Information[NetModel["LeNet Trained on MNIST Data"],"InputPorts"]

Information[NetModel["LeNet Trained on MNIST Data"],"OutputPorts"]

Obtain the layer type counts:

Information[NetModel["LeNet Trained on MNIST Data"],"LayerTypeCounts"]

Statistics on the Models in the WNNR

Distribution of NetModels by TaskType and InputDomain

Statistics on the Models in the WNNR

NetModels Parameters Count Over the Years

Statistics on the Models in the WNNR

Evolution of Layer Usability in WNNR Over the Years

Statistics on the Models in the WNNR

Recent Additions Summary By TaskType

Highlights on the Latest Models

Classification

Application  Specific  Classification

NetModel["Optic-Net Classifier for Retinal Diseases"][]

NetExtract[
 NetModel["Optic-Net Classifier for Retinal Diseases"], {"Output", "Labels"}]

Vision Transformer

NetModel["Vision Transformer Trained on ImageNet Competition Data"]

Classify an image:

NetModel["Vision Transformer Trained on ImageNet Competition Data"][]

Obtain the probabilities of the 10 most likely entities predicted by the net.

NetModel["Vision Transformer Trained on ImageNet Competition Data"][,{"TopProbabilities",10}]

Define a function to visualize the attention matrix on an image:

visualizeAttention[img_Image,attentionMatrix_]:=Block[{heatmap,wh},
	wh=ImageDimensions[img];
	heatmap=ImageApply[{#,1-#,1-#}&,ImageAdjust@Image[attentionMatrix]];
	heatmap=ImageResize[heatmap,wh*256/Min[wh]];
	ImageCrop[ImageCompose[img,{ColorConvert[heatmap,"RGB"],0.4}],ImageDimensions[heatmap]]
]

Extract attention weights:

attentionMatrix=Transpose@NetModel["Vision Transformer Trained on ImageNet Competition Data"][,NetPort[{"encoder",-1,"attention","attention","AttentionWeights"}]];

Dimensions[attentionMatrix]

Extract the attention weights between the "classification token" and the input patches. These weights can be interpreted as which patches in the original image the net is "looked at" in order to perform the classification:

classifierAttention=attentionMatrix[[All,1,2;;]];
{numHeads,numPatches}=Dimensions[classifierAttention];
classifierAttention=ArrayReshape[classifierAttention,{numHeads,Sqrt[numPatches],Sqrt[numPatches]}];

visualizeAttention[,Mean[classifierAttention]]

Map[visualizeAttention[,#]&,classifierAttention]

Highlights on the Latest Models

Classification

Action Recognition

3D-Inflated ResNet-50 Trained on Kinetics 400 Data

Channel-Separated Video Action Classification Net Trained on Kinetics-400 Data

SlowFast Video Action Classification Trained on Kinetics-400 Data

X3D Video Action Classification Trained on Kinetics-400 Data

NetModel[
   "SlowFast Video Action Classification Trained on Kinetics-400 Data"]

video = ResourceData["Sample Video: Barbecuing"]

NetModel[
  "SlowFast Video Action Classification Trained on Kinetics-400 Data"][video, {"TopProbabilities", 5}]

Highlights on the Latest Models

Speech Recognition

Wav2Vec2 Trained on Multiple Datasets

Wav2Vec2 XLSR-53 Trained on Multilingual Data

Wav2Vec2 Trained on LibriSpeech Data

NetModel["Wav2Vec2 XLSR-53 Trained on Multilingual Data","ParametersInformation"]

NetModel["Wav2Vec2 Trained on LibriSpeech Data"]

netevaluate[audio_]:=Module[{chars},chars=NetModel["Wav2Vec2 Trained on LibriSpeech Data"][audio];
StringReplace[StringJoin@chars,"|"->" "]]

record=AudioCapture[]

netevaluate[record]

Highlights on the Latest Models

Semantic Segmentation

U2-Net Trained on DUTS-TR Data

DoubleU-Net Trained on Medical Image Segmentation Datasets

YOLO V8 Segment Trained on MS-COCO Data

Define  evaluation function:

netevaluate[net_,img_,device_:"CPU"]:=Round@ArrayResample[net[img,TargetDevice->device],Reverse@ImageDimensions[img],Resampling->"Bilinear"];

NetModel["U2-Net Trained on DUTS-TR Data"]

mask=netevaluate[NetModel["U2-Net Trained on DUTS-TR Data"],];

ArrayPlot[mask]

NetModel["YOLO V8 Segment Trained on MS-COCO Data"]

Define  evaluation function:

yolov8evaluate[net_,img_,detectionThreshold_:.45,overlapThreshold_:.45]:=Module[
];

testImage=;

detection=yolov8evaluate[NetModel["YOLO V8 Segment Trained on MS-COCO Data"],]

HighlightImage[testImage,GroupBy[Thread[detection["Classes"]->detection["Masks"]],First->Last]]

Highlights on the Latest Models

Multimodal Feature Extractors

CLIP Multi-domain Feature Extractor

OpenCLIP Multi-domain Feature Extractor Trained on LAION-2B Data

OpenCLIP Multi-domain Feature Extractor Trained on DataComp-1B Data

CoCa Image Captioning Nets Trained on LAION-2B Data

images={,,,,,,,,};

descriptions={"a photo of Henry Ford","a photo of Audrey Hepburn","a woman sitting on ottoman in front of paintings","a dog looking out from a car window","a man working out","a clos-up photo of a cheeseburger","a photo of a church","a metro sign in Paris","Niccolo Macchiavelli Concrete Statue"};

textFeatures=NetModel[{"OpenCLIP Multi-domain Feature Extractor Trained on LAION-2B Data","InputDomain"->"Text"}][descriptions];
imgFeatures=NetModel["OpenCLIP Multi-domain Feature Extractor Trained on LAION-2B Data"][images];

ArrayPlot[
DistanceMatrix[textFeatures,imgFeatures,DistanceFunction->CosineDistance],
ColorFunction->"BlueGreenYellow",
PlotLegends->Automatic,
FrameTicks->{
{Transpose[{Range[Length[descriptions]],descriptions}],None},{None,Transpose[{Range[Length[images]],ImageResize[#,60]&/@images}]}
}
,ImageSize->Full
]

descriptions={"Blossoming rose on textbook among delicate petals","Photo of foggy forest","A woman with eyeglasses smiling","Calm body of lake between mountains","A portrait of a man with red hair taking a picture with an analog camera","A portrait of a woman with red hair taking a picture with an analog camera","Smiling man surfing on wave in ocean","A portrait of a woman with red hair taking a picture with a phone","Elderly woman carrying a baby"};

Note  that the scores are relative:

textFeatures=NetModel[{"OpenCLIP Multi-domain Feature Extractor Trained on LAION-2B Data","InputDomain"->"Text"}][descriptions];

ArrayPlot[
DistanceMatrix[textFeatures,imgFeatures,DistanceFunction->CosineDistance],
ColorFunction->"BlueGreenYellow",
PlotLegends->Automatic,
FrameTicks->{
{Transpose[{Range[Length[descriptions]],descriptions}],None},{None,Transpose[{Range[Length[images]],ImageResize[#,60]&/@images}]}
}
,ImageSize->Full
]

Highlights on the Latest Models

Zero-shot image classification

By using the text and image feature extractors together, it's possible to perform generic image classification between any set of classes without having to explicitly train any model for those particular classes (zero-shot classification). Obtain the FashionMNIST test data, which contains ten thousand test images and 10 classes:

testData=ResourceData["FashionMNIST","TestData"];

Display a few random examples from the set:

RandomChoice[testData,5]

Get a mapping between class IDs and labels:

idToLabel=ResourceData["FashionMNIST","ClassLabels"]

Generate the text templates for the FashionMNIST labels and embed them. The text templates will effectively act as classification labels:

labelTemplates="This is a photo of a "<>#&/@ToLowerCase[Values@idToLabel]

textEmbeddings=NetModel["OpenCLIP Multi-domain Feature Extractor Trained on LAION-2B Data","InputDomain"->"Text"][labelTemplates];

Dimensions[textEmbeddings]

Classify an image from the test set. Obtain its embedding:

img=testData[[5634,1]]

imgFeatures=NetModel["OpenCLIP Multi-domain Feature Extractor Trained on LAION-2B Data","InputDomain"->"Image"][img];

Dimensions[imgFeatures]

The result of the classification is the description of the embedding that is closest to the image embedding:

Nearest[textEmbeddings->labelTemplates,imgFeatures,DistanceFunction->CosineDistance]

Find the top 10 description nearest to the image embedding:

SortBy[Rule@@@MapAt[labelTemplates[[#]]&,Nearest[textEmbeddings->{"Index","Distance"},imgFeatures,10,DistanceFunction->CosineDistance],{All,1}],Last]

Obtain the accuracy of this procedure on the entire test set. Extract the features for all the images (if a GPU is available, setting TargetDevice -> "GPU" is recommended as the computation will take several minutes on CPU):

imageEmbeddings=NetModel["OpenCLIP Multi-domain Feature Extractor Trained on LAION-2B Data","InputDomain"->"Image"][testData[[All,1]],TargetDevice->"CPU"];

Dimensions[imageEmbeddings]

Calculate the distance matrix between the computed text and image embeddings:

distanceMatrix=DistanceMatrix[imageEmbeddings,textEmbeddings,DistanceFunction->CosineDistance];

Obtain the top-1 predictions:

predictedClassIDs=Flatten[Ordering[#,1]&/@distanceMatrix]-1;

Obtain the final classification results:

ClassifierMeasurements[idToLabel/@predictedClassIDs,idToLabel/@testData[[All,2]]]

Highlights on the Latest Models

Evaluation function

Define an evaluation function that uses all model parts to obtain the image features and automate the caption generation:

decodeStep[textDecoder_, textEncoderBare_, imageEmbeddings_, temperature_, tokenDict_,targetDevice_,topK_][textTokens_]:= 
Append[
]
Options[netevaluate] ={

};
netevaluate[input:(_?ImageQ|_?VideoQ), opts:OptionsPattern[]] :=

Define a test image:

img=;

Generate an image caption:

Highlights on the Latest Models

Object Detection

YOLO V8 Detect Trained on MS-COCO Data

YOLO V8 Pose Trained on MS-COCO Data

CenterNet Pose Estimation Nets Trained on MS-COCO Data

CenterNet Trained on MS-COCO Data

NetModel["YOLO V8 Pose Trained on MS-COCO Data"]

Define  evaluation function:

netevaluate[net_,img_,detectionThreshold_:.25,overlapThreshold_:.5]:=;

Evaluate a net on an image:

testImage=;

predictions=netevaluate[NetModel["YOLO V8 Pose Trained on MS-COCO Data"],testImage];

keypoints=predictions["KeypointEstimation"];

keypoints//Dimensions

HighlightImage[testImage,keypoints]

HighlightImage[testImage,AssociationThread[Range[Length[keypoints]]->keypoints],ImageLabels->None]

HighlightImage[testImage,AssociationThread[Range[Length[Transpose@keypoints]]->Transpose@keypoints],ImageLabels->None,ImageLegends->labels]

getSkeleton[personKeypoints_]:=;
HighlightImage[testImage,AssociationThread[Range[Length[#]]->#]&/@{keypoints,Map[getSkeleton,keypoints],predictions["ObjectDetection"][[;;,1]]},ImageLabels->None]

Stable Diffusion-v1

Generating  an  image  from  a  prompt

NetModel["Stable Diffusion V1","ParametersAllowedValues"]

NetModel["Stable Diffusion V1","UninitializedEvaluationNet"]

NetModel[{"Stable Diffusion V1","Part"->"Encoder"},"UninitializedEvaluationNet"]

NetModel[{"Stable Diffusion V1","Part"->"Decoder"},"UninitializedEvaluationNet"]

StableDiffusionSynthesize=ResourceFunction["StableDiffusionSynthesize"];

StableDiffusionSynthesize["A cat in party hat"]

StableDiffusionSynthesize["A cat wearing glasses"->]

Grid@StableDiffusionSynthesize[{
"A cat in party hat",
{"image cut out, bad anatomy, bad composition, odd looking, weird, odd, unnatural","A cat in party hat"}},3,BatchSize->3]

How to Contribute?

Further the advancement of computational research by contributing to the Wolfram Function Repository. Utilizing the foundational work from Wolfram Neural Nets examples provides a robust starting point:

ResourceFunction["ImageGeoLocation"]

ResourceFunction["ImageSuperResolution"]

ResourceFunction["StableDiffusionSynthesize"]

ResourceFunction["ImageContentsRemove"]

ResourceFunction["DrawSketch"]

Related  Resources

What Is ChatGPT Doing … and Why Does It Work?

Generative AI Space and the Mental Imagery of Alien Minds

An Elementary Introduction to the Wolfram Language: Machine Learning