Tensor Decomposition definitions of Neural Net Architectures.

Dr. Bheemaiah Anil K, UW Madison.

(RowDefault[, (Spacer1), , (Spacer1), ])  #WolframTechConf

ABSTRACT

This paper describes complexity theory of neural networks, defined by Tensor Decompositions, with a review of simplification of the Tensor Decomposition for simpler Neural Network architectures.
The concept of Z-Completeness for a network N, is defined in the existence of a Tensor Decomposition for that particular N.

Wolfram Neural Repository.
Prediction Modelling in Wolfram.

Classification, Taxonomy, Ontologies, 
Generation and Segmentation and Modeling 
including rule extraction are the normal 
tasks, for deep learning networks. There is a 
need to evaluate the fit of a model based on 
a trained net. Overfitting is a symptom of 
the need for a different architecture of the 
neural network, in transfer learning 
evaluation, based on a repository of a 
number of ready to use pre-trained 
networks.

Wolfram defines the tasks as: 
[classification, feature extraction, Image processing, Image Segmentation, 
Regression, Language Modeling, Speech Recognition, Audio Processing]
(“Wolfram Neural Net Repository of Neural Network 
Models” n.d.)

They presently support the following inputs 
[Text, Image, Audio] 
Arbitrary 1D waveforms can be used instead of audio, which can be any 1D data. And any 2D data can be used instead of images

Contd..

For 2.x D data, including video streams, event streams and data streams and higher dimensional data, we need custom tensor decompositions.
We consider a number of examples,  including cycleGAN, attnGAN, resnet-50.

Tensor Decomposition simplification illustrated  with ResNet

The NetChain command allows for a Tensorization of a Neural Network 
architecture, defined as named parameters of a NetChain command. The mapping from string names to Tensor Symbols is left as 
an exercise to the readers.

Theoretical constructions in Tensorization. 
Regression Expansions. 
Tensor Canonicalization 
TensorReduce — convert any polynomial tensor expression into a canonical form 
TensorExpand — expand out products, sums, and other tensor operations 
(Symbolic Tensors—Wolfram Language Doc...; Symbolic Tensors—Wolfram 
Language Doc...) 
TensorReduce and TensorExpand are two symbolic Tensor simplification and reducing operations defined in WolfRam, it is thus possible to simplify Tensor Architectures

Resnet-50

The NetChain depicts the architecture, as an open exercise, please write a script to convert this to a tensor expression and use TensorReduce and TensorExpand on them.


Think Problem.
Theorem: A Network N is defines as Z-Complete if there exists a tensor decomposition T for the network.
Are all networks Z-Complete? Can we think of one non complete network, we can only approximate on Wolfram

Thank You For Viewing this Presentation.

For Questions, Please use this academic CRM system, developed by the author, Dr Bheemaiah Anil K by clicking the link below.
Ask the SME