Introducing Hadamard Binary Neural Networks

Yash Akhauri

Contents

Demystifying Parallel and Distributed Deep Learning

Binary Neural Networks

Hadamard Binary Neural Networks (HBNNs)

Performance on MNIST, CIFAR-10

xHBNN benchmarking

Rise of distributed neural network training

(Legended)

Binary Neural Networks

W = RandomReal[{-1, 1}, {4,4}];
Wb = Mean[Mean[Abs[W]]]*Sign[W];
W//MatrixForm 
Wb//MatrixForm

Hadamard Binary Neural Networks

divider[x_, binAgg_, index_] := Append[Partition[x[[index]], binAgg], x[[index,  -1*Mod[Dimensions@x[[2]], binAgg][[1]];;]]];
hbGradForward[x_, binAgg_]:=ArrayReshape[Map[Flatten[Map[Max, Abs[divider[x, binAgg, #]], {1}]*Sign[divider[x, binAgg, #]]]&, Range[Dimensions@x][[1]]], Dimensions@x];

W = RandomReal[{-1,1}, {3,9}];
Wb = hbGradForward[W, 4];
Wb//MatrixForm

matrix = RandomVariate[NormalDistribution[], {64,512}];
f[x_] := VectorAngle[Flatten@matrix, Flatten@hbGradForward[matrix, x]]/Degree;ListLinePlot[Transpose[{2^(Range@10 - 1), Map[f, 2^(Range@10 - 1)]}], ScalingFunctions->{"Log", "Log"}]

BackPropagation

= Full precision weights  
 = Binarized weights
   = Full precision matrix tied to 
   = Binarization Aggression

= Abs[hbGradForward[W, ]]

= Clip[, {-1, 1}]

Binarizing Convolutions

Import["E:\\no_padding_no_strides.gif", "Animation"]

Performance

MNIST

LeNet

lenet=NetGraph[{ConvolutionLayer[20,5],Ramp,PoolingLayer[2,2],ConvolutionLayer[50,5],Ramp,PoolingLayer[2,2],FlattenLayer[],500,Ramp,10,SoftmaxLayer[]}, {1->2->3->4->5->6->7->8->9->10->11}]

CIFAR-10

AlexNet inspired network

alexnet=NetGraph[{ConvolutionLayer[128, 3],Ramp, BatchNormalizationLayer[], 
ConvolutionLayer[128, 3],Ramp, PoolingLayer[2,2], BatchNormalizationLayer[],
ConvolutionLayer[256, 3],Ramp, BatchNormalizationLayer[],
ConvolutionLayer[256, 3],Ramp, PoolingLayer[2,2], BatchNormalizationLayer[],
ConvolutionLayer[512, 3],Ramp, BatchNormalizationLayer[],
ConvolutionLayer[512, 3],Ramp, PoolingLayer[2,2], BatchNormalizationLayer[],
1024, Ramp, BatchNormalizationLayer[], 1024, Ramp, BatchNormalizationLayer[], 
10}, {1->2->3->4->5->6->7->8->9->10->11->12->13->14->15->16->17->18->19->20->21->22->23->24->25->26->27->28}]

RESULTS

TextGrid[{{"Dataset", "Network","",  "HBNN Accuracy",  "", "XNOR-Net", "Full Precision"},
                  {"", "", "b=4", "b=8", "b=16","", ""}, {"MNIST", "LeNet5", "99.40%", "99.23%", "99.31%", "99.23%", "99.34%"},
{"CIFAR10", "AlexNet Inspired", "88.64%", "88.54%", "N/A", "86.46%", "89.53%"}}, Frame->All]

xHBNN benchmarking

Benchmarking CMMA and xHBNN kernels in C using Intel® OpenMP.(GEMM)

Test bench specifications : Intel® AI DevCloud
Intel® Xeon® Gold 6128 processor clocked at 3.40 GHz, a 19.25 MB cache, 24 cores with 2 - way hyper - threading.

Thank you