Neural network research with the Wolfram Language

Ian Wright, GitHub

(RowDefault[, (Spacer1), , (Spacer1), ])  #WolframTechConf

ABSTRACT

Wolfram Language is an advanced framework for rapidly prototyping neural network applications and exploring research ideas. In this talk, I demonstrate the language features that helped me develop a novel neural network that learns compact non-differentiable Boolean functions.

Lossless hardening with ∂𝔹 nets. I. Wright. In “Differentiable Almost Everything: Differentiable Relaxations, Algorithms, Operators, and Simulators”, ICML 2023 Workshop, Honolulu, 2023.

R&D goals

Neural nets are differentiable for backpropagation

Can’t learn discrete functions (e.g. boolean functions)

Huge nets (e.g. 32-bit weights) that are expensive to query

∂𝔹-nets

Hard and soft bits

A hard-bit is x ∈ {True, False}
A soft-bit is x ∈ [0, 1]

Conversion

Hard[x_]:=If[x>1/2,True,False]
Hard[x_List]:=Hard/@x

Hard[{0.2,0.6,0.45}]

Hard equivalence

Hard(SoftNet(x)) = HardNet(Hard(x))

Learning to mask

Aim: mask input x to False if a learnable weight w is False; otherwise pass-through x.

We need a differentiable version of the boolean function:

And[x,w]

ResourceFunction["TruthTable"][And[x,w],{x,w}]

Product logic

ProductAnd[x_Real,w_Real]:=x w

Manipulate[BPlot[ProductAnd,w],{{w,1},0,1}]

Product logic is

gradient rich

but not hard equivalent

Gödel logic

GodelAnd[x_Real,w_Real]:=Min[x,w]

Manipulate[BPlot[GodelAnd,w],{{w,1},0,1}]

Godel logic is

hard equivalent

but not gradient rich

Margin packing

Idea: pack the hard-equivalent margin with gradient-rich information

MarginPacking[]

DAnd[x_,w_]:=MarginPack[Min[x,w],{x,w}]

DAnd[x,w]//PiecewiseExpand

Manipulate[BPlot[DAnd,w],{{w,1},0,1}]

∂𝔹 logic is

hard equivalent

and gradient rich

∂𝔹-net layers

E.g. Boolean majority

Majority[True,True,False]

Majority[False, False,True]

BooleanConvert[Majority[x1,x2,x3,x4,x5,x6],"DNF"]

If we sort(x) the input bits then the ‘middle’ soft-bit is representative and therefore hard-equivalent to Boolean majority.

NetChain[{
   FunctionLayer[ReverseSort /@ # &],
   PartLayer[{All, Ceiling[(inputSize + 1) / 2]}]
}]

The soft net

{softNet,hardNet}=HardNeuralMajority[8];

softNet

softInput={{0.52,0.8,0.3,0.7,0.2,0.7,0.1,0.99}};

softOutput=softNet[softInput]

The hard net

hardNet

Hard equivalence

hardOutput=First[hardNet[{Harden[softInput],{}}]]

Harden[softOutput]==hardOutput

A classification problem

data=ResourceData["663653b1-6151-48ad-b693-3ee813b191c6"]

Length[data]

Feature encoding

{trainData,testData}=[data,"TestSetSize"->Scaled[0.1],"Shuffle"->True];

Input encoder

encoders=Encoders[trainData]

inputEncoder=InputEncoder[InputEncoders[encoders]]

Define ∂𝔹-net

inputSize=Total[First[#["Output"]]&/@Normal/@Values[InputEncoders[encoders]]]

classes=Normal[DeleteDuplicates[data[All,"Acceptability"]]]

{softNet,hardNet}=Net[inputSize,classes,inputEncoder,OutputEncoder[encoders]];

NetFlatten[softNet]

Train ∂𝔹-net

result=TrainNet[softNet,trainData,testData]

trainedNet=GetTrainedNet[result]

Evaluate ∂𝔹-net

softNetClassifier=NetGraph[
\[LeftAssociation]"TrainedNet"->trainedNet\[RightAssociation],{},"Output"->NetDecoder[OutputEncoder[encoders]]
];

ClassifierMeasurements[softNetClassifier,testData->"Acceptability"]

Bind weights with hard net

booleanClassifier=HardNetFunction[hardNet,trainedNet]

HardNetBooleanExpression[booleanClassifier,inputSize][[1]]

Evaluate boolean classifier

{predictions,summary}=EvaluateNet[booleanClassifier,testData,inputEncoder,OutputEncoder[encoders]];

summary["Accuracy"]

Size of boolean classifier

weights=GetNetWeights[result["TrainedNet"]];

booleanWeightSize=Quantity[Length[weights]/8/1024//N,"Kilobytes"]

The size of a (minimal) MLP for this problem is ~16 kB

∂𝔹-net is ~80 times smaller

Conclusion

Wolfram Language for Neural Network research

Rapidly prototype your research ideas

Neural networks “just work”

High-level yet flexible

Arbitrary code into your network. E.g. FunctionLayer

Different forward and backward passes. E.g. CompiledFunction

More on ∂𝔹-nets

Lossless hardening with ∂𝔹 nets. I. Wright. In "Differentiable Almost Everything: Differentiable Relaxations, Algorithms, Operators, and Simulators", ICML 2023 Workshop, Honolulu, 2023.

GitHub repo: https://github.com/Z80coder/db-nets

Initialization

BPlot[f_,y_]:=Plot[f[x,y],{x,0,1},PlotRange->{{0,1},{0,1}},GridLines->{{1/2,1},{1/2,1}},AxesLabel->{"x","output"}]
ActivationPlot[f_,contours_:10]:=ContourPlot[f[x,y],{x,0,1},{y,0,1},FrameLabel->{x,w},PlotLegends->BarLegend[{Automatic,{0,1}}],PlotRange->All,ColorFunction->(If[#>1/2,Lighter[Red,#],Lighter[Darker[Gray,0.5],#]]&),ColorFunctionScaling->False,GridLines->{{1/2},{1/2}},GridLinesStyle->Black,Contours->Range[0,1,1/contours]]
MarginPacking[]:=Manipulate[
Block[{m,eps,thresholdLine,marginLine,representativeLine,augmentation},
m=Min[x,w];
eps=0.01;
augmentation=Mean[{x,w}]Abs[m-1/2];
thresholdLine=Line[{{1/2,-1},{1/2,1}}];
marginLine={Line[{{m,0.2},{1/2,0.2}}],Text[Style["margin",Bold,FontFamily->"Helvetica"],{m+(1/2-m)/2,0.3}]};
representativeLine={Line[{{m,-0.8},{m,1}}],Text[Style["representative bit",Bold,FontFamily->"Helvetica"],{m,-0.9}]};
Labeled[Plot[
{
Callout[If[m<=1/2,
If[x>=(m+augmentation-eps)&&x<=(m+augmentation+eps),1,Nothing],
If[x>=(1/2+augmentation-eps)&&x<=(1/2+augmentation+eps),1,Nothing]
],Style["augmented bit",Bold,FontColor->Gray],{If[m>1/2,1/2+augmentation,m+augmentation],1.2},CalloutStyle->{Gray},Background->Transparent],
If[m<=1/2,
If[x>m && x<=(m+augmentation),1,Nothing],
If[x>1/2&&x<(1/2+augmentation),1,Nothing]
]
},
{x,0,1},
PlotRange->{{If[m>1/2,0.45,0],If[m>1/2,1,0.55]},{-1,1}},
PlotStyle->Transparent,
Filling->{1->1,2->-0.8},
FillingStyle->LightGray,
Axes->{True,False},
Ticks->{True,False},
Epilog->{Directive[Black],representativeLine,Directive[Gray,Dashed],thresholdLine,marginLine},
ImagePadding->{{0,0},{0,30}},
AspectRatio->2/3
],If[m>1/2,Style["high margin",FontFamily->"Helvetica"],Style["low margin",FontFamily->"Helvetica"]],Bottom]
],{x,0,1},{w,0,1}
]
MarginPack[representativeBit_,x_List]:=Module[{μ=Mean[x],δ},
δ=μ Abs[(representativeBit-1/2)];
If[representativeBit>1/2,Evaluate[1/2+δ],Evaluate[representativeBit+δ]]
]
Get["neural-logic.m",Path->SetDirectory["/home/wright/mywork/db-nets/prototype"]]

Get this file from https://github.com/Z80coder/db-nets.

Get["neural-logic.m",Path->SetDirectory["/home/wright/mywork/db-nets/prototype"]]

Encoders[data_]:=Block[{features=Normal[Keys@First[data]],featureValues},
featureValues={#,Normal[DeleteDuplicates[data[All,#]]]}&/@features;
Association[First[#]->NetEncoder[{"Class",Last[#],"IndicatorVector"}]&/@featureValues]
]
InputEncoders[encoders_]:=Drop[encoders,-1]
OutputEncoder[encoders_]:=Last[encoders]
InputEncoder[inputEncoders_]:=NetGraph[\[LeftAssociation]"Catenate"->CatenateLayer[]\[RightAssociation],
Map[NetPort[First[#]]->"Catenate"&,Normal[inputEncoders]],Sequence@@Normal[Normal[inputEncoders]]
];
Net[inputSize_,classes_,inputEncoder_,outputEncoder_]:=Module[{softNet,hardNet,net,trainableNet},
{softNet,hardNet}=HardNeuralChain[{
HardNeuralOR[inputSize,64],
HardNeuralNOT[64,Length[classes]]
}];
trainableNet=NetGraph[
\[LeftAssociation]"Input"->inputEncoder,"Net"->softNet,"Loss"->HardClassificationLoss[]\[RightAssociation],
{"Input"->"Net"->"Loss",NetPort["Acceptability"]->NetPort["Loss","Target"]},
"Acceptability"->outputEncoder
];
{trainableNet,hardNet}
]
TrainNet[net_,trainData_,testData_]:=NetTrain[net,trainData,All,
ValidationSet->testData,
LossFunction->"Loss",
TrainingStoppingCriterion->Function[#ValidationLoss<0.02],
Method->{"ADAM"},
TargetDevice->"GPU",
WorkingPrecision->"Real64",
MaxTrainingRounds->Infinity]
GetTrainedNet[result_]:=NetGraph[
\[LeftAssociation]"TrainedNet"->NetDelete[NetFlatten[result["TrainedNet"]],"Loss/Error"]\[RightAssociation],
{}
]
EvaluateNet[hardNetFunction_,testData_,inputEncoder_,outputEncoder_]:=Module[{predictions,eval},
predictions=HardNetClassify[hardNetFunction,testData,NetDecoder[outputEncoder],inputEncoder[KeyDrop[#,"Acceptability"]]&,#["Acceptability"]&];eval=HardNetClassifyEvaluation[predictions];
{predictions,eval}
]
GetNetWeights[trainedNeuralLogicNet_]:=Flatten[ExtractWeights[trainedNeuralLogicNet]]