Time-dependent Differential Equation Networks: TDDiffyQNetworks

Presenter: Cody A Trevillian, Oakland University
Co-author(s): Dr. Vasyl Tyberkevych, Oakland University
Preliminary Code Work: James Voorheis, North American Lighting

Abstract

In this project, coupled sets of time-dependent 2nd order ordinary differential equations (TDDiffyQs) simulated based on given parameters describe the spiking behavior of materials forming a physical neuron network. These TDDiffyQNets are trained through a stochastic-gradient-descent-like method. This implemented functionality consists of a set of functions: TDDiffyQNet, TDDiffyQNetSim, TDDiffyQNetTrain, & TDDiffyQNetPlot.
	In the presented case, TDDiffyQNet dictates the behavior of a bilayer of an antiferromagnet & a normal metal using a TDDiffyQ described in [1-2]. Then, DiffyQNetSim simulates the results of an application of TDDiffyQNetTrain. Finally, the resulting actions of the system are shown using TDDiffyQNetPlot.
	We find that AND, OR, and XOR gates can be simulated using NDSolve, with over 200 iterative computations occurring in under 3 seconds. By demonstrating this system with simple gates, we show that this method is not overly computationally intensive to classify larger systems like the MNIST database. These results thus provide impetus to the continued development of this project for generalized time-dependent DiffyQNets.
[1] Roman Khymyn et al., “Ultra-fast artificial neuron: generation of picosecond-duration spikes in a current-driven antiferromagnetic auto-oscillator”, Scientific Reports 8, (Oct. 2018).
[2] Olga Sulymenko et al., “Ultra-fast logic devices using artificial “neurons” based on antiferromagnetic pulse generators”, Journal of Applied Physics 124, 152115 (July 2018).

Time-dependent differential equation...neurons?

Describe the action of physical neurons

Physical neurons in biological systems exhibit spiking behavior only when interacted with:

Physical neurons are not limited to biological systems

Other systems exist that can be used

But, the brain is awesome & so are spiking neural networks!

Networks...for Neuromorphic Computing?

Neuromorphic chips of digital neurons emulate spiking behavior of physical neurons in biological systems

DeepBlue=<|"Name"->"Deep Blue; IBM","Elements"->1.5 10^6 ,"Power"->Quantity[900.,"Watts"],"Year"->1997,"Opponent"->GarryKasparov|>;
AlphaGo=<|"Name"->"AlphaGo; Google","Elements"->17328.,"Power"->Quantity[1000000.,"Watts"],"Year"->2016,"Opponent"->LeeSedol|>;
Human=<|"Name"->"Human Being; Biological","Elements"->84. 10^9,"Power"->Quantity[20.,"Watts"],"Year"->0,"Opponent"->Machines,"Neurons"->84. 10^9|>;
LeeSedol=<|"Type"->Human|>;
GarryKasparov=<|"Type"->Human|>;
TrueNorth=<|"Name"->"True North; IBM","Elements"->5.4 10^9,"Power"->Quantity[70.,"mW"],"Year"->2014,"Opponent"->"N/A","Neurons"->1.0 10^6|>;
Loihi=<|"Name"->"Loihi; Intel","Elements"->2. 10^9,"Power"->Quantity[110.,"mW"],"Year"->2017,"Opponent"->"N/A","Neurons"->131062.|>;
SpiNNaker=<|"Name"->"SpiNNaker; APT at Manchester","Elements"->57.6 10^9,"Power"->Quantity[100. ,"kW"],"Year"->2019,"Opponent"->"N/A","Neurons"->1.0 10^9|>;

Circumvent the von-neumann bottleneck

Various attempts currently exist with varying efficiencies:

ListPlot[Callout[{#[[2]],#[[3]]},#[[1]],Appearance->"Balloon",LabelStyle->Directive[Bold],Background->LightBlue,FrameMargins->7]&/@({#["Power"]/#["Elements"]&/@#//UnitConvert[#,"nW"]&/@#&,#["Elements"]&/@#,ToString[#["Name"]]&/@#}&@{DeepBlue,AlphaGo,Human}//Transpose//Reverse/@#&),PlotRange->{Full,{-100,6.5 10^10}},ImageSize->600,AxesLabel->{"Element Count","Power/Element"}]

ListPlot[Callout[{#[[2]],#[[3]]},#[[1]],Appearance->"Balloon",LabelStyle->Directive[Bold],Background->LightBlue,FrameMargins->7]&/@({#["Power"]/#["Elements"]&/@#//UnitConvert[#,"nW"]&/@#&,#["Elements"]&/@#,ToString[#["Name"]]&/@#}&@{DeepBlue,AlphaGo}//Transpose//Reverse/@#&),PlotRange->{Full,{-100,6.5 10^10}},ImageSize->600,AxesLabel->{"Element Count","Power/Element"}]

ListPlot[Callout[{#[[2]],#[[3]]},#[[1]],Appearance->"Balloon",LabelStyle->Directive[Bold],Background->LightBlue,FrameMargins->7]&/@({#["Power"]/#["Neurons"]&/@#//UnitConvert[#,"nW"]&/@#&,#["Elements"]/#["Neurons"]&/@#,ToString[#["Name"]]&/@#}&@{SpiNNaker,Loihi,TrueNorth,Human}//Transpose//Reverse/@#&),ImageSize->600,PlotRange->{Full,{-100,120000}},AxesLabel->{"Elements/Neuron","Power/Neuron"}]

Specialized hardware is more efficient than generalized approaches

Current hardware is getting better over time.

The brain wins every time!

Digital Neurons vs Physical Neurons

Digital Neurons

Input/Output (I/O)

Functional time-independent behavior; no memory of internal state

F[input]=output

Many-element

Physical Neurons

Stimulus/Response (S/R)

Dynamical time-dependent behavior; memory of internal state

F[stimulus[t],response[t]]dt=d[output]

Single-element

Time-dependent Differential Equation...Neuron Network?

Physical neurons are not limited to biological systems

Other systems exist that can be used

One such example is that of an Antiferromagnet & Normal Metal Bilayer:

(1/(Subscript[ω, ex]))(Overscript[ϕ, ..])+(Subscript[α, eff])(OverDot[ϕ])+((Subscript[ω, e])/2)Sin[2 ϕ]=σ𝒿

σ 𝒿 dictates the bias of the neuron

The above description can be extended to a coupled network of physical neurons:

(1/(Subscript[ω, ex]))(Subscript[(Overscript[ϕ, ..]), n])+(Subscript[α, eff])(Subscript[(OverDot[ϕ]), n])+((Subscript[ω, e])/2)Sin[2 (Subscript[ϕ, n])]=(Subscript[σ𝒿, n])+(∑_m)(Subscript[κ, nm]) (Subscript[(OverDot[ϕ]), n])

(∑_m)(Subscript[κ, nm]) (Subscript[(OverDot[ϕ]), n]) dictates the coupling between physical neurons

TDDiffyQNetworks

Initialization Code, Evaluate this first!

ClearAll["Global`*"]

Global Parameters

Default Options

Options[TDDiffyQOptions]={
"ClockPeriod"->2500.(*number of picoseconds during single clock period of simulation*),
"AnisotropicFrequency"->2Pi*1.75*10^-3(*THz*),
"EffectiveDamping"->.1,
"ExchangeFrequency"->2Pi*27.5(*THz*)};

Usage & Error Messages

TDDiffyQNetworks::usage="TDDiffyQNeuralNetworks is a package that simulates physical neurons that behave according to time-dependent differential functions defined by the user.";

Neural Network

Options[TDDiffyQNet]=Options[TDDiffyQOptions];

TDDiffyQNet::usage="TDDiffyQNet[ConnectivityMatrix,CurrentFloor,Options] is a function that simulates the behavior of a real physical neuron dictated by a user defined function.";
TDDiffyQNet::argerr="I'm sorry, I can't let you do that, Dave.";

(*Creates neural network and initial conditions for a given: Coupling matrix, current list (j/(Subscript[j, thresh]))*)
TDDiffyQNet[κ_,currents_,OptionsPattern[]]:=
Block[{ωe=OptionValue["AnisotropicFrequency"],ωex=OptionValue["ExchangeFrequency"],
αeff=OptionValue["EffectiveDamping"],Tc=OptionValue["ClockPeriod"],
numNeurons=Length[κ],phidotList,rightlist,NN,int,ϕ,t},
(*Creates (OverDot[ϕ]) list*)
phidotList=Array[(Subscript[ϕ, #])'[t]&,numNeurons];
(*((ωe)/2)((j/(Subscript[j, th])))+κ*(OverDot[ϕ])*)
rightlist=((ωe)/2)*currents+κ.phidotList;
(*Outputs equation list (1/(ωex))(Overscript[ϕ, ..])+(Subscript[α, eff])(OverDot[ϕ])+((ωe)/2)sin2ϕ = ((ωe)/2)((j/(Subscript[j, th])))+κ*(OverDot[ϕ]), initial conditions, starting Options, number of neurons, connectivity matrix, and current list*)
<|"Equations"->Array[(1/(ωex))(Subscript[ϕ, #])''[t]+αeff (Subscript[ϕ, #])'[t]+((ωe)/2)Sin[2 (Subscript[ϕ, #])[t]]==Part[rightlist,#]&,numNeurons],
"InitialConditions"->Join[{(1/2)ArcSin[currents]},{Array[0&,Length[currents]]}],"ClockPeriod"->Tc,
"AnisotropicFrequecy"->ωe,"EffectiveDamping"->αeff,"ExchangeFrequency"->ωex,"NumberNeurons"->numNeurons,
"ConnectivityMatrix"->κ,"NetworkCurrent"->currents|>
]

Simulation

Multiple Clock Period Simulation

TDDiffyQNetSim::usage="TDDiffyQNetSim[TDDiffyQNet[],BinaryInput] is a multiple clock period simulator that uses TDDiffyQNetSimSingle.";
TDDiffyQNetSim::argerr="I'm sorry, I can't let you do that, Dave.";

(*Runs simulation given a neural network and binary input. Outputs each end of period conditions and change of spin list. 
Simulation length (not period time) is equal to the length of the given binary input*)
TDDiffyQNetSim[NeuralNetwork_,BinaryInput_,OptionsPattern[]]:=
Block[{ωex=NeuralNetwork["ExchangeFrequency"],κδ,EndTime=NeuralNetwork["ClockPeriod"],i,
int=NeuralNetwork["InitialConditions"],numNeurons=NeuralNetwork["NumberNeurons"],SimTime=Length[BinaryInput],numInputs=Length[BinaryInput[[1]]],
NN=NeuralNetwork["Equations"],sol,intStart,ϕ,ϕstart={},ϕend={},ϕdotend={},BinaryOut={},EndConds={},Δϕdot,FullInput={}},
(*Creates list of delta pulse strength to be used for input neurons, Note issues will occur if no buffer neurons exist; It must be noted that this is what is interpretted as the input to the network!!*)
κδ=Join[(Transpose[NeuralNetwork["ConnectivityMatrix"]].Array[1&,numNeurons])[[1;;numInputs]],Array[0&,numNeurons-numInputs]];
(*Calculates addition of change in (OverDot[ϕ]) due to delta spike approximation*)
Δϕdot=ωex*Pi*κδ;
(*Simulation loop, loops through each element in binary input list*)
Table[
(*Creates input list for all neurons*)
FullInput=Join[BinaryInput[[i]],Array[0&,numNeurons-Length[BinaryInput[[1]]]]];

(*Adds change in (OverDot[ϕ]) due to delta spike to (OverDot[ϕ]) based on binary input list, and give new initial condition list to be used at start of period*)
intStart=Join[{int[[1]]},{int[[2]]+(FullInput*Δϕdot)}];
ϕstart=intStart[[1]];

(*Solves single clock period simulation, leaving blank uses default clock period and initial conditions from Neural Network*)


sol=NDSolve[
{
Join[NeuralNetwork,<|"InitialConditions"->intStart|>]["Equations"],
Array[(Subscript[ϕ, #])[0]==Join[NeuralNetwork,<|"InitialConditions"->intStart|>]["InitialConditions"][[1]][[#]]&,numNeurons],
Array[(Subscript[ϕ, #])'[0]==Join[NeuralNetwork,<|"InitialConditions"->intStart|>]["InitialConditions"][[2]][[#]]&,numNeurons]
},
Array[(Subscript[ϕ, #])&,numNeurons],
{t,0,EndTime},
MaxSteps->10^8];

(*Finds end of period conditions to be used at the start of next period (with the addition of the delta spike inputs)*)
int=Join[{Array[N[(Subscript[ϕ, #])[EndTime]/.sol][[1]]&,numNeurons]},{Array[N[(Subscript[ϕ, #])'[EndTime]/.sol][[1]]&,numNeurons]}];
ϕend=int[[1]];ϕdotend=int[[2]];

(*Creates list for values at end of each period*)
EndConds=Append[EndConds,{ϕend,ϕdotend}];

(*Creates binary list of Δϕ*)
BinaryOut=Append[BinaryOut,Round[(ϕend-ϕstart)/Pi]],{i,1,SimTime}];

(*Returns end conditions and binary output, and delta strength list*)
<|"SimSingle"->sol,"EndConditions"->EndConds,"BinaryOutput"->BinaryOut,"DeltaStrength"->κδ|>

]

Training Network

TDDiffyQNetTrain::usage="TDDiffyQNetTrain[TDDiffyQNet[],Input,Output,numberIterations] is a synapse or current based neural network trainer. At each iteration, by selecting random periods of input and comparing a random selection of their simulated outputs to their intended outputs, error calculations can be made and propagated to the next iteration.";
TDDiffyQNetTrain::argerr="I'm sorry, I can't let you do that, Dave.";

Options[TDDiffyQNetTrain]={"Method"->__};

TDDiffyQNetTrain[NeuralNetwork_,BinaryInput_,DesiredOutput_,numberIterations_,OptionsPattern[]]:=Switch[OptionValue["Method"],

"Synapse",

Block[{κcollect,κ=NeuralNetwork["ConnectivityMatrix"],currents=NeuralNetwork["NetworkCurrent"],
i,NN,TrueOutput,Error,Correction,κ1,(*Correction to connectivity matrix*)Δκ=.005,numNeurons=NeuralNetwork["NumberNeurons"],PeriodNumber,FullInput,
whichOUT,whichSYNAPSE,whichPERIOD,rowOUT,whichWEIGHT,
numOutputs=Length[DesiredOutput[[1]]],WhichOutput,simNN,weightStrength,EndTime=NeuralNetwork["ClockPeriod"]},

(*Gives status bar*)
Monitor[
κ1=κ;
(*set the number of runs*)
κcollect=Prepend[Table[
NN=TDDiffyQNet[κ,currents,"ClockPeriod"->EndTime];


(*Choose random desired binary input period from those supplied by the user*)
whichPERIOD=RandomInteger[{1,Length[DesiredOutput]}];


(*Choose random output neuron from those supplied by the user*)
whichOUT=RandomInteger[{1,numOutputs}];
rowOUT=κ[[numNeurons-numOutputs+whichOUT]];


(*Simulates network for chosen random binary input from those supplied by the user, chosen at random in previous step*)
simNN=(TDDiffyQNetSim[NN,{BinaryInput[[whichPERIOD]]}]["BinaryOutput"])[[1]];


(*Choose random nonzero weight to chosen output neuron to check the outcome, and feed to subsequenct error calculation*)
whichWEIGHT=numNeurons-Length@BinaryInput[[1]]-numOutputs+RandomChoice[SparseArray[simNN[[numNeurons-Length@BinaryInput[[1]]-numOutputs+1;;numNeurons-numOutputs]]]["NonzeroPositions"]][[1]];
weightStrength=rowOUT[[whichWEIGHT]];


(*Calculate error*)
Error=(DesiredOutput[[whichPERIOD]]-(simNN)[[numNeurons-numOutputs+1;;numNeurons]])[[whichOUT]];

(*error checking*)


If[Error≠0(*&&weightStrength+Error*Δκ≥0*),

Correction=
Join[Table[0,{i,1,numNeurons+whichOUT-numOutputs-1},
{j,1,numNeurons}],

{ReplacePart[Array[0&,numNeurons],
whichWEIGHT->Error*Δκ]},

Table[0,{i,1,numOutputs-whichOUT},
{j,1,numNeurons}]];

κ=κ+Correction,κ=NN["ConnectivityMatrix"]],{i, 1, numberIterations}],κ1];,

{"Progress: ",ProgressIndicator[i,{1,numberIterations}],i}];

(*Output trained neural network*)
<|"TrainedNet"->NN,"ConnectivityMatrices"->κcollect|>]
,

"Current",

Block[{currentscollect,κ=NeuralNetwork["ConnectivityMatrix"],currents=NeuralNetwork["NetworkCurrent"],
i,numi=Length[BinaryInput],NN,TrueOutput,Error,Correction,currents1,Δcurrent,numNeurons,PeriodNumber,
numOutputs=Length[DesiredOutput[[1]]],(*κδ=OptionValue["DeltaStrength"],*)ΔκRandom,whichOUT,whichPERIOD,rowOUT,simNN,whichCURRENT,currentStrength,EndTime=NeuralNetwork["ClockPeriod"]},
numNeurons=Length[κ];
(*Change current by constant increment*)
Monitor[Δcurrent=(currents/.x_/;x>0->.0001);
currents1=currents;
currentscollect=Prepend[Table[NN=TDDiffyQNet[κ,currents,"ClockPeriod"->EndTime];

(*Choose a binary period at random, and if an error exists, manipulate the current matrix*)
whichPERIOD=RandomInteger[{1,Length[BinaryInput]}];
whichOUT=RandomInteger[{1,numOutputs}];
rowOUT=currents[[numNeurons-numOutputs+whichOUT]];

(*compute true output*)
simNN=(TDDiffyQNetSim[NN,{BinaryInput[[whichPERIOD]]}]["BinaryOutput"])[[1]];


whichCURRENT=numNeurons-Length@BinaryInput[[1]]-numOutputs+RandomChoice[SparseArray[simNN[[numNeurons-Length@BinaryInput[[1]]-numOutputs+1;;numNeurons-numOutputs]]]["NonzeroPositions"]][[1]];
currentStrength=currents[[whichCURRENT]];


(*compute error*)
Error=(DesiredOutput[[whichPERIOD]]-(simNN)[[numNeurons-numOutputs+1;;numNeurons]])[[whichOUT]];


(*compute correction*)
Correction=ReplacePart[Δcurrent*BinaryInput[[whichPERIOD]][[whichOUT]],whichCURRENT->.0001]*Error;
(*error checking*)
If[Norm[Error]≠0,

currents=currents+Correction,currents=NN["NetworkCurrent"]]
,{i,1,numberIterations}],currents1];,
{"Progress: ",ProgressIndicator[i,{1,numberIterations}],i}];
(*Outputs the trained neural network*)
<|"TrainedNet"->NN,"NetworkCurrents"->currentscollect|>]

]

Plot Simulation

TDDiffyQNetPlot::usage="TDDiffyQNetPlotState[TDDiffyQNet[],InitialConditions,ClockTime,PlotHeight,PlotDepth] is a plot the state or rate of the simulated network.";
TDDiffyQNetPlotState::argerr="I'm sorry, I can't let you do that, Dave.";

(*Plots all ϕ and (OverDot[ϕ]) given a neural network, initial conditions, clock period, and height of graph. Includes two horizontal lines: One at π/4 and one at π.*)

TDDENetPlot

TDDiffyQNetPlot[NeuralNetwork_,BinaryInput_,ClockTime_,RateHeight_,StateHeight_,PlotDepth_]:=
Block[{
pdatlist={},pvdatlist={},
t,p,i,solution,
numNeurons=NeuralNetwork["NumberNeurons"],
plist={},pvlist={},
l1,l2,ϕ
},

(*Solves one period for given inputs*)
solution=TDDiffyQNetSim[Join[NeuralNetwork,<|"ClockPeriod"->ClockTime|>],BinaryInput]["SimSingle"];

(*Loops through number of neurons*)
Table[

(*ϕ list for visualization*)
pdatlist=Append[pdatlist,(Subscript[ϕ, i])[t]/.solution];
(*Plots ϕ*)
(Subscript[p, i])=Plot[(Subscript[ϕ, i])[t]/.solution,{t,0,ClockTime},PlotRange->{{0,ClockTime},{PlotDepth,StateHeight}},PlotStyle->{Thickness[0.0075], Dashed,ColorData[i,"ColorList"](*Black*)}, PlotLegends->LineLegend[{(Subscript["ϕ", i])}]];
plist=Append[plist,{(Subscript[p, i])}];

(*(OverDot[ϕ]) list for visualization*)
pvdatlist=Append[pvdatlist,(Subscript[ϕ, i])'[t]/.solution];
(*Plots (OverDot[ϕ])*)
(Subscript[p, i,i])=Plot[(Subscript[ϕ, i])'[t]/.solution,{t,0,ClockTime},PlotRange->{{0,ClockTime},{PlotDepth,RateHeight}},PlotStyle->{Thickness[0.0075], ColorData[i,"ColorList"]}, PlotLegends->LineLegend[{(Subscript["(OverDot[ϕ])", i])}],PlotPoints->200];
pvlist=Append[pvlist,{(Subscript[p, i,i])}];

,{i,1,numNeurons}];

(*Horizontal lines π/4 and π*)
l1=Plot[(Pi/4),{t,0,ClockTime},PlotStyle->{Thickness[0.005],Gray, Dotted},PlotLegends->LineLegend[{"π/4"}]];
l2=Plot[Pi,{t,0,ClockTime},PlotStyle->{Thickness[0.005],Black, Dotted},PlotLegends->LineLegend[{"π"}]];

plist=Append[plist,{l1,l2}];plist=Flatten[plist];pvlist=Flatten[pvlist];
(*Outputs plot with all ϕ and (OverDot[ϕ])*)

<|"State"->(Show[plist,Frame->True,FrameLabel->{Style["Time (picoseconds)",14],Style["ϕ (radians)",14]},PlotRange->{{0,ClockTime},{PlotDepth,StateHeight}}]),

"Rate"->(Show[pvlist,Frame->True,FrameLabel->{Style["Time (picoseconds)",14],Style["(OverDot[ϕ]) (radians/picosecond)",14]},PlotRange->{{0,ClockTime},{PlotDepth,RateHeight}}]),

"StateData"->pdatlist,"RateData"->pvdatlist,

"ClockPeriod"->ClockTime|>]

GraphMakers

GraphLayerList

GraphLayerList[inpmat_]:=Module[{glist=Join@@MapIndexed[Thread@DirectedEdge[First/@Position[#1,Except[0|0.],{1},Heads->False],First@#2]&,inpmat],classes={},classvrtcs},
classvrtcs=Cases[Thread[{VertexList@glist,VertexInDegree@glist}],{v_,0}:>v];
While[Length@classvrtcs>0
,
AppendTo[classes,Union@classvrtcs];
classvrtcs=Cases[EdgeList@glist,(Alternatives@@Last[classes])\[DirectedEdge]v_:>v]
];
{glist,classes}
]

ColPacityStyle

ColPacity[adjmat_,poscol_,negcol_]:=Block[{styleEdge},
(
styleEdge[n_Real]:={Opacity[50 Abs@n],If[Positive[n],poscol,negcol],Thick};
Map[#->styleEdge@Part[adjmat,#[[2]],#[[1]]]&,GraphLayerList[adjmat][[1]]]
)]

ADJmatFCN

ADJmatFCN[numinput_,numoutput_,numhidden_,defaultweight_:.01]:=Module[{numtotal=2 numinput+numhidden+numoutput},defaultweight*Normal@SparseArray[{Band[{numinput+1,1},{numinput+numinput,numinput}]->1,Band[{numinput+numinput+1,numinput+1},{numinput+numinput+numhidden,numinput+numinput},{1,1}]->{{1,1},{1,1}},If[numhidden≠0,Band[{numinput+numinput+numhidden+1,numinput+numinput+1},{numtotal,numinput+numinput+numhidden},{1,1}]->{{1,1},{1,1}},Nothing]},{numtotal,numtotal}]]

StyledGraphPlotter

Options[StyledADJPlot]={"PositiveColor"->Darker@Red,"NegativeColor"->Darker@Blue};
StyledADJPlot[adjmat_,OptionsPattern[]]:=Block[{amat=adjmat,poscol=OptionValue["PositiveColor"],negcol=OptionValue["NegativeColor"]},
LayeredGraphPlot[GraphLayerList[adjmat][[1]],Left,VertexStyle->Flatten@MapIndexed[#1->ColorData[63][#2[[1]]]&,GraphLayerList[adjmat][[2]],{2}],VertexSize->0.2,EdgeStyle->ColPacity[amat,poscol,negcol]]]

Underlying framework consists of four main functions

TDDiffyQNet[] creates the physical neuron network

TDDiffyQNetSim[] simulates the current state of the physical neuron network

TDDiffyQNetTrain[] modifies the physical neuron network according to some learning algorithm

TDDiffyQNetPlot[] plots the current state of the physical neuron network

Helper functions that assist with visualization

GraphLayerList[] organizes adjacency matrix for StyledADJPlot[]

ColPacity[] colors the edges according to their weight for StyledADJPlot[]

Thanks go to Jofre Espigule for the help on making this stylization function!

StyledADJPlot[] styles the plot using the above two functions.

Thanks go to Bob Nachbar for the help on making this & the above GraphLayerList[] happen!

ADJmatFCN[] creates an adjacency matrix for a fully connected network

Simple Use Case: 4-Neuron Chain

SynapseMatrix=.01*Normal[SparseArray[Band[{2,1}]->1,{4,4}]];
Current=ConstantArray[.974,4];
TDDiffyQNeuralNetwork=TDDiffyQNet[SynapseMatrix,Current];
TDDiffyQNeuralNetwork;

TDDiffyQNeuralNetwork["Equations"]

TDDiffyQNeuralNetwork["ConnectivityMatrix"]

The connectivity matrix can be visualized using the visualization helper functions:

StyledADJPlot@TDDiffyQNeuralNetwork["ConnectivityMatrix"]

Simulation of the network is readily shown using TDDiffyQNetSim and TDDiffyQNetPlot:

TDDiffyQNetSim[TDDiffyQNeuralNetwork,{{1}}]

StartingConditions=Join[{TDDiffyQNeuralNetwork["InitialConditions"][[1]],TDDiffyQNeuralNetwork["ExchangeFrequency"]*Pi*{.01,0,0,0}}];
rate=TDDiffyQNetPlot[TDDiffyQNeuralNetwork,{{1}},1000,.14,Pi+1,0]["Rate"]
state=TDDiffyQNetPlot[TDDiffyQNeuralNetwork,{{1}},1000,.14,Pi+1,0]["State"]

AND, OR: Application of TDDiffyQNetTrain

TDDiffyQNetTrain

TDDiffyQNetTrain trains the network for the user defined inputs and outputs over a set number of iterations. This is currently done using a stochastic-gradient-descent-like method following the algorithm:
(1)	Output from a randomly selected input vector is simulated.
(2)	Error is computed by comparison of simulated output against desired outcome.
(3)	Update to current vector/connectivity array is made by a combination of specified learning rate with the error computed in (2) to aid in convergence.
(4)	Method is ceased after a defined number of iterations.

AND/OR

ANDmat=.01 {{0,0,0,0,0},{0,0,0,0,0},{1,0,0,0,0},{0,1,0,0,0},{0,0,1,1,0}};

ORmat=ANDmat;

Synapse Based

TrainedANDsyn=TDDiffyQNetTrain[TDDiffyQNet[ANDmat,ConstantArray[.974,Length@ANDmat]],{{0,1},{1,0},{1,1}},{{0},{0},{1}},30,"Method"->"Synapse"];
(TDDiffyQNetSim[TrainedANDsyn["TrainedNet"],{{0,1},{1,0},{1,1}}]["BinaryOutput"])[[1;;,{1,2,5}]]

TrainedORsyn=TDDiffyQNetTrain[TDDiffyQNet[ANDmat,ConstantArray[.974,Length@ORmat]],{{0,1},{1,0},{1,1}},{{1},{1},{1}},30,"Method"->"Synapse"];
(TDDiffyQNetSim[TrainedORsyn["TrainedNet"],{{0,1},{1,0},{1,1}}]["BinaryOutput"])[[1;;,{1,2,5}]]

Current Based

TrainedANDcur=TDDiffyQNetTrain[TDDiffyQNet[ANDmat,ConstantArray[.974,Length@ANDmat]],{{0,1},{1,0},{1,1}},{{0},{0},{1}},30,"Method"->"Current"];
(TDDiffyQNetSim[TrainedANDcur["TrainedNet"],{{0,1},{1,0},{1,1}}]["BinaryOutput"])[[1;;,{1,2,5}]]

TrainedORcur=TDDiffyQNetTrain[TDDiffyQNet[ANDmat,ConstantArray[.974,Length@ORmat]],{{0,1},{1,0},{1,1}},{{1},{1},{1}},30,"Method"->"Current"];
(TDDiffyQNetSim[TrainedORcur["TrainedNet"],{{0,1},{1,0},{1,1}}]["BinaryOutput"])[[1;;,{1,2,5}]]

Plots

TrainedANDStart=Join[
{
(TrainedANDsyn["TrainedNet"]["InitialConditions"])[[1]],TrainedANDsyn["TrainedNet"]["ExchangeFrequency"]*Pi*{.01,.01,0,0,0}
}
];
TDDiffyQNetPlot[TrainedANDsyn["TrainedNet"],{{0,1}},450,.15,4,0]["Rate"]
TDDiffyQNetPlot[TrainedANDsyn["TrainedNet"],{{0,1}},450,.15,4,0]["State"]

TrainedORStart=Join[
{
(TrainedORsyn["TrainedNet"]["InitialConditions"])[[1]],TrainedORsyn["TrainedNet"]["ExchangeFrequency"] Pi {.0,.0,0,0,0}
}
];
TDDiffyQNetPlot[TrainedORsyn["TrainedNet"],{{0,1}},450,.15,4,0]["Rate"]
TDDiffyQNetPlot[TrainedORsyn["TrainedNet"],{{0,1}},450,.15,4,0]["State"]

XOR: Advanced Application

XORmatP1cur={{0,0,0,0,0,0},{0,0,0,0,0,0},{.01,0,0,0,0,0},{0,.01,0,0,0,0},{0,0,.01,.01,0,0},{0,0,.01,.01,0,0}};

XORPart1cur=TDDiffyQNet[XORmatP1cur,ConstantArray[.974,6]];
TDDiffyQNetSim[XORPart1cur,{{1,1},{1,0},{0,1}}]["BinaryOutput"]

TrainedXORpart1syn=TDDiffyQNetTrain[TDDiffyQNet[XORmatP1cur,ConstantArray[0.974,6]],{{1,0},{0,1},{1,1}},{{1,0},{0,1},{0,0}},200,"Method"->"Synapse"];
(TDDiffyQNetSim[TrainedXORpart1syn["TrainedNet"],{{1,1},{1,0},{0,1}}]["BinaryOutput"])[[1;;,{3,4,5,6}]]

XORFULL=TDDiffyQNet[Append[Transpose[Append[Transpose[TrainedXORpart1syn["TrainedNet"]["ConnectivityMatrix"]],ConstantArray[0,6]]],{0,0,0,0,0.01,0.01,0}],ConstantArray[0.974,7]];
(TDDiffyQNetSim[XORFULL,{{1,1},{1,0},{0,1}}]["BinaryOutput"])[[1;;,{3,4,7}]]

XOR Plot

XORStart=Join[{(XORFULL["InitialConditions"])[[1]],TrainedANDsyn["TrainedNet"]["ExchangeFrequency"] Pi {.01,.012,0,0,0,0,0}}];
TDDiffyQNetPlot[XORFULL,{{0,1}},700,.15,4,0]["Rate"]
TDDiffyQNetPlot[XORFULL,{{0,1}},700,.15,4,0]["State"]

XORStart=Join[{(XORFULL["InitialConditions"])[[1]],TrainedANDsyn["TrainedNet"]["ExchangeFrequency"] Pi {.01,0*.012,0,0,0,0,0}}];
TDDiffyQNetPlot[XORFULL,{{1,1}},700,.15,4,0]["Rate"]
TDDiffyQNetPlot[XORFULL,{{1,1}},700,.15,4,0]["State"]

Summary & Key Takeaways

Spiking neural networks (SNN) with neurmorphic computing is becoming popular

Physical neuron networks are central to efficient SNN

Single element implementation of specialized hardware is best

The Wolfram Language Can provide:

A better understanding of complex systems such as these

A tool for further exploration and development

Time-dependent physical neuron network simulation

This is a work in progress, but the structures are there for further development

You can help! Ask me how, or contact me at: 1:eJxTTMoPChZjYGAoKUoty8zJyUzMc8hPzM5JzEvRS00pBQCXeApu

Special Thanks

Professor Vasyl Tyberkevych, Research Advisor, OU Department of Physics

Professor Andrei Slavin, Senior Research Advisor, OU Department of Physics

Professor Steven Louis, OU Department of Electrical & Computer Engineering

Mom, Bob, & Zoey, too!