Neural Networks for Images & Audio Workshop

Carlo Giacometti, Markus van Almsick
Wolfram Research

Join the Conversation #WolframTechConf

ABSTRACT

Deep neural networks (DNNs) are potent tools that help to achieve amazing results in artificial intelligence. The design of DNN architectures, however, still appears to be a black art. In this workshop we discuss the properties of image and audio data and show how these properties imply the appropriate neural layer types and network architectures. We perform experiments to determine the limitations of current networks and demonstrate DNN solutions for a range of application tasks

Outline

Building an Audio Classifier

Audio Encoders

Details about layers

Training

Net  Surgery for Feature extraction

Audio Event Detection

Transfer Learning

Building an Audio Classifier

Building an Audio Classifier

You can learn about the details of the neural net framework by building a small audio classifier yourself. It all starts with labeled data that you will need for training a network.

Spoken Digit Commands Dataset

For speed and simplicity, the Spoken Digit Commands Dataset can be used. The dataset consists of 10,000 short mono recordings of the digits from 0 to 9. This is a toy problem, but with slightly different data it can become a very useful application for tasks like keyword spotting, where the available resources are limited (e.g. a phone that needs to continuously monitor with minimal resources for the word “OK ...”, so then it can activate more advanced voice-related functionalities).

"Spoken Digit Commands" is available as a Wolfram resource:

spokenDigitsObj=ResourceObject["Spoken Digit Commands"]

spokenDigitsTraining=RandomSample@ResourceData[spokenDigitsObj,"TrainingData"];

spokenDigitsTesting=RandomSample@ResourceData[spokenDigitsObj,"TestData"];

spokenDigitsClasses=Sort@Union[spokenDigitsTraining[[All,"Output"]]]

A random sample of labeled training audios:

RandomSample[spokenDigitsTraining,6]

Building an Audio Classifier

Next, you need to construct a NetEncoder that convert your incoming audio data into arrays.

NetEncoder

Convert the signal into an array of neural activities:

enc=NetEncoder[{"AudioMFCC","TargetLength"->50,"WindowSize"->(Quantity[40, "ms", milliseconds, "Milliseconds"]),"Offset"->(Quantity[20, "ms", milliseconds, "Milliseconds"]),"NumberOfCoefficients"->32}]

enc[]//Normal//MatrixPlot

Building an Audio Classifier

Encoders: types

There are several types of audio NetEncoder that provide a fast and efficient way of transforming an Audio object or file into an array of neural activities.

Audio

The "Audio" encoder provides basic audio conforming (such as channel averaging and resampling) and is used as the basis for all other encoders.

An equivalent implementation would be:

conform[audio_]:= AudioResample[AudioChannelMix[audio, "Mono"], 16000]

NetEncoder["Audio"][]

%[[;;5000]]//Flatten//ListLinePlot

Short-Time Fourier transform

The "AudioSTFT" encoder computes the Fourier transform on overlapping windows of audio data. This provides both time and frequency information.

An equivalent implementation would be:

stft[a_]:=Map[ReIm@*Fourier,Partition[Flatten[AudioData[a]],400,133]]

NetEncoder["AudioSTFT"][]

%//formatForMP//Log//MatrixPlot

Spectrogram

The "AudioSpectrogram" encoder computes the power spectrum on overlapping windows of audio data. This provides both time and frequency information.

An equivalent implementation would be:

spectrogram[a_]:=Abs[SpectrogramArray[a, 400, 133, HannWindow][[All,;;201]]]^2

NetEncoder["AudioSpectrogram"][]

%//formatForMP//Log//MatrixPlot

Mel-spectrogram

The "AudioMelSpectrogram" encoder computes the power spectrum on overlapping windows of audio data and filters it with logarithmically spaced windows. This rescaling in the frequency domain approximates human perception of pitch. In addition to that, the number of filters is usually smaller than the number of spectrum frequency bins, so the size of the output is reduced:

NetEncoder["AudioMelSpectrogram"][]

%//formatForMP//Log//MatrixPlot

MFCC (Mel-frequency cepstral coefficients)

The "AudioMFCC" encoder returns a selection of coefficients from the DCT of Mel-spectrogram. This reduces the dimensionality of the output further, while preserving most of the information:

NetEncoder["AudioMFCC"][]

%//formatForMP//MatrixPlot

Building an Audio Classifier

Encoders: data augmentation

Data augmentation can be very useful when dealing with limited or reduced-size datasets, and to make a net more robust against artificial/irrelevant trends in the training data.

As an example, if you were to classify recordings of cats and dogs, and in the training data all the dogs were recorded with a noisy microphone, the network might end up recognizing the noise rather than the dog!

Another convenient usage of data augmentation for audio is extracting segments from the recordings in the training data.

Look at the possible augmentations.

a=;AudioPlot[a]

Time shift

Trim each example by an amount randomly sampled from a normal distribution.

Improve robustness with respect to locality of an audio event.

Extract a random subset of the signal.

NetEncoder[{"Audio","Augmentation"->{"TimeShift"->NormalDistribution[0,.5]}}][{a,a}]

Column[Spectrogram[Audio[Flatten@#,SampleRate->16000],ImageSize->Medium,FrameTicks->None]&/@%]

Noise

Add white noise to each example with a randomly selected maximum amplitude.

Can also specify a model (file or in-core Audio) to use as the noise.

NetEncoder[{"Audio","Augmentation"->{"Noise"->UniformDistribution[{0,.4}]}}][{a,a}]

Column[Spectrogram[Audio[Flatten@#,SampleRate->16000],ImageSize->Medium,FrameTicks->None]&/@%]

Specify a noise signal.

noiseSignal=AudioGenerator[TriangleWave[1000+800Sin[2Pi#]]&,4];NetEncoder[{"Audio","Augmentation"->{"Noise"->{UniformDistribution[{0,.2}],noiseSignal}}}][{a,a}]

Column[Spectrogram[Audio[Flatten@#,SampleRate->16000],ImageSize->Medium,FrameTicks->None]&/@%]

Convolution

Convolve examples with another signal with a randomly sampled mixture level.

impulseResponse=ExampleData[{"Audio","IRRailwayTunnel"}];NetEncoder[{"Audio","Augmentation"->{"Convolution"->{UniformDistribution[{0,1}],impulseResponse}}}][{a,a}]

Column[Spectrogram[Audio[Flatten@#,SampleRate->16000],ImageSize->Medium,FrameTicks->None]&/@%]

Volume

Randomly adjust the amplitude of each example.

NetEncoder[{"Audio","Augmentation"->{"Volume"->UniformDistribution[{.1,2}]}}][{a,a}]

Column[plotData/@%]

VTLP (vocal tract length perturbation)

Randomly adjust (warp) the spacing of the filter bank centers when computing the mel-spectrum.

idea based on modeling the frequency-filtering effects of different vocal tract lengths

only applicable to “MelSpectrogram” and “MFCC”

NetEncoder[{"AudioMelSpectrogram","Augmentation"->{"VTLP"->UniformDistribution[{.3,1.5}]}}][{a,a}]//Short

Column[MatrixPlot[formatForMP[Log@#],ImageSize->Medium]&/@%]

Building an Audio Classifier

Encoders: performance

Using NetEncoder instead of other top-level code brings a significant performance advantage, since the computations are done in parallel at C++ level. This is especially true for datasets of out-of-core audio objects.

In core data:

topLevelTimings={{10,0.649567`},{20,1.448076`},{40,2.849806`},{80,6.827706`},{160,12.761704`},{320,25.169323`},{640,38.931099`},{1280,95.534567`},{2560,182.959472`},{5120,331.763654`}};
topLevelTimingsParallel={{10,0.03178`},{20,0.029938`},{40,0.033842`},{80,0.052382`},{160,0.089185`},{320,0.171205`},{640,0.359299`},{1280,0.78154`},{2560,1.937559`},{5120,5.387118`}};
encTimings={{10,0.00731`},{20,0.013991`},{40,0.017921`},{80,0.037618`},{160,0.056577`},{320,0.085983`},{640,0.180181`},{1280,0.382492`},{2560,0.873882`},{5120,1.659201`}};ListPlot[{topLevelTimings,topLevelTimingsParallel,encTimings},PlotMarkers->Automatic,Joined->True,ScalingFunctions->"Log",AxesLabel->{"# of audios","AbsoluteTiming"},ImageSize->Large,PlotLegends->Placed[{Style["Map[Abs[SpectrogramArray[AudioNormalize[#],400,133,HannWindow][[All,;;201]]]^2&,data]","InputForm"],Style["ParallelMap[Abs[SpectrogramArray[AudioNormalize[#],400,133,HannWindow][[All,;;201]]]^2&,data]","InputForm"],Style["NetEncoder[{\"AudioSpectrogram\",\"Normalization\"->\"Max\"}]","InputForm"]},Below]]

Out of core data:

topLevelTimingsParallelOutOfCore={{10,0.778977`},{20,0.199009`},{40,0.388209`},{80,0.891259`},{160,1.563454`},{320,3.429179`},{640,7.936796`},{1280,18.277928`},{2000,39.215791`}};
encTimingsOutOfCore={{10,0.060214`},{20,0.089742`},{40,0.126471`},{80,0.253023`},{160,0.574914`},{320,1.1976`},{640,2.487388`},{1280,6.596795`},{2000,13.207445`}};
ListPlot[{topLevelTimingsParallelOutOfCore,encTimingsOutOfCore},PlotMarkers->Automatic,Joined->True,ScalingFunctions->"Log",AxesLabel->{"# of audios","AbsoluteTiming"},ImageSize->Large,PlotLegends->Placed[{Style["ParallelMap[Abs[SpectrogramArray[AudioNormalize[Import[#]],400,133,HannWindow][[All,;;201]]]^2&,data]","InputForm"],Style["NetEncoder[{\"AudioSpectrogram\",\"Normalization\"->\"Max\"}]","InputForm"]},Below]]

Building an Audio Classifier

We also need to convert the the network result into tags / labels. For this, use a NetDecoder.

NetDecoder

Decode probability tensors as class labels:

dec=NetDecoder[{"Class",spokenDigitsClasses}]

dec[{0.8,0.1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1}]

Building an Audio Classifier

Start from a simple multilayer perceptron network:

mlp=NetChain[
{
FlattenLayer[],
LinearLayer[400],
ElementwiseLayer[Ramp],
LinearLayer[],
SoftmaxLayer[]
},
"Input"->enc,
"Output"->dec
];

Every layer is a function that processes an array of data. The data can be of any rank, and the rank and dimensions can change layer by layer.

For audio signals, the input array is usually of rank 2, where the first dimension is time. This means that the first dimension can be a dynamic one (in this case a fixed length is used). The interpretation of the other dimension depends on the specifics of the encoder used.

The output array is a list of rank 1 with a dimension equal to the number of classes.

Building an Audio Classifier

NetTrain

There are a lot of hyperparameters in training neural nets (e.g. what optimizer is used, batch size, learning rate, regularization...), but NetTrain can worry about finding a reasonable combination of those:

resultMLP=NetTrain[
mlp,
spokenDigitsTraining,
All,
ValidationSet->Scaled[.05],
MaxTrainingRounds->20
];

resultMLP

Results

Training time:

Quantity[resultMLP["TotalTrainingTime"],"s"]

Training throughput:

Quantity[resultMLP["TotalBatches"] resultMLP["BatchSize"],"Item"] / Quantity[resultMLP["TotalTrainingTime"],"s"]

Grid[
{{Style["Cross Entropy Loss","Text"],Style["Error Rate","Text"]},
{Lookup[resultMLP["FinalPlots"],"Loss"], Lookup[resultMLP["FinalPlots"],"ErrorRate"]}}
]

With[
{vms=resultMLP["ValidationMeasurementsLists"]},
Grid[{
{Style["Lowest Validation Error Rate:","Text"],Style[ToString[100Min[Lookup[vms,"ErrorRate"]]]<>"%","Text"]},
{Style["Final Validation Error Rate:","Text"],Style[ToString[100Last[Lookup[vms,"ErrorRate"]]]<>"%","Text"]}},
Alignment->Left
]
]

Row[{Style["Accuracy: "<>ToString[100NetMeasurements[resultMLP["TrainedNet"],spokenDigitsTesting,"Accuracy"]]<>"%","Text"],"\t",NetMeasurements[resultMLP["TrainedNet"],spokenDigitsTesting,"ConfusionMatrixPlot"]}]

This is already a very impressive result, and it demonstrates the power of extracting meaningful features using NetEncoder, compared to using the plain data. However, the complexity and quality of the model can be improved to build a much more powerful network.

Building an Audio Classifier

A More Audio-Specific Example

Up to now, you have used an encoder with the setting “TargetLength”→50, thus ignoring the variable-length nature of audio signals. From now on,  “TargetLength”→All will be used, so you will need to define a network that will accept variable-length data as input:

enc=NetEncoder[{"AudioMFCC","TargetLength"->All,"WindowSize"->(Quantity[40, "ms", milliseconds, "Milliseconds"]),"Offset"->(Quantity[20, "ms", milliseconds, "Milliseconds"]),"NumberOfCoefficients"->32}];

If you try to use the same multilayer perceptron, you get an error message because of the dynamic time dimension:

NetInitialize[NetChain[{LinearLayer[400],Ramp,LinearLayer[],SoftmaxLayer[]},"Input"->enc,"Output"->dec]]

In this example, the network will be based on a few convolutional layers. Notice how the “Interleaving” option allows us to apply convolutions to the time and frequency dimensions, despite the time dimension being of variable length. Before the classification we solve the variable length problem by aggregating the result on the time dimension.

spokenDigitsNetConv=
NetChain[
{
ReshapeLayer[{Inherited,Automatic,1}],
ConvolutionLayer[64,3,"PaddingSize"->1,"Interleaving"->True],
Ramp,
ConvolutionLayer[64,3,"PaddingSize"->1,"Interleaving"->True],
Ramp,
AggregationLayer[Mean,1],
FlattenLayer[],
LinearLayer[],
SoftmaxLayer[]
},
"Input"->enc,
"Output"->dec
];

Building an Audio Classifier

NetTrain

resultConv=NetTrain[
spokenDigitsNetConv,
spokenDigitsTraining,
All,
ValidationSet->Scaled[.05]
];

resultConv

Results

Training time:

Quantity[resultConv["TotalTrainingTime"],"s"]

Training throughput:

Quantity[resultConv["TotalBatches"] resultConv["BatchSize"],"Item"] / Quantity[resultConv["TotalTrainingTime"],"s"]

Grid[
{{Style["Cross Entropy Loss","Text"],Style["Error Rate","Text"]},
{Lookup[resultConv["FinalPlots"],"Loss"], Lookup[resultConv["FinalPlots"],"ErrorRate"]}}
]

With[
{vms=resultConv["ValidationMeasurementsLists"]},
Grid[{
{Style["Lowest Validation Error Rate:","Text"],Style[ToString[100Min[Lookup[vms,"ErrorRate"]]]<>"%","Text"]},
{Style["Final Validation Error Rate:","Text"],Style[ToString[100Last[Lookup[vms,"ErrorRate"]]]<>"%","Text"]}},
Alignment->Left
]
]

Row[{Style["Accuracy: "<>ToString[100NetMeasurements[resultConv["TrainedNet"],spokenDigitsTesting,"Accuracy"]]<>"%","Text"],"\t",NetMeasurements[resultConv["TrainedNet"],spokenDigitsTesting,"ConfusionMatrixPlot"]}]

Building an Audio Classifier

Using Recurrent Layers

In this example, the network will be based on a couple of recurrent layers. These layers are optimized for variable-length sequences and have the added advantage that the number of trainable parameters is very low:

spokenDigitsNet=
NetChain[
{
GatedRecurrentLayer[32],
GatedRecurrentLayer[64],
SequenceLastLayer[],
LinearLayer[64],
ElementwiseLayer[Ramp],
LinearLayer[],
SoftmaxLayer[]
},
"Input"->enc,
"Output"->dec
];

Building an Audio Classifier

Using Recurrent Layers

spokenDigitsNet=
NetChain[
{
GatedRecurrentLayer[32],
GatedRecurrentLayer[64],
SequenceLastLayer[],
LinearLayer[64],
ElementwiseLayer[Ramp],
LinearLayer[],
SoftmaxLayer[]
},
"Input"->enc,
"Output"->dec
];

In a recurrent layer, the neuron holds a single set of weights for all time steps, but at each time step it is fed both the input and its own state from the previous step:

Block[
{h=4,r=0.3,spacing=2,layer0Pts,layer1Pts,rnnPts,n=5},
layer0Pts=Thread[{spacing Range[0,n],h}];
layer1Pts=Thread[{spacing Range[0,n],0}];
rnnPts=Thread@{Thread[{ spacing Range[0,n]-.5,h/2-.4}],Thread[{.5+ spacing Range[0,n],h/2+.4}]};
Graphics[
{{Gray,Arrowheads[Medium],
MapThread[Arrow[{#1,#2+{0,r}}]&,{layer0Pts,layer1Pts}],
MapThread[Arrow[{#1,#2+{0,.4}}]&,{layer0Pts,Mean/@rnnPts}],
MapIndexed[Text[Subscript["h",ToString[First[#2]]],#+{1,.2}]&,Mean/@Most@rnnPts],
Text[Subscript["h","0"],{-1,2.2}],
Arrowheads[Small],
Arrow[{{-1.5,h/2},{-.5,h/2}}],
Arrow[{#[[1]]+{.5,0},#[[2]]-{.5,0}}]&/@Partition[Mean/@rnnPts,2,1]},
{FaceForm[Orange],Map[Disk[#,r]&,layer0Pts],Map[Disk[#,r]&,layer1Pts]},
{Gray,MapIndexed[Text[Subscript["in",ToString[First[#2]]],#+{0,.5}]&,layer0Pts],
MapIndexed[Text[Subscript["out",ToString[First[#2]]],#-{0,.5}]&,layer1Pts]},
{FaceForm[White],EdgeForm[Black(*Directive[Thick,Lighter@Lighter@Blue]*)],Map[{Rectangle[Sequence@@#,RoundingRadius->.05],Gray,Text["GRU",Mean[#]]}&,rnnPts]}
}
]
]

This means that you can process sequences of arbitrary length with a very small number of parameters, while keeping track of short- and long-term dependencies:

Building an Audio Classifier

Using Recurrent Layers

The details of the operation computed at each time step depend on which specific layer is used. The most common ones are BasicRecurrentLayer, GatedRecurrentLayer and LongShortTermMemoryLayer. You can use NetFoldOperator to create custom recurrent layers.

You can look at the total number of elements in the various recurrent layers. The LongShortTermMemoryLayer is the most complex one, providing a solution to both the vanishing gradient and exploding gradient problems:

Head[#]->NetInformation[
#,
"ArraysTotalElementCount"
]&/@{BasicRecurrentLayer[32,"Input"-> {"Varying",28}],GatedRecurrentLayer[32,"Input"-> {"Varying",28}],LongShortTermMemoryLayer[32,"Input"-> {"Varying",28}]}//Association//Dataset

In this net, the GatedRecurrentLayer is used, since it proves to be a good compromise between efficiency and resilience to gradient problems. Given a sequence of inputs {(Subscript[x, 1]),(Subscript[x, 2]),...,(Subscript[x, t])}, the GatedRecurrentLayer computes a sequence of states {(Subscript[s, 1]),(Subscript[s, 2]),...,(Subscript[s, t])} using the following recurrence relation:

TextGrid[{
{TraditionalForm["input gate:"],TraditionalForm[HoldForm[(Subscript[i, t])=LogisticSigmoid[(Subscript[W, ix])·(Subscript[x, t])+(Subscript[W, is])·(Subscript[s, t-1])+(Subscript[b, i])]]]},
{TraditionalForm["reset gate:"],TraditionalForm[HoldForm[(Subscript[r, t])=LogisticSigmoid[(Subscript[W, rx])·(Subscript[x, t])+(Subscript[W, rs])·(Subscript[s, t-1])+(Subscript[b, r])]]]},
{TraditionalForm["memory gate:"],TraditionalForm[HoldForm[(Subscript[m, t])=Tanh[(Subscript[W, mx])·(Subscript[x, t])+(Subscript[r, t])*((Subscript[W, ms])·(Subscript[s, t-1]))+(Subscript[b, m])]]]},
{TraditionalForm["state:"],TraditionalForm[HoldForm[(Subscript[s, t])=(1-(Subscript[i, t]))*(Subscript[m, t])+(Subscript[i, t])*(Subscript[s, t-1])]]}
}]

In the recurrent layers, nonlinearities are included in the recurrence relations:

Animate[
Column[{
	TableForm[{x,ss[[;;tt]]},TableHeadings->{{"input:","state:"}}],
	Spacer[1],
	TextGrid[{
		{TraditionalForm["input gate:"],TraditionalForm[HoldForm[(Subscript[i, t])=LogisticSigmoid[(Subscript[W, ix])·(Subscript[x, t])+(Subscript[W, is])·(Subscript[s, t-1])+(Subscript[b, i])]]],ii[[tt]]},
		{TraditionalForm["reset gate:"],TraditionalForm[HoldForm[(Subscript[r, t])=LogisticSigmoid[(Subscript[W, rx])·(Subscript[x, t])+(Subscript[W, rs])·(Subscript[s, t-1])+(Subscript[b, r])]]],rr[[tt]]},
		{TraditionalForm["memory gate:"],TraditionalForm[HoldForm[(Subscript[m, t])=Tanh[(Subscript[W, mx])·(Subscript[x, t])+(Subscript[r, t])*((Subscript[W, ms])·(Subscript[s, t-1]))+(Subscript[b, m])]]],mm[[tt]]},
		{TraditionalForm["state:"],TraditionalForm[HoldForm[(Subscript[s, t])=(1-(Subscript[i, t]))*(Subscript[m, t])+(Subscript[i, t])*(Subscript[s, t-1])]],ss[[tt]]}
	}]
}],
{{tt,1,"time:"},Range[1,6],AppearanceElements->"PlayPauseButton"},
Paneled-> False,
FrameMargins-> 0,
ControlPlacement->Right,
AnimationRunning->False,
Initialization:>(dim = 1;
	gru = NetInitialize@GatedRecurrentLayer[dim, "Input" -> {"Varying", 1}];
	x = RandomReal[1, {5, 1}];
	ClearAll[i,r,m,s];
{(Subscript[b, i]),(Subscript[W, ix]),(Subscript[W, is]),(Subscript[b, m]),(Subscript[W, mx]),(Subscript[W, ms]),(Subscript[b, r]),(Subscript[W, rx]),(Subscript[W, rs])} = 
	Normal/@Values[NetInformation[gru,"Arrays"]];
ii = mm = rr = ss = ConstantArray[0., {Length[x], dim}];
Table[
	ii[[t]]=LogisticSigmoid[(Subscript[W, ix]).x[[t]]+(Subscript[W, is]).If[t==1,ConstantArray[0., dim],ss[[t-1]]]+(Subscript[b, i])];
	rr[[t]]=LogisticSigmoid[(Subscript[W, rx]).x[[t]]+(Subscript[W, rs]).If[t==1,ConstantArray[0., dim],ss[[t-1]]]+(Subscript[b, r])];
	mm[[t]]=Tanh[(Subscript[W, mx]).x[[t]]+rr[[t]]*((Subscript[W, ms]).If[t==1,ConstantArray[0., dim],ss[[t-1]]])+(Subscript[b, m])];
	ss[[t]]=(1-ii[[t]])*mm[[t]]+ii[[t]]*If[t==1,ConstantArray[0., dim],ss[[t-1]]];,
	{t, Length[x]}
];
PrependTo[x,Spacer[.1]];
PrependTo[ss,ConstantArray[0., dim]];
PrependTo[mm,ConstantArray[0., dim]];
PrependTo[rr,ConstantArray[0., dim]];
PrependTo[ii,ConstantArray[0., dim]];)
]

The trainable parameters in that recurrent relation are these:

NetInformation[GatedRecurrentLayer[32,"Input"->{"Varying",28}],"ArraysElementCounts"]//Dataset

Summary

You should use a recurrent layer if:

the data has variable length

the machine learning task needs to to take into account short- and long-term dependencies

the number of weights and biases needs to stay small.

Building an Audio Classifier

NetTrain

result=NetTrain[
spokenDigitsNet,
spokenDigitsTraining,
All,
ValidationSet->Scaled[.05],
MaxTrainingRounds->12
];

Results

Training time:

Quantity[result["TotalTrainingTime"],"s"]

Training throughput:

Quantity[result["TotalBatches"] result["BatchSize"],"Item"] / Quantity[result["TotalTrainingTime"],"s"]

Grid[
{{Style["Cross Entropy Loss","Text"],Style["Error Rate","Text"]},
{Lookup[result["FinalPlots"],"Loss"], Lookup[result["FinalPlots"],"ErrorRate"]}}
]

With[
{vms=result["ValidationMeasurementsLists"]},
Grid[{
{Style["Lowest Validation Error Rate:","Text"],Style[ToString[100Min[Lookup[vms,"ErrorRate"]]]<>"%","Text"]},
{Style["Final Validation Error Rate:","Text"],Style[ToString[100Last[Lookup[vms,"ErrorRate"]]]<>"%","Text"]}},
Alignment->Left
]
]

Row[{Style["Accuracy: "<>ToString[100NetMeasurements[result["TrainedNet"],spokenDigitsTesting,"Accuracy"]]<>"%","Text"],"\t",NetMeasurements[result["TrainedNet"],spokenDigitsTesting,"ConfusionMatrixPlot"]}]

From the start, you get a very accurate result with a much smaller net. You can try to improve the result by adding some regularization in the form of dropout.

Building an Audio Classifier

Adding Dropout

spokenDigitsNetDropout=
NetChain[
{
GatedRecurrentLayer[32,"Dropout"->.2],
GatedRecurrentLayer[64,"Dropout"->.2],
SequenceLastLayer[],
LinearLayer[64],
ElementwiseLayer[Ramp],
LinearLayer[],
SoftmaxLayer[]
},
"Input"->enc,
"Output"->dec
];

In this case, the dropout is included in the definition of the recurrent layer. In this specific implementation, the dropout is applied to the gate contributions from the input, using the same pattern of units at each sequence step.

resultDropout=NetTrain[
spokenDigitsNetDropout,
spokenDigitsTraining,
All,
ValidationSet->Scaled[.05],
MaxTrainingRounds->12
];

Results

Grid[
{{Style["Cross Entropy Loss","Text"],Style["Error Rate","Text"]},
{Insert[Lookup[resultDropout["FinalPlots"],"Loss"],Epilog->{Dashed,Lookup[result["FinalPlots"],"Loss"][[1,1,1]]},{1,1,-1}], 
Insert[Lookup[resultDropout["FinalPlots"],"ErrorRate"],Epilog->{Dashed,Lookup[result["FinalPlots"],"ErrorRate"][[1,1,1]]},{1,1,-1}]}}
]

With[
{vms=resultDropout["ValidationMeasurementsLists"]},
Grid[{
{Style["Lowest Validation Error Rate:","Text"],Style[ToString[100Min[Lookup[vms,"ErrorRate"]]]<>"%","Text"]},
{Style["Final Validation Error Rate:","Text"],Style[ToString[100Last[Lookup[vms,"ErrorRate"]]]<>"%","Text"]}},
Alignment->Left
]
]

Row[{Style["Accuracy: "<>ToString[100NetMeasurements[resultDropout["TrainedNet"],spokenDigitsTesting,"Accuracy"]]<>"%","Text"],"\t",NetMeasurements[resultDropout["TrainedNet"],spokenDigitsTesting,"ConfusionMatrixPlot"]}]

If you are wondering why you cannot seem to get a similar level of accuracy as LeNet on MNIST, one explanation is that the spoken digits dataset is less clean: the recordings have been contributed by users, recorded on a variety of devices, and some of them are even misclassified. You can get an idea of where the network fails using the “WorstClassifiedExamples” property on ClassifierMeasurements:

ClassifierMeasurements[resultDropout["TrainedNet"],#Input->#Output&/@spokenDigitsTesting,"WorstClassifiedExamples"]

Net Surgery for Feature Extraction

Spoken Digits Features

You can take the first recurrent network that you trained and chop off the last few layers that are in charge of the classification:

spokenDigitsFeatures=NetTake[resultDropout["TrainedNet"],{1,5}]

This will produce a net that, for a variable-length input, will produce a single vector of length 64 that lives in a space that is semantically significant for spoken digits from 0 to 9. You can use it to produce a semantic visualization of the test dataset of the spoken digits dataset:

With[{colors=AssociationMap[ColorData[97],Range[0,9]]},FeatureSpacePlot[Callout[Style[#Input,colors[#Output]],#Output]&/@spokenDigitsTesting,FeatureExtractor->spokenDigitsFeatures,LabelingFunction->None]]

The recordings corresponding to different digits are clustered, even though they are recorded by a lot of different speakers.

Audio Event Detection

What if you want to construct a network capable of computing a valid prediction per each time stamp, and then aggregating them over the time axis to compute a single result? Essentially, you can shift the problem from classification:

ttt={\[LeftAssociation]"Input"->,"SpeakerID"->"aa753bb9","Output"->7\[RightAssociation],\[LeftAssociation]"Input"->,"SpeakerID"->"a5609cce","Output"->4\[RightAssociation],\[LeftAssociation]"Input"->,"SpeakerID"->"5677ec77","Output"->4\[RightAssociation]};Graphics[{Inset[//Rasterize,{0,0},{0,0},{.5,Automatic}],
Arrowheads[Small],
Arrow[{{.7,.3},{1,.3}}],
Style[Text["dog",{1.5,.3}],Large]
},PlotRange->{{0,2},{0,.8}},ImageSize->Tiny]

to a sound event detection one:

AudioPlot[AudioJoin[AudioNormalize/@ttt[[All,"Input"]]],Frame->False,Ticks->None,Epilog->{
{EdgeForm[Black],Opacity[0],Rectangle[{.1,-1},{.5,1}]},
{EdgeForm[Black],Opacity[0],Rectangle[{1.4,-1},{1.7,1}]},
{EdgeForm[Black],Opacity[0],Rectangle[{2.4,-1},{2.8,1}]},
Style[Text["cat",{.3,-1.6}],18],
Style[Text["dog",{1.55,-1.6}],18],
Style[Text["cat",{2.6,-1.6}],18]
},
PlotRange->{All,{-2.2,1}},Axes->False]

Data and encoders

You need to keep in mind that you only have access to a weakly labeled dataset. This means that the network will have to infer the temporal localization of the events on its own.

ro=ResourceObject["Audio Cats and Dogs"];
data=ResourceData[ro];
classes=Normal[data[Union,"Label"]];
RandomSample[data,2]

The duration of the signals varies between 1 and 18 seconds.

data=Map[#Audio->#Label&,Normal[data]];
{trainingData,testingData}=TakeDrop[RandomSample[data],Round[.8Length[data]]];
Histogram[Duration/@data[[All,1]]]

Use the “AudioMelSpectrogram” encoder to feed the audio signal into the network.

enc=NetEncoder[{"AudioMelSpectrogram","WindowSize"->(Quantity[40, "ms", milliseconds, "Milliseconds"]),"Offset"->(Quantity[20, "ms", milliseconds, "Milliseconds"]),"NumberOfFilters"->64,"Augmentation"->{"TimeShift"->Scaled[{-.5,0}],"VTLP"->{0.8,1.2}}}]

dec=NetDecoder[{"Class",Union[data[[All,2]]]}];

Audio Event Detection

Network and Training

The net will be based on a stack of recurrent layers ((RefLink[GatedRecurrentLayer, paclet:ref/GatedRecurrentLayer])), and an (RefLink[AggregationLayer, paclet:ref/AggregationLayer]) to pool the result in the time dimension. This allows the net to output a single classification result instead of a sequence.

net=
NetChain[
{
ElementwiseLayer[Log[#+.01]&],
GatedRecurrentLayer[64,"Dropout"->{"VariationalInput"->.2}],
GatedRecurrentLayer[64,"Dropout"->{"VariationalInput"->.2}],
GatedRecurrentLayer[2],
AggregationLayer[Max,1],
SoftmaxLayer[]
},
"Input"->enc,
"Output"->NetDecoder[{"Class",classes}]
];

trainedNet=NetTrain[
net,
trainingData,
All,
ValidationSet->Scaled[.08],
MaxTrainingRounds->40
];

To use the net in a reproducible fashion, the encoder with the augmentations needs to be replaced with one without them.

finalNet=NetReplacePart[trainedNet["TrainedNet"],"Input"->NetEncoder[{"AudioMelSpectrogram","WindowSize"->(Quantity[40, "ms", milliseconds, "Milliseconds"]),"Offset"->(Quantity[20, "ms", milliseconds, "Milliseconds"]),"NumberOfFilters"->64}]]

Results

Grid[
{{Style["Cross Entropy Loss","Text"],Style["Error Rate","Text"]},
{Insert[Lookup[trainedNet["FinalPlots"],"Loss"],Epilog->{Dashed,Lookup[trainedNet["FinalPlots"],"Loss"][[1,1,1]]},{1,1,-1}], 
Insert[Lookup[trainedNet["FinalPlots"],"ErrorRate"],Epilog->{Dashed,Lookup[trainedNet["FinalPlots"],"ErrorRate"][[1,1,1]]},{1,1,-1}]}}
]

With[
{vms=trainedNet["ValidationMeasurementsLists"]},
Grid[{
{Style["Lowest Validation Error Rate:","Text"],Style[ToString[100Min[Lookup[vms,"ErrorRate"]]]<>"%","Text"]},
{Style["Final Validation Error Rate:","Text"],Style[ToString[100Last[Lookup[vms,"ErrorRate"]]]<>"%","Text"]}},
Alignment->Left
]
]

Row[{Style["Accuracy: "<>ToString[100NetMeasurements[trainedNet["TrainedNet"],testingData,"Accuracy"]]<>"%","Text"],"\t",NetMeasurements[trainedNet["TrainedNet"],testingData,"ConfusionMatrixPlot"]}]

Audio Event Detection

Time-Resolved Results

By removing the (RefLink[AggregationLayer, paclet:ref/AggregationLayer]) and reattaching the (RefLink[SoftmaxLayer, paclet:ref/SoftmaxLayer]) to the chopped net, a network that returns a sequence of class probabilities instead of a single classification result is obtained.

timeResolvedNet=NetAppend[NetTake[finalNet,4],SoftmaxLayer[]]

generateTimeSeries[in_Audio]:=Association[MapThread[#2->TimeSeries[#1,{0,QuantityMagnitude[Duration[in],"Seconds"]}]&,{Transpose[Normal[timeResolvedNet[in]]],classes}]]

Test this time-resolved net on a signal that contains both cat and dog noises.

aud=;
aud//AudioAnnotationLookup//Dataset

We can define some functions to aggregate the time resolved result into labeled intervals:

getIntervals[in_,thr_:.51]:=Map[If[#[[1,2]]===1,{#[[1,1]],#[[-1,1]]},Nothing]&,SplitBy[MapAt[If[#>thr,1,0]&,Normal[in],{All,2}],#[[2]]&]]
getRectangles[in_]:=Map[Rectangle[{#[[1]],-1},{#[[2]],1}]&,in]

intervals=getIntervals/@generateTimeSeries[aud];

And finally plot the recognition result of the time-resolved net:

Legended[AudioPlot[aud,ImageSize->Medium,Epilog->{,getRectangles[intervals["cat"]],,getRectangles[intervals["dog"]]},AnnotationRules->"animal",AspectRatio->1/2],SwatchLegend[{,},{"cat","dog"}]]

Transfer Learning

Sometimes the amount of data available to train a network is insufficient for the task at hand. Transfer learning is a possible solution to this problem. Instead of training a network from scratch, it is possible to use as a starting point a net that has already been trained on a different but related task.

dir="/Files/Data/ESC-50-master";
rawMetadata=Import[FileNameJoin[{dir,"meta","esc50.csv"}]];
metaData=Map[Append[AssociationThread[rawMetadata[[1]]->MapAt[File@FileNameJoin[{dir,"audio",#}]&,#,1]],"features"->None]&,Rest[rawMetadata]];

RandomChoice[metaData]//Dataset

classes=DeleteDuplicates[metaData[[All,"category"]]];
classes//Short

Use the network from (RefLink[AudioIdentify, paclet:ref/AudioIdentify]) as a starting point.

net=NetModel["Wolfram AudioIdentify V1 Trained on AudioSet Data","Size"->"Large"]

The network was trained on the AudioSet dataset, where each audio signal is annotated with the sound classes/sources that are present in the recording. The labels are organized in an ontology of about 632 classes that span a very wide domain of sound types or sources, from musical instruments and music types, to animal, mechanical and human sounds. Some example of classes:

AudioIdentify;AudioIdentify`Private`Ontology["Data"]//Keys//Short

The ontology graph for AudioSet:

AudioIdentify`Private`Ontology["Graph"]

The main inspiration was take from "CNN architectures for large-scale audio classification". Like the architectures presented in that paper, out net has a CNN architecture (based on MobileNet v2):

mainNet=NetExtract[net,{1,"Net"}]
NetInformation[mainNet,"FullSummaryGraphic"]

Transfer Learning

From that, we can construct a feature extractor that produces a sequence of semantically significant vectors:

featureExtractor=NetChain[{NetMapOperator[NetDrop[mainNet,-3]],ReshapeLayer[{Inherited,Automatic}]},"Input"->net[["Input"]]]

Instead of retraining the full net and specifying a (RefLink[LearningRateMultipliers, paclet:ref/LearningRateMultipliers]) option in (RefLink[NetTrain, paclet:ref/NetTrain]) to train only the classification layers, precompute the results of the feature extractor net and train the classifier. This avoids redundant evaluation of the full net during training.

It is also possible to divide the dataset into the folds defined by its creator.

metaData[[All,"features"]]=featureExtractor[metaData[[All,"filename"]]];

folds=Table[<|"Train"->(#features->#category&/@Select[metaData,#fold=!=i&]),"Test"->(#features->#category&/@Select[metaData,#fold===i&])|>,{i,5}];

Construct a simple recurrent classifier network that will be attached to the feature extractor and train it on each of the folds.

Define a simple recurrent classifier and train it on each of the folds:

classifier=NetChain[{GatedRecurrentLayer[128,"Dropout"->{"VariationalInput"->.1}],AggregationLayer[Mean,1],LinearLayer[],SoftmaxLayer[]},"Input"->{"Varying",1664},"Output"->NetDecoder[{"Class",classes}]]

trainedClassifiers=Table[
NetTrain[classifier,data["Train"],ValidationSet->Scaled[.05],MaxTrainingRounds->20],
{data,folds}];

After training, it is easy to measure the accuracy of the classifiers on each of the folds and average them to obtain a cross-validated result.

Measure the performance on each fold and compute their average:

With[{accuracies=MapThread[NetMeasurements[#1,#2,"Accuracy"]&,{trainedClassifiers,folds[[All,"Test"]]}]},Around[Mean[accuracies],StandardDeviation[accuracies]]]

As a final test, construct a net that joins the original feature extractor and an ensemble average of the trained classifiers and run it on a signal that was not in the original dataset.

Test the final net on an unrelated example:

finalNet=NetGraph[Join[trainedClassifiers,{TotalLayer[],ElementwiseLayer[#/5&],featureExtractor}],{NetPort["Input"]->8->Range[5]->6->7},"Output"->NetDecoder[{"Class",classes}]]

finalNet[,{"TopProbabilities",3}]//Reverse

Thanks!

Init

formatForMP[data_]:=Reverse[Transpose[If[ArrayDepth[data]==3,Map[Norm,Normal[data],{2}],Normal[data]]]];
conform[audio_]:= AudioResample[AudioChannelMix[audio, "Mono"], 16000];
plotData[data_]:=AudioPlot[Audio[Flatten[data],SampleRate->16000],ImageSize->Medium];

{spokenDigitsTraining,spokenDigitsTesting} = Import[FileNameJoin[{NotebookDirectory[],"spokenDigitsData.wxf"}]];

resultConv=Import[FileNameJoin[{NotebookDirectory[],"resultConv.wxf"}]];

result = Import[FileNameJoin[{NotebookDirectory[],"result.wxf"}]];

resultDropout=Import[FileNameJoin[{NotebookDirectory[],"resultDropout.wxf"}]];

trainedNet=Import[FileNameJoin[{NotebookDirectory[],"trainedNet.wxf"}]];

metaData=Import[FileNameJoin[{NotebookDirectory[],"metaData.wxf"}]];

trainedClassifiers=Import[Import[FileNameJoin[{NotebookDirectory[],"trainedClassifiers.wxf"}]]];

audioIdentifyNet=NetModel["Wolfram AudioIdentify V1 Trained On AudioSet","Size"->"Large"];
coreNet=NetExtract[audioIdentifyNet,{1,"Net"}];