Neural Networks for Audio

Carlo Giacometti, Wolfram Research

Join the Conversation #WolframTechConf

ABSTRACT

In this talk I’ll cover the new capabilities of the Wolfram Language at the intersection of audio analysis and neural networks.

NetEncoder Updates

NetEncoder is the entry point for an audio object into the world of NN!

The audio encoders were introduced in 11.3 to allow fast, parallel feature extraction from audio objects and files. Several different features are supported in the audio encoders:

a=Import["ExampleData/rule30.wav"];

Audio

NetEncoder["Audio"][a][[;;5000]]//Flatten//ListLinePlot

Short-Time Fourier transform

NetEncoder["AudioSTFT"][a][[All,All,1]]//formatForMP//MatrixPlot

Spectrogram

NetEncoder["AudioSpectrogram"][a]//formatForMP//MatrixPlot

Mel-spectrogram

NetEncoder["AudioMelSpectrogram"][a]//formatForMP//MatrixPlot

MFCC

NetEncoder["AudioMFCC"][a]//Transpose//MatrixPlot

NetEncoder Updates

The audio encoders have been updated in 12: we get better performance, and the “Augmentation” parameter was added. This can turn out to be very useful if training on small amounts of data.

Augmentation in the audio encoders

Let’s start with a basic MFCC audio encoder:

enc=NetEncoder["AudioMFCC"]

TimeShift

enc=NetEncoder[{"AudioMelSpectrogram","Augmentation"->{"TimeShift"->NormalDistribution[0,.5]}}]

Table[MatrixPlot[formatForMP[enc[a]],ImageSize->Medium],2]//Column

Noise

enc=NetEncoder[{"AudioMelSpectrogram","Augmentation"->{"Noise"->UniformDistribution[{0,.1}]}}]

Table[MatrixPlot[formatForMP[enc[a]],ImageSize->Medium],2]//Column

noiseSignal=AudioGenerator[TriangleWave[1000+800Sin[2Pi#]]&,4]

enc=NetEncoder[{"AudioMelSpectrogram","Augmentation"->{"Noise"->{UniformDistribution[{0,.2}],noiseSignal}}}]

Table[MatrixPlot[formatForMP[enc[a]],ImageSize->Medium],2]//Column

Convolution

ir=ExampleData[{"Audio","IRRailwayTunnel"}];

enc=NetEncoder[{"AudioMelSpectrogram","Augmentation"->{"Convolution"->{UniformDistribution[{0,1}],ir}}}]

Table[MatrixPlot[formatForMP[enc[a]],ImageSize->Medium],2]//Column

Volume

enc=NetEncoder[{"AudioMelSpectrogram","Augmentation"->{"Volume"->UniformDistribution[{.5,1}]}}]

Table[enc[a]//MinMax,2]

VTLP

enc=NetEncoder[{"AudioMelSpectrogram","Augmentation"->{"VTLP"->UniformDistribution[{.3,1.5}]}}]

Table[MatrixPlot[formatForMP[enc[a]],ImageSize->Medium],2]//Column

NetEncoder Updates

Does it really help?

We can try it out on a reduced version of the Spoken Digit Commands dataset:

SeedRandom[42];
ro=ResourceObject["Spoken Digit Commands"];
trainingData=#Input->#Output&/@ResourceData[ro,"TrainingData"];
testingData=#Input->#Output&/@ResourceData[ro,"TestData"];
trainingDataReduced=RandomSample[trainingData,800];

We can define two encoders that are completely the same except for the augmentation:

encPlain=enc=NetEncoder[{"AudioMFCC","TargetLength"->All,"SampleRate"->16000,"WindowSize" -> 1024,"Offset"-> 570,"NumberOfCoefficients"->28,"Normalization"->"RMS"}];
encAugmentation=enc=NetEncoder[{"AudioMFCC","TargetLength"->All,"SampleRate"->16000,"WindowSize" -> 1024,"Offset"-> 570,"NumberOfCoefficients"->28,"Normalization"->"RMS","Augmentation"->{"Noise"->{.001,.01}}}];

Let’s use a simple recurrent net centered around two GatedRecurrentLayer:

net=NetChain[{GatedRecurrentLayer[32],GatedRecurrentLayer[64],SequenceLastLayer[],LinearLayer[64],Ramp,LinearLayer[],SoftmaxLayer[]},"Output"->NetDecoder[{"Class",Union[trainingData[[All,2]]]}]]

First we train the net without augmentation:

res=NetTrain[NetReplacePart[net,"Input"->encPlain],trainingDataReduced,"ResultsObject",MaxTrainingRounds->20,ValidationSet->Scaled[.05]]

And then with augmentation:

resAugmentation=NetTrain[NetReplacePart[net,"Input"->encAugmentation],trainingDataReduced,"ResultsObject",MaxTrainingRounds->20,ValidationSet->Scaled[.05]]

We can then compare the results:

cm=ClassifierMeasurements[res["TrainedNet"],testingData];
cm["Accuracy"]

cm=ClassifierMeasurements[resAugmentation["TrainedNet"],testingData];
cm["Accuracy"]

AudioIdentify

Another important addition to version 12 for audio analysis using neural networks in AudioIdentify.

a=Import["ExampleData/rule30.wav"]

The goal is to figure out what sounds are contained in a signal:

AudioIdentify[a]

It’s possible to restrict the categories to a subset of the available ones:

AudioIdentify[a,"Musical instrument"|"Animal"]

Return multiple results:

AudioIdentify[a,"Musical instrument",4]

Return the probability  of the result:

AudioIdentify[a,"Musical instrument",4,"Probability"]

Everything revolves around a neural net (of course...):

audioIdentifyNet=NetModel["HiddenAudioIdentifyMobileNetDepth1-3"]

AudioIdentify

How was the network trained?

Data

The data used in the training was AudioSet.

AudioSet consists of an expanding ontology of 632 audio event classes and a collection of 2,084,320 human-labeled 10-second sound clips drawn from YouTube videos. The ontology is specified as a hierarchical graph of event categories, covering a wide range of human and animal sounds, musical instruments and genres, and common everyday environmental sounds.

Some example of classes:

AudioIdentify`Private`Ontology["Data"]//Keys//Short

and how they are represented in the Wolfram Language:

soundEntity=RandomEntity["Sound"]

A description is usually available:

soundEntity["Description"]

The full ontology graph is available in a private context:

AudioIdentify`Private`Ontology["Graph"]

One very important thing to note in this dataset is that the classes are not mutually exclusive. We can look at the metainformation from one entry of the training data:

Import["http://storage.googleapis.com/us_audioset/youtube_corpus/v1/csv/balanced_train_segments.csv"][[4;;]]//RandomChoice

The training process needed to reflect this aspect. This was done by using one classifier per class:

mainNet=ConvolutionLayer[10,10];
net=NetGraph[
<|"mainNet"->{audioIdentifyNet[[1,"Net"]],FlattenLayer[]},"classifier"->{LinearLayer[527],LogisticSigmoid},"loss"->CrossEntropyLossLayer["Binary"]|>,
{"mainNet"->"classifier"->NetPort["loss","Input"]}
];
NetInformation[net,"SummaryGraphic"]

AudioIdentify

How was the network trained?

Network characteristics

The main inspiration was take from "CNN architectures for large-scale audio classification". Like the architectures presented in that paper, out net has a CNN architecture (based on MobileNet v2):

singleFrameNet=audioIdentifyNet[[1,"Net"]]
NetInformation[singleFrameNet,"FullSummaryGraphic"]

As you can see, the net takes a fixed-dimension input. How does this reconcile with the inherently variable length of audio signals?

Encoding

The encoder is simply a mel-spectrogram (the Log of that is computed by the network itself. The full mel-spectrogram (whose length depends on the length of the signal) is then partitioned in ~1 second chunks:

enc=audioIdentifyNet[["Input"]];
enc[["Function"]]

This way the encoder produces a variable length sequence of fixed size matrices:

enc[a]//Dimensions

MatrixPlot@*formatForMP/@enc[a]

Result from network

The network is then applied to the encoded data using a NetMapOperator:

multiFrameNet=NetMapOperator[singleFrameNet]

result=multiFrameNet[enc[a]];
Dimensions[result]

So we get the probability distribution over all the classes for every chunk of the signal:

ListLinePlot[#,PlotRange->All,ImageSize->Medium]&/@multiFrameNet[enc[a]]//Row

To get a single result per input we need to collapse the time dimension somehow. Since the goal is to figure out which sounds were present in the signal and not what the average sound in the signal was, AggregationLayer[Max] is used:

collapsedNet=NetChain[{multiFrameNet,AggregationLayer[Max,1]}];
result=collapsedNet[enc[a]];
Dimensions[result]

ListLinePlot[result,PlotRange->All]

The result can then be passed to the class decoder:

dec=audioIdentifyNet[["Output"]]

dec[result,"TopProbabilities"]

Ontology scoring

The result of the network is then scored using information from the AudioSet ontology.

AudioIdentify`Private`Ontology["Graph"]

It is possible to control the weight that is given to the depth in the branch for a result using the SpecificityGoal option:

AudioIdentify[a,All,2,SpecificityGoal->#]&/@{0,.4,.7,1}

AudioIdentify

It’s possible to run AudioIdentify in real time on an AudioStream by using the “CurrentAudio” property:

a=Audio["test.mp3"]

stream=AudioStream[a];
stream["BufferSize"]=(Quantity[0.5, "s", seconds, "Seconds"]);

AudioPlay[stream];

oldBuffer=Table[AudioGenerator["Silence",1],2];
Dynamic[
newBuffer=Join[oldBuffer,{stream["CurrentAudio"]}];
res=AudioIdentify[AudioJoin[newBuffer[[;;2]]],All,10,"Probability",AcceptanceThreshold->0];
oldBuffer=newBuffer[[2;;]];
res//Column
]

AudioStop[]

Feature extraction using the AudioIdentify Net

By deleting the last layers used for classification, we can create a semantically meaningful feature extractor for audio signals:

featureExtractorNet=NetMapOperator[NetChain[{NetDelete[singleFrameNet,{158,159,160}],ReshapeLayer[{Automatic}]}],"Input"->enc]

names=Delete[ExampleData["Audio"],21];
exampleData=ExampleData/@names;
labels=AssociationThread[exampleData->names[[All,2]]];

FeatureSpacePlot[exampleData,FeatureExtractor->featureExtractorNet]

By the way, a similar network is the default feature extractor for audio objects for machine learning functions, so now FeatureExtract, FeatureSpacePlot, Classify, etc.. will have a semantically significant behaviour in version 12:

FeatureSpacePlot[exampleData]

Feature extraction using the AudioIdentify Net

This can be a very good starting point for classifying collections of audio objects.

Data

We will be using the IRMAS dataset. It consists of a collections of audio snippets that are labeled with the main instrument present.

instrumentDataNames//RandomChoice

These are the available classes:

instrumentDataNames[[All,2]]//DeleteDuplicates

Using Classify

In this example I am pre encoding the data with the feature extractor net instead of using the FeatureExtractor option in Classify to make the process faster. The commented code is completely equivalent, just slightly less efficient.

(*Classify[instrumentTrainingData,FeatureExtractor->featureExtractorNet]*)

clInstruments=Classify[encodedInstrumentTrainingData(*PerformanceGoal->"DirectTraining"*)]

ClassifierMeasurements[clInstruments,encodedInstrumentTestData]

%["Accuracy"]
%%["ConfusionMatrixPlot"]

Feature extraction using the AudioIdentify Net

Using transfer learning

We can build a simple classifier that takes the data pre encoded with the feature extractor net as an input:

instrumentNet=NetChain[{NetMapOperator[{DropoutLayer[.3],LinearLayer[128],Ramp,DropoutLayer[.3],LinearLayer[128],Ramp}],AggregationLayer[Mean,1],LinearLayer[],SoftmaxLayer[]},"Input"->{"Varying",1664},"Output"->NetDecoder[{"Class",instrumentClasses}]]

Train the net:

instrumentNetTrained=NetTrain[instrumentNet,encodedInstrumentTrainingData,ValidationSet->Scaled[.1],MaxTrainingRounds->30]

And measure the results!

ClassifierMeasurements[instrumentNetTrained,encodedInstrumentTestData]

%["Accuracy"]
%%["ConfusionMatrixPlot"]

With some surgery, we can now build a net that computes the result directly from an audio object or file:

finalNet=NetChain[{featureExtractorNet,instrumentNetTrained}]

in=instrumentTestData//RandomChoice

finalNet[in[[1]]]

PitchRecognize

PitchRecognize is another function introduced in version 12. The goal is to find the fundamental frequency of a monophonic signal.

The default algorithm used is a variation of the classical YIN algorithm:

a=ExampleData[{"Audio","Piano"}];
PitchRecognize[a]//ListLinePlot

For a potentially more precise, but more expensive result it is possible to use the “CREPE” method, which is based on the network released with the paper "CREPE: A Convolutional Representation for Pitch Estimation":

PitchRecognize[a,Method->"CREPE"]//ListLinePlot

PitchRecognize

The network can be found in Wolfram Neural Net Repository by just calling NetModel:

crepe=NetModel["CREPE Pitch Detection Net Trained on Monophonic Signal Data"]

The net operates directly on fixed size partitions of the raw waveform:

crepe[["Input"]][["Function"]]

The architecture is based on a chain of six convolution stacks, followed by a classifier. The net outputs a vector that represents the probability of the pitch being in one of 360 nonlinearly spaced frequency classes.

crepe[["Net"]]

The result is the probability of the signal to be in one of 360 pitch classes, so it needs to be transformed into a frequency. The most basic (and fast) strategy uses a simple interpolation:

findPrediction["Interpolation",salience_,center_: None] :=
Module[
	{c, a, endpoints, cents},
	If[center === None,
		c = First@Ordering[salience, -1],
		c = center
	];
	endpoints = {Max[1, c - 4], Min[Length@salience, c + 5]};
	a = Take[salience, endpoints];
	cents = Range[0, 7180, 20] + 1997.3794084376191;
	{10.*2^(Total[a*Take[cents, endpoints]]/Total[a]/1200.), salience[[c]]}
]
findPrediction["Interpolation", salience_, center_: None] /; ArrayDepth[salience] == 2:=
Map[Function[in, findPrediction["Interpolation", in, center]], salience]

To get a more accurate result one case use a HiddenMarkovProcess for the decoding:

HMP = 
Module[
	{starting,transition,emission,xx,yy,selfEmission},
	starting = ConstantArray[1./360,360];
	yy = ConstantArray[Range[360],360]; xx=Transpose@yy;
	transition = Map[Max[#,0.]&, 12-Abs[xx-yy], {2}];
	transition = N@#/Total[#]& /@ transition; selfEmission=0.1;
	emission = IdentityMatrix[360]*selfEmission + ConstantArray[(1. - selfEmission)/360., {360, 360}];
	HiddenMarkovProcess[starting, transition, emission]
];
findPrediction["Viterbi",salience_] :=
MapThread[
	findPrediction["Interpolation", ##]&,
	{salience, FindHiddenMarkovStates[First[Ordering[#, -1]]& /@ salience, HMP]}
]

We can then compute the result of the net:

res=NetModel["CREPE Pitch Detection Net Trained on Monophonic Signal Data"][AudioPad[a,{0.032,0.032}]];

And use the Hidden Markov process decoder to filter the result:

res=findPrediction["Viterbi",res];

This allows us to compute both a prediction for the pitch and the confidence in such prediction:

times={Range[0.,QuantityMagnitude[Duration[a],"Seconds"],.01]};
res=<|"Prediction"->TimeSeries[res[[All,1]],times],"Confidence"->TimeSeries[res[[All,2]],times]|>

Visualize the result:

ListLinePlot[res["Prediction"],ColorFunction->Function[{x,y},Directive@Opacity@res["Confidence"][x]],ColorFunctionScaling->False,PlotRange->{500,550}]

Team

Grid[Partition[{({{}, {"Shadi Ashnai"}, {"shadi"}}),({{}, {"Piotr Wendykier"}, {"piotrw"}}),({{}, {"Carlo Giacometti"}, {"carlog"}}),({{}, {"Mariusz Jankowski"}, {"mariuszj"}}),({{}, {"Chien-Yu Chen"}, {"chienyuc"}}),({{}, {"Rebecca Frederick"}, {"rebeccaf"}})},3]]

Init

GP@NF;
GP@ML;
SetOptions[Audio,AudioOutputDevice->"System Setting"];
Get["/Volumes/Files/Workspaces/Mathematica/Kernel/Source/Kernel/StartUp/Visualization/ChartLabeling.m"];

SeedRandom[42];

Formatting encoder output

formatForMP[x_]:=Log[Reverse[Transpose[x]]+.0001]

Dynamic AudioIdentify

(*enc=NetEncoder[{"AudioMelSpectrogram","WindowSize"->400,"Offset"->160,"WindowFunction"->HannWindow,"NumberOfFilters"->64,"TargetLength"->96,"MinimumFrequency"->20}];dec=audioIdentifyNet[["Output"]];
n=NetReplacePart[audioIdentifyNet[[1,"Net"]],{"Input"->enc,"Output"->dec}];
oldBuffer=AudioGenerator["Silence",.5];
Dynamic[
newBuffer=stream["CurrentAudio"];
res=n[AudioJoin[{oldBuffer,newBuffer}],{"TopProbabilities",10}];
oldBuffer=newBuffer;
res//Association//ListPlot[#,Filling->Axis,PlotRange->{0,1}]&
]*)

Instrument classification

instrumentFileNames=FileNames["*.wav","/Users/carlogico/Desktop/AudioDatasets/IRMAS-TrainingData/",2];
instrumentDataNames=Normal@AssociationMap[StringSplit[FileNameSplit[#][[-2]]][[-1]]&,File/@instrumentFileNames]/.{"cel"->"cello","cla"->"clarinet","flu"->"flute","gac"->"acoustic guitar","gel"->"electric guitar","org"->"organ","pia"->"piano","sax"->"saxophone","tru"->"trumpet","vio"->"violin","voi"->"voice"};
instrumentClasses=instrumentDataNames//Values//DeleteDuplicates
RandomSeed[2387];
instrumentTrainingData=RandomSample[instrumentDataNames,Floor[.8Length@instrumentDataNames]];
instrumentTestData=RandomSample[Complement[instrumentDataNames,instrumentTrainingData]];

(*encodedInstrumentTrainingData=instrumentTrainingData;
encodedInstrumentTrainingData[[All,1]]=featureExtractorNet[encodedInstrumentTrainingData[[All,1]]];
encodedInstrumentTestData=instrumentTestData;
encodedInstrumentTestData[[All,1]]=featureExtractorNet[encodedInstrumentTestData[[All,1]]];*)

cl = Import["/Users/carlogico/Desktop/WTC2018/NeuralNetsForAudio/Classifier.wmlf"];
encodedInstrumentTestData = Import["/Users/carlogico/Desktop/WTC2018/NeuralNetsForAudio/InstrumentTestData.mx"];
encodedInstrumentTrainingData = Import["/Users/carlogico/Desktop/WTC2018/NeuralNetsForAudio/InstrumentData.mx"];

instrumentNetTrained = Import["/Users/carlogico/Desktop/WTC2018/NeuralNetsForAudio/InstrumentNetTrained.mx"];