Understanding Speech

Carlo Giacometti, Wolfram Research

Join the Conversation #WolframTechConf

ABSTRACT

In this talk I’ll cover some possible applications that revolve around understanding various aspects of speech, specifically:

Keyword spotting

Speech recognition

Speaker recognition and identification

Gender classification

Keyword Spotting

We can start with a relatively simple problem: can we classify a recording of a single spoken word as a number from 0 to 10?

The Data

The dataset we’ll be using is a subset of the Speech Commands Dataset released by Google. It essentially is a sort of “spoken MNIST”:

ro=ResourceObject["Spoken Digit Commands"]

trainingData=ResourceData[ro,"TrainingData"];
testingData=ResourceData[ro,"TestData"];
RandomSample[trainingData,3]//Dataset

One important thing we made sure of is that the speakers in the training and testing sets are different. This means that in the testing phase, the trained classifier/network will encounter speakers that it will never have heard before.

Intersection[trainingData[[All,"SpeakerID"]],testingData[[All,"SpeakerID"]]]

The possible output values are the digits from 0 to 9:

classes=Union[trainingData[[All,"Output"]]]

Conveniently, the length of all of the input data is between .5 and 1 seconds, with the majority for the signals being one second long.

Dataset[trainingData][Histogram[#,ScalingFunctions->"Log"]&@*Duration,"Input"]

There are several approaches to this problem, that range from a fully automated solution to a complete neural net.

Keyword Spotting

Automated machine learning

First of all, let’s format the data to something that would make Classify happy:

classifyTrainingData=#Input->#Output&/@trainingData;
classifyTestingData=#Input->#Output&/@testingData;

We’ll guide Classify a bit here by specifying what features should be used. In this case we’ll first extract the MFCC features using a NetEncoder, and then compute some statistics on them to produce a matrix of fixed dimensions even if the input is inherently of variable length:

encoder=NetEncoder[{"AudioMFCC","TargetLength"->28,"SampleRate"->16000,"WindowSize" -> 1024,"Offset"-> 570,"NumberOfCoefficients"->28,"Normalization"->True}];
clDigits=Classify[classifyTrainingData,FeatureExtractor->(Flatten[{Mean[#],StandardDeviation[#],Kurtosis[#]}]&@*encoder),PerformanceGoal->"DirectTraining"];

Let’s inspect the results:

cmDigits=ClassifierMeasurements[clDigits,classifyTestingData];
cmDigits["Accuracy"]
cmDigits["ConfusionMatrixPlot"]

Keyword Spotting

Recurrent Neural Network

An more time consuming approach is to train a neural network from scratch. Again we’ll be defining our encoder to extract the MFCC features:

encoder=NetEncoder[{"AudioMFCC","TargetLength"->All,"NumberOfCoefficients"->28,"SampleRate"->16000,"WindowSize" -> 1024,"Offset"-> 571,"Normalization"->True}]

We’ll use an architecture based on the GatedRecurrentLayer. Used on its own, it returns its state per each time step, but we are only interested in the classification of the entire sequence; i.e. we want a single output for all time steps. We can use SequenceLastLayer to extract the last state for the sequence. After that we can add a couple of fully connected layers to do the classification.

rnn=
NetChain[{
GatedRecurrentLayer[32],
GatedRecurrentLayer[64],
SequenceLastLayer[],
LinearLayer[64],
Ramp,
LinearLayer[Length@classes],
SoftmaxLayer[]},
"Input"->encoder,
"Output"->NetDecoder[{"Class",classes}]
]

Let’s leave NetTrain in charge of all hyperparameters:

resultObjectRNN=NetTrain[
rnn,
trainingData,
All,
ValidationSet->Scaled[.05]
]

... and measure the performance:

cm=ClassifierMeasurements[resultObjectRNN["TrainedNet"],classifyTestingData];
cm["Accuracy"]
cm["ConfusionMatrixPlot"]

Keyword Spotting

Feature extraction

Now that we have some a network, we can play with it a bit. Let’s chop off the last two layers:

choppedNet=NetTake[resultObjectRNN["TrainedNet"],{1,5}]

This leaves us with something that produces a vector of 64 number per each input signal. We can try to use it as feature extractor and plot the results:

FeatureSpacePlot[Style[#Input,ColorData[97][#Output+1]]->#Output&/@testingData,FeatureExtractor->choppedNet]

It looks like that the various classes get properly separated!

We can also record a signal, and test the trained network on it:

a=AudioTrim@AudioCapture[]

resultObjectRNN["TrainedNet"][a]

SpeechRecognize

Speech recognition is a much more general (and more difficult) task than a simple classification. In version 12 this is embedded in the language:

a=ExampleData[{"Audio","MaleVoice"}]
SpeechRecognize[a]

The operation is two-fold: there’s a an acoustic model for speech, whose result is then filtered by a character-level language model. It’s possible to skip the language model post-processing:

SpeechRecognize[a,"LanguageModel"->False]

SpeechRecognize

Acoustic model

The acoustic model is the Deep Speech 2 network released by Baidu:

deepSpeech=NetModel["Deep Speech 2 Trained on Baidu English Data"]

The network was trained on an unreleased Baidu dataset of 8628 hours of labeled speech.

Encoder

The encoder takes an audio or file object and produces a spectrogram:

encoder=deepSpeech[["Input"]]

result=encoder[a];
result//formatForMP//MatrixPlot

Net

The network consists of a few convolutional layers over both time and frequency, followed by gated recurrent unit (GRU) layers (modified with an additional batch normalization).

net=NetReplacePart[deepSpeech,{"Input"->None,"Output"->None}]

result=net[result];

The output of the net at each frame is the probability distribution over all characters. An additional “blank” character is included for the CTC training/decoding.

labels=deepSpeech[["Output","Labels"]];ArrayPlot[Transpose@result,DataReversed->True,FrameTicks->{Thread[{Range@Length[labels],labels}],None}]

Decoder

The decoder uses a CTC beam search:

decoder=deepSpeech[["Output"]]

Simplifying a lot, at each step the decoder gets as an input the acoustic model result and a collection of n possible sequences from the previous step. It then computes the likelihoods of all the possible one-step continuations of those sequences given the acoustic model result. Then only the top n sequences are kept and passed to the next decoding step. This avoids the combinatorial hell of computing the likelihoods of all possible sequences given the acoustic model result.

Table[MapAt[StringJoin,decoder[result[[;;i]],{"TopNegativeLogLikelihoods",4}],{All,1}],{i,10,Length@result,5}]//Grid

SpeechRecognize

Scoring results with a language model

As mentioned before, the acoustic model result is an array of probabilities with dimensions {timeSteps, classes}, where each inner list is a probability distribution over the set Alphabet ⋃ { blank } at a given time step:

result//Dimensions

There are 29 classes; a special blank character plus our alphabet:

alphabet = Characters@"' abcdefghijklmnopqrstuvwxyz";

Language model

The language model is based on a character level language model that was trained on a dataset of transcripts from TED talks:

net = NetModel["SpeechRecognizeCharactersLevelLanguageModelTrainedOnTeds_v1"]

Some surgery is required to exploit batch evaluation in the CTC computations:

net = NetGraph[NetExtract[net,{{1, 1}, {1, 2}, {1, 3}, {2}, {3}, {4}}],{1 -> 2 -> 3 -> 4 -> 5 -> 6,NetPort["State2In"] -> NetPort[2, "State"],NetPort["State3In"] -> NetPort[3, "State"],NetPort[2, "State"] -> NetPort["State2Out"],NetPort[3, "State"] -> NetPort["State3Out"]}]

The following functions massages the output of the net to the format the the CTC beam search needs:

getNetResult[states_, suffixes_]:=
Transpose[net[<|"Input" -> suffixes,Transpose[states /. None -> <|"State2In" -> ConstantArray[0., 256],"State3In"-> ConstantArray[0., 256]|>,AllowedHeads -> All]|>,{"Output", "State2Out", "State3Out"}],AllowedHeads -> All]

This net actually operates on a larger alphabet than the acoustic model dealt with, so it will be necessary to post-process the probability distributions from the net to output a list of length 28 (the number of classes from the acoustic model, minus the blank character:

restrictedVocabulary[probabilities_] := Join[probabilities[[1 ;; 2]],probabilities[[3 ;; 28]] + probabilities[[29 ;; 54]]]/(1 - Total[probabilities[[55;;]]])

We also need to convert the raw probabilities to log likelihoods for the decoder to use. This is how our final language model function will end up looking:

languageModelFun[states_, suffixes_]:=
{Log@restrictedVocabulary[#Output],KeyMap[<|"State2Out" -> "State2In","State3Out" -> "State3In"|>,KeyDrop[#, "Output"]]}& /@ getNetResult[states, suffixes]

At each iteration, we prune beams from the search based on a score computed from combining the log likelihoods from the acoustic and language model input for each sequence. The mixture between the two likelihoods is controlled by the “MixtureLevel” parameter.

languageModel = <|"LanguageModelFun" -> languageModelFun,"EmptyString" -> 62,"MixtureLevel" -> 0.4|>;

Running the decoder

The decoder expects input from the acoustic model needs to be a NumericArray, and does its internal calculations in single precision:

res = NumericArrayUtilities`CTCBeamSearchDecode2[NumericArray[result,"Real32"],False,-1,languageModel,"Alphabet"->alphabet,"BeamSize"->100,"PostProcessOptions"-> <|"UpTo" -> 4|>];
res//Column

Extract the decoded string:

StringJoin[res[[1]]["Text"]]

Speaker recognition

We can move on to a very different problem: while speech recognition seeks to understand what words are spoken while ignoring completely who is speaking, speaker recognition does the complete opposite: the goal is to ignore the specific words spoken and figuring out the acoustic characteristics of different speakers.

Data

The network was trained on the VoxCeleb2 dataset. The structure of the data is pretty simple: every audio file is associated with a speaker id.

voxCelebDataset=Rest@Import["/Users/carlogico/Downloads/vox2_meta.csv"];
RandomChoice[voxCelebDataset]

There are a total of 5994 speaker in the development dataset for a total than more of 2000 hours of speech.

labels=Import["/Users/carlogico/Desktop/VoxCeleb/SingleFrameNet.wlnet"][["Output","Labels"]];
RandomSample[labels,10]

The task is well defined: it’s a simple classification with mutually exclusive classes.

Speaker recognition

Net

As in the case of AudioIdentify, we have an architecture that was conceived for image processing being mapped onto fixed length chunks of the input. Then the result is averaged in the time dimension to produce a single output:

net=Import["/Users/carlogico/Desktop/VoxCeleb/MultiFrameNet.wlnet"]

a=ExampleData[{"Audio","MaleVoice"}]
net[a,"TopProbabilities"]

Encoder

The encoder uses a mel-spectrogram, partitioned into fixed length chunks

encoder=net[["Input"]]

result=encoder[a];
MatrixPlot@*formatForMP/@result

Main network

The main network is the mapped onto each partition.

singleFrameNet=net[[1,"Net"]]

The result is a list of fixed length vectors:

result=singleFrameNet/@result;
ListLinePlot[result]

The result is then averaged in the time dimension:

result=Mean[result];
ListLinePlot[result]

and fed to the decoder:

decoder=net[["Output"]]

decoder[result]

Speaker recognition

While classifying a voice between a list of 6000 “celebrities” is amusing, it does not really serve any practical purpose. What can be interesting instead is the use of the same net as a feature extractor to compute features which are semantically significant to differentiate voices.

Spoken digits commands dataset

We have already seen this data: it’s a collection of spoken digits with information about what digit is spoken and a unique id of the speaker:

data=ResourceData[ResourceObject["Spoken Digit Commands"],"TrainingData"];
RandomChoice[data]

I’ll ignore the digit information and focus on the speaker label.

I will use the same network structure as the speaker classification, but since the dataset contains only signals of less than one seconds, I’ll use a version that was trained on a single fixed length audio chunk at a time:

singleFrameNet=Import["/Users/carlogico/Desktop/VoxCeleb/SingleFrameNet.wlnet"]

singleFrameNet[RandomChoice[data]["Input"],"TopProbabilities"]

By dropping the last few layer that are in charge of classifying the various celebrities, we can produce a network that returns a semantically meaningful vector of length 1280:

featureExtractorSingle=NetChain[{singleFrameNet[[1]],ReshapeLayer[{Automatic}]},"Input"->singleFrameNet[["Input"]]]

This can be applied to any (short) audio object:

featureExtractorSingle[RandomChoice[data]["Input"]]//ListLinePlot

We can select 20 speakers from our dataset:

speakers=Values[(Reverse@SortBy[GroupBy[data,#SpeakerID&],Length])[[;;20]]]//Flatten;
idsTest=DeleteDuplicates[speakers[[All,"SpeakerID"]]];
colors=Association@Table[idsTest[[i]]->ColorData[97][i],{i,Length[idsTest]}];

And use FeatureSpacePlot with our network as feature extractor to visualize the embeddings in a 2-D space:

FeatureSpacePlot[Style[#Input,colors[#SpeakerID]]&/@speakers,LabelingFunction->None,FeatureExtractor->featureExtractorSingle]

Speaker recognition

VoxCeleb test dataset

In this section I’ll use a random subset of the VoxCeleb test dataset (the network was never exposed to the speakers in this subset during training).

names=FileNames["*.flac","/Users/carlogico/Desktop/VoxCeleb/test",3];
testSet=File[#]->FileNameSplit[#][[-3]]&/@names;
gender=#[[1]]->#[[3]]&/@voxCelebDataset//Association;voxCeleb=Map[<|"Input"->#[[1]],"Gender"->gender[#[[2]]],"SpeakerID"->#[[2]]|>&,testSet];
RandomChoice[voxCeleb]

The net that we will use is the same one that was used for the speaker classification. The main network that is mapped on a variable length list of chunks of the audio signal:

net[RandomChoice[voxCeleb]["Input"],"TopProbabilities"]

By dropping the last few layer that are in charge of classifying the various celebrities, we can produce a network that returns a semantically meaningful vector of length 1280, independently of the length of the input signal:

featureExtractorMulti=NetChain[{
NetMapOperator[{
NetDelete[net[[1,"Net"]],{2,3}],
ReshapeLayer[{Automatic}]
}],
AggregationLayer[Mean,1]
},"Input"->net[["Input"]]]

Again, this can be applied to any audio object:

featureExtractorMulti[RandomChoice[voxCeleb]["Input"]]//ListLinePlot

We can select 20 speakers from our dataset:

SeedRandom[42];speakers=Values[GroupBy[RandomSample[voxCeleb,1500],#SpeakerID&][[;;20]]]//Flatten;idsTest=DeleteDuplicates[speakers[[All,"SpeakerID"]]];
colors=Association@Table[idsTest[[i]]->ColorData[97][i],{i,Length[idsTest]}];

And use FeatureSpacePlot with our net as feature extractor to visualize the embeddings in a 2-D space:

FeatureSpacePlot[Style[#Input,colors[#SpeakerID]]&/@speakers,LabelingFunction->None,FeatureExtractor->featureExtractorMulti]

Speaker recognition

Gender classification

Another application that leverages the speaker classification network is gender classification.

First of all, we need to make sure that the speakers in our training and testing data do not overlap:

{trainGenderData,testGenderData}=Flatten@*Values/@TakeDrop[GroupBy[voxCeleb,#SpeakerID&],50];

From here, we can use Classify even on an extremely limited subset of the data to produce a powerful classifier:

SeedRandom[42];
trainingDataSubset=#Input->#Gender&/@RandomSample[trainGenderData,400];

clGender=Classify[trainingDataSubset,FeatureExtractor->featureExtractorMulti,PerformanceGoal->"DirectTraining"]

We can now evaluate the result:

cmGender=ClassifierMeasurements[clGender,#Input->#Gender&/@RandomSample[testGenderData,400]];
cmGender["Accuracy"]
cmGender["ConfusionMatrixPlot"]

The ClassifierFunction is now easy to use on any audio input:

clGender[ExampleData[{"Audio","MaleVoice"}]]

clGender[ExampleData[{"Audio","Scream"}]]

Team

Grid[Reverse@Partition[{({{}, {"Shadi Ashnai"}, {"shadi"}}),({{}, {"Piotr Wendykier"}, {"piotrw"}}),({{}, {"Carlo Giacometti"}, {"carlog"}}),({{}, {"Mariusz Jankowski"}, {"mariuszj"}}),({{}, {"Chien-Yu Chen"}, {"chienyuc"}}),({{}, {"Rebecca Frederick"}, {"rebeccaf"}}),Column@{,"Timothee Verdier","timotheev"}},UpTo@4]]

Init

Formatting encoder output

formatForMP[x_]:=Log[Reverse[Transpose[x]]+.0001]

Details about the CTC decoding with a language model

Acoustic Model Probabilities

Audio input

audio=SpeechSynthesize["sally sells sea shells by the sea shore"];

Encoded input

For this net we will use the power spectrum:

specgram = NetEncoder[{"AudioSpectrogram","WindowSize"->320}][audio];
Dimensions[specgram]

Net model

net = NetReplacePart[NetModel["Deep Speech 2 Trained on Baidu English Data"],
		{"Output" -> None, "Input" -> None}];

Result is an array of acoustic model probabilities with dimensions { timeSteps, classes }, where each inner list is a probability distribution over the set Alphabet ⋃ { blank } at a given time step:

acousticModelResult = net[specgram];
Dimensions[acousticModelResult]

There are 29 classes; a special blank character plus our alphabet:

alphabet = Characters@"' abcdefghijklmnopqrstuvwxyz";

The acoustic model probabilities here are actual probabilities rather than log likelihoods, so

logitQ = False;

The net outputs a probability distribution over Alphabet ⋃ { blank }, with the probability of blank returned as the last element in the list. Negative indexing is supported, so

blankIndex = -1;

Language Model Probabilities

Interface

We want to use the language model interface like this:

languageModelFunction[
	{(Subscript[state, 1]), (Subscript[state, 2]), ...},
	{{(Subscript[suffix, 1])}, {(Subscript[suffix, 2])}, ...}
]
-->
{ {(Subscript[logProbDist, 1]), (Subscript[newState, 1])}, {(Subscript[logProbDist, 2]), (Subscript[newState, 2])}, ... }

where it will accept the suffix and internal net state for each input sequence and output the resulting probability distribution over the characters and new state.

[ Saving internal net states is an optimization to reduce the time spent querying the language model during decoding, since it avoids  redoing the net evaluation “from scratch” for each sequence. We can do this since the internal state of this net is the same for any identical sequence of characters. ]

Net model

net = NetModel["SpeechRecognizeCharactersLevelLanguageModelTrainedOnTeds_v1"];
net = NetGraph[
		NetExtract[net,{{1, 1}, {1, 2}, {1, 3}, {2}, {3}, {4}}],
		{
			1 -> 2 -> 3 -> 4 -> 5 -> 6,
			NetPort["State2In"] -> NetPort[2, "State"],
			NetPort["State3In"] -> NetPort[3, "State"],
			NetPort[2, "State"] -> NetPort["State2Out"],
			NetPort[3, "State"] -> NetPort["State3Out"]
		}
		];

getNetResult[states_, suffixes_]:=
Transpose[
	net[
		<|
		"Input" -> suffixes,
		Transpose[states /. 
					None -> 
					<|"State2In" -> ConstantArray[0., 256],
					"State3In"-> ConstantArray[0., 256]|>,
			AllowedHeads -> All]
		|>
		,
		{"Output", "State2Out", "State3Out"}
	],
AllowedHeads -> All]

Output

This net will output an association with Output and States, and the decoder expects the language model to return the list  { languageModelProbabilities, newStates }, so we will need to massage the output a bit to get it into that form:

processOutput[netResult_]:=
{
	#Output,
	KeyMap[
			<|
			"State2Out" -> "State2In",
			"State3Out" -> "State3In"
			|>,
			KeyDrop[#, "Output"]
	]
}& /@ netResult

This net actually operates on a larger alphabet than the acoustic model dealt with, so it will be necessary to post-process the probability distributions from the net to output a list of length 28 (the number of classes from the acoustic model, minus the blank character:

restrictedVocabulary[probabilities_] := 
Join[
	probabilities[[1 ;; 2]],
	probabilities[[3 ;; 28]] + probabilities[[29 ;; 54]]
]/(1 - Total[probabilities[[55;;]]])

We also need to convert the raw probabilities to log likelihoods for the decoder to use.

This is how our final language model function will end up looking:

languageModelFun[states_, suffixes_]:=
{
	Log@restrictedVocabulary[#Output],
	KeyMap[
			<|
			"State2Out" -> "State2In",
			"State3Out" -> "State3In"
			|>,
			KeyDrop[#, "Output"]
	]
}& /@ getNetResult[states, suffixes]

Other characteristics of the language model

The language model expects all sequences to begin with a special “empty string” character, which we need to define:

epsilon = 62;

Decode

Scoring beams

At each iteration, we prune beams from the search based on a score computed from combining the log likelihoods from the acoustic and language model input for each sequence. The mixture is computed (in log-space) in the form: acousticModelLogLikelihood + α * languageModelLogLikelihood. The parameter α needs to be specified to the decoder.

α = 0.4;

Expressing the language model

The necessary components of the language model are:
• a function which computes { suffixes, states} --> { logProbDists, newStates }
• an empty string character
• a mixture parameter

languageModel = <|
	"LanguageModelFun" -> languageModelFun,
	"EmptyString" -> epsilon,
	"MixtureLevel" -> α
|>;

Additional configuration

The number of beams to keep at each time step:

beamSize = 100;

The number of top beams to return at the end of the search; 1 ≤ topN ≤ beamSize.

topN = 1;

Running the decoder

The decoder expects input from the acoustic model needs to be a NumericArray, and does its internal calculations in single precision:

acousticModelResult = NumericArray[acousticModelResult,"Real32"];

res=NumericArrayUtilities`CTCBeamSearchDecode2[
	acousticModelResult,
	logitQ,
	blankIndex,
	languageModel
	,
	"Alphabet"->alphabet,
	"BeamSize"->beamSize,
	"PostProcessOptions"-> <| "UpTo" -> topN |>
]

Extract the decoded string:

StringJoin[res[[1]]["Text"]]