Understanding Speech
Speech & Speaker Analysis

Rebecca Fredrick and Carlo Giacometti
Wolfram Research

Join the Conversation #WolframTechConf

Speaker Analysis

Motivation

The idea behind this project is pretty simple: to get something similar to AudioIdentify, but with speakers as the target.

The result will be two-fold: a net that performs some kind of classification

NetModel["Wolfram AudioIdentify V1 Trained on AudioSet Data"][]

and a derived net to perform feature extraction:

FeatureSpacePlot[ExampleData[#]->#[[2]]&/@ExampleData["Audio"],FeatureExtractor->"AudioFeatureVector"]

Speaker Analysis

Dataset

The dataset used is VoxCeleb 2. It is composed by about about 1 million samples from roughly 6000 celebrities, sourced from YouTube videos.

files=FileNames["*txt","/Files/Data/VoxCeleb/txt/",3];
Tally[FileNameSplit[#][[-3]]&/@files][[All,2]]//Histogram

Each recording is labeled with the main speaker. Of course the tagging is not perfect, so the data will have label noise.

Speaker Analysis

Architecture

The input data is encoded into a spectrogram:

encoder=NetModel["SpeakerClassificationAlex"][["Input"]]

a=ExampleData[{"Audio","MaleVoice"}]

encoder[a]//Log//MatrixPlot

Then the spectrogram is fed into a net inspired by VGG. (Disclaimer: this is still a work in progress result, the plan is to train similar but more modern nets adapted for the idiosyncrasies of the audio data).

net=NetModel["SpeakerClassificationAlex"](* Temporary name! *)

net[a]

Speaker Analysis

Architecture

One important difference though is that this net has been adapted to deal with Audio data, which is inherently of variable length. Normally 2-D convolution layers operate on the last two dimensions. However, the spectrogram computed by the encoder does not have a channel dimension:

NetChain[
{ConvolutionLayer[96,{1,1}]},
"Input"->encoder
]

This is why we need to use a ReshapeLayer to provide that. We end up having a tensor with shape {time, frequency, channels} .That is not enough though, since the first dimension is dynamic the convolution will fail:

NetChain[
{
ReshapeLayer[{Inherited,Automatic,1}],
ConvolutionLayer[96,{1,1}]
},
"Input"->encoder
]//NetInitialize

The solution is to use the “Interleaving” option to make sure that the convolution will succeed, and will be applied correctly to the time and frequency dimensions.

NetChain[
{
ReshapeLayer[{Inherited,Automatic,1}],
ConvolutionLayer[96,{1,1},"Interleaving"->True]
},
"Input"->encoder
]

By using a simple chain of convolutions, batch normalizations, nonlinearities and poolings we end up with a network that generates a variable length sequence of vectors:

NetTake[net,20]

To produce a single classification, we aggregate the sequence over the time dimension:

NetDrop[net,20]

Speaker Analysis

Training for classification

The network was at first trained for the classification task.

The results were computed on a subset of the data that was withheld from the training.

classResults=Import["/Users/carlogico/Nextcloud/carlog/WorkUnsorted/VoxCeleb/CNNAlex2-19.07.24--21.37.13/ClassifierMeasurements.wxf"];
<|"Accuracy" ->classResults["Identity"][\[LeftAssociation]"Measurement"->"Accuracy","Key"->"Accuracy"\[RightAssociation]],"Top3Accuracy"->classResults["Identity"][\[LeftAssociation]"Measurement"->"Accuracy"->3,"Key"->"Top3Accuracy"\[RightAssociation]],"AUC"->(Around[Mean[#],StandardDeviation[#]]&[classResults["Identity"][\[LeftAssociation]"Measurement"->"AreaUnderROCCurve","Key"->"AreaUnderROCCurve"\[RightAssociation]]]),"F1"->(Select[classResults["Identity"][\[LeftAssociation]"Measurement"->"F1Score","Key"->"F1Score"\[RightAssociation]],#=!=Indeterminate&]//Around[Mean[#],StandardDeviation[#]]&)|>//Dataset

The byproduct of the classification training is that the net develops a internal representation that is semantically significant in the space of voices. If we use the net as feature extractor for some data that the network has never seen, we can see that this representation captures factors that differentiate speakers from each other.

With[
{featureNet=NetAppend[NetTake[net,19],{NetMapOperator[NetTake[net[[20,"Net"]],3]],AggregationLayer[Mean,1]}]},VisualizeDigits[featureNet,5,100]
]

Speaker Analysis

Optimizing the embedding

After the classification training was finished, the net was chopped so that it behaved as a feature extractor, and it was further trained to optimize the resulting embedding. There are several different strategies and losses to achieve that result. Some examples are:

Contrastive loss (Dimensionality Reduction by Learning an Invariant Mapping)

Triplet loss (FaceNet: A Unified Embedding for Face Recognition and Clustering)

CosFace (CosFace: Large Margin Cosine Loss for Deep Face Recognition)

ArcFace (ArcFace: Additive Angular Margin Loss for Deep Face Recognition)

...

We will look at a simple example using the contrastive loss.

Train for classification

Let’s start with a minimal network:

classNet=NetChain[
{
GatedRecurrentLayer[64,"Dropout"->{"VariationalInput"->.2}],
GatedRecurrentLayer[64,"Dropout"->{"VariationalInput"->.2}],
SequenceLastLayer[],
LinearLayer[2],
Ramp,
LinearLayer[],
SoftmaxLayer[]
},
"Input"->NetEncoder[{"AudioMelSpectrogram","WindowSize"->(Quantity[40, "ms", milliseconds, "Milliseconds"]),"Offset"->(Quantity[20, "ms", milliseconds, "Milliseconds"])}],
"Output"->NetDecoder[{"Class",classes}]
]

That we’ll train on a tiny dataset containing signals from 5 different speakers.

#Input->#SpeakerID&/@data[[;;2]]

classificationNet=NetTrain[classNet,#Input->#SpeakerID&/@data]

By discarding the last classification layers we gain a net that produces a 2-dimensional embedding of the input signal:

embeddingNet=NetTake[classificationNet,4]

That we can easily visualize!

ListPlot[embeddingNet[Flatten[#]]&/@groupedData[[All,All,"Input"]]]

Contrastive Loss training

Using the network that we just trained, we build another one that applies it to two inputs at the same time, computes the squared euclidean distance between the results and feeds that to the loss function. One important thing to note is that the two copies of the network share their parameters, so at each step of the training they will be updated in the same way

pairEmbeddingNet=
With[
{netWithSharedArrays = NetInsertSharedArrays[embeddingNet]},
NetGraph[
<|"net1" -> netWithSharedArrays,
"net2" -> netWithSharedArrays,
"distance" -> MeanSquaredLossLayer[],
"loss"->ContrastiveLossLayer[]
|>,
{NetPort["Input1"] -> "net1" -> NetPort["distance", "Input"],NetPort["Input2"] -> "net2" -> NetPort["distance", "Target"],"distance" -> NetPort["loss","Input"]}
]
]

The loss function will minimize the distance if the label is False (the two inputs belong to the same class), and maximize it if the label is True (the two inputs belong to different classes).

Plot[{ContrastiveLossLayer[][<|"Input"->x,"Target"->True|>],ContrastiveLossLayer[][<|"Input"->x,"Target"->False|>]},{x,0,1},PlotLegends->{True,False}]

The data is now divided in pairs whose label reflect if they belong to the same speaker or not.

RandomSample[embeddingData,2]

trainedEmbedding=NetTrain[pairEmbeddingNet,embeddingData]

Once the training is done, we can extract the net that computes the embedding

embeddingContrastive=NetTake[trainedEmbedding,"net1"]

And again, use it to visualize the training data!

ListPlot[embeddingContrastive/@groupedData[[All,All,"Input"]]]

Speaker Analysis

Embedding results

The speaker classification network was fine-tuned using a distance based loss.

NetModel["AlexFineTunedSpeakerClassification"]

We can again use this to visualize the spoken digits data:

VisualizeDigits[NetModel["AlexFineTunedSpeakerClassification"],5,100]

Since the net fully supports variable length input, we can do some minor surgery on it to produce a time resolved result:

featuresTimeResolved=NetAppend[NetTake[NetModel["AlexFineTunedSpeakerClassification"],19],{NetMapOperator[NetJoin[NetModel["AlexFineTunedSpeakerClassification"][[20]][["Net"]],{NetModel["AlexFineTunedSpeakerClassification"][[22]]}]] }]

This allow us to figure out what pieces of a single recording belong to what speaker, giving us a foothold into the problem of speaker diarization.

debate=;

We can compute the time resolved features on a recording:

debateFeatures=computeTimeResolvedFeatures[debate];

Find clusters that we assume belong to the same speaker:

clusters=FindClusters[debateFeatures,4];

And visualize them using FeatureSpacePlot!

FeatureSpacePlot[Table[With[{col=ColorData[97][i]},Style[#,col]&/@clusters[[i]]],{i,Length@clusters}]//Flatten,LabelingFunction->None]

Speaker Analysis

New functionality depending on net

Classify

We will add a named classifier based on the net:

cl=Classify["NotableSpeaker"]

cl[AudioCapture[]]

ML functions

Two new feature extractors will be available in all machine learning functions:

”SpeakerFeatures": a time resolved sequence of features

"SpeakerFeatureVector": a single vector of semantically significant features

FeatureSpacePlot[featData,FeatureExtractor->"SpeakerFeatures",LabelingFunction->Callout]

These features will be available in all of the functions that support the FeatureExtractor option.

SpeakerMatchQ

The main idea is very simple: after discarding all of the non voiced segments, we compute the speaker features for two signals and measure the distance between two of them.

Using the testing subset of the VoxCeleb 2 dataset, we can get an estimate of the distribution of the embedding distances between same and different people:

This allows us to make a probabilistic decision if two recordings were produced by the same speaker or not:

SpeakerMatchQ[AudioCapture[],AudioCapture[]]

SpeakerMatchQ[AudioCapture[],ExampleData[{"Audio","MaleVoice"}]]

SpeakerRecognize

The SpeakerRecognize function will, at a first look, seem very similar to Classify:

SpeakerRecognize[{(Subscript[example, 1])->(Subscript[name, 1]),(Subscript[example, 1])->(Subscript[name, 2]),…}]

However, the operations performed are significantly different. The pipeline will be:

voice activity detection and filtering

feature extraction

modeling of the distribution of the features corresponding to each class

Row[{waves,Style["⟹",40] ,f,Style["⟹",40],points,Style["⟹",40],cont}]

Once the modeling in complete, once a new input will be passed to the classifier function, SpeakerRecognize will compute the probability of the new features belonging to each of the classes seen at training time.

Given the access to time resolved features as well, this will allow us also to tackle the problem of speaker diarization, where instead of outputting a single recognition per recording we can return multiple recognized intervals.

Thanks!

Init

ro=ResourceObject["Spoken Digit Commands"];

{trainingSet,testingSet}={ResourceData[ro,"TrainingData"],ResourceData[ro,"TestData"]};

groupedTrain = GroupBy[trainingSet, #SpeakerID &];
groupedTrain = Select[groupedTrain, Length[#] ≥ 2 &];
groupedTrain = SortBy[groupedTrain, Minus@*Length];
groupedTest = SortBy[GroupBy[testingSet, #SpeakerID&], Minus@*Length];
groupedData = groupedTrain[[;;5, ;;UpTo[200]]];

colorTr = AssociationThread[
	Keys[groupedTrain]
		->
	(ColorData[97]/@Range[Length[groupedTrain]])
];
VisualizeDigits[net_, numSpeakers_, numDataPoints_]:=
FeatureSpacePlot[
	Flatten @ Values @ Map[Style[#Input, colorTr[#SpeakerID]] -> #SpeakerID &,
	groupedTrain[[;;numSpeakers, ;;UpTo[numDataPoints]]], {2}],
	FeatureExtractor -> net,
	LabelingFunction -> Callout
]

featData = Flatten@Values@Map[
	Style[#Input, colorTr[#SpeakerID]] -> #SpeakerID &,
	groupedTrain[[;; 5, ;; UpTo[50]]],
	{2}
];

embeddingData = 
Flatten@Table[
	With[
	{speaker = RandomChoice[Keys[groupedData]]},
	{
		<|
			"Input1" -> RandomChoice[groupedData[speaker]]["Input"],
			"Input2" -> RandomChoice[RandomChoice[KeyDrop[groupedData,speaker]]]["Input"],
			"Target" -> True
		|>,
		<|
			"Input1" -> #[[1,"Input"]],
			"Input2" -> #[[2,"Input"]],
			"Target" -> False
		|> & @ RandomSample[groupedData[speaker], 2]}
	],
	5000
];

data = Flatten@Values@groupedData;
classes = Union[data[[All, "SpeakerID"]]];

$enc = NetModel["AlexFineTunedSpeakerClassificationTimeResolved"][["Input"]];
$network = NetReplacePart[NetModel["AlexFineTunedSpeakerClassification"], "Input" -> None];

computeFeatures[in_] :=
Module[
	{encoded, voiceActivity},
	encoded = $enc[in];
	Needs["WebRTCLink`"];
	WebRTCLink`VADSetMode[3]; (* lowest false-positive rate *)
	voiceActivity = oVoiceActivity[in, 10, 10];
	encoded = Pick[encoded, PadRight[voiceActivity, Length[encoded], 0], 1];
	If[Length[encoded < 39],
		encoded = PadRight[encoded, {39, Dimensions[encoded][[2]]}, Last@encoded];
	];
	$network[encoded]
]


oVoiceActivity[audio_, wSize_, offset_] :=
Block[{},
	Needs["WebRTCLink`"];
	WebRTCLink`VADSetMode[3];
	Normal @ WebRTCLink`VADProcess[
		With[
			{
				a = If[Audio`Utilities`AudioSampleRate[audio] =!= 32000,
					AudioResample[audio, 32000]
					,
					audio
				]
			},
			If[AudioChannels[a] =!= 1,
				Mean @ AudioChannelSeparate @ a,
				a
			]
		],
		wSize,
		offset,
		"ChannelQ" -> False
	]
];

computeTimeResolvedFeatures[a_] :=
Module[
	{voiceActivity, encoded},
	encoded = $enc[a];
	voiceActivity = PadRight[oVoiceActivity[a, 10, 10], Length[encoded], 0];
	featuresTimeResolved[Pick[encoded, voiceActivity, 1]]
]