Audio Analysis

Carlo Giacometti, Wolfram Research

Join the Conversation #WolframTechConf

Main Applications

Information retrival

Speech analisis

Machine learning on audio

Many more!

Some basic concepts and tools

{a,b,c}={ExampleData[{"Audio","Piano"}],ExampleData[{"Audio","ThaiBells"}],AudioTrim[ExampleData[{"Audio","Drums"}],3]}

AudioPlot[#,ImageSize->Medium]&/@{a,b,c}

Periodogram[#,4000,ImageSize->Medium,PlotRange->{{0,10000},All}]&/@{a,b,c}

Spectrogram[#,2048,2048,ImageSize->Medium,PlotRange->{All,{0,10000}}]&/@{a,b,c}

Global features

AudioMeasurements computes global properties of an Audio object.

audio=ExampleData[{"Audio","Drums"}]

AudioMeasurements[audio,"Properties"]//Multicolumn[#,4]&

Time domain properties

AudioMeasurements[audio,"TimeDomainProperties"]//Keys//Multicolumn

AudioMeasurements[audio,{"MinMax","Mean","StandardDeviation","RMSAmplitude","ZeroCrossings"},"Dataset"]

Simple statistical properties like the ones above can also be computed by applying the corresponding function directly to the audio object.

names={"MinMax","Mean","StandardDeviation","RMSAmplitude","ZeroCrossings"};
props={MinMax,Mean,StandardDeviation,Sqrt@Mean[#^2]&,Total@CrossingDetect[#,CornerNeighbors->None]&};

AssociationThread[names ->Through[props@audio]]//Dataset

There is an important difference though: AudioMeasurements operates on the downmixed version of the signal, while the corresponding function is computed on the flattened version:

data={{0.32,-0.61,-0.56,0.65,-0.43},{-0.05,0.,0.87,0.93,-0.35}};

MinMax[Audio[data]]
MinMax[Flatten[data]]

AudioMeasurements[Audio[data],"MinMax"]
MinMax[Mean[data]]

Statistical functions

In addition to the the properties in AudioMeasurements, there is a set of statistical functions that can be directly applied to an Audio object:

Moment[audio,2]

Quantile[audio,{.1,.9}]

Entropy[audio]//N

Frequency domain properties

AudioMeasurements[audio,"FrequencyDomainProperties"]//Keys//Multicolumn

Some properties have a simple interpretation in relation to the power spectrum:

AudioMeasurements[audio,{"SpectralCentroid","SpectralFlatness","SpectralRollOff","SpectralSpread"},"Dataset"]

One simple application is clustering a list of Audio objects based on their spectral centroid and spread:

list={,,,,,,,,};

props=AudioMeasurements[list,{"SpectralCentroid","SpectralSpread"},"List"];

clusters=FindClusters[props->props]

ListPlot[Partition[props,1],PlotMarkers->list,ImageSize->300,Prolog->({Opacity[.3,],Disk[Mean@#,400]&/@clusters}),PlotRange->{{00,4000},{0,4000}},AspectRatio->1]

Local features

A lot of features can be computed locally by mapping the measurement on a partitioned version of the signal. 
This is what AudioLocalMeasurements does.

Row[{Audio[audio,Appearance->"Basic"],Spacer[10],Style["⟶",Bold,15],Spacer[10],Text[Style["AudioLocalMeasurements",Bold,15],FormatType->StandardForm],Spacer[10],Style["⟶",Bold,15],Spacer[10],Style[TimeSeries[],Bold,15]}]//Panel

The output is a TimeSeries, and timestamps are placed at the center of each partition.

AudioLocalMeasurements[audio,"RMSAmplitude"]

Show[
AudioPlot[audio,Appearance->"ContinuousAbs",PlotLayout->"Averaged",AspectRatio->1/4],
ListLinePlot[%,PlotRange->All,PlotStyle->Red]]

Defining the partitioning scheme

partitionSize=.5;
offset=.01;
window=DirichletWindow;

minmax=AudioLocalMeasurements[audio,"MaxAbs",PartitionGranularity->{partitionSize,offset,window}]

(Row)

Show[AudioPlot[audio,PlotLayout->"Averaged",Appearance->"ContinuousAbs",AspectRatio->1/4],ListLinePlot[AudioLocalMeasurements[audio,"MaxAbs",PartitionGranularity->{partitionSize,offset,window}],PlotStyle->Red]]

Multiple properties

It’s possible to compute multiple properties at the same time:

AudioLocalMeasurements[audio,{"Power","RMSAmplitude"}]

ListLinePlot[%,PlotRange->All]

Local features

There are a lot of properties available for computation:

AudioLocalMeasurements[audio,"Properties"]//Multicolumn[#,3]&

Time Domain Properties

audio=ExampleData[{"Audio","FemaleVoice"}]//AudioNormalize

AudioLocalMeasurements[audio,"TimeDomain"]//Multicolumn

props={"Min","Max","Mean","StandardDeviation"};Show[AudioPlot[audio,PlotLayout->"Averaged",PlotRange->All,AspectRatio->1/2,PlotLabel->#1,ImageSize->Medium],ListLinePlot[AudioLocalMeasurements[audio,#],PlotRange->All,PlotStyle->{Red,Opacity[.7]}]]&/@props//Multicolumn

Most are analogous to the ones found in AudioMeasurements, but we have some new ones:

AudioLocalMeasurements[audio,"LPC",PartitionGranularity->{.02,.01}]

ArrayPlot[%["Values"]//Transpose,AspectRatio->1/3]

Frequency Domain

AudioLocalMeasurements[audio,"FrequencyDomain"]//Multicolumn

Many of the properties are in common with AudioMeasurements, but there some new ones.

AudioLocalMeasurements[audio,"MFCC"]

ArrayPlot[%%["Values"][[All,2;;]]//Transpose,AspectRatio->1/3]

Fundamental frequency detection can be useful to extract the pitch of a sound:

AudioLocalMeasurements[ExampleData[{"Audio","PianoScale"}],"FundamentalFrequency",PartitionGranularity->{.05,.01,HannWindow}]//ListLinePlot

Properties computed on neighbouring partitions

All of the following properties are based on the comparison of the spectrum of neighbouring frames

props={"HighFrequencyContent","Novelty","ModifiedKullbackLeibler","PhaseDeviation","ComplexDomainDistance"};

Show[AudioPlot[audio,PlotLayout->"Averaged",Appearance->"ContinuousAbs",AspectRatio->1/2,PlotLabel->#1,ImageSize->Medium],ListLinePlot[Rescale@AudioLocalMeasurements[audio,#,PartitionGranularity->{.04,.01}],PlotRange->All,PlotStyle->{Red,Opacity[.7]}]]&/@props//Multicolumn[#,2]&

Interval detection

With AudioIntervals it is possible to find sections of an audio object that satisfy a user defined criterion.

audio=ExampleData[{"Audio","MaleVoice"}]

crit=#RMSAmplitude<.03&;

int=AudioIntervals[audio,crit]

AudioPlot[audio,Epilog->{,Rectangle[{#[[1]],-1},{#[[2]],1}]&/@int}]

AudioDelete[audio,int]

AudioPlot[%]

A criterion could me based on multiple local properties:

a=ExampleData[{"Audio","FluteScale"},"Audio"]

int=AudioIntervals[a,#RMSAmplitude>.1&&(#FundamentalFrequency<270||#FundamentalFrequency>300)&,PartitionGranularity->{.04,.01}]

AudioPlot[a,Epilog->{,Rectangle[{#[[1]],-1},{#[[2]],1}]&/@int}]

Spectrogram[a,Epilog->{,Rectangle[{#[[1]],0},{#[[2]],20000}]&/@int},PlotRange->{All,{0,1000}}]

AudioBlockMap

AudioBlockMap is a broader version of AudioLocalMeasurements, where the user is free to determine whatever the measurement to be computed on each partition:

audio=Audio["ExampleData/rule30.wav"];

AudioBlockMap[Max,audio,{.1,.01}]//ListLinePlot

It’s possible to define more complicated functions

dBRMS=AudioBlockMap[20Log10[Sqrt@Mean[#^2]]&,audio,{.05,.01}]

ListLinePlot[dBRMS]

The output doesn’t need to be a number:

m=AudioBlockMap[{Hue[Max@#],Opacity[.5],Disk[{0,0},Max@#]}&,audio,{.1,.01,HannWindow}]

Graphics@MapThread[{Sequence@@#1[[;;2]],Translate[#1[[3]],{#2,0}]}&,{m["Values"],m["Times"]}]

Example Applications

LPC

a=ExampleData[#]&/@ExampleData["Audio"];

lpc=AudioLocalMeasurements[#+$MachineEpsilon,"LPC"]["Values"]&/@a;

MapThread[MatrixPlot[DistanceMatrix[#1], PlotLabel -> #2,FrameTicks->None]&,{lpc,ExampleData["Audio"]}]

MFCC

alice={,,,};

AudioPlot[alice,ImageSize->Medium]

mfcc=AudioLocalMeasurements[#,"MFCC",PartitionGranularity->{.05,.01}]["Values"][[All,2;;]]&/@alice;

Column[MatrixPlot[#,PlotTheme->"Minimal",ImageSize->Medium]&/@Transpose/@mfcc]

DistanceMatrix[mfcc,DistanceFunction->WarpingDistance]//MatrixPlot

{n,m}=WarpingCorrespondence[mfcc[[1]],mfcc[[2]]];

dur=QuantityMagnitude[Duration[alice[[1]]],"s"];
s={n,m}\[Transpose]/Max[{n,m}]dur;
Labeled[
ListLinePlot[
s,
PlotRange->{{0,dur},{0,dur}},AspectRatio->1,Axes->False,PlotStyle->Thickness[.01],ImageSize->Medium,Frame->True,FrameTicks->None,Prolog->{,{Line[{{#[[1]],0},#}],Line[{{0,#[[2]]},#}]}&/@(s[[;;;;100]])}
],
AudioPlot[#,PlotStyle->,Frame->False,Axes->False,ImageSize->Medium,AspectRatio->1/15]&/@(alice[[;;2]]),{Bottom,Left},RotateLabel->True,Spacings->{0, 0}]

Transient Detection

a=;

properties={"ComplexDomainDistance","HighFrequencyContent","ModifiedKullbackLeibler","Novelty" , "PhaseDeviation"};detectionFunctions=Rescale/@AudioLocalMeasurements[a,properties,PartitionGranularity->{.02,.005}];
detectionFunction=Mean[Lookup[detectionFunctions,{"HighFrequencyContent","ModifiedKullbackLeibler","Novelty"}]]

ListLinePlot[detectionFunction,ImageSize->Medium]

Filter the detection function using an adaptive threshold.

filteredDetectionFunction=(detectionFunction-MedianFilter[detectionFunction, .02])

Find the peaks of the filtered detection function.

peaks=FindPeaks[filteredDetectionFunction,0,0,0.07];
ListLinePlot[filteredDetectionFunction,PlotRange->All,ImageSize->Medium,Epilog-> {Red,PointSize[0.02],Point[peaks//Normal]}]

Plot the detected transient on the waveform.

AudioPlot[a,ColorFunction->Function[{x,y},If[AnyTrue[peaks["Times"],Abs[x-#]<.015&],,]],PlotRange->{All,All},ImageSize->Medium]

Split the original audio signal at the detected transients.

samples=Select[AudioSplit[a,peaks["Times"]],Duration[#]>Quantity[50,"ms"]&];

Create an audio object by randomly sampling the list and joining the result.

AudioFade[#,Quantity[20,"Samples"]]&/@RandomChoice[samples,50]//AudioJoin

Pitch extraction

a=ExampleData[{"Audio","PianoScale"}]

Mean[{.3a,AudioNormalize@AudioGenerator[AudioLocalMeasurements[a,"RMSAmplitude",PartitionGranularity->{.03,.01}]]AudioGenerator[{"Triangle",AudioLocalMeasurements[a,"FundamentalFrequency",PartitionGranularity->{.05,.01,HannWindow},MissingDataMethod->{"Interpolation",InterpolationOrder->0}]}]}]

Upcoming features

More features in AudioMeasurements

More features in AudioLocalMeasurements

More special feature extraction functions

Speech-related analysis functions