Introduction to Audio in the Wolfram Language

Lambert Chao and Piotr Wendykier, Wolfram Research

Join the Conversation #WolframTechConf

Computing with Audio in the Wolfram Language

Audio In the Wolfram Language

Audio Sources

Audio Framework

Audio Properties

Basic Manipulation

Arithmetical and Statistical Operations

Audio Visualization

Audio Editing Operations

Audio Processing

Audio Filtering

Audio Analysis

Audio Effects

Bringing Audio into the Wolfram Language

Seamless integration of audio sources:

Audio["Bach-BWV1007_Prelude.mp3"]

Challenges of Integrating Audio

Audio Data Size

ByteCount[]

ByteCount[Import["Bach-BWV1007_Prelude.mp3","Data"]]

tedTalk=Audio["StephenWolfram_2010.mp3"]

{ByteCount[tedTalk],FileByteCount["StephenWolfram_2010.mp3"]}

ByteCount[Import["StephenWolfram_2010.mp3","Data"]]

Smooth Gapless Playback

Handling multiple streams in the kernel at the same time as other evaluations

Pause[3]

Solve the Poisson equation over a Disk

NDSolve[{-Laplacian[u[x,y],{x,y}]==1,DirichletCondition[u[x,y]==0,True]},u,{x,y}∈Disk[]]

Plot3D[Evaluate[u[x,y]/.%],{x,y}∈Disk[]]

Audio Typesetting

Audio["Bach-BWV1007_Prelude.mp3",Appearance->"Basic"]

Audio["Bach-BWV1007_Prelude.mp3",Appearance->"Detailed"]

Grid[{#,AudioGenerator[#,Appearance->"Minimal"]}&/@{"Sin","Triangle","Sawtooth","Square","Pulse","Impulse"},Frame->All]

Audio Sources

Audio for computation:

Audio File

a=Audio["Bach-BWV1007_Prelude.mp3"]

Audio Data

Supported types: SignedInteger8, SignedInteger16, SignedInteger32, Real32, Real64.

f=1000;t=5;sampleRate=44100;
audioData=Table[IntegerPart[127*N[Sin[2π f t/sampleRate]]],{t,0,sampleRate*5}];
incoreAudio=Audio[audioData,"SignedInteger8"]

ByteCount[incoreAudio]

ByteCount[Audio[incoreAudio,"Real32"]]

Internet Audio

Stream audio from a URL:

URLAudio = 
Audio[URL["http://www.music.army.mil/music/play.asp?TheStarSpangledBanner_BandOnly.mp3"]]

Stream audio from the Wolfram Cloud:

cloudAudio = Audio[CloudObject["https://programming.wolframcloud.com/app/objects/fabe54ff-c06e-4d01-b543-ee899eda0fab"]]

Audio Generation

Oscillators

The simplest way to generate a signal is to use one of the native oscillator models supported in AudioGenerator.

AudioGenerator["Sin"]

All oscillator models accept time-varying parameters for frequency and phase: the input can be a TimeSeries, another Audio object, or even a pure function.

Use a pure function to control the frequency.

AudioGenerator[{"Sin",440+10Sin[2Pi 6#]&},4]

Use a time series to control the frequency.

vib=TimeSeries[Table[{t,Sin[2Pi 6t]},{t,0,4,1/200}]];
AudioGenerator[{"Sin",440+10vib}]

Noise Generators

In addition to oscillators, AudioGenerator allows for the creation of different  noise signals.

Generate one second of pink noise.

AudioGenerator["Pink"]

Different types of noise have different spectral contents.

noises={"White","PeriodicRandomNoise","Pink","Brown","Blue",{"Color",1.5`}};
Periodogram[AudioGenerator/@noises,2000,ScalingFunctions->{"Log10","dB"},PlotStyle->{,,,,,},PlotLegends->Placed[noises,Right]]

Processes and Time Series

AudioGenerator can be used to generate a signal by sampling a random process.

AudioGenerator[FractionalBrownianMotionProcess[.5]]//AudioNormalize

AudioGenerator can also be used to create an Audio object from a TimeSeries:

Audio Synthesis Talk: tomorrow at 10:00am

Import/Export

a=Import["Bach-BWV1007_Prelude.mp3"]

ByteCount[a]

b=Import["Bach-BWV1007_Prelude.mp3","AudioFile"]

ByteCount[b]

ExampleData

ExampleData["Audio"]

ExampleData[{"Audio","Clapping"}]

ExampleData[{"Sound","Apollo13Problem"}]

Sound

Sound[Table[SoundNote[0,0.3,i],{i,15}]]

Audio[Sound[Table[SoundNote[0,0.3,i],{i,15}]]]

Meeting the Challenge: New Audio Technology in the Wolfram Language

New audio features in Mathematica 11:

Support for new audio sources

Efficient out-of-core algorithms

New updated interface

Suite of powerful audio processing functions

New technology:

Working behind the scenes to make Audio possible:

New ways to stream audio from data sources in core, in files, on the internet or from your microphone

New system for buffered gapless playback

New audio devices capabilities

Audio Properties

Options

Options[Audio]

Options[audio=ExampleData[{"Sound", "Bassoon"}, "AudioFile"]]

Length and Duration

Length is defined as the total number of samples. Duration is the length divided by the sample rate.

AudioLength[audio]

Duration[audio]

Channels

The number of channels can be arbitrary.

AudioChannels[audio]

Type

Query the type.

AudioType[audio]

Sample Rate

Sample rate can be arbitrary.

AudioSampleRate[audio]

Arithmetical and Statistical Operations

Arithmetical operators as well as statistical functions work directly with audio objects.

Create a linear combination of two audio objects.

0.3×AudioGenerator["Sin"]+0.7×AudioGenerator["Triangle"]

Compute the mean of a list of audio objects.

Mean[{,,,,}]

Compute statistics of an audio object.

a=ExampleData[{"Audio","Bird"}];
stats={Total,Min, Max,Mean,Median,StandardDeviation};Dataset[AssociationThread[ToString/@stats ->Through[stats[a]]]]

A system option called "IndeterminateValue" is used to replace values that can result from arithmetical operation but cannot be stored in an audio object. These include ComplexInfinity and Indeterminate.

This displays the current value of "IndeterminateValue" option and changes it to 2.

SystemOptions["AudioProcessingOptions"->"IndeterminateValue"]

SetSystemOptions["AudioProcessingOptions"->"IndeterminateValue"->2. ]

This shows how "IndeterminateValue" is used to replace unacceptable values in the result of an arithmetical operation.

data= {-2,-1,0,1,2};
a=Audio[data];

data^data

AudioData[a^a]

Complex numbers in the result of an arithmetical operation are replaced by the real parts of these values.

data= {-2,-1,0,1,2};
a=Audio[data];

Sqrt[data]

AudioData[Sqrt[a]]

Re[Sqrt[data]]

Audio Visualization

Audio can be visualized in a variety of ways.

a=ExampleData[{"Audio","Bird"},"Audio"];

Visualize the waveform of an audio signal.

AudioPlot[a,Appearance->"Discrete",PlotRange->All,ImageSize->Medium]

Plot the power spectrum of the audio signal.

Periodogram[a,300,200,HammingWindow,GridLines->Automatic]

Visualize the spectrogram of the audio signal.

Spectrogram[a,PlotRange->{All,{0,15000}}]

Audio Editing Operations

Padding

Adds samples at the beginning and at the end of audio signal using some predetermined scheme.

AudioPlot[a=Import["ExampleData/car.mp3"]]

AudioPlot[b=AudioPad[a,3, 0]]

AudioPlot[AudioPad[a, 3, "Periodic"]]

Cropping

Removes silence from the beginning and from the end of audio.

AudioPlot[AudioTrim[b]]

Take the samples between third and fourth second.

AudioPlot[AudioTrim[a, {(Quantity[3, "s", seconds, "Seconds"]),(Quantity[4, "s", seconds, "Seconds"])}]]

Channel Operations

Separate audio channels.

AudioChannelSeparate[c=Audio["ExampleData/drums.ogg"]]

Combine audio channels.

AudioChannelCombine[%]

Joining

Join multiple audio signals.

AudioPlot[AudioJoin[{a,c}]]

Overlaying

Overlay multiple audio signals.

AudioPlot[AudioOverlay[{a,c}]]

Splitting

Split the audio object using a list of split points.

Duration/@AudioSplit[,{.4,.8}]

Partition audio into overlapping blocks.

AudioPlot[AudioPartition[ExampleData[{"Audio","Apollo11ReturnSafely"}],1,.5]]

Resampling

Resample audio signals.

{AudioSampleRate[a],AudioSampleRate[AudioResample[a, (Quantity[8000, "Hz", hertz, "Hertz"])]]}

Scaling

All samples are rescaled using the same scaling factor s /(Abs[peak]), with peak being the sample with the highest absolute value.

{MinMax[a],MinMax[AudioNormalize[a, 0.5]]}

Conforming

All audio objects in the list are made to have conforming properties, including duration, data type, and number of channels.

a=Audio["ExampleData/car.mp3"];
b=Audio["ExampleData/drums.ogg"];

Dataset[AssociationThread[{a,b},Association@@@Transpose[Outer[Rule[ToString[#],#[#2]]&,{AudioLength,AudioChannels,AudioType,Duration,AudioSampleRate},{a,b}]]]]

{aa,bb}= ConformAudio[{a,b}];

Dataset[AssociationThread[{aa,bb},Association@@@Transpose[Outer[Rule[ToString[#],#[#2]]&,{AudioLength,AudioChannels,AudioType,Duration,AudioSampleRate},{aa,bb}]]]]

Audio Filtering

Many of the filtering functions present in the Wolfram Language can be immediately used on audio objects.

FIR Filters

a=ExampleData[{"Audio","Apollo11SmallStep"},"Audio"]

lp=LowpassFilter[a,(Quantity[800, "Hz", hertz, "Hertz"]),101]

hp=HighpassFilter[a,(Quantity[1600, "Hz", hertz, "Hertz"]),101]

bs=BandstopFilter[a,{(Quantity[800, "Hz", hertz, "Hertz"]),(Quantity[1600, "Hz", hertz, "Hertz"])},91]

Plot the periodogram of the original and processed signals.

Periodogram[{a,bs,lp,hp},2000,PlotRange->{{0,3000},Automatic},Frame->True,ImageSize->Medium,PlotLegends->{"Original","Bandstop","Lowpass","Highpass"}]

RecurrenceFilter

Apply a resonant biquadratic filter to an audio object.

filterModel=BiquadraticFilterModel[{ω,q}]

discreteFilter=ToDiscreteTimeModel[filterModel,1/22050,z,Method->{"BilinearTransform","CriticalFrequency"->ω}]

a=;

RecurrenceFilter[discreteFilter/.{ω->1000 2 Pi,q->7},a]

WienerFilter

Denoise a recording.

a=ExampleData[{"Audio","Apollo11SmallStep"}]

WienerFilter[a,25]

Spectrogram[#,ImageSize->Small,PlotRange->{All,{0,5000}},FrameTicks->None]&/@{a,%}

Audio Analysis

Global Properties

Both time domain and frequency domain properties can be measured with AudioMesurements. The properties are computed on the average sample values over the channels of the audio object.

Compute time domain properties of a recording.

AudioMeasurements[ExampleData[{"Audio","Drums"}],{"MinMax","Mean","StandardDeviation","RMSAmplitude","ZeroCrossings"},"Dataset"]

Compute frequency domain properties of a recording.

AudioMeasurements[ExampleData[{"Audio","Drums"}],{"SpectralCentroid","SpectralFlatness","SpectralRollOff","SpectralSpread"},"Dataset"]

Local Properties

In AudioLocalMeasurements properties are computed locally: the signal is partitioned according to the PartitionGranularity specification, and the requested property is computed on each partition. The result is returned as TimeSeries whose timestamps correspond to the center time of each partition.

Compute the RMS amplitude on 40 ms partitions with an offset of 1 ms.

a=;
res=AudioLocalMeasurements[a,"RMSAmplitude",PartitionGranularity->{(Quantity[40, "ms", milliseconds, "Milliseconds"]),(Quantity[1, "ms", milliseconds, "Milliseconds"])}];
Show[AudioPlot[a,PlotRange->All],ListLinePlot[res,PlotStyle->Red,PlotRange->All]]

Use the "SpectralCentroid" and "SpectralSpread" measurements to find clusters of similar audio objects in a list.

samples={,,,,,,,,};
props=AudioMeasurements[#,{"SpectralCentroid","SpectralSpread"},"List"]&/@samples;
clusters=FindClusters[props]

ListPlot[props,PlotMarkers->samples,ImageSize->300,Prolog->({Opacity[.3,],Disk[Mean@#,400]&/@clusters}),PlotRange->{{00,4000},{0,4000}},AspectRatio->1]

Audio Analysis Talk: tomorrow at 10:30am

Audio Effects

Time Stretching

Slow down an audio object.

a=ExampleData[{"Audio","MaleVoice"}]

AudioTimeStretch[a,2]

Pitch Shifting

Shift the pitch of an audio object up.

AudioPitchShift[a,(Quantity[2, "semitones", semitones, IndependentUnit["semitones"]])]

Reverberation

Add a reverberation.

AudioReverb[a,ExampleData[{"Audio","IRStMarysChurch"}]]

Delay

Add a delay effect.

AudioDelay[a,.7,.8,.4,PaddingSize->4]

Audio Effects Talk: tomorrow at 9:30am

Conclusion

Various Audio sources: data, file, internet, synthesis.  More to come...

Audio arithmetic, basic properties.

Advanced signal processing, effects.

What’s to come: audio capture, programmatic control, speech...

Talks

Init

$AllowDataUpdates=False;
PacletManager`Package`$checkedForUpdates=True;
SetOptions[PacletSiteUpdate,"Force"->False];

SetDirectory[NotebookDirectory[]];