Neural Networks for Audio Effects

Carlo Giacometti, WRI

(RowDefault[, (Spacer1), , (Spacer1), ])  #WolframTechConf

Introduction

The Acoustic Era

Recording of sounds was jump started by the invention of the phonograph in 1877. Initially the recording was simply mechanical (a stylus converting sound to mechanical incisions onto a soft medium).

Introduction

The Electrical Era

With the invention of transducers (as microphones) and the vacuum tube, sounds could be converted into electrical signals, and amplified and filtered. The recording still happened onto a mechanical medium like wax, but the stylus was now driven electrically.

GraphicsRow[{,Spacer[40],}]

Introduction

The Magnetic Era

Magnetic tape was the key new innovation: now the signal could be record directly on tape, increasing dramatically the level of fidelity and unlocking innovative recording techniques.

Introduction

The Digital Era

In the 1970s compart solutions for digital recording started to appear). These sample the signal at a fixed frequency through a ADC (Analog to Digital Converter) and store the result. The recorded data can then be transformed to a continuous analog signal by a DAC (Digital to Analog Converter) and played back.

This unlocks a universe of possibilities for modifying the recorded signal through digital signal processing.

Imitation of preexisting analog effects

define new transfer functions that “sound similar”

physical modeling: simulate the analog circuit and process the signal using the circuit simulation

https://blog.wolfram.com/2021/12/29/digital-vintage-sound-modeling-analog-drums-with-the-wolfram-language-and-system-modeler/

https://blog.wolfram.com/2020/07/23/digital-vintage-sound-modeling-analog-synthesizers-with-the-wolfram-language-and-system-modeler/

neural networks ?!?!?

Case Study: Simulating an Analog Compressor

The goal is to produce a network that mimics the behaviour of the famous Teletronix LA-2A:

ListLinePlot[AudioLoudness[AudioTrim[#,20]]&/@a,PlotLegends->{"Input","Compressed Signal"},AxesLabel->{"time(s)","loudness(LUFS)"}]

There are already digital solutions that exploit physical modeling, but the requirement is to produce a network with a minimal amount of resources to possibly enable real time processing.

Another requirement is that the network needs to be able not only to process the signal, but also take into account the values of the knobs on the compressor, and produce the appropriate behaviour.

Net Architecture

Requirements & Constraints

As fast as possible to enable real time processing

Allow conditioning on knobs positions

Minimal training data

Net Architecture

Input

Many audio networks use high level features as input, however that is a very big computational cost, and implies a reconstruction cost at the output of the network.
A higher fidelity strategy is to use the raw samples an inputs for the network. Reducing the sampling frequency of the signal will reduce the amount of data to be processed at the expense of potentially serious information loss. In this paper the input is the unprocessed signal at full  resolution.

length=88200;
NetEncoder[{"Audio","TargetLength"->length,"SampleRate"->44100}]

Net Architecture

Architecture

The proposed architecture is based around a stack of Temporal Convolutional Network (TCN) blocks, and a multi-layer perceptron (MLP) to handle the values of the “knobs”:

Net Architecture

TCN

In general, TCN are made of 1D convolutional layers, with padding that can be causal (which have applications in fields like time series forecasting or real time signal processing) or non-causal (where the network has access to the whole signal, and looking at the “future” can enable better performance).

Net Architecture

TCN

Dilation

One important characteristic of these convolutions is the use of dilation (which was a crucial element in the development of WaveNet). Dilation, as compared to stride, injects “holes” in the convolutional kernel (while maintaining the operation  as efficient as possible), rather than skipping elements of the input.

input={{0,0,0,0,0,1,0,0,0,0,0}};conv2=ConvolutionLayer["Weights"->{{{1,2,3}}},"Biases"->None,"Dilation"->2];
convk=ConvolutionLayer["Weights"->{{{1,0,2,0,3}}},"Biases"->None];
convs=ConvolutionLayer["Weights"->{{{1,2,3}}},"Stride"->2,"Biases"->None];
Grid[{{"Input: ", input},
{"Weights -> {1, 0, 2, 0, 3}: ", convk[input]},
{"Dilation -> 2, Weights -> {1, 2, 3}: ", conv2[input]},
{"Stride -> 2, Weights -> {1, 2, 3}: ", convs[input]}
}]

This, combined with the stacking of layers with increasing dilation factors, allows for rapid growth of the receptive field of the network with minimal increase of network parameters. While this can be convenient in image processing, it shines with audio signals, since the desired receptive file can span between hundreds of milliseconds to a few seconds, which translates to tens of thousands of samples.

Growth of the receptive field as a function of the dilation factor:

ClearAll[r];
r[K_,d_][0]:=K
r[K_,d_][n_]:=r[K,d][n-1]+(K-1)*d^n

ListPlot[UnitConvert[Quantity[N@Table[Table[r[13,d][n],{n,1,10}],{d,{2,3,10}}]/44100,"Seconds"],"Milliseconds"],Joined->True,PlotMarkers->Automatic,PlotRange->{All,{0,310}},AxesLabel->{"n (Layer Index)","Receptive Field (ms)"}]

Net Architecture

TCN

Feature-wise Linear Modulation (FiLM)

This layer allows conditioning the layer on a set of two parameters for scaling (γ) and bias (β). The operation applies an affine transform to the intermediate activations h:

FiLM[γ_,β_][h_]:=γ h+β

Generally the desired parameters are mapped to the FiLM parameters by means of a multilayer perceptron which is trained in conjunction to them main network.

Net Architecture

TCN

Parametric ReLU (PReLU)

The nonlinearity itself contains a trainable parameter α, which controls the slope for x < 0.

Piecewise[{{x,x>=0}},α *x]

Manipulate[Plot[Piecewise[{{x,x>0}},α x],{x,-2,2},PlotRange->{All,{-2,2}},PlotLabel->Dynamic@Which[α===0., "α == 0, ReLU", α>0,"α > 0, Leaky ReLU"]],{α,0.,1.}]

Net Architecture

TCN

Residual Connection

The residual is scaled with a single parameter by means of a single 1 x 1 convolution. However only the input is casually/non-casually padded, so each convolution loses some samples. This needs to be addressed in the residual connection, where the signal is cropped to conform to the dimensions of the output of the TCN. This can done two ways:

non causal cropping, which is done by dropping samples at both ends of the signal

causal cropping, which is done by dropping all the necessary samples starting at the beginning of the signal

Net Architecture

Handling of processor parameters

The parameters are transformed into a feature vector by a MLP that is trained alongside to the main network. This feature vector is then independently scaled for each TCN block, and reduced to two parameter (γ and β) for each block.

Loss Functions

The loss function needs to take into consideration both  the actual values of the signal, but also its perceptual characteristics. To achieve that, a combination of two losses is used.

Mean Absolute Error (MAE)

The MAE loss only takes into account differences in the time domain.

MAE[y1_,y2_]:=Mean[Abs[y1-y2]]

aud=Normal@Audio@"ExampleData/rule30.wav";
Manipulate[Module[{res=LowpassFilter[aud,Quantity[f,"Hertz"],400]},Grid[{{"Input: ", aud,"LowpassFilter[Input, f]: ",res},{"MAE: ",MAE[aud,res],SpanFromLeft},{"Abs[y1 - y2]: "AudioPlot[Abs[aud - res],PlotRange->{{.2,.4},{0,.2}},ImageSize->400],SpanFromLeft}}]],{f,100,20000}]

Multi-Resolution short-time Fourier Transform Error:

The multi-resolution STFT loss takes into account differences in the frequency domain. It is composed of two elements.

The spectral convergence term:

lSC[n_,d_][y1_,y2_]:=With[{stft=SpectrogramArray[y1,n,d]},Norm[Abs[stft] - Abs[SpectrogramArray[y2,n,d]], "Frobenius"]/Norm[Abs[stft], "Frobenius"]]

And the spectral log-magnitude term:

lSM[n_,d_][y1_,y2_]:=1/Length[y1]*Norm[Log@Abs@SpectrogramArray[y1,n,d] - Log@Abs@SpectrogramArray[y2,n,d], "Frobenius"]

Both of these can be combined and computed at different resolutions:

lMR[ns_][y1_,y2_]:=Mean@Table[lSC[n,Automatic][y1,y2]+lSM[n,Automatic][y1,y2],{n,ns}]

aud=Normal@Audio@"ExampleData/rule30.wav";
ListLinePlot[Table[Module[{res=LowpassFilter[aud,Quantity[f,"Hertz"],400]},(*Grid[{{"Input: ", aud,"LowpassFilter[Input, f]: ",res},{"MAE: ",MAE[aud,res],SpanFromLeft},{"Abs[y1 - y2]: "AudioPlot[Abs[aud - res],PlotRange->{{.2,.4},{0,.2}},ImageSize->400],SpanFromLeft}}]*)
{f,lMR[{200,500,1000}][First@AudioData@aud,First@AudioData@res]}
],{f,100,20000,1000}],
AxesLabel->{"f(Hz)","lMR"},
PlotLabel->"lMR[in, LowpassFilter[in, f]"]

For optimal performance, the appropriate partition size, window type, and offset must be selected. Often there is no clear choice. To address it is possible to randomly select these parameters each time the loss is computed, ensuring the model is not biased by a fixed set of parameters.

Metrics

In addition to the losses used in training, a metric that was used in evaluating the result of the experiment is the comparison of the perceptual loudness of the target - output pairs:

ListLinePlot[AudioLoudness/@{,},]

Net Architecture

WL Implementation

FiLM

fiLM[par_] := FunctionLayer[
	Slot[par][[1]] * #Input + Slot[par][[2]] &,
	par -> 2
]

fiLM["par"]

PReLU

prelu = FunctionLayer[
	Block[
		{array},
		array = NetArray["Array" -> {1}];
		If[# >= 0., array * #,#]
	] &
]

TCN

computeSize[n_, d_, k_] := Floor[(n - d * (k - 1) - 1)]

TCN[c_, k_, d_, crop: "Causal" | "Center", nInputs_, inChannels_, i_] :=
With[{par = "par" <> ToString[i]},
	Module[
		{size = nInputs - computeSize[nInputs, d, k]},
		NetGraph[
			<|
				"conv" ->
					ConvolutionLayer[c, {k},
						"Input" -> {nInputs, inChannels},
						"Dilation" -> d,
						Interleaving -> True
					],
				"batchNorm" -> BatchNormalizationLayer[
					Interleaving -> True
				],
				"FiLM" -> fiLM[par],
				"PReLU" -> prelu,
				"skip" -> ConvolutionLayer[
					c, {1},
					"Input" ->{nInputs, inChannels},
					Interleaving -> True
				],
				"crop" -> If[crop === "Causal",
					FunctionLayer[
						#[[size ;; - 1]] &
					],
					FunctionLayer[
						#[[size / 2 ;; - size / 2]] &
					]
				],
				"plus" -> ThreadingLayer[Plus]
			|>,
			{
				NetPort["Input"] -> "conv" -> "batchNorm" -> NetPort["FiLM", "Input"],
				NetPort["FiLM", "Output"] -> "PReLU",
				NetPort["Input"] -> "skip" -> "crop",
				{"crop", "PReLU"} -> "plus"
			}
			
		]
	]]

TCN[32, 13, 2^5,"Causal" , 884200, 1, 1]

MLP

mlp[nparams_] := NetChain[
	{
		LinearLayer[16],
		ElementwiseLayer @ "ReLU",
		LinearLayer[32],
		ElementwiseLayer @ "ReLU",
		LinearLayer[32],
		ElementwiseLayer @ "ReLU"
	},
	"Input" -> nparams
]

mlp[2]

Full Net

d = 10;
k = 13;
c = 32;
nParams = 2;
nSamples = 88200;
nLayers = 4;

net = Module[
	{n = nSamples, outDim, tcn},
	NetGraph[
		Association @ Join[
			Table[
				outDim = computeSize[n, d, k];
				tcn = Rule[
					"tcn" <> ToString[i],
					TCN[
						If[i === nLayers, 1, c], k, d,
						"Causal", n, If[i === 1, 1, 32], i
					]
				];
				n = outDim + 1;
				tcn,
				{i, nLayers}
			],
			Table[
				Rule[
					"linear" <> ToString[i],
					LinearLayer[2]
				],
				{i, nLayers}
			],
			{"mlp" -> mlp[nParams]}
		],
		Join[
			{Fold[#1 -> #2&,
				Table[
					"tcn" <> ToString[i],
					{i, nLayers}
				]
			]}
			,
			Table[
				"linear" <> ToString[i] -> NetPort["tcn" <> ToString[i], "par"  <> ToString[i]],
				{i, nLayers}
			],
			{NetPort["Parameters"] -> "mlp" -> Table[
				"linear" <> ToString[i],
				{i, nLayers}
			]}
		]
	]
]

Results

The results show that even with few layers this architecture can mimic the behaviour of an analog compressor pretty accurately if judged by humans.

Other directions

LSTM based networks

WaveNet based networks

Differentiable audio effects

...

References

WL Guides

Audio Representation

Audio Processing

Neural Networks

Courses

Wolfram U | Audio & NN Courses

Repositories

Wolfram Neural Net Repository

Wolfram Function Repository | Sound & Video

Papers

Steinmetz, C. J., & Reiss, J. D. (2021). Efficient neural networks for real-time analog audio effect modeling. arXiv preprint arXiv:2102.06200.

Steinmetz, C. J. (2020). Learning to mix with neural audio effects in the waveform domain. MS thesis.

Perez, E., Strub, F., De Vries, H., Dumoulin, V., & Courville, A. (2018, April). Film: Visual reasoning with a general conditioning layer. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 32, No. 1).

Steinmetz, C. J., Pons, J., Pascual, S., & Serrà, J. (2021, June). Automatic multitrack mixing with a differentiable mixing console of neural audio effects. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 71-75). IEEE.

Steinmetz, C. J., & Reiss, J. D. (2020, December). auraloss: Audio focused loss functions in PyTorch. In Digital Music Research Network One-day Workshop.

Steinmetz, C. J., & Reiss, J. D. (2021). Steerable discovery of neural audio effects. arXiv preprint arXiv:2112.02926.

Steinmetz, C. J., Bryan, N. J., & Reiss, J. D. (2022). Style Transfer of Audio Effects with Differentiable Signal Processing. arXiv preprint arXiv:2207.08759.

SignalTrain LA2A Dataset (Hawley, S. H., Colburn, B., & Mimilakis, S. I. (2019). SignalTrain: Profiling audio compressors with deep neural networks. arXiv preprint arXiv:1905.11928.)

Thank you!