New in Image & Audio Processing

Shadi Ashnai
Manager of Image & Signal Processing, Wolfram Research

Join the Conversation #WolframTechConf

ABSTRACT

In this talk I will present new and improved image, audio and signal processing capabilities and show application examples highlighting their use. One of our most important efforts are on having a tighter integration with neural nets and machine learning and building high-level functions and applications. We also have various other updates including new way of storing annotations, image pyramids, and many more.

Image Updates

NN-based applications

Color

Image pyramids

Audio Updates

NN-based applications

Speech

WebAudioSearch

Annotations

Upcoming Video

Extensive video/audio codec support

A video player

Aiming for a strong processing & analysis tool

What’s new for Images

NN-based Image

Color

Image Pyramids

NN-based Image Computations

Face Analysis

FindFaces is now a lot more accurate, with less false positives.

Return all faces in an image:

FindFaces[,Automatic,"Image"]

Extract facial features:

FacialFeatures[]//Dataset

Select what faces should be returned:

FindFaces[,#Age>30&,"Image"]

Specify sorting, maximum allowed overlap and various other settings:

Options[FindFaces]//Column

i=;
HighlightImage[i,KeyValueMap[{RandomColor[Hue[_]],Tooltip[#2,#1]}&,First@FacialFeatures[i,"Landmarks"]]]

More coming

Face recognition

Face morphing

Effects and transformations

Image Cases

ImageContents[]

ImageCases[,(Entity["dog", Entity["Species","Infraspecies:CanisLupusFamiliaris"], "Entity[\"Species\", \"Infraspecies:CanisLupusFamiliaris\"]", "species specification"])]

A lot more from the Neural Net Framework

More than half of the networks in the repo are working with images [link].

Image Translation

NetModel["CycleGAN Horse-to-Zebra Translation Trained on ImageNet Competition Data"][]

Semantic Segmentation

net=NetModel["Ademxapp Model A1 Trained on PASCAL VOC2012 and MS-COCO Data"]

netevaluate[img_,device_: "CPU"]:=Block[{net,resized,encData,dec,mean,var,prob},net=NetModel["Ademxapp Model A1 Trained on PASCAL VOC2012 and MS-COCO Data"];
resized=ImageResize[img,{504}];
encData=Normal@NetExtract[net,"Input"];
dec=NetExtract[net,"Output"];
{mean,var}=Lookup[encData,{"MeanImage","VarianceImage"}];
prob=NetReplacePart[net,{"Input"->NetEncoder[{"Image",ImageDimensions@resized,"MeanImage"->mean,"VarianceImage"->var}],"Output"->Automatic}][resized,TargetDevice->device];
prob=ArrayResample[prob,Append[Reverse@ImageDimensions@img,21]];
dec[prob]]

labels={"background","airplane","bicycle","bird","boat","bottle","bus","car","cat","chair","cow","table","dog","horse","motorcycle","person","plant","sheep","sofa","train","television"};

mask=netevaluate[];
Colorize[mask]

detected=DeleteDuplicates@Flatten@mask

labels[[detected]]

Neural Networks for Images, Markus van Almsick [Wednesday 10-10:30am]

(we will always love) Color

Find Matching Color

?FindMatchingColor

Perform a foreground-background segmentation based on the color of the background:

i=;

Binarize[ColorDistance[i,Blue]]

The image seems to contain blue, but the actual pixel values are very far from it:

Min@ColorDistance[i,Blue]

Find the matching blue in the image and use it to perform the segmentation:

c=FindMatchingColor[i,Blue,MaxColorDistance->1]

Binarize[ColorDistance[i,c]]

Colors Near

?ColorsNear

We use to have color directives forever. Now, ColorsNear can define color regions.

Red

ColorsNear[Red]

ChromaticityPlot3D[ColorsNear[Red],"LAB",PlotRange->All]

ChromaticityPlot3D[{"sRGB",ColorsNear[Red]},"LAB"]

ChromaticityPlot3D[{"sRGB",ColorsNear[#]&/@{Red,Green,Blue}},"LAB",BoxRatios->Automatic]

ChromaticityPlot3D[{"sRGB",ColorsNear[#,.2,"CIE76"]&/@{Red,Green,Blue}},"LAB",BoxRatios->Automatic]

Color Detect

?ColorDetect

ColorDetect[,ColorsNear[,.15]]

Image Recolor

?ImageRecolor

ImageRecolor[,ColorsNear[,.15]->]

Playing with Colors, Giulio Alessandrini [Thursday 9:30-10am]

Image Constructs

Image pyramids

Creation and reconstruction

Convert a large image into a pyramid:

img=ExampleData[{"TestImage","Volubilis"}];

pd=ImagePyramid[img,"Gaussian"]

InverseImagePyramid[pd,Automatic,{320}]

Fast Viewing

View the image with width 320:

ImageResize[InverseImagePyramid[pd,Automatic,{320}],{320}]

Compare the timings for image retrieval:

ImageResize[InverseImagePyramid[pd,Automatic,{320}],320];//RepeatedTiming

ImageResize[img,320];//RepeatedTiming

Multi-scale analysis

Apply a gradient filter at several scales simultaneously:

img = ;

pyramid=ImagePyramid[img,"Gaussian",4]

pyramidG=ImagePyramidApply[img|->GradientFilter[img,1.4],pyramid];

Reconstruct assuming a Laplacian pyramid to add up features extracted from all levels:

InverseImagePyramid[pyramidG,"Laplacian"]//ImageAdjust

Blending

Merge two images at just one scale:

{apple, orange}={, };

splitMerge[a_?NumericQ][i1_Image,i2_Image]:=
 With[{c = Floor[a First@ImageDimensions[i1]]},
ImageAssemble[{{ImageTake[i1,All,{1,c}],ImageTake[i2,All,{c+1,-1}]}}]]

splitMerge[0.5][apple, orange]

Merge the two images at all scales:

pyramids=Map[ImagePyramid[#,"Laplacian",{2,2}]&,{apple, orange}];

pFruit=ImagePyramidApply[splitMerge[0.5],pyramids]

InverseImagePyramid[pFruit]

Minor but important

The default image type is now “Real32”

The data is stored in (now documented, but experimental) NumericArray

Image[RandomReal[1,{5,5}],ImageSize->100]//InputForm

NumericArray – Compact Representation of “Numeric” Arrays, Piotr Wendykier [Wednesday 2:30-3pm]

What’s new for Audio

NN-based Audio

Speech

WebAudioSearch

Annotations

NN-based Audio

Now we have a large number of audio encoders with various parameters and controls, e.g. "AudioMFCC".

Build a Spoken Digit Commands Identifier

ro=ResourceObject["Spoken Digit Commands"];
trainingData=ResourceData[ro,"TrainingData"];
testingData=ResourceData[ro,"TestData"];

Define the net:

rnn=NetChain[{GatedRecurrentLayer[32],GatedRecurrentLayer[64],SequenceLastLayer[],LinearLayer[64],Ramp,LinearLayer[Length@Union[trainingData[[All,"Output"]]]],SoftmaxLayer[]},"Input"->NetEncoder[{"AudioMFCC","TargetLength"->All,"NumberOfCoefficients"->28,"SampleRate"->16000,"WindowSize"->1024,"Offset"->571,"Normalization"->True}],"Output"->NetDecoder[{"Class",Union[trainingData[[All,"Output"]]]}]];

Train:

trainedRNN=NetTrain[rnn,trainingData,ValidationSet->Scaled[.05]];

Evaluate:

ClassifierMeasurements[trainedRNN,#Input->#Output&/@testingData]["Accuracy"]

trainedRNN[AudioTrim[AudioCapture[]]]

Audio Identification

{#,AudioIdentify[ExampleData[#],All,4]}&/@ExampleData["Audio"][[;;10]]//TextGrid

Audio Classification

FeatureSpacePlot[ExampleData/@ExampleData["Audio"]]

Neural Networks for Audio, Carlo Giacometti [Wednesday 5-5:30pm]

Speech

Speech Synthesis

We used to have a list of available voice names:

$VoiceStyles//Short

Now, we extract and return voice properties:

VoiceStyleData[]//Dataset

Perform selection on the voices:

(voices=VoiceStyleData[#Gender=="Female"&])//Dataset

Synthesize using a specific voice or for a specific language:

AudioJoin[{SpeechSynthesize[#Language,Language->"English"],SpeechSynthesize[With[{res=Quiet@TextTranslation["Hello",#Language]},If[StringQ[res],res,"cannot synthesize"]],Language->#Language]}]&/@voices[[;;10]]//AudioJoin

Speech Recognition

a=ExampleData[{"Audio","MaleVoice"}]

SpeechRecognize[a]

More

Notable person voice classifier

Voice feature extractor

Speaker identification

...

Understanding Speech, Carlo Giacometti [Wednesday 1:30-2pm]

WebAudioSearch [Experimental]

Connect to sound repositories and search and extract for sound tracks. We are using Freesound only for now.

WebAudioSearch["dog","RandomSample"]

WebAudioSearch["dog","RandomSample",10<#Duration<20&]

WebAudioSearch["dog","Samples"]

WebAudioSearch["dog","Dataset"]

WebAudioSearch["dog","Gallery"]

Annotations [Experimental]

Add custom annotations to an audio recording for any time or time interval:

a=Import["ExampleData/rule30.wav"];res=AudioAnnotate[a,{"tag1"->{.4->"v11",{1,1.5}->"v12"},"tag2"->{.1->"v21",{.7,1.5}->"v22"}}];

Options[res,AnnotationRules]

AudioPlot[res,AnnotationRules->{"tag1","tag2"}]

AudioAnnotationLookup[res]//Dataset

AudioAnnotationLookup[res,All->(#Time==.4&)]//Dataset

AudioAnnotationLookup[res,All->(#Time>.5&)]

Create pre-defined annotations:

res2=AudioAnnotate[a,"Loud"];

AudioAnnotationLookup[res2]

ShortTimeFourier

Spectrogram & SpectrogramArray (from version 9)

We have had Spectrogram and SpectrogramArray since version 9, but they didn’t store information necessary for reconstructing the signal.

a=ExampleData[{"Audio","Bird"}]

Spectrogram[a]

SpectrogramArray[a]//Short

ShortTimeFourier

stft=ShortTimeFourier[a]

InverseShortTimeFourier[stft]

data=stft["Data"];
data[[All,20;;-50]]=0+I;
data[[250;;350,All]]=0;
stft["Data"]=data;

InverseShortTimeFourier[stft]

Spectrogram[%]

Inverse Magnitude Spectrogram

Construct an audio object from an image:

img=;

Reconstruct the signal with the assumption that the image was its spectrogram:

InverseSpectrogram[img]//Audio

Spectrogram[%,PlotRange->All,AspectRatio->1]

Upcoming Video Project [Future]

Why

Video creation has become easier

cell phones, action cameras, security cameras, medical procedures, microscopes, ...

Various video services and repositories are now available

YouTube, Video, Twitch, Facebook, ...

Applications in various industries

medicine, material science, security, automobile, video games, remote sensing, online training, ...

What

Extensive format support

Video object (out-of-core only perhaps)

Video capture and generation

Video streaming (similar to audio streaming)

A player in the FrontEnd

Computation on video frames

Computation on audio frames

Computation on video+audio

Video: What is coming first

Extensive video format support

GetAvailableContainers[]

GetAvailableDecoders["Video"]//Short

GetAvailableDecoders["Audio"]//Short

A Player

video1=OpenVideo[FileNameJoin[{videopath,"/MKV/samples/haruhi-vp9.mkv"}]];

VideoPlayer[video1]

Aiming for a strong processing & analysis tool

Stabilization

Object tracking

Activity recognition

Reenactment

Interesting snippet

Video (alpha) – Introduction of Video and Video Processing, Shadi Ashnai, Piotr Wendykier, Chien-Yu Chen [Friday 4-4:30pm]

Thanks!

Init

WebAudioSearch

Needs["PacletManager`"]
PacletDirectoryAdd["~/workspace/EXTSERV/Freesound/build/ServiceConnection_Freesound"];
WebAudioSearch["dog","RandomSample"];

Video

Needs["FFmpegTools`"];
videopath="/Users/shadi/ownCloud/ImportExport/Video";