New in Image and Audio Processing

Shadi Ashnai, Manager of Sound & Vision, Algorithms R&D

Abstract

Recent advancements in machine learning and neural networks have made significant improvements in applications such as computer vision and audio analysis. Utilizing these technologies to achieve higher accuracy results has been a priority for us in latest releases of the Wolfram Language. In this talk, I will mention new additions and ongoing efforts to make smarter built-in functions including but not limited to facial analysis and speech understanding. Also, I will talk about other important additions and updates to provide practical utilities such as annotating objects, bridging different data domains and many more. At the end, I will briefly highlight our achievements towards having built-in and efficient video support, processing and analysis.

Outline

Video – introduction, import/export, streaming, processing, analysis

Audio – several updates, major focus on speech & speaker analysis

Image – several updates, major focus on text and face analysis

Annotations – attaching local information to images, audio and eventually video

Video

Video Import

$VideoDecoders//Keys

$VideoDecoders//Dataset

Video Object & Player

v=Video["ExampleData/Caminandes.mp4"]

Head[v]

VideoQ[v]

Information[v]

Streaming, Playing & a Player

v=Import["ExampleData/Caminandes.mp4","Video"]

str=Import["ExampleData/Caminandes.mp4","VideoStream"]

Dynamic[str["Position"]]

Dynamic[str["CurrentFrame"]]

VideoPlay[str]

VideoStop[str]

Video Analysis

ts=VideoMap[ImageMeasurements[#,"MeanIntensity"]&,v]

ListLinePlot[ts]

fr=QuantityMagnitude@Import["ExampleData/Caminandes.mp4","FrameRate"][1]

diffs=VideoBlockMap[ImageDistance@@#&,v,{2/fr,1/fr}]

ListLinePlot[diffs,PlotRange->All]

Video Processing

VideoApply[ColorNegate,v,FileNameJoin[{NotebookDirectory[],"CaminandesNegate.mp4"}]]

Video[FileNameJoin[{NotebookDirectory[],"CaminandesNegate.mp4"}]]

Video Export

$VideoEncoders//Keys

$VideoEncoders//Dataset

Export Manipulates with or without Sound

Export[FileNameJoin[{NotebookDirectory[],"file.mp4"}],Manipulate[Plot[Sin[a x],{x,0,2π}],{a,1,10}]]

Video[FileNameJoin[{NotebookDirectory[],"file.mp4"}]]

Export[FileNameJoin[{NotebookDirectory[],"file2.mp4"}],{"Animation"->Manipulate[Plot[Sin[a x],{x,0,2π}],{a,1,10}],"Audio"->Audio@Sound[Table[SoundNote[i],{i,4}]]},"Rules"]

Video[FileNameJoin[{NotebookDirectory[],"file2.mp4"}]]

More on Video

Video Import/Export, Piotr Wendykier [Thursday 1-1:30pm]

Video Processing and Analysis, Chien-Yu Chen, Shadi Ashnai [Thursday 3:30-4pm]

Audio Updates

Machine Learning & Audio

https://www.wolfram.com/language/12/machine-learning-for-audio/

Audio in the Cloud

https://www.wolframcloud.com/env/f576ab2a-d8e9-47bf-8804-289fa844986d

Audio Streaming Updates

Infinite generated streams are coming AudioStream[func,dur]

Stream collections are coming AudioStream[{(Subscript[audio, 1]),(Subscript[audio, 2]),…}]

Audio – Speech

Speech Recognition

SpeechRecognize[ExampleData[{"Audio","MaleVoice"}]]

Speech Interpretation

Interpret spoken recording (Interpreter for speech).

SpeechInterpreter["SemanticNumber"][]

SpeechInterpreter["Date"][]

SpeechInterpreter["Country"][]

Speech Case Detection

Find cases in a spoken recording (TextCases for speech).

SpeechCases[,"Word"]

SpeechCases[,{"City","Country"}]

SpeechCases[,{"City","Country"}->"Interpretation"]

Notable Speaker Classification

net=NetModel["SpeakerClassificationAlex"]

net[AudioCapture[]]

Interpreter["Person"][%]

%["PropertyAssociation"]

Speaker Feature Extraction

samples=Normal@ResourceData["Spoken Digit Commands","TestDataset"];

RandomSample[samples,2]

grouped=ReverseSortBy[GroupBy[samples,#SpeakerID&],Length];
color=AssociationThread[Keys[grouped]->(ColorData[97]/@Range[Length[grouped]])];

net=NetModel["AlexFineTunedSpeakerClassification"]

net[samples[[1,"Input"]]]

FeatureSpacePlot[Flatten@Values@Map[Style[#Input,color[#SpeakerID]]->#SpeakerID&,grouped[[;;5,;;UpTo[40]]],{2}],FeatureExtractor->net,LabelingFunction->Callout]

Speech Synthesis

Integrating external services such as Amazon Polly, Google Text-to-Speech, etc

Working on representation & synthesis of speech synthesis markup language (SSML)

More on Speech

Understanding Speech—Speech and Speaker Analysis, Carlo Giacometti, Rebecca Frederick [Thursday 2-2:30pm]

Image Updates

Machine Learning & Images

https://www.wolfram.com/language/12/machine-learning-for-images/

ImageAssemble Updates

list=Thumbnail/@WebImageSearch["fish","Thumbnails",MaxItems->4]

Manipulate[ImageAssemble[Partition[list,2],fit,Spacings->s,PaddingSize->s,Background->bg],{fit,{"Fill","Fit","Stretch"}},{s,0,10},{bg,White}]

ImageAssemble[Table[EdgeDetect[list[[1]],r,t],{r,{2,4,6}},{t,{.01,.05,.1}}],Background->Green,Spacings->5]

Faster & Special Transformations

i=;
ImageTransformation[i,FractionalPart[2 # Total[#]]&]

(Legended[, (SwatchLegend["12.1", "12.0"])])

Thin plate splines

⟶

DynamicModule[
{sources ={{173,195},{149,195},{156,207},{180,236},{144,236}}},Manipulate[
ImageTransformation[
,
Image`RadialBasisTransform[targets->sources,"ThinPlateSpline"],
Padding-> White,DataRange-> Full
],
{{targets,sources},Locator,Appearance->{Graphics[{Red, Point[{0,0}]}]}, 
ImageSize-> 300}
]
]

Image Highlighting

Default highlight color changed to magenta!

Labeling & legending is coming

i=;
boxes=ImageBoundingBoxes[i]

HighlightImage[i,boxes]

HighlightImage[i,boxes,ImageLegends->All,ImageLabels->None]

HighlightImage[i,boxes,ImageLegends->All]

Updated Image Toolbar

ExampleData[{"TestImage","Flower"}]

Image – Face & Text

Face Analysis

i=;

Manipulate[HighlightImage[i,{EdgeForm[AbsoluteThickness[1.5]],FaceForm[],FindFaces[#,PaddingSize->p,MaxFeatures->item]&}],{{p,0},-10,10},{{item,5},1,5}]

FindFaces[i,"Image"]

FaceAlign[i,Automatic,80]

HighlightImage[i,MapThread[Labeled,{FindFaces[i],FacialFeatures[i,"Age"]}]]

Text Analysis

Text detection is coming

Text recognition is being improved

We are also looking into both handwritten and math OCR.

image=;

(*FindText[image]*)
HighlightImage[image,{Green,detectedtext}]

(*FindText[image, {"AlignedImage", "BoundingBox","Strength","DeskewAngle"}]*)
Dataset[findtextdataset]

More on Images

Computer Vision, Giulio Alessandrini, Mikayel Egibyan [Thursday 2:30-3pm]

Neural Networks for Images and Audio Workshop, Carlo Giacometti, Markus van Almsick [Wednesday 12:30-1:30pm]

Image Processing MOOC

https://www.devel.wolfram.com/wolfram-u/introduction-to-image-processing/course-overview.html

Annotations

Audio Annotation

a=AudioAnnotate[Import["ExampleData/rule30.wav"],{"Loud","Transients","Max"}];

AudioPlot[a,AnnotationRules->"Loud",PlotLayout->"Averaged"]

AudioPlot[a,AnnotationRules->{"Loud","Transients"},PlotLayout->"Averaged"]

AudioPlot[a,AnnotationRules->{"Loud","Transients","Max"},PlotLayout->"Averaged"]

Image Annotations

Detect all elephants and and annotate the image with the corresponding bounding boxes:

i=;

boxes=ImageBoundingBoxes[i,"elephant"];

ii=ImageAnnotate[i,"elephant"->boxes]

Lookup annotations and highlight them:

HighlightImage[i,ImageAnnotationLookup[ii],ImageLabels->None]

Size of each bounding box:

ImageAnnotationLookup[ii,"elephant"->(RegionMeasure[#RegionOfInterest]&)]

Total size of all boxes:

ImageAnnotationLookup[ii,"elephant"->(RegionMeasure[#RegionOfInterest]&),Total]

Import/Export

On top of all of the upcoming video format support:

"WAV" — now supports large WAV files (RF64)

“HEIF” — mac & windows only

“DICOM” (updated) — backend switch, speed improvement, bug fixes, new elements

"DICOMDIR" — new support for DICOMDIRs

“RLE” (updated) — new elements such as “CellularAutomaton”, “Rule”

a lot of framework updates for more efficient & more robust Import/Export

Import/Export—Updates and Applications, Sean Cheren [Wednesday 2:30-3pm]

LibraryLink Utilities, Rafal Chojna [Wednesday 4:30-5pm]

Thanks!

Supporting Data

FindText

detectedtext={Parallelogram[{51.19019719241669`,122.38068204150748`},{{83.98612604102186`,8.39934551238035`},{-2.8169903965583956`,28.167445922182836`}}],Parallelogram[{125.00169368813901`,130.41666000725226`},{{49.67077851261665`,4.394095192969104`},{-2.0355855666290523`,23.01022517338569`}}],Parallelogram[{56.04104472949241`,100.75383151266456`},{{80.33830831811166`,10.418292553620432`},{-2.7468595690886985`,21.18178673062077`}}],Parallelogram[{131.8553059107525`,109.42936798957406`},{{40.594482797535676`,3.6382896147975146`},{-2.0124920799781307`,22.454527750780912`}}],Parallelogram[{75.09381233463084`,79.89304133996814`},{{84.62044467865768`,14.809597137389204`},{-3.5782414670600105`,20.445686759822962`}}],Parallelogram[{47.75423605593989`,53.48598492747056`},{{60.448601748237486`,11.022993346942123`},{-3.712460212656339`,20.358628717062082`}}],Parallelogram[{102.57785712610351`,62.217759568502835`},{{78.66427670264699`,12.879268915835915`},{-3.4467609743852847`,21.052201082900602`}}],Parallelogram[{38.36975061017342`,29.962653170123836`},{{44.91313020059621`,7.461408891054411`},{-3.4799400757282086`,20.947116555807305`}}],Parallelogram[{79.78392455815553`,37.30215627140255`},{{48.71997826211539`,9.741922031420572`},{-4.020096665535846`,20.10476182470012`}}],Parallelogram[{127.74423860853204`,44.977403448515936`},{{61.06487755003877`,11.18417380828302`},{-3.62024929835286`,19.76633087915777`}}]};

findtextdataset={{,Rectangle[{48.373206795858295`,122.38068204150748`},{135.17632323343855`,158.94747347607066`}],0.9999375343322754`,5.711088180541992`},{,Rectangle[{122.96610812150996`,130.41666000725226`},{174.67247220075566`,157.82098037360706`}],0.9999514818191528`,5.05547571182251`},{,Rectangle[{53.294185160403714`,100.75383151266456`},{136.37935304760407`,132.35391079690575`}],0.9999735355377197`,7.3888959884643555`},{,Rectangle[{129.84281383077436`,109.42936798957406`},{172.44978870828817`,135.5221853551525`}],0.9999265670776367`,5.121463298797607`},{,Rectangle[{71.51557086757083`,79.89304133996814`},{159.71425701328852`,115.14832523718032`}],0.9999717473983765`,9.926915168762207`},{,Rectangle[{44.04177584328355`,53.48598492747056`},{108.20283780417738`,84.86760699147477`}],0.9999722242355347`,10.334514617919922`},{,Rectangle[{99.13109615171822`,62.217759568502835`},{181.2421338287505`,96.14922956723933`}],0.999969482421875`,9.298226356506348`},{,Rectangle[{34.88981053444521`,29.962653170123836`},{83.28288081076963`,58.371178616985546`}],0.999953031539917`,9.432390213012695`},{,Rectangle[{75.76382789261969`,37.30215627140255`},{128.50390282027092`,67.14884012752324`}],0.9999629259109497`,11.307587623596191`},{,Rectangle[{124.12398931017918`,44.977403448515936`},{188.80911615857082`,75.92790813595673`}],0.9999581575393677`,10.378823280334473`}};