Video Processing & Analysis

Chien-Yu Chen, Software Engineer, WRI
Shadi Ashnai, Manager of Sound & Vision, WRI

Outline

Video Object in the Wolfram Language

Video Stream & Programmatic Playback

Analysis

VideoMap

VideoBlockMap

Processing

VideoApply

VideoFilter

Use FFmpegTools functions

Video Object in the Wolfram Language

v=Video["ExampleData/Caminandes.mp4"]

Head[v]

VideoQ[v]

Information[v]

Video["ExampleData/Caminandes.mp4",Appearance->"Minimal"]

Video["ExampleData/Caminandes.mp4",Appearance->"Static"]

Video Streams & Programmatic Playback

str=Import["ExampleData/Caminandes.mp4","VideoStream"]

TextGrid[{#,str[#]}&/@ str["Properties"],Frame->All]

Dynamic[str["Position"]]

Dynamic[str["CurrentFrame"]]

VideoPlay[str]

VideoStop[str]

Video Analysis

Mean intensity of each frame

ts=VideoMap[ImageMeasurements[#,"MeanIntensity"]&,v]

ListLinePlot[ts]

Mean color for each channel of each frame

ts=VideoMap[ImageMeasurements[#,"Mean"]&,v]

ListLinePlot[ts,PlotStyle->{Red,Green,Blue}]

Difference of consecutive frames (simple scene detection)

v=Video[FileNameJoin[{NotebookDirectory[],"Musician.mp4"}]]

diffs=VideoBlockMap[ImageDistance@@#&,v,{2/fr,1/fr}]

ListLinePlot[diffs,PlotRange->All]

times=FindPeaks[diffs,Automatic,Automatic,150]["Times"]

Video Processing

Negate every image frame

VideoApply[ColorNegate,v,FileNameJoin[{NotebookDirectory[],"CaminandesNegate.mp4"}]];

Video[FileNameJoin[{NotebookDirectory[],"CaminandesNegate.mp4"}]]

Highlight detected faces

VideoApply[Rasterize@HighlightImage[#,FindFaces]&,Video[FileNameJoin[{NotebookDirectory[],"Girl640.mov"}]],FileNameJoin[{NotebookDirectory[],"GirlFaceHighlighted.mp4"}]];

Video[FileNameJoin[{NotebookDirectory[],"GirlFaceHighlighted.mp4"}]]

Highlight detected objects

v=Video[FileNameJoin[{NotebookDirectory[],"Restaurant - 910.mp4"}]];

VideoApply[Rasterize@HighlightImage[#,ImageBoundingBoxes]&,v,FileNameJoin[{NotebookDirectory[],"RestaurantHighlighted.mp4"}]];

Video[FileNameJoin[{NotebookDirectory[],"RestaurantHighlighted.mp4"}]]

Semantic segmentation

v=Video[FileNameJoin[{NotebookDirectory[],"Street - 22516.mp4"}]];

segment[img_,device_:"CPU"]:=Block[{net,encData,dec,mean,var,prob},net=NetModel["Dilated ResNet-38 Trained on Cityscapes Data"];
encData=Normal@NetExtract[net,"input_0"];
dec=NetExtract[net,"Output"];
{mean,var}=Lookup[encData,{"MeanImage","VarianceImage"}];
Colorize@NetReplacePart[net,{"input_0"->NetEncoder[{"Image",ImageDimensions@img,"MeanImage"->mean,"VarianceImage"->var}],"Output"->dec}][img,TargetDevice->device]]

VideoApply[segment,v,FileNameJoin[{NotebookDirectory[],"StreetColorized.mp4"}]];

Video[FileNameJoin[{NotebookDirectory[],"StreetColorized.mp4"}]]

Scene Segmentation

v=Video[FileNameJoin[{NotebookDirectory[],"Musician.mp4"}]]

diffs=VideoBlockMap[ImageDistance@@#&,v,{2/fr,1/fr}]

ListLinePlot[diffs,PlotRange->All]

times=FindPeaks[diffs,Automatic,Automatic,150]["Times"]

indices=Round[Information[v,"FrameRate"]times];
paddedindices=Sort@Flatten[Table[indices+i,{i,-5,5}]]

size=Import[Information[v,"Location"],{"ImageSize",1}];
i=1;
VideoApply[If[MemberQ[paddedindices,i++],ConstantImage[Red,size],#]&,v,FileNameJoin[{NotebookDirectory[],"MusicianScenes.mp4"}]];

Video[FileNameJoin[{NotebookDirectory[],"MusicianScenes.mp4"}]]

Trim the first scene from the video:

list=Import[FileNameJoin[{NotebookDirectory[],"Musician.mp4"}],{"ImageList",1,Quantity[times[[1]],"seconds"];;Quantity[times[[2]],"seconds"]}];

Export[FileNameJoin[{NotebookDirectory[],"MusicianScene1.mp4"}],list]

Video[FileNameJoin[{NotebookDirectory[],"MusicianScene1.mp4"}]]

Stabilization

https://www.wolfram.com/language/12/new-in-image-processing/keypoint-based-video-stabilization.html?product=language

v=Video[FileNameJoin[{NotebookDirectory[],"soap_bubble.mp4"}]]

fr=QuantityMagnitude[Import[FileNameJoin[{NotebookDirectory[],"soap_bubble.mp4"}],"FrameRate"][1]];

mask = ;

f=Identity;
VideoFilter[Module[{tmp},tmp=Last@FindGeometricTransform[##,TransformationClass->"Rigid"]&@@ImageCorrespondingPoints[Sequence@@#,MaxFeatures->25,Method->"ORB", Masking-> mask];
f=Composition[tmp,f];
ImagePerspectiveTransformation[#[[2]],f,DataRange->Full,Padding->"Fixed"]]&,v,{2/fr,1/fr},FileNameJoin[{NotebookDirectory[],"soap_stabilized.mp4"}]];

Video[FileNameJoin[{NotebookDirectory[],"soap_stabilized.mp4"}]]

Using FFmpegTools Filters

ColorNegate

v=Video[FileNameJoin[{NotebookDirectory[],"soap_bubble.mp4"}]];

VideoApply[{{"lutrgb","r","negval","g","negval","b","negval"}}, v]

VideoApply[{{"deshake"}}, v]

AbsoluteTiming[VideoApply[{{"lutrgb","r","negval","g","negval","b","negval"}}, v]//VideoQ]

AbsoluteTiming[VideoApply[ColorNegate, v]//VideoQ]

ImageReflect

AbsoluteTiming[VideoApply[{{"vflip"}}, v]//VideoQ]

AbsoluteTiming[VideoApply[ImageReflect, v]//VideoQ]

EdgeDetect

AbsoluteTiming[VideoApply[{{"edgedetect"}}, v]//VideoQ]

AbsoluteTiming[VideoApply[EdgeDetect, v]//VideoQ]

ImageResize

AbsoluteTiming[VideoApply[{{"scale", "w","iw/2","h","ih/2"}}, v]//VideoQ]

AbsoluteTiming[VideoApply[ImageResize[#,Scaled[0.5]]&, v]//VideoQ]

Thanks & Happy Halloween!

Video[FileNameJoin[{NotebookDirectory[],"Vampire - 28388.mp4"}]]