Video Processing & Analysis

Shadi Ashnai, Wolfram Research, Inc.

(RowDefault[, (Spacer1), , (Spacer1), ])  #WolframTechConf

ABSTRACT

We have continued development of the video computation in the Wolfram Language full speed to allow for applications ranging from fun but easy video generation to advanced and flexible processing and analysis of videos. In this talk, I will demonstrate what can be done now with videos in the Wolfram Language.

Outline

Video Object

Video Creation

Video Editing

Advanced Processing

Video Analysis

Video Object

The Video object (new in version 12.1) is now a first-class citizen in the Wolfram Language with a lot of functionality to .

Video[ file ]

Create a Video from a local file:

video=Video["ExampleData/fish.mp4"]

Information[video]

Check specific video properties:

Duration[video]

ImageDimensions[video]

AudioChannels[video]

Other Video Sources

Create a Video from a remote file:

Video["http://exampledata.wolfram.com/musician.ogg"]

Video[CloudObject[(HyperlinkURL["https://www.wolframcloud.com/obj/documentation/dog", https://www.wolframcloud.com/obj/documentation/dog])]]

In the Wolfram Data Repository:

ResourceData["Sample Video: Citrus in Water"]

Let’s also switch the video appearance...

SetOptions[Video,Appearance->"Minimal"]

Video Creation

In addition to linking to an existing video file, there are ways to create video files in the Wolfram Language.

From a Manipulate...

ResourceData["Demonstrations Project: Bicycle Wheels Using Curves of Constant Width"]

Video[%]

VideoCombine[{%,SpeechSynthesize[ResourceObject["Demonstrations Project: Bicycle Wheels Using Curves of Constant Width"]["Title"]]}]

From an evolving expression...

n=128;
AnimationVideo[
ImageResize[ImageCrop[
Image[CellularAutomaton[30,{{1},0},Floor[i]]],
{2n-1,n},Top
],Scaled[2]],
{i,1,n,1}
]

From a webcam...

Through VideoStream capabilities, you can record from a stream connected to a screen or a connected imaging device (like the built-in webcam) to capture video.

stream=VideoRecord[$DefaultImagingDevice];
Pause[5]
VideoStop[];

stream["RecordedVideo"]

From navigating on a 2D Scene...

TourVideo[,Table[<|"Position"->Scaled[{.5,1-1/(i+1)}],"Width"->Scaled[1/i]|>,{i,1,5}],AspectRatio->1]

More!

Video Generation
12:30-1pm on Wednesday, October 13  [next up]

https://wtc2021.pathable.co/meetings/virtual/8ZSESS8eY6vaNaqLx

Video Editing

Temporal Editing

Trim, split, join delete in the time axis using VideoTrim, VideoSplit, etc.

video=ResourceData["Sample Video: Citrus in Water"]

Information[video]

VideoTrim[video,{3,5}]

Spatial Editing

Perform frame-by-frame cropping, resizing, reflecting, etc by directly applying many image processing functions such as ImageResize.

ImageCrop[video,{720,720}]

ImageResize[video,300]

Color Processing

Make the video darker, color, create effects, etc by directly applying image functions to every frame.

ColorNegate[video]

ImageEffect[video,"Stippling"]

Combining Videos

Overlay and image on top of a video:

OverlayVideo[video,]

ImageReflect[video,Left]

OverlayVideo[%,]

Create a grid of videos:

small=ImageResize[video,200]

list=Table[ImageEffect[small,e],{e,{"Sepia","MotionBlur","Jitter","Embossing"}}]

GridVideo[list,Spacings->5,Background->White]

Advanced Processing

Processing a video using Neural Networks

This net is an image colorization net.

net=NetModel["ColorNet Image Colorization Trained on Places Data"]

colorize[img_Image]:=Image[Prepend[ArrayResample[NetModel["ColorNet Image Colorization Trained on Places Data"][img],Prepend[Reverse@ImageDimensions@img,2]],ImageData[ColorSeparate[img,"L"]]],Interleaving->False,ColorSpace->"LAB"]

colorize[]

Apply it to frames of Charlie Chaplin's "The Good For Nothing".

file=URLDownload["https://archive.org/download/CC_1914_08_31_TheGoodforNothing/CC_1914_08_31_TheGoodforNothing_512kb.mp4"];

video=Video[file]

trim=VideoTrim[video,{90,100}]

VideoFrameMap[colorize,trim]

Denoising using Neural Networks

This examples shows how to use the “VideoFrames” net encoder to prepare a few subsequent frames of a video for a neural network. Using FastDVDNet, multiple frames at a time are used to perform denoising.

net=NetModel["FastDVDNet Trained on DAVIS Data"]

video=ResourceData["Sample Video: Horse Riding (Noisy)"]

noise=ConstantArray[1/8.,{1,256,256}];

denoised=VideoFrameMap[net[<|"Video"->#,"Noise"->noise|>]&,video,5,1]

Video Analysis

Object Recognition

video=ResourceData["Sample Video: Small Home Interior"]

VideoFrameMap[HighlightImage[#,ImageBoundingBoxes]&,video]

Frame-by-frame Analysis

video=ResourceData["Sample Video: Friends at the Beach"]

VideoMapTimeSeries[Length[FindFaces[#Image]]&,video]

DateListPlot[%]

StarTrek Face Recognition Example

This example trains a face recognizer and performs recognition on frames of a StarTrek promo.

Find training data for a face recognizer:

faces=AssociationMap[Flatten[FindFaces[#,"Image"]&/@WebImageSearch["star trek "<>#]]&,{"Jean-Luc Picard","William Riker","Phillipa Louvois","Data"}];

Image[#,ImageSize->30]&/@faces

recognizer=FaceRecognize[faces]

Now, recognize faces in the StarTrek promo:

file=URLDownload["https://ia802900.us.archive.org/7/items/2000-promo-for-star-trek-the-next-generation/2000%20promo%20for%20Star%20Trek%20-%20The%20Next%20Generation.ia.mp4"];

v=Video[file]

complete=VideoMapList[recognizer[FindFaces[#Image,"Image"]]&,v]/.m_Missing:>"Other";

ListPlot[Catenate[MapIndexed[{First[#2],#1}&,ArrayComponents[%],{2}]],]

Video Classification

There are several video classification networks available from the Wolfram Neural Net Repository. Let’s show how one can use them to analyze a video collection.

net=NetModel["MobileNet-3D V2 Trained on Video Datasets"]

Class information:

NetExtract[net, "Output"][["Labels"]]//Length

Text/@RandomSample[NetExtract[net, "Output"][["Labels"]],5]//Column

Identify the action in one video:

video=ResourceData["Sample Video: Barbecuing"]

net[video]

net[bbq,"TopProbabilities"->10]

Use the net as a feature extractor:

extractor=NetTake[net,{1,-3}]

videos=Join[ResourceData["Tooth Brushing Video Samples"],ResourceData["Cheerleading Video Samples"]];

videos[[{1,-1}]]

FeatureSpacePlot[videos,FeatureExtractor->extractor,LabelingFunction->(Callout[Thumbnail@VideoExtractFrames[#1,Quantity[1,"Frames"]]]&),LabelingSize->50, ImageSize->600]

Before we go...

Video functionality will greatly benefit from FFmpeg installation

Video and all video functions continue to be under active development.

They all remain "Experimental" in 13.0 to allow for design updates.

... DO SEND US FEEDBAK :)

By default Video appearance (most probably) will switch to be “Basic” in 13.0, using an in-notebook player, but...

... DO EXPECT LIMITATIONS!

Based on our tests, the player plays in real time up to 1080p at 30fps on most machines

Video objects linked to local files will show GUIs with missing files.

You need to download corresponding files or re-evaluate results.

Thanks!

AnimationVideo[Graphics[Text[Style["Thanks!",Hue[t/(2π)],32],{Cos[t],Sin[t]Cos[t]}],],{t,0,4π}]//AnimatedImage

Meet-up: Video Processing
10-11am on Thursday, October 14

https://wtc2021.pathable.co/meetings/virtual/DmeCGRSemWCbtYKgm

Initialization

SetOptions[Video,Appearance->"Basic"];

denoised=Video["/Users/shadi/Documents/Wolfram/Video/VideoFrameMap-da1338d4-417f-4364-975b-5c89ab2c2615.mp4"];