Image Understanding Using Deep Neural Networks

Learning to Tackle Real-World Computer Vision Applications

Wolfram Training
training@wolfram.com

Developing Your Application

What You Need and What You Do Not Necessarily Need

Select an existing network architecture and only make tweaks necessary to your application.

Select an already trained network, then either use its features directly or fine-tune it to your data.

Augment your training set by synthesizing new data.

Set up small training jobs on a laptop or workstation with a recent single NVIDIA GPU. Save expensive training in the cloud for big jobs.

Trust the Wolfram Language to choose proper values for the hyperparameters or pick such values from existing publications.

Developing Your Application

Examples

Out-of-core image retrieval that directly uses features from an already trained network

Blind image quality assessment that tweaks an already trained network, with data augmentation

Colorization of black-and-white pictures without user intervention, by training an existing network architecture

TextGrid[{
{"Object and Face Recognition","Action Recognition","Object Classification"},{"Image Segmentation","Object and Face Detection","Scene Parsing"},
{"Optical Character Recognition","Facial Landmarks Detection","Edge Detection"},
{"Microscopy Image Diagnosis","Super-Resolution","Depth Estimation"},
{"Self-driving cars", "Visual Question Answering","..."}},
Spacings->7,ItemStyle->Directive[Gray,Italic]]

({{Object and Face Recognition, Action Recognition, Object Classification}, {Image Segmentation, Object and Face Detection, Scene Parsing}, {Optical Character Recognition, Facial Landmarks Detection, Edge Detection}, {Microscopy Image Diagnosis, Super-Resolution, Depth Estimation}, {Self-Driving Cars, Visual Question Answering, ...}})

Content-Based Image Retrieval

Definition

Given a query image, find the most visually similar images from a dataset.

Lookup[,]->(({{}, {}, {}}))

Content-Based Image Retrieval

Approach: Nearest Features as Output by a Late Layer of an Already Trained General Recognition Network

Grid[{{,"->",,"->",Nearest[RandomInteger[1,{500,1024}]],"->",ImageCrop@ImageAssemble[Transpose@{ImagePad[#,5,White]&/@ConformImages[{,,},Max]}]},{"","",Text@"Stripped network","",Text@"k-NN classifier","",Text@"Nearest neighbors"}}]

Content-Based Image Retrieval

Already Trained General Recognition Network: ImageIdentify

Conceptual visualization of such an underlying network.

ImageIdentify[{,,,,,}]

Content-Based Image Retrieval

Access to High-Level Features by Stripping the Last Layers of the Classification Network

->

Built into the Wolfram Language.

feid=FeatureExtraction[{RandomImage[1,{256,256},ColorSpace->"RGB"]},"ImageFeatures"];
feid[[1,"Processor",2,"Processors"]]=Most@feid[[1,"Processor",2,"Processors"]];
feid

extractImageIdentifyFeatures[image_?ImageQ]:=feid[image];

extractImageIdentifyFeatures[]//Short

Out-of-core version where the image is read from disk on demand.

extractImageIdentifyFeatures[image_File]:=extractImageIdentifyFeatures[Import@First@image];

File[FileNameJoin[{NotebookDirectory[],"data","city.jpg"}]]//extractImageIdentifyFeatures//Short

Content-Based Image Retrieval

Extract Features for Monument Images, Offline and Out of Core

FormObject["Search images (top 500)"-><|"Hint"->"type of images",
"Help" -> "Get the hyperlinks"|>]

urlmonuments//Length

URL/@RandomSample[urlmonuments,4]

Thumbnail/@%

featuremonuments=extractImageIdentifyFeatures/@urlmonuments;

Dimensions@featuremonuments

Content-Based Image Retrieval

Retrieve Top Five Images as Nearest Neighbors in the Feature Space

nf=Nearest[featuremonuments->urlmonuments];

retrieveImage[image_,top_:5]:=Import/@nf[extractImageIdentifyFeatures[image],top];

query=(Entity["Taj Mahal", Entity["Building","TajMahal::zw4mm"], "Entity[\"Building\", \"TajMahal::zw4mm\"]", "building"])[(EntityProperty["image", EntityProperty["Building","Image"], "EntityProperty[\"Building\", \"Image\"]"])]

retrieveImage[query]

Blind Image Quality Assessment

Definition

Given an image, predict its quality, where quality is a score between 1 (bad) and 100 (excellent).

GraphicsGrid[{{,Style[Text@"->",50],Style[Text@75,Blue,50]}} ,BaselinePosition->Center]

References

[1] S. Bianco, L. Celona, P. Napoletano and R. Schettini, “On the Use of Deep Learning for Blind Image Quality Assessment,” arxiv.org/abs/1602.05531.

[2] D. Ghadiyaram and A. C. Bovik, “Massive Online Crowdsourced Study of Subjective and Objective Picture Quality,” IEEE Transactions on Image Processing, 25(1), 2016, pp. 372–387. doi:10.1109/TIP.2015.2500021.

[3] D. Ghadiyaram and A. C. Bovik. “LIVE In the Wild Image Quality Challenge Database.” (Dec 2, 2016) live.ece.utexas.edu/research/ChallengeDB/index.html.

Blind Image Quality Assessment

Approach: Replace the Last Layers of a Classification Network by Training on a Small, Application-Specific Dataset

Grid[{Text/@{,"->",,"->",,"->",Predict[{1->1.3,2->2.4,3->6.4,4->10.1}],"->",Style[75,Bold,Blue]},Text/@{"","","Stripped network","","Added layers","","Regression",""},
Text/@{"","","(1)","","(2)","","(3)",""}}]

Blind Image Quality Assessment

Dataset: LIVE in the Wild

1,162 images assessed by 7,000 different persons

Blind Image Quality Assessment

Data Augmentation: Taking Random Crops within the Image

crop[image_?ImageQ]:=Module[
{img=RemoveAlphaChannel@ImageResize[image,{350,350}],
rc=RandomInteger[{0,50},2]},
Image[
ImageTake[img,rc[[1]]+{1,299},rc[[2]]+{1,299}],
Magnification->.25]
];

Table[crop[],{10}]

Other possibility: horizontal reflection

ImageReflect[,Left->Right]

Blind Image Quality Assessment

(1) Access to Features of an Already Trained Classification Network: Stripping Google’s Inception3

inceptionv3=Import[FileNameJoin[{NotebookDirectory[],"data","inceptionv3.wlnet"}]]

inceptionv3[]

Strip the last layers.

featnet=Drop[inceptionv3,-2]

extractInceptionFeatures[image_?ImageQ]:=featnet[image];

extractInceptionFeatures[]//Short

Blind Image Quality Assessment

(2) Extracting Inception3’s Late-Stage Features on Random Image Crops

Compute for each image of the dataset.

Table[extractInceptionFeatures@crop[],{4}]//Shallow

LIVEinTheWildDatasetInceptionv3Augmented=Import@FileNameJoin[{NotebookDirectory[],"data","LIVEinTheWild-dataset-AugmentedCNNfeatures-nomean.mx"}]

n=Length@LIVEinTheWildDatasetInceptionv3Augmented

No validation test is needed, because you will do cross-validation inside Predict.

ntest=Ceiling[n/5]

SeedRandom[20];
idx=RandomSample[Range[n]];

{testidx,trainingidx}=TakeDrop[idx,ntest];

training=Flatten[MapThread[Thread[#2->#1]&,{Normal@LIVEinTheWildDatasetInceptionv3Augmented[trainingidx,"Class"],Normal@LIVEinTheWildDatasetInceptionv3Augmented[trainingidx,"CropsCNNFeatures"]}],1];

testing=Flatten[MapThread[Thread[#2->#1]&,{Normal@LIVEinTheWildDatasetInceptionv3Augmented[testidx,"Class"],Normal@LIVEinTheWildDatasetInceptionv3Augmented[testidx,"CropsCNNFeatures"]}],1];

Blind Image Quality Assessment

(2) Toward Application-Specific Features: Training Added Layers

addednet=NetChain[{
DotPlusLayer[100],
ElementwiseLayer[Ramp],
DotPlusLayer[5],
SoftmaxLayer[]},
"Input"-> 2048,
"Output"->NetDecoder[{"Class",{"Bad","Poor","Fair","Good","Excellent"}}]
]

Automatic training

trainedaddednet=NetTrain[addednet,training,ValidationSet->testing,TargetDevice->"GPU"]

ClassifierMeasurements[trainedaddednet,testing,"Accuracy"]

ClassifierMeasurements[trainedaddednet,testing,"ConfusionMatrixPlot"]

finetunedfeatureextractor=Drop[trainedaddednet,-1]

extractFineTunedFeatures[cnnfeat_]:=finetunedfeatureextractor[cnnfeat];

extractFineTunedFeatures[training[[1,1]]]

Blind Image Quality Assessment

(3) Training the Final Predictor

training=Flatten[MapThread[Thread[#2->#1]&,{Normal@LIVEinTheWildDatasetInceptionv3Augmented[trainingidx,"MOS"],extractFineTunedFeatures/@Normal@LIVEinTheWildDatasetInceptionv3Augmented[trainingidx,"CropsCNNFeatures"]}],1];
testing=Flatten[MapThread[#2->#1&,{Normal@LIVEinTheWildDatasetInceptionv3Augmented[testidx,"MOS"],extractFineTunedFeatures/@Normal@LIVEinTheWildDatasetInceptionv3Augmented[testidx,"CropsCNNFeatures",1]}],1];

prFT=Predict[training,ValidationSet->testing];

PredictorInformation[prFT]

Evaluating the performance of the end-to-end predictor

pmFT=PredictorMeasurements[prFT,testing];

predictorMeasurementsLCCandSROCC[gt_,pred_]:=
{Correlation[gt,pred],
SpearmanRho[gt,pred]};

lccsrocc=predictorMeasurementsLCCandSROCC[prFT/@testing[[All,1]],testing[[All,2]]];

Labeled[pmFT["ComparisonPlot"],{Text@"LCC"->NumberForm[lccsrocc[[1]],2],Text@"SROCC"->NumberForm[lccsrocc[[2]],2]}]

Blind Image Quality Assessment

Putting It All Together: From an Image to Its Quality

predictImageQuality[image_]:=prFT@extractFineTunedFeatures@extractInceptionFeatures[image];

predictImageQuality[]

Automatic Colorization of Black-and-White Pictures

Definition

Given a luminance channel L, find chrominance channels A and B whose combination forms a realistic color picture.

Grid[{Text/@{,"->",MatrixForm@{{},{}},"->",MatrixForm@{{},{},{}},"->",},Text/@{"Luminance","",Style["Estimated A and B chrominance channels",Bold],"","Combination of L & AB chrominance","",""}}]

References

[1] S. Iizuka, E. Simo-Serra and H. Ishikawa, “Let There Be Color!: Joint End-to-End Learning of Global and Local Image Priors for Automatic Image Colorization with Simultaneous Classification,” ACM Transactions on Graphics (Proceedings of SIGGRAPH 2016), 35(4), 2016. doi:10.1145/2897824.2925974.

[2] S. Giollo. “Image Colorization.” Wolfram Summer School, 2016. education.wolfram.com/summer/school/alumni/2016/giollo.

Automatic Colorization of Black-and-White Pictures

Approach: Training an Existing Network Architecture Using Your Own Data

Constraining an ill-defined luminance-to-color process: prior knowledge about an image’s type

Image[#,ImageSize->{Automatic,200}]&/@{,,}

The shirt color could be any of these.

RandomColor[LABColor[UniformDistribution[{.4,.6}],_,_],20]

In an image of a road, objects are more likely to be red, white, black, navy blue, graphite, ....

Table[
ColorReplace[,White->color,.15],
{color,ColorConvert[RandomColor[LABColor[UniformDistribution[{.8,1}],_,_],5],"RGB"]}]

In an outdoor scene, flat regions on the top are likely to be sky color, while texture on the bottom may point to vegetation colors.

->

Automatic Colorization of Black-and-White Pictures

Approach: Training an Existing Network Architecture Using Your Own Data

Overview of the network architecture

Automatic Colorization of Black-and-White Pictures

Network 1: Low-Level Features

lowlevel=NetChain[{
conv[64,3,2,1],bn,relu,
conv[128,3,1,1],bn,relu,
conv[128,3,2,1],bn,relu,
conv[256,3,1,1],bn,relu,
conv[256,3,2,1],bn,relu,
conv[512,3,1,1],bn,relu}]

trainedlowlevel=NetExtract[trainedcolorizationnet, "LowLev"];

bird=;

lfeat=trainedlowlevel[{ImageData@bird}];

Dimensions@lfeat

Image[#,ImageSize->100]&/@RandomSample[lfeat,10]

Automatic Colorization of Black-and-White Pictures

Network 2: Mid-Level Features

midlevel=NetChain[{
conv[512,3,1,1],bn,relu,
conv[256,3,1,1],bn,relu}]

trainedmidlevel=NetExtract[trainedcolorizationnet, "MidLev"];

mfeat=trainedmidlevel[lfeat];

Dimensions@mfeat

Image[#,ImageSize->100]&/@RandomSample[mfeat,10]

Automatic Colorization of Black-and-White Pictures

Network 3: Global-Level Features

globlevel=NetChain[{
conv[512,3,2,1],bn,relu,
conv[512,3,1,1],bn,relu,
conv[512,3,2,1],bn,relu,
conv[512,3,1,1],bn,relu,
FlattenLayer[],
fc[1024],bn,relu,
fc[512],bn,relu}]

trainedgloblevel=NetExtract[trainedcolorizationnet, "GlobLev"];

gfeat=trainedgloblevel[lfeat];

gfeat//Short

Automatic Colorization of Black-and-White Pictures

Network 4: Fusion of Global- and Mid-Level Features

fusion=NetGraph[
<|
"fc"->fc[256],
"batchn"->bn,
"relu"->relu,
"Reshape"->ReshapeLayer[{256,1,1}],
"Broadcast"->NeuralNetworks`BroadcastPlusLayer[]|>,
{"fc"->"batchn"->"relu"->"Reshape",NetPort["mid"]->NetPort["Broadcast","LHS"],
"Reshape"->NetPort["Broadcast","RHS"],
NetPort["glob"]->NetPort["fc","Input"]}
]

fusiontrained=NetGraph[
<|
"fc"->Take[trainedcolorizationnet,{"GlobLev2","GlobLev2"}],
"Reshape"->ReshapeLayer[{256,1,1}],
"Broadcast"->NeuralNetworks`BroadcastPlusLayer[]|>,
{"fc"->"Reshape",NetPort["mid"]->NetPort["Broadcast","LHS"],
"Reshape"->NetPort["Broadcast","RHS"],
NetPort["glob"]->NetPort["fc","Input"]}
];

ffeat=fusiontrained[<|"mid"->mfeat,"glob"->gfeat|>];

Dimensions@ffeat

ImageAdjust@Image[#,ImageSize->100]&/@RandomSample[ffeat,10]

Automatic Colorization of Black-and-White Pictures

Network 5: Generation of Chrominance Channels

chrom=NetChain[{
conv[256,3,1,1],bn,relu,
conv[128,3,1,1],bn,relu,
upSampl,
conv[64,3,1,1],bn,relu,
conv[64,3,1,1],bn,relu,
upSampl,
conv[32,3,1,1],bn,relu,
conv[2,3,1,1],
ElementwiseLayer[LogisticSigmoid],
upSampl}]

trainedchrom=NetExtract[trainedcolorizationnet, "ColNet"];

cfeat=trainedchrom[ffeat];

Dimensions@cfeat

ImageAdjust@Image[#,ImageSize->100]&/@cfeat

Construct the colored image.

Image[{ImageData@bird,Rescale[cfeat[[1]],{0,1},{-0.8,0.94}],Rescale[cfeat[[2]],{0,1},{-1.13,0.94}]},ColorSpace->"LAB",Interleaving->False]

Automatic Colorization of Black-and-White Pictures

Training: Complete Network with Loss Functions

NetGraph[
<|
"SplitL"->sl,"LowLev"->lln,"MidLev"->mln,"GlobLev"->gln,"GlobLev2"->gln2,"ColNet"->coln,"Sigmoid"->σ1,"TimesL1"->tl1,"TimesL2"->tl2,"CatL"->cl,"LossMS"->lossMS,"LossCE"->lossCE,"Broadcast"->bl,"ReshapeL"->rshL,"ClassN"->classn, "timesLoss"->timesLoss|>,
{
NetPort["Image"]->"SplitL","SplitL"->{"LowLev","TimesL1","TimesL2"},{"TimesL1","TimesL2"}->"CatL","CatL"->"Sigmoid","LowLev"->"MidLev","LowLev"->"GlobLev","GlobLev"->"GlobLev2","GlobLev"->"ClassN","MidLev"->NetPort["Broadcast","LHS"],"GlobLev2"->"ReshapeL","ReshapeL"->NetPort["Broadcast","RHS"],"Broadcast"->"ColNet","ColNet"->NetPort["LossMS","Input"],"Sigmoid"->NetPort["LossMS","Target"],"ClassN"->NetPort["LossCE","Input"],NetPort["Class"]->NetPort["LossCE","Target"],"LossCE"->"timesLoss"},"Image"->NetEncoder[{"Image",{224,224},"ColorSpace"->"LAB","Parallelize"->False}]]

Automatic Colorization of Black-and-White Pictures

Training & Training Data

NetTrain[
colorizationnet,
<|"Image"->$trainPathsFile,"Class"->$trainClasses|>,
ValidationSet-><|"Image"->$testPathsFile,"Class"->$testClasses|>
]

Dataset: 350,000 images from ImageIdentify’s dataset (roughly 5% of the dataset)

Duration: A checkpoint was saved after 5 epochs (14 hours on a single Titan GPU)

Automatic Colorization of Black-and-White Pictures

Putting It All Together: From a Grayscale Image to Its Colorized Version

trainedcolorizationnet=Import[FileNameJoin[{NotebookDirectory[],"data","squared_loss_epoch_5.wlnet"}]]

fullcolnet = NetChain[
{Take[trainedcolorizationnet, {"LowLev", "ColNet"}]},
"Input"->NetEncoder["Image",{224,224},"ColorSpace"->"Grayscale"]]

Clear@colorize;
colorize[img_?ImageQ]:=
Module[{l,a,b,rescale},
rescale[x_,channel_] :=Rescale[x,{0,1},If[channel==="A",{-0.8,0.94},{-1.13,0.94}]];
l=ColorSeparate[img,"L"];
{a,b}=fullcolnet[l];
a=rescale[a,"A"];
b=rescale[b,"B"];
{a,b}=ImageResize[Image@#,ImageDimensions@img]&/@{a,b};
Image[ColorConvert[ColorCombine[{l,a,b},"LAB"],"RGB"],"Byte"]
];

colorize@

Thank you!

colorize/@{,,,,}

Init

urlmonuments=Import[FileNameJoin[{NotebookDirectory[],"data","urlsmonument.mx"}]];
featuremonuments=Import[FileNameJoin[{NotebookDirectory[],"data","featuremonument.mx"}]];
SeedRandom[10];

trainedaddednet=Import[FileNameJoin[{NotebookDirectory[],"data","net2048to5.wlnet"}]];
SeedRandom[10];

$numClasses=4314;
$α=1/300.;
conv[out_Integer,k_Integer,str_Integer,p_Integer]:=ConvolutionLayer[out,k,"Stride"->str,"PaddingSize"->p]; (* Convolution layer *)
fc[n_Integer]:=DotPlusLayer[n]; (* Fully connected layer *)
relu=ElementwiseLayer[Ramp]; (* Ramp activation function *)
σ=ElementwiseLayer[LogisticSigmoid];(* Sigmoid activation function *)
σ1=ElementwiseLayer[LogisticSigmoid];
tl1=NeuralNetworks`ScalarTimesLayer[100];  (* This layer multiplies elementwise the input tensor by a scalar number *)
tl2=NeuralNetworks`ScalarTimesLayer[100];
timesLoss=NeuralNetworks`ScalarTimesLayer[$α];
bn=BatchNormalizationLayer[]; (* Batch Normalizaion layer *)
upSampl=NeuralNetworks`UpsampleLayer[2]; (* Upsampling using the nearest neighbor techique *)
sl=NeuralNetworks`SplitLayer[False];  (* This layer splits the input tensor into its channels *)
cl=CatenateLayer[]; (* This layer catenates the input tensors and outputs a new tensor *)

(* "Fusion" layer *)
rshL=ReshapeLayer[{256,1,1}]; (* This layer reinterprets the input to be an array of the specified dimensions *)
bl=NeuralNetworks`BroadcastPlusLayer[]; (* This layer catenates a vector all along the corresponding dimension of a tensor *)

(* Loss functions *)
lossMS=MeanSquaredLossLayer[]; 
lossCE=CrossEntropyLossLayer["Index"]; 
(* Low-Level Features Network *)
lln=NetChain[{conv[64,3,2,1],bn,relu,conv[128,3,1,1],bn,relu,conv[128,3,2,1],bn,relu,conv[256,3,1,1],bn,relu,conv[256,3,2,1],bn,relu,conv[512,3,1,1],bn,relu} ];
(* Mid-Level Features Network *)
mln=NetChain[{conv[512,3,1,1],bn,relu,conv[256,3,1,1],bn,relu}];
(* Colorization Network *)
coln = NetChain[{conv[256,3,1,1],bn,relu,conv[128,3,1,1],bn,relu,upSampl,conv[64,3,1,1],bn,relu,conv[64,3,1,1],bn,relu,upSampl,conv[32,3,1,1],bn,relu,conv[2,3,1,1],σ,upSampl}];
(* Global Features Network *)
gln=NetChain[{conv[512,3,2,1],bn,relu,conv[512,3,1,1],bn,relu,conv[512,3,2,1],bn,relu,conv[512,3,1,1],bn,relu,FlattenLayer[],fc[1024],bn,relu,fc[512],bn,relu}];
gln2=NetChain[{fc[256],bn,relu}];
(* Classification Network *)
classn=NetChain[{fc[256],bn,relu,fc[$numClasses],bn,relu}];

trainedcolorizationnet=Import[FileNameJoin[{NotebookDirectory[],"data","squared_loss_epoch_5.wlnet"}]];