Neural Networks for Computer Vision

Computer Vision tasks

Introduction

Two major goals:

A computer vision application

A multi-task learning example

Understanding training failures

Task

Image of digits and plus sign:

Semantic Segmentation:

pixel-level classification

Source: https://devblogs.nvidia.com/parallelforall/image-segmentation-using-digits-5/

Addition:

Regression

Character Counting:

Classification

\[LeftAssociation]"Input"->,"TargetSum"->91,"TargetCount"->4,"TargetMask"->\[RightAssociation]

Dataset Generation (Run Entire Block)

dictionary=Join[{" ","+"},ToString/@Range[0,9]];
classNum=Length[dictionary];

imgMap=AssociationThread[dictionary,Rasterize[#,RasterSize->Floor[{8,18}]]&/@dictionary];
class=0;
maskMap=Map[1+class++*(1-ImageData[Binarize[#,0.7]])&,imgMap];

colorRules=Thread[Range[classNum]->Prepend[RandomColor[classNum-1],White]];
legend=Thread[dictionary->colorRules[[All,2]]];

createDataset[nExamples_,maxDigits_]:=Block[{lhs,rhs,countList,resultList,stringList,imgList,maskList,dataset},{lhs,rhs}=RandomInteger[{1,10^maxDigits-1},{2,nExamples}];
countList=Map[Length@*IntegerDigits,lhs]+Map[Length@*IntegerDigits,rhs]+1;
resultList=lhs+rhs;
stringList=MapThread[StringPadRight[StringJoin[ToString@#1,"+",ToString@#2],2*maxDigits+1]&,{lhs,rhs}];
imgList=Map[ImageAssemble[Lookup[imgMap,#]]&,Characters[stringList]];
maskList=Map[MapThread[Join,Lookup[maskMap,#]]&,Characters[stringList]];
dataset=MapThread[<|"Input"->#1,"TargetSum"->#2,"TargetCount"->#3,"TargetMask"->#4|>&,{imgList,resultList,countList,maskList}];
AssociationThread[{"TrainingData","ValidationData","TestData"},Dataset/@TakeList[dataset,{Floor[nExamples*0.85],Floor[nExamples*0.1],All}]]]

Look at data

Generate the dataset:

datasetTemp=createDataset[500,2];
{testData,trainData,valData}=Lookup[datasetTemp,{"TestData","TrainingData","TestData"}];

RandomSample[trainData,2]

Convert the masks to images:

MapAt[ArrayPlot[#,ColorRules->colorRules]&,RandomSample[trainData,10],{All,Key["TargetMask"]}]

Sum Task

Define an architecture:

sumNet=NetChain[{
ConvolutionLayer[8,1],Ramp,
ConvolutionLayer[8,3,"Stride" -> 2,"PaddingSize"->1],Ramp,
ConvolutionLayer[16,1],Ramp,
ConvolutionLayer[8,1],Ramp,
ConvolutionLayer[8,3,"PaddingSize"->1],Ramp,
ConvolutionLayer[16,1],Ramp,
BatchNormalizationLayer[],
ConvolutionLayer[8,1],Ramp,
ConvolutionLayer[8,3,"PaddingSize"->1],Ramp,
ConvolutionLayer[16,1],Ramp,
ConvolutionLayer[8,1],Ramp,
ConvolutionLayer[8,3,"Stride"->2,"PaddingSize"->1],Ramp,
ConvolutionLayer[16,1],Ramp,
ConvolutionLayer[8,1],Ramp,
ConvolutionLayer[8,3,"Stride"->2,"PaddingSize"->1],Ramp,
ConvolutionLayer[16,1],Ramp,
FlattenLayer[],
LinearLayer[]
},
"Input"->NetEncoder[{"Image",{40,18},"Grayscale"}],
"Output"->NetDecoder[{"Scalar"}]
]

Train the net:

sumTrainSet=Normal[trainData[[All,"Input"]]]->Normal[trainData[[All,"TargetSum"]]];
sumValSet=Normal[valData[[All,"Input"]]]->Normal[valData[[All,"TargetSum"]]];

trainedSumnet=NetTrain[sumNet,sumTrainSet,ValidationSet->sumValSet,MaxTrainingRounds->200]

Get 4 samples from the test set:

sample=RandomSample[testData,4];
testIms=Normal@sample[[All,"Input"]]
Normal@sample[[All,"TargetSum"]]

Evaluate trained net on test images:

trainedSumnet[testIms]

Counting Task

Define an architecture:

countNet=NetChain[{
ConvolutionLayer[8,1],Ramp,
ConvolutionLayer[8,3,"Stride" -> 2,"PaddingSize"->1],Ramp,
ConvolutionLayer[16,1],Ramp,
ConvolutionLayer[8,1],Ramp,
ConvolutionLayer[8,3,"PaddingSize"->1],Ramp,
ConvolutionLayer[16,1],Ramp,
BatchNormalizationLayer[],
ConvolutionLayer[8,1],Ramp,
ConvolutionLayer[8,3,"PaddingSize"->1],Ramp,
ConvolutionLayer[16,1],Ramp,
ConvolutionLayer[8,1],Ramp,
ConvolutionLayer[8,3,"Stride"->2,"PaddingSize"->1],Ramp,
ConvolutionLayer[16,1],Ramp,
ConvolutionLayer[8,1],Ramp,
ConvolutionLayer[8,3,"Stride"->2,"PaddingSize"->1],Ramp,
ConvolutionLayer[16,1],Ramp,
FlattenLayer[],
LinearLayer[],
SoftmaxLayer[]
},
"Input"->NetEncoder[{"Image",{40,18},"Grayscale"}],
"Output"->NetDecoder[{"Class",{3,4,5}}]
]

Train the net:

countTrainSet=Normal[trainData[[All,"Input"]]]->Normal[trainData[[All,"TargetCount"]]];
countValSet=Normal[valData[[All,"Input"]]]->Normal[valData[[All,"TargetCount"]]];

trainedCountNet=NetTrain[countNet,countTrainSet,ValidationSet->countValSet,MaxTrainingRounds->200]

Get 4 samples from the test set:

testIms=Normal@RandomSample[testData,4][[All,"Input"]]

Evaluate trained net on test images:

trainedCountNet[testIms]

Create a ClassifierMeasurements and query it for properties:

cm=ClassifierMeasurements[trainedCountNet,Normal[testData[[All,"Input"]]]->Normal[testData[[All,"TargetCount"]]]]

cm["Accuracy"]

cm["ConfusionMatrixPlot"]

Segmentation

Define an architecture:

segNet=NetChain[{
ConvolutionLayer[16,1],Ramp,
ConvolutionLayer[16,3,"Stride" -> 2,"PaddingSize"->1],Ramp,
ConvolutionLayer[32,1],Ramp,
ConvolutionLayer[16,1],Ramp,
ConvolutionLayer[16,3,"PaddingSize"->1],Ramp,
ConvolutionLayer[32,1],Ramp,
BatchNormalizationLayer[],
ConvolutionLayer[16,1],Ramp,
ConvolutionLayer[16,3,"PaddingSize"->1],Ramp,
ConvolutionLayer[32,1],Ramp,
ConvolutionLayer[16,1],Ramp,
DeconvolutionLayer[16,{3, 2},"Stride"->2, "PaddingSize"->1],Ramp,
ConvolutionLayer[32,1],Ramp,
PaddingLayer[{{0,0},{0,1},{1,1}}],
(* be careful: no ramp after this! *)
ConvolutionLayer[classNum,1],
TransposeLayer[{1->3,1->2}],
SoftmaxLayer[]
},
"Input"->NetEncoder[{"Image",{40,18},"Grayscale"}]
(* no NetDecoder for segmentation yet. Coming soon! *)
]

Train the net:

segTrainSet=Normal[trainData[[All,"Input"]]]->Normal[trainData[[All,"TargetMask"]]];
segValSet=Normal[valData[[All,"Input"]]]->Normal[valData[[All,"TargetMask"]]];

trainedSegNet=NetTrain[segNet,segTrainSet,ValidationSet->segValSet,MaxTrainingRounds->300,TargetDevice->"CPU"]

Get 4 samples from the test set:

sample=RandomSample[testData,4];
testIms=Normal@sample[[All,"Input"]]
Map[ArrayPlot[#,ColorRules->colorRules]&,Normal@sample[[All,"TargetMask"]]]

Evaluate trained net on test images:

pixelPredict[array_]:=Map[First@Ordering[#,-1]&,array,{2}]

Map[ArrayPlot[pixelPredict@#,ColorRules->colorRules]&,trainedSegNet[testIms]]

Training Failures

Overfitting

Sum Net

Train the net:

sumTrainSetSmall=Normal[trainData[[1;;10,"Input"]]]->Normal[trainData[[1;;10,"TargetSum"]]];

NetTrain[sumNet,sumTrainSetSmall,ValidationSet->sumValSet,MaxTrainingRounds->2000]

Regression Example

meanSquaredError[input_,net_]:=Mean[(input[[;;,2]]-net[input[[;;,1]]])^2]

$trainData=#->(Sin[#]+RandomVariate[NormalDistribution[0,0.09]])&/@RandomReal[{0,5},20];
$testData=#->(Sin[#]+RandomVariate[NormalDistribution[0,0.09]])&/@RandomReal[{0,5},20];
plot=ListPlot[List@@@$trainData,Filling->Axis]

net=NetChain[{LinearLayer[300],Tanh,LinearLayer[200],Tanh,LinearLayer[100],Tanh,LinearLayer[1]}]

trainedNet=NetTrain[net,$trainData,MaxTrainingRounds->10000]

Show[plot,Plot[trainedNet[x],{x,0,5},PlotStyle->Black],Plot[Sin[x],{x,0,5},PlotStyle->Red]]

How do you know overfitting is happening?

NetTrain[net,$trainData,Automatic,"LossEvolutionPlot",ValidationSet->$testData]

Compare mean-squared error of test set versus training set:

meanSquaredError[$trainData,trainedNet]

meanSquaredError[$testData,trainedNet]

Solution 1: Reduce net capacity

littleNet=NetChain[{LinearLayer[3],Tanh,LinearLayer[1]}]

trainedLittleNet=NetTrain[littleNet,$trainData,MaxTrainingRounds->10000]

Visualize

Show[plot,Plot[trainedLittleNet[x],{x,0,5},PlotStyle->Black],Plot[Sin[x],{x,0,5},PlotStyle->Red]]

Train with validation set:

NetTrain[littleNet,$trainData,Automatic,"LossEvolutionPlot",ValidationSet->$testData]

Compare mean-squared error of test set versus training set:

meanSquaredError[$trainData,trainedLittleNet]

meanSquaredError[$testData,trainedLittleNet]

Downsides

hard to tweak net for perfect capacity

Solution 2: Dropout

dropoutNet=NetChain[{LinearLayer[300],Tanh,DropoutLayer[0.3],LinearLayer[200],Tanh,DropoutLayer[0.3],LinearLayer[100],Tanh,DropoutLayer[0.3],LinearLayer[1]}]

trainedDropNet=NetTrain[dropoutNet,$trainData,MaxTrainingRounds->10000]

Visualize

Show[plot,Plot[trainedDropNet[x],{x,0,5},PlotStyle->Black],Plot[Sin[x],{x,0,5},PlotStyle->Red]]

Train with validation set:

NetTrain[dropoutNet,$trainData,Automatic,"LossEvolutionPlot",ValidationSet->$testData]

Compare mean-squared error of test set versus training set:

meanSquaredError[$trainData,trainedDropNet]

meanSquaredError[$testData,trainedDropNet]

Downsides

Slows down training (often need to increase MaxTrainingRounds)

Solution 3: L2 Regularization

trainedL2Net=NetTrain[net,$trainData,Method->{"ADAM","L2Regularization"->0.02}]

Visualize

Show[plot,Plot[trainedL2Net[x],{x,0,5},PlotStyle->Black],Plot[Sin[x],{x,0,5},PlotStyle->Red]]

Train with validation set:

NetTrain[net,$trainData,Automatic,"LossEvolutionPlot",ValidationSet->$testData,Method->{"ADAM","L2Regularization"->0.02}]

Compare mean-squared error of test set versus training set:

meanSquaredError[$trainData,trainedL2Net]

meanSquaredError[$testData,trainedL2Net]

Pros and Cons

Hard to set good value

Not popular anymore

dropout more popular

Solution 4: ‘Early Stopping’

Use ValidationSet in NetTrain, and automatically select best net throughout training

trainedEarlyNet=NetTrain[net,$trainData,MaxTrainingRounds->10000,ValidationSet->$testData]

Visualize

Show[plot,Plot[trainedEarlyNet[x],{x,0,5},PlotStyle->Black],Plot[Sin[x],{x,0,5},PlotStyle->Red]]

Compare mean-squared error of test set versus training set:

meanSquaredError[$trainData,trainedEarlyNet]

meanSquaredError[$testData,trainedEarlyNet]

Downsides:

Sometimes overfits before validation set loss is calculated (need to tweak)

Solution 5: More data!

The obvious solution if you can get more data!

Usually you can’t do this easily

One exception: data augmentation!

Image Augmentation

The obvious solution if you can get more data!

im=;

ImageReflect[im,Left->Right]

Rotations, crops, etc

Can use:

ImageAugmentationLayer[{100,100}]

Recommendation

Net capacity

make sure it is big enough that it would normally overfit

Use ~10% of training data for ValidationSet (for early stopping)

unless: you have truly tiny dataset (<30 examples)

NetTrain[net,ValidationSet->Scaled[0.1]]

(Optional): Try add dropout

if you don’t mind slowing down training

Sometimes improves performance on ValidationSet

(Optional): Data Augmentation

if possible, use data augmentation

Non-Normalized Data

newSumNet=NetChain[{
ConvolutionLayer[8,1],Ramp,
ConvolutionLayer[8,3,"Stride" -> 2,"PaddingSize"->1],Ramp,
ConvolutionLayer[16,1],Ramp,
ConvolutionLayer[8,1],Ramp,
ConvolutionLayer[8,3,"PaddingSize"->1],Ramp,
ConvolutionLayer[16,1],Ramp,
ConvolutionLayer[8,1],Ramp,
ConvolutionLayer[8,3,"PaddingSize"->1],Ramp,
ConvolutionLayer[16,1],Ramp,
ConvolutionLayer[8,1],Ramp,
ConvolutionLayer[8,3,"Stride"->2,"PaddingSize"->1],Ramp,
ConvolutionLayer[16,1],Ramp,
ConvolutionLayer[8,1],Ramp,
ConvolutionLayer[8,3,"Stride"->2,"PaddingSize"->1],Ramp,
ConvolutionLayer[16,1],Ramp,
FlattenLayer[],
LinearLayer[]
},
"Output"->NetDecoder[{"Scalar"}]
];

Add 100 to all pixel values:

addition=100;
sumTrainDataNorm=MapAt[{First@ImageData[#,Interleaving->False]+addition}&,trainData,{All,1}];
sumValDataNorm=MapAt[{First@ImageData[#,Interleaving->False]+addition}&,valData,{All,1}];

Train the “sum net” on this shifted dataset:

sumTrainSetNorm=Normal[sumTrainDataNorm[[All,"Input"]]]->Normal[sumTrainDataNorm[[All,"TargetSum"]]];
sumValSetNorm=Normal[sumValDataNorm[[All,"Input"]]]->Normal[sumValDataNorm[[All,"TargetSum"]]];

sumNetTrained=NetTrain[
newSumNet,
sumTrainSetNorm,ValidationSet->sumValSetNorm,MaxTrainingRounds->200,TargetDevice->"CPU"]

Recommendations

Normalize data with Mean of 0 and Variance 1.

inserting BatchNormalization after convolution helps a great deal

Pooling + Class imbalance

Get pooling net with extra PoolingLayer:

countNetPool=NetChain[{
PoolingLayer[3,"Stride"->2,"PaddingSize"->1],
ConvolutionLayer[16,1],Ramp,
ConvolutionLayer[16,3,"PaddingSize"->1],Ramp,
ConvolutionLayer[32,1],Ramp,
ConvolutionLayer[16,1],Ramp,
ConvolutionLayer[16,3,"PaddingSize"->1],Ramp,
ConvolutionLayer[32,1],Ramp,
ConvolutionLayer[16,1],Ramp,
ConvolutionLayer[16,3,"Stride"->2,"PaddingSize"->1],Ramp,
ConvolutionLayer[32,1],Ramp,
ConvolutionLayer[16,1],Ramp,
ConvolutionLayer[16,3,"Stride"->2,"PaddingSize"->1],Ramp,
ConvolutionLayer[32,1],Ramp,
FlattenLayer[],
LinearLayer[3],
SoftmaxLayer[]
},
"Input"->NetEncoder[{"Image",{40,18},"Grayscale"}],
"Output"->NetDecoder[{"Class",{3,4,5}}]
]

Get data:

countTrainSet=Normal[trainData[[All,"Input"]]]->Normal[trainData[[All,"TargetCount"]]];
countValSet=Normal[valData[[All,"Input"]]]->Normal[valData[[All,"TargetCount"]]];

Train the net:

countNetPoolTrained=NetTrain[countNetPool,countTrainSet,ValidationSet->countValSet,MaxTrainingRounds->200,TargetDevice->"CPU"];

Obtain a classifier measurements object

cm=ClassifierMeasurements[countNetPoolTrained,countTestData]

Accuracy is not bad:

cm["Accuracy"]

But:

cm["ConfusionMatrixPlot"]

Useful Info

Oversampling is an option, see: "A systematic study of the class imbalance problem in convolutional neural networks"

https://arxiv.org/abs/1710.05381

Summary of Things Going Wrong

Overfitting

Early stopping with validation set mostly solves this

Class imbalance can mask bad training

oversampling is an option

Architectural choices prevent training happening

Pooling too aggressively at the beginning can destroy training

many other scenarious possible (eg put Ramp at end of net for Regression)

Unnormalized Training Data

The learning rates, intialization procedure, etc. are all tuned under assumption of normalized data

Multitask Learning

Architecture

sharedStem=NetChain[{
ConvolutionLayer[16,1],Ramp,
ConvolutionLayer[16,3,"Stride" -> 2,"PaddingSize"->1],Ramp,
ConvolutionLayer[32,1],Ramp,
ConvolutionLayer[16,1],Ramp,
ConvolutionLayer[16,3,"PaddingSize"->1],Ramp,
ConvolutionLayer[32,1],Ramp,
ConvolutionLayer[16,1],Ramp,
ConvolutionLayer[16,3,"PaddingSize"->1],Ramp,
ConvolutionLayer[32,1],Ramp
}]

nonsegBranch=NetChain[{
ConvolutionLayer[16,1],Ramp,
ConvolutionLayer[16,3,"Stride"->2,"PaddingSize"->1],Ramp,
ConvolutionLayer[32,1],Ramp,
ConvolutionLayer[16,1],Ramp,
ConvolutionLayer[16,3,"Stride"->2,"PaddingSize"->1],Ramp,
ConvolutionLayer[32,1],Ramp,
FlattenLayer[]
}]

segBranch=NetChain[{
ConvolutionLayer[16,1],Ramp,
DeconvolutionLayer[16,{3, 2},"Stride"->2, "PaddingSize"->1],Ramp,
ConvolutionLayer[32,1],Ramp,
PaddingLayer[{{0,0},{0,1},{1,1}}],Ramp,
ConvolutionLayer[classNum,1],
TransposeLayer[{1->3,1->2}],
SoftmaxLayer[]
}
]

multiNet=NetGraph[
<|"shared"->sharedStem,"nonsegBranch"->nonsegBranch,"segBranch"->segBranch,"count"->NetChain[{LinearLayer[3],SoftmaxLayer[]}],"sum"->NetChain[{LinearLayer[1],PartLayer[1]}]|>,{"shared"->{"nonsegBranch","segBranch"},"nonsegBranch"->{"count","sum"},"segBranch"->NetPort["Segmentation"],"count"->NetPort["Count"],"sum"->NetPort["Sum"]},
"Input"->NetEncoder[{"Image",{40,18},"Grayscale"}],
"Count"->NetDecoder[{"Class",{3,4,5}}]
]

multiNetTrain=NetGraph[{multiNet,CrossEntropyLossLayer["Index"],CrossEntropyLossLayer["Index"],MeanSquaredLossLayer[]},{NetPort[1,"Segmentation"]->2,NetPort["TargetMask"]->NetPort[2,"Target"],NetPort[1,"Count"]->3,NetPort["TargetCount"]->NetPort[3,"Target"],NetPort[1,"Sum"]->4,NetPort["TargetSum"]->NetPort[4,"Target"]},
"TargetCount"->NetEncoder[{"Class",{3,4,5}}]
]

Training & evaluation

Train the multi loss net:

trainedMultiNet=NetTrain[multiNetTrain,trainData,ValidationSet->valData,MaxTrainingRounds->500,TargetDevice->"CPU"];

Extract the evaluation part:

evalMultiNet=NetReplacePart[
NetExtract[trainedMultiNet,1],
{"Input"->NetEncoder[{"Image",{40,18},"Grayscale"}],"Count"->NetDecoder[{"Class",{3,4,5}}]}
]

sample=RandomSample[testData,4];
MapAt[ArrayPlot[#,ColorRules->colorRules]&,sample,{All,Key["TargetMask"]}]
MapAt[ArrayPlot[pixelPredict@#,ColorRules->colorRules]&,evalMultiNet[Normal@sample[[All,"Input"]]],{Key["Segmentation"],All}]

How to automatically weigh multi-task losses:

“Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics”, https://arxiv.org/pdf/1705.07115.pdf, 2017

Extra Reading

One of the best resources for learning about convnets for image processing:

CS231n Convolutional Neural Networks for Visual Recognition

http://cs231n.github.io/