Neural Networks for Cancer Diagnosis

Tuseeta Banerjee, Research Scientist, Advanced Research Group, Wolfram Research

Join the Conversation #WolframTechConf

Outline

In this talk, we will see how we can apply the advances in neural network research, especially in the field of computer vision, to digital pathology. Tasks in digital pathology range from image segmentation and tissue classification.

Image Segmentation

One architecture for all image segmentation tasks

Unified workflow for all the tasks

Image pre-processing

Implementing the SegNet architecture in wolfram language

Why is the work timely?

Recent growth in machine learning and neural network algorithms especially in fields on computer vision

Availability of hardware (GPU’s) to train more complex and deeper models

Publically accessible medical images through various challenges (https://grand-challenge.org/)

Growing interest in Inter-disciplinary research

https : // arxiv.org/pdf/1702.05747.pdf

Why is the work relevant?

Problem

Most of the algorithms are hand-crafted for particular applications: e.g tubule/cell/nuclei segmentation models

Lots of feature engineering required for hand-crafting these algorithms, which require subject matter expertise

Attempts to create a single model needs task specific image pre-processing (patch generation) : See https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4977982/#ref9

https : // arxiv.org/pdf/1702.05747.pdf

Solution

Aim: To generate an universal architecture that will work on similar tasks (segmentation) and requires same treatment for image pre-processing

Plan:

Deeper and more complex models

Larger input/crop size

Better loss functions to handle class imbalance

Pixel-wise Image Segmentation

SegNet

It is a Convolution neural network trained for Image (semantic pixel-wise) Segmentation

Encoder network is identical to the first 13 layers of VGG-network

The role of the decoder is to map the low resolution encoder to the full input resolution feature for pixel-wise classification

Decoder upsamples the lower resolution using deconvolution layer

Architecture is similar to the paper below

https://arxiv.org/pdf/1511.00561.pdf

Pre-Process Curated Images

Pre-process Curated Images: Code Snippet for Training Dataset

Images

traindatasetone=FileNames[<all files> ]

Get["MXNetLink`"]
Get["MXNetLink`"]//ParallelEvaluate
fastImport=MXNetLink`FastImageImport;

image1=fastImport[traindatasetone[[1]]];

imageparts=Flatten[ImagePartition[ImageResize[image1,2048], 256]];

partImages := Function[
	Block[
		{inputImage, imageParts, fileName, export},
		inputImage = fastImport[#];
		imageParts =Flatten[ImagePartition[ImageResize[inputImage,2048], 256]];
		fileName = FileBaseName[#];
		export = Function[Export[StringJoin["<name of directory>", fileName, ToString[#1], ".png"], #2]];
		MapThread[export, {Range[Length[imageparts]], imageParts}]
	]
]

ParallelMap[partImages,traindatasetone];

Labels

traindatasetone=FileNames[<all files> ]

label1=fastImport[traindatasetone[[1]]];

imageparts=Flatten[ImagePartition[ImageResize[label1,2048], 256]];

partImages := Function[
	Block[
		{inputImage, imageParts, fileName, export},
		inputImage = fastImport[#];
		imageParts =Flatten[ImagePartition[ImageResize[inputImage,2048], 256]];
		fileName = FileBaseName[#];
		export = Function[Export[StringJoin["<name of labels directory>", fileName, ToString[#1], ".png"], #2]];
		MapThread[export, {Range[Length[imageparts]], imageParts}]
	]
]

ParallelMap[partImages,traindatasetone];

Images -> Labels

trainPartImages=FileNames["all part images"];

trainPartLabels=FileNames["all labels images"];

traindata=Thread[Table[fastImport[trainPartImages[[i]]]-> fastImport[trainPartLabels[[i]]],{i,1,Length[trainPartImages]}]];

DumpSave["traindata.mx",traindata];

Constructing the Network

SegNet with Batch Normalization

Encoder

encChain[index_,nLayers_,nChannels_]:=Module[{tags,names,layers},
tags=Table[ToString[index]<>"_"<>ToString[j],{j,nLayers}];names=Append[Flatten[{"conv"<>#,"conv"<>#<>"bn","relu"<>#}&/@tags],"pool"<>ToString[index]];layers=Append[Flatten@Table[{ConvolutionLayer[nChannels,{3,3},"PaddingSize"->{1,1}],BatchNormalizationLayer[],ElementwiseLayer[Ramp]},{nLayers}],PoolingLayer[{2,2},"Stride"->{2,2}]];AssociationThread[names->layers]]

encoder=NetChain[Join[
encChain[1,2,64],
encChain[2,2,128],
encChain[3,3,256],
encChain[4,3,512],
encChain[5,3,512]],"Input"->NetEncoder[{"Image",{256,256},"MeanImage"->ConstantImage[{0.698916,0.50909,0.656692}(*enter the mean image of the dataset here*),{256,256}] }]]

Decoder

decChain[index_,nLayersmax_,nLayersmin_,nChannels_]:=Module[{tags,names,layers,nlayers},
nlayers=nLayersmax-nLayersmin+1;
tags=Table[ToString[index]<>"_"<>ToString[j]<>"_"<>"D",{j,nLayersmax,nLayersmin,-1}];names=Flatten@Append[{"deconv"<>ToString[index],"deconv"<>ToString[index]<>"pad"},Flatten[{"conv"<>#,"conv"<>#<>"bn","relu"<>#}&/@tags]];layers=Flatten@Append[{DeconvolutionLayer[nChannels,{3,3},"Stride"->2,"PaddingSize"->1],PaddingLayer[{{0,0}, {0,1}, {0,1}}]},Flatten@Table[{ConvolutionLayer[nChannels,{3,3},"PaddingSize"->{1,1}],BatchNormalizationLayer[],ElementwiseLayer[Ramp]},{nlayers}]];AssociationThread[names->layers]]

decChain2[index_,nLayersmax_,nLayersmin_,nChannels_]:=Module[{tags,names,layers,nlayers},
nlayers=nLayersmax-nLayersmin+1;
tags=Table[ToString[index]<>"_"<>ToString[j]<>"_"<>"D",{j,nLayersmax,nLayersmin,-1}];names=Flatten[{"conv"<>#,"conv"<>#<>"bn","relu"<>#}&/@tags];layers=Flatten@Table[{ConvolutionLayer[nChannels,{3,3},"PaddingSize"->{1,1}],BatchNormalizationLayer[],ElementwiseLayer[Ramp]},{nlayers}];AssociationThread[names->layers]]

decoder=NetChain[Join[
decChain[5,3,1,512],
decChain[4,3,2,512],
decChain2[4,1,1,256],
decChain[3,3,2,256],
decChain2[3,1,1,128],
decChain[2,2,2,128],
decChain2[2,1,1,64],
decChain[1,2,2,64]]]

Final layers

chain=NetChain[{encoder,decoder,ConvolutionLayer[3,{3,3},"PaddingSize"->1,"Input"-> {64,256,256},"Output"->NetDecoder[{"Image","MeanImage"->ConstantImage[{0.698916,0.50909,0.656692},{256,256}]}]]}]

Stromal - Epithelial Segmentation

Why important?

Stromal (ST) tissue are the includes the fatty and fibrous connective tissues while Epithelial (EP) tissue is the cellular tissue lining and found in the ductal and lobular system of the breast milk ducts. About 80% breast tumors originate in the breast EP cells.

Changes in the stroma tend to drive tumor invasion and metastasis. Tumor - stroma ratio is important in pathology for diagnosing cancer.

Spatial arrangement of stromal cell in tumors is a prognostic factor in breast cancer

Xu, J., Luo, X., Wang, G., Gilmore, H., & Madabhushi, A. (2016). A deep convolutional neural network for segmenting and classifying epithelial and stromal regions in histopathological images. Neurocomputing, 191, 214-223

Dataset

The dataset consist of 42 images ER+ BCa images scanned at 20x. Each image is 1,000 x 1,000. The epithelium region was manually annotated across all images by an expert pathologist. It contains 1735 regions

9 were kept for testing set and the rest for training

The images and the lables (ground truth)  were then partitioned into 16 images to be fed into the network, each of dimensions 256 x 256

Example of a randomly selected training data

traindata[[500]]

Results: Visualizations

GraphicsGrid[Partition[Table[HighlightImage[fullimage[[i]],{{EdgeForm[AbsoluteThickness[3]],trainedfullimage[[i]]},{EdgeForm[AbsoluteThickness[0.5]],RGBColor[0,0,1],labelfullimage[[i]]}}],{i,Length@trainp}],4],ImageSize->1500]

Results: Accuracy Calculations

Pixel-wise accuracy

Trained and test images are converted to Grayscale

Post-processing filtering for trained images to smooth the edges and diminish the noise

trainedimages2=Table[ColorConvert[Round@MedianFilter[trainedimages[[i]],7],"GrayScale"],{i,1,Length[testPartImages]}];
testlabels=Table[ColorConvert[testlabels[[i]],"GrayScale"],{i,1,Length@testPartLabels}];

Accuracy Calculations

accuracy=Table[1.0-ImageMeasurements[ImageDifference[trainedimages2[[i]],testlabels[[i]]],"Mean"],{i,1,Length[testPartImages]}]

Basic statistical analysis

Histogram[accuracy]

Through[{Mean,Variance,StandardDeviation,Median, Skewness, Kurtosis}[accuracy]]

Outlier treatment

Needs["StatisticalPlots`"]

BoxWhiskerPlot[accuracy,BoxOutliers-> All]

Through[{WinsorizedMean,WinsorizedVariance,QuartileDeviation,QuartileSkewness}[accuracy]]

Tubule Segmentation

Why is it important?

Tubule structure and organization is correlated with the aggressiveness of the cancer

The Nottingham breast cancer grading criteria is based on tubule area to the power field of view

Thus tubule segmentation will lead to (a) automatic area estimation which in turn decreases the error in inter/intra reader estimation (b) greater specificity

Why is it difficult?

Tubules are the most complex structures. They have various components  (e.g., nuclei, epithelium and lumen)  and the structure of each of the individual elements determines the tubule boundaries.

Large variance in tubule structures depending on the aggressiveness of the cancer

Benign cases: the tubules are well-organized fashion with similar size and morphological properties

Cancer cases: the organizational structure breaks and identifying the boundaries become exceedingly difficult even for experts

Tubules as an entity are much larger compared to their individual components, thus requiring a greater viewing area to provide sufficient context to make an accurate assessment.

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4977982/#ref9

Dataset

The dataset consist of 85 images ColoRectal images scanned at 40x., where each image is 775 x 522. The epithelium region was manually annotated across all images by an expert pathologist.

It contains a total of 795 regions

Benign and malignant tubules was separately considered

Benign: Dataset and Result Visualizations

Dataset

4 were kept for testing set and 33 for training

The images and the lables (ground truth)  were then partitioned into 16 images to be fed into the network, each of dimensions 256 x 256

Example of a randomly selected training data

traindata[[100]]

Results

GraphicsRow[Table[HighlightImage[fullimage[[i]],{{EdgeForm[AbsoluteThickness[3]],trainedfullimage[[i]]},{EdgeForm[AbsoluteThickness[0.5]],RGBColor[0,0,1],labelfullimage[[i]]}}],{i,Length@trainp}],ImageSize->1500]

Benign:  Pixel-wise Accuracy

accuracy=Table[1.0-ImageMeasurements[ImageDifference[trainedimages[[i]],testlabels[[i]]],"Mean"],{i,1,Length[testPartImages]}]

Through[{Mean,Variance,StandardDeviation,Median, Skewness, Kurtosis}[accuracy]]

Histogram[accuracy]

Needs["StatisticalPlots`"]

BoxWhiskerPlot[accuracy,BoxOutliers-> All]

Through[{WinsorizedMean,WinsorizedVariance,QuartileDeviation,QuartileSkewness}[accuracy]]

Malignant: Dataset and Result Visualizations

Dataset

5 images were kept for testing set and 43 for training

The images and the lables (ground truth)  were then partitioned into 16 images to be fed into the network, each of dimensions 256 x 256

Example of a randomly selected training data

testdata[[11]]

Results

GraphicsRow[Table[HighlightImage[fullimage[[i]],{{EdgeForm[AbsoluteThickness[3]],trainedfullimage[[i]]},{EdgeForm[AbsoluteThickness[0.5]],RGBColor[0,0,1],labelfullimage[[i]]}}],{i,Length@trainp}],ImageSize->1500]

Malignant:  Pixel-wise Accuracy

This is a text cell.

accuracy=Table[1.0-ImageMeasurements[ImageDifference[trainedimages[[i]],testlabels[[i]]],"Mean"],{i,1,Length[testPartImages]}]

Through[{Mean,Variance,StandardDeviation,Median, Skewness, Kurtosis}[accuracy]]

Histogram[accuracy]

Needs["StatisticalPlots`"]

BoxWhiskerPlot[accuracy,BoxOutliers-> All]

Through[{WinsorizedMean,WinsorizedVariance,QuartileDeviation,QuartileSkewness}[accuracy]]

Conclusion and Future Work

Results are already as good or better than existing literature for tasks performed (in terms of accuracy)

Results also match our intuition for the difficulty of the tasks

Complete the work for nuclei and invasive ductal carcinoma segmentation

Explore other segmentation models available in the Wolfram Neural Net Repository

Move to other tasks (mitosis detection, tissue classifcation) and come up with one-stop model designed for all.