Automated Structure Discovery

Unsupervised Machine Learning

© Wolfram Research, Inc.

Outline

Unsupervised Learning with the Wolfram Language™

Find clusters in data

FindClusters

ClusteringComponents

ClusteringTree

Clusters are potential classes—so it is possible to classify with the help of clustering

ClusterClassify

Directly learn a distribution or formula from data

FindDistribution

Moving to Feature Space

Extract features from data and project to lower dimensions

FeatureExtraction

DimensionReduction

Explore data in a meaningful space

FeatureSpacePlot

FeatureNearest

Clusters in Data

Framed[Manipulate[With[{axislabels = ExampleData[{"MachineLearning","FisherIris"},"VariableDescriptions"][[1]],
data = ExampleData[{"MachineLearning","FisherIris"},"Data"]},ListPlot[FindClusters[data[[All,1,{x,y}]],n, Method->"KMeans"],PlotMarkers->Automatic,AspectRatio -> 1,Axes->False,Frame-> {{True,False},{True,False}},FrameTicks -> Automatic,FrameLabel->{{axislabels[[y]],None},{axislabels[[x]],None}},FrameTicks->None]],
Style[Image[ ,ImageSize->Small]],
Style["Three species of Iris flower:\nSetosa, Versicolor, Virginica", 14],
"",
Delimiter,
"",
Style["Features", 12, Bold],
"",
{{x,1,"Feature 1 (x axis)"},{1->"Sepal Length",2->"Sepal Width",3->"Petal Length",4->"Petal Width"}},
{{y,2,"Feature 2 (y axis)"},{1->"Sepal Length",2->"Sepal Width",3->"Petal Length",4->"Petal Width"}},
"",
Delimiter,
Style["Number of Clusters", 12, Bold],
{{n,2,"k"},1,5,1,ControlType->Setter},
ControlType->PopupMenu,
ControlPlacement->Left,FrameLabel->{{None,None},{None,Style["Fisher's Iris Data: K-Means Clustering",18]}},SynchronousUpdating->False,SaveDefinitions->True],Background -> GrayLevel[.9], BaseStyle -> {FontColor -> GrayLevel[.2]},FrameStyle->GrayLevel[.9],FrameMargins->None]

How Is the Data Organized

FeatureSpacePlot[countryFlags[[All,2]],PlotLabel->Style["Flags of Asian Countries",12,Bold]]

Clustering

FindClusters

The goal is to partition a dataset into clusters of similar elements (all sorts of data: numerical, textual, image, as well as dates and times).

FindClusters[{-10,-9,-8,-7,5,6,7,8}]

The number of clusters can be explicitly specified.

FindClusters[{-10,-9,-8,-7,5,6,7,8},3]

ClusteringComponents

Get the indices of the clusters to which each sample belongs.

FindClusters[{10,4,5,6,11}, 2]

ClusteringComponents[{10,4,5,6,11}, 2]

Nearest

Find the elements closest to a given sample.

Nearest[{1,2,10,12,3,1,13,25},8]

Find the 5 nearest data points to 8.

Nearest[{1,2,10,12,3,1,13,25},8,5]

Customizing How Clusters Are Defined

Option: DistanceFunction

Specify an appropriate distance function for the data.

FindClusters[data,numOfClusters,DistanceFunction->EuclideanDistance]

Euclidean distance for numeric data

Jaccard dissimilarity for Boolean data

Edit distance for string data, etc.

Example data

p1 = Table[RandomReal[{0,10}]{Cos[a=RandomReal[{π/6,π/4}]],Sin[a]},{200}];
p2 = Table[RandomReal[{0,10}]{Cos[a=RandomReal[{((11 π)/24),((13 π)/24)}]],Sin[a]},{200}];
p3 = Table[RandomReal[{0,10}]{Cos[a=RandomReal[{((17 π)/24),((19 π)/24)}]],Sin[a]},{200}];
ListPlot[{p1,p2,p3},PlotStyle->Directive[Gray,PointSize[Medium]]]

Euclidean versus Cosine

TabView[{"Euclidean"->ListPlot[FindClusters[Join[p1,p2,p3],3,DistanceFunction->EuclideanDistance],PlotRange->All,PlotLabel->"Euclidean Distance",PlotMarkers->Automatic],
"Cosine"->ListPlot[FindClusters[Join[p1,p2,p3],3,DistanceFunction->CosineDistance],PlotRange->All,PlotLabel->"Cosine Distance",PlotMarkers->Automatic]}]

Customizing How Clusters Are Created

Option: Method

FindClusters[data,numOfClusters,Method->"DBSCAN"]

Certain clustering algorithms perform better than others, depending on the natural clusters in the data.

SeedRandom[2];
data1 = RandomVariate[MultinormalDistribution[{0,0},{{.1,0},{0,.1}}],1500];
data2 = RandomVariate[UniformDistribution[{{-2,2}, {-2,2}}],500];
test = Join[data1,data2];
ListPlot[{data1,data2},PlotRange->All,PlotStyle->Directive[Gray,PointSize[Medium]]]

K-Means versus DBSCAN

kmClusters = FindClusters[test,2, Method->"KMeans"];dbscanClusters = FindClusters[test, Method->"DBSCAN"];TabView[{"KMeans"->ListPlot[kmClusters,PlotLabel->"KMeans",ImageSize->Medium,PlotMarkers->Automatic],
"DBSCAN"->ListPlot[{First@dbscanClusters,Join@@Rest@dbscanClusters},PlotLabel->"DBSCAN",ImageSize->Medium,PlotMarkers->Automatic]}]

Customizing How Clusters Are Evaluated

Option: CriterionFunction

FindClusters[data,numOfClusters,CriterionFunction->"StandardDeviation"]

The specified CriterionFunction is used to evaluate the clusters obtained, to decide on the best clusters.

SeedRandom[1234];
pts1 = Table[RandomReal[{0,0.15}]{Cos[a=RandomReal[{0,2π}]],Sin[a]},{500}];
pts2 = Table[RandomReal[{0.5,0.65}]{Cos[a=RandomReal[{0,2π}]],Sin[a]},{500}];
ListPlot[{pts1,pts2},PlotRange->Full,PlotStyle-> Directive[Gray,PointSize[Medium]],AspectRatio->1]

Standard deviation versus Calinski–Harabasz index

TabView[#->ListPlot[Quiet@FindClusters[Join[pts1,pts2],CriterionFunction->#, RandomSeeding->0],PlotRange->Full,PlotStyle-> PointSize[Medium],PlotLabel->#,AspectRatio->1]&/@ {"StandardDeviation","CalinskiHarabasz"}]

Clustering Visualizations

ClusteringTree

A weighted tree from the hierarchical clustering of data elements.

colors = RandomColor[30]

ClusteringTree[colors]

ClusteringTree[colors,ClusterDissimilarityFunction->"Centroid"]

Dendrogram

Dendrogram—a tree where the heights of nodes are proportional to intercluster distances.

Dendrogram[colors, ClusterDissimilarityFunction->"Centroid"]

Classification with Clustering

Clusters group data into potential classes. So in the absence of pre-labeled training data, clustering can be used as the basis for assigning labels to new samples.

ClusterClassify

c=ClusterClassify[{-10,-9,-8,-7,5,6,7,8}]

ClassifierInformation[c]

{c[-4],c[10]}

Cluster colors into 3 groups and then classify a new color.

colors=RandomColor[50]

c=ClusterClassify[colors,3]

c[{"White","Purple","Blue","Green","Yellow","Orange","Red"}]

Finding a Distribution from the Data

Given a large amount of data, is it possible to create a simple interpretable model of how the data was generated?

FindDistribution

For a list of possible outcomes from a univariate distribution, FindDistribution attempts to learn the continuous or discrete distribution from the data.

d = MixtureDistribution[{0.3,0.7},{ExponentialDistribution[1],NormalDistribution[5,0.8]}];

data=RandomVariate[d, 1000];

Histogram of the data.

histogram = Histogram[data,30,"PDF"]

Finding a simple distribution to fit this data.

dist =FindDistribution[data]

Comparing the fitted distribution to the histogram.

plot=Plot[PDF[dist,x],{x,0,7},Filling->Axis];
Show[histogram, plot]

Moving to Feature Space

Automate the process of extracting useful features from raw data—find a meaningful and useful representation.

FeatureExtract

Transform data into numeric vectors to:

find distances between samples (for clustering or K nearest neighbors)

apply transformations like standardizations, affine transforms, etc. (while preprocessing data for further analysis).

FeatureExtract[{{1.4,"A"},{1.5,"A"},{2.3,"B"},{5.4,"B"}}]

FeatureExtract[{,, ,,,}]

FeatureExtractorFunction

fe = FeatureExtraction[{{1.4,"A"},{1.5,"A"},{2.3,"B"},{5.4,"B"}}]

fe[{3.2,"A"}]

Moving to Feature Spaces

DimensionReduce

When faced with too many features, projecting data to lower dimensions helps to:

Avoid the "Curse of Dimensionality"—models overfitted on training data perform poorly

Allow intuitive visualization of data points in two or three dimensions

Make models more interpretable

DimensionReduce[{"the cat is grey","my cat is fast","this dog is scary","the big dog"}]

RandomSample[irisData,5]//Column

features=DimensionReduce[irisData[[All,1]],2];
byspecies=GroupBy[Thread[features->irisData[[All,2]]],Last->First];
ListPlot[Values[byspecies], PlotMarkers-> Automatic,PlotLegends->Keys[byspecies]]

Exploring the Feature Space

FeatureSpacePlot

Automating FeatureExtraction and DimensionalityReduction—given any collection of objects, lay them out in an appropriate “feature space” for exploratory analysis.

FeatureSpacePlot uses sophisticated pre-trained feature extractors for specific types of data—like photographs, texts, etc.

creatures

FeatureSpacePlot[Flatten@Values@creatures]

FeatureNearest

Find the training sample that is the closest match in feature space.

testImages ={,,,};

nf = FeatureNearest[Flatten@Values@creatures]

Grid[{#,nf[#][[1]]}& /@ testImages]

Thank You

Initialization

irisData ={{4.7`,3.2`,1.3`,0.2`}->"setosa",{4.6`,3.1`,1.5`,0.2`}->"setosa",{5.`,3.6`,1.4`,0.2`}->"setosa",{5.4`,3.9`,1.7`,0.4`}->"setosa",{5.`,3.4`,1.5`,0.2`}->"setosa",{4.9`,3.1`,1.5`,0.1`}->"setosa",{5.4`,3.7`,1.5`,0.2`}->"setosa",{4.8`,3.4`,1.6`,0.2`}->"setosa",{4.8`,3.`,1.4`,0.1`}->"setosa",{5.7`,4.4`,1.5`,0.4`}->"setosa",{5.4`,3.9`,1.3`,0.4`}->"setosa",{5.4`,3.4`,1.7`,0.2`}->"setosa",{5.1`,3.7`,1.5`,0.4`}->"setosa",{4.6`,3.6`,1.`,0.2`}->"setosa",{5.1`,3.3`,1.7`,0.5`}->"setosa",{4.8`,3.4`,1.9`,0.2`}->"setosa",{5.`,3.`,1.6`,0.2`}->"setosa",{5.`,3.4`,1.6`,0.4`}->"setosa",{5.2`,3.5`,1.5`,0.2`}->"setosa",{4.7`,3.2`,1.6`,0.2`}->"setosa",{4.8`,3.1`,1.6`,0.2`}->"setosa",{5.4`,3.4`,1.5`,0.4`}->"setosa",{5.2`,4.1`,1.5`,0.1`}->"setosa",{4.9`,3.1`,1.5`,0.2`}->"setosa",{5.`,3.2`,1.2`,0.2`}->"setosa",{5.5`,3.5`,1.3`,0.2`}->"setosa",{4.9`,3.6`,1.4`,0.1`}->"setosa",{4.4`,3.`,1.3`,0.2`}->"setosa",{5.`,3.5`,1.3`,0.3`}->"setosa",{4.4`,3.2`,1.3`,0.2`}->"setosa",{5.`,3.5`,1.6`,0.6`}->"setosa",{5.1`,3.8`,1.9`,0.4`}->"setosa",{4.6`,3.2`,1.4`,0.2`}->"setosa",{5.3`,3.7`,1.5`,0.2`}->"setosa",{5.`,3.3`,1.4`,0.2`}->"setosa",{7.`,3.2`,4.7`,1.4`}->"versicolor",{6.4`,3.2`,4.5`,1.5`}->"versicolor",{6.9`,3.1`,4.9`,1.5`}->"versicolor",{5.5`,2.3`,4.`,1.3`}->"versicolor",{6.5`,2.8`,4.6`,1.5`}->"versicolor",{6.3`,3.3`,4.7`,1.6`}->"versicolor",{4.9`,2.4`,3.3`,1.`}->"versicolor",{6.6`,2.9`,4.6`,1.3`}->"versicolor",{5.2`,2.7`,3.9`,1.4`}->"versicolor",{5.9`,3.`,4.2`,1.5`}->"versicolor",{6.`,2.2`,4.`,1.`}->"versicolor",{6.7`,3.1`,4.4`,1.4`}->"versicolor",{5.6`,3.`,4.5`,1.5`}->"versicolor",{5.6`,2.5`,3.9`,1.1`}->"versicolor",{6.3`,2.5`,4.9`,1.5`}->"versicolor",{6.1`,2.8`,4.7`,1.2`}->"versicolor",{6.6`,3.`,4.4`,1.4`}->"versicolor",{6.8`,2.8`,4.8`,1.4`}->"versicolor",{6.7`,3.`,5.`,1.7`}->"versicolor",{5.7`,2.6`,3.5`,1.`}->"versicolor",{5.5`,2.4`,3.8`,1.1`}->"versicolor",{5.5`,2.4`,3.7`,1.`}->"versicolor",{5.8`,2.7`,3.9`,1.2`}->"versicolor",{6.`,2.7`,5.1`,1.6`}->"versicolor",{6.7`,3.1`,4.7`,1.5`}->"versicolor",{6.3`,2.3`,4.4`,1.3`}->"versicolor",{5.6`,3.`,4.1`,1.3`}->"versicolor",{5.5`,2.5`,4.`,1.3`}->"versicolor",{5.5`,2.6`,4.4`,1.2`}->"versicolor",{6.1`,3.`,4.6`,1.4`}->"versicolor",{5.8`,2.6`,4.`,1.2`}->"versicolor",{5.`,2.3`,3.3`,1.`}->"versicolor",{5.7`,2.9`,4.2`,1.3`}->"versicolor",{5.1`,2.5`,3.`,1.1`}->"versicolor",{5.7`,2.8`,4.1`,1.3`}->"versicolor",{5.8`,2.7`,5.1`,1.9`}->"virginica",{6.3`,2.9`,5.6`,1.8`}->"virginica",{4.9`,2.5`,4.5`,1.7`}->"virginica",{7.3`,2.9`,6.3`,1.8`}->"virginica",{6.7`,2.5`,5.8`,1.8`}->"virginica",{7.2`,3.6`,6.1`,2.5`}->"virginica",{6.5`,3.2`,5.1`,2.`}->"virginica",{6.4`,2.7`,5.3`,1.9`}->"virginica",{6.8`,3.`,5.5`,2.1`}->"virginica",{5.7`,2.5`,5.`,2.`}->"virginica",{5.8`,2.8`,5.1`,2.4`}->"virginica",{6.5`,3.`,5.5`,1.8`}->"virginica",{7.7`,2.6`,6.9`,2.3`}->"virginica",{6.`,2.2`,5.`,1.5`}->"virginica",{5.6`,2.8`,4.9`,2.`}->"virginica",{7.7`,2.8`,6.7`,2.`}->"virginica",{6.3`,2.7`,4.9`,1.8`}->"virginica",{6.7`,3.3`,5.7`,2.1`}->"virginica",{6.2`,2.8`,4.8`,1.8`}->"virginica",{6.1`,3.`,4.9`,1.8`}->"virginica",{6.3`,2.8`,5.1`,1.5`}->"virginica",{6.1`,2.6`,5.6`,1.4`}->"virginica",{7.7`,3.`,6.1`,2.3`}->"virginica",{6.4`,3.1`,5.5`,1.8`}->"virginica",{6.`,3.`,4.8`,1.8`}->"virginica",{6.9`,3.1`,5.4`,2.1`}->"virginica",{6.7`,3.1`,5.6`,2.4`}->"virginica",{6.9`,3.1`,5.1`,2.3`}->"virginica",{5.8`,2.7`,5.1`,1.9`}->"virginica",{6.8`,3.2`,5.9`,2.3`}->"virginica",{6.7`,3.`,5.2`,2.3`}->"virginica",{6.3`,2.5`,5.`,1.9`}->"virginica",{6.5`,3.`,5.2`,2.`}->"virginica",{6.2`,3.4`,5.4`,2.3`}->"virginica",{5.9`,3.`,5.1`,1.8`}->"virginica"};

creatures=\[LeftAssociation]"Griffin"->{,,,,,,,},"Centaur"->{,,,,,,,},"Dragon"->{,,,,,,,},"Unicorn"->{,,,,,,,}\[RightAssociation];