Apples and Oranges: Teaching Computational Thinking via Colorimetry

Flip Phillips
Skidmore College -> Rochester Institute of Technology

ABSTRACT

In our “Computational Methods for Psychology and Neuroscience” course, we teach undergraduate students the fundamentals of computational thinking (as opposed to traditional “programming”) using a project-based approach. Over the years project topics have ranged from linguistics, video image analysis, Dynamic[] driven data collection,  analysis and presentation, machine learning, and beyond. Most recently, we chose colorimetry and psychophysics as our project  theme. Using the Connected Devices framework and an Arduino for data collection, we build a machine learning model from publicly available hyperspectral data that could reliably discriminate fruit types from simple, low-dimensional spectral scans.

The resulting project was well received by students and covered a broad range of topics that are useful in neuroscience including: procedural programming of the Arduino, basic electronics, sensor based data acquisition, functional programming in Wolfram Language, instrument calibration, analysis, visualization, and machine learning. Here we discuss the various challenges and successes in this 15-week class.

Biting off more…

Offering this class at Skidmore since 1998

Specialized software (RSVP, NN software, eel, World Building)

Robots (Lego Logo)

Python

Mathematica

Analysis & modeling

Stimulus synthesis

Experiment control

Data acquisition

Psychophysics

WolframAlpha["Gustav Fechner",IncludePods->{"Image:PeopleData","BasicInformation:PeopleData"},AppearanceElements->{"Pods"}]

ListAnimate[(ImageResize[#1,Scaled[2]]&)/@Import["~/Desktop/mof/*.jpg"],AnimationDirection->ForwardBackward,AnimationRunning->False,ContentSize->800,Deployed->True,AppearanceElements->{"PlayPauseButton"},
Paneled-> False, FrameMargins-> 0,ControlPlacement->Right]

Previous projects

More details at my blog at flipphillps.com

Cartoon Face Recognition

Rat Behavior Monitoring

Evolution of a Sensing Creature

Simulating Visual Systems of Other Animals

Modeling Prosopagnosia

Predicting Voter Behavior*

Simulating an Asteroid Strike*

Spring 2019 Project

Goals

Project-based familiarity with ‘computation’

Programming

Data

Hardware

situated in psychology & neuroscience.

Programming

You do not become a ‘programmer’ nor do you ‘learn a language’ in a single class / semester / year / career.

Data

Getting it, manipulating it, looking at it, reasoning about it.

Hardware

Controlling it, not throwing it out the window.

Spring 2019 Project - Identifruit™

Goal

Teach a program to distinguish fruit from spectral information.

Phases

Collect spectral data paired with category data.

Teach ML.

Deployment.

Collect spectral data paired with category data

Hardware

Arduino
Spectral sensor
Case
Buttons
Indicator

Firmware

Trigger scan.

Store scan data wolfram cloud
	6-channels of spectra
	Category
	Concordance - hygiene, spectral, repeated entry
	Indicator - LEDs, Voice output, screen, electrical shock, destructive testing

Location services based anti-theft

Calibration

Diagnostics

Documentation

Operating instructions

Advertising / attract mode

Theory of operation

Observation

Watch and see... timestamps

Teach ML

Convert scan data to Dataset for training

Design network

Feature detection... 
	6 channels
	Ambient light - structure
	Weight
	Temperature
	Time of day

Iterative vs bulk

Feedback / status?

Deployment

Same hardware

Status indicator

Deploy all hardware at once
	Put fruit on scanner
	Press ‘guess’ button
	Spectra transmitted to cloud
	ML guesses
	Sends back to scanner
	Scanner displays guess
	Person confirms or denies guess. 
	Data is added to ML

WL

Flipped class

EIWL chapters

Streaming notebooks or video sharing

In-class work recorded with ScreenFlow / Ensemble streaming

Charrette code sharing and review

Hardware

IO and Calibration

Very simple procedural programming in Arduino

Basic electronics

Mathematica interface to argyll-cms

meas=Table[Pause[1];
<|"V"->getArduinoV[],"lx"->getAmbient[]|>,{5}];

ListPlot[meas,AxesLabel->Automatic,PlotLabel->"Photoresistor Calibration"]

AS7262

Extend code to use AS7262 to measure spectra.

Sensor

6 channel I2C device via Device Framework

Plot[vf,{λ,350,750},PlotStyle->styles,Evaluated->True,PlotRange->All]

Device

arduino=DeviceOpen["Arduino",{Last[FileNames["/dev/cu.usbmodem*"]],"BoardType"->"Uno"}]

DeviceConfigure[arduino,"Upload"->{"Libraries"->{"Wire","/Users/flip/Documents/Arduino/libraries/SparkFun_AS726X"},Initialization->" 
AS726X sensor;
byte GAIN = 2;
byte MEASUREMENT_MODE = 0;

void sensorSetup() { sensor.begin(Wire, GAIN, MEASUREMENT_MODE);}
byte sensorInitQ = 0;",

"Functions"-><|
"SetupSensor"-><|
"ArgumentTypes"->{},
"ReturnType"->Integer,
"Code"->"
	int SetupSensor(void) {
		if(!sensorInitQ) {
			sensorInitQ = 1;
			sensorSetup();
			return 1;
		}
		return 0;
	}"|>,

"GetCalibratedR"-><|
"ArgumentTypes"->{},
"ReturnType"->Real,
"Code"->"
float GetCalibratedR(void) {
	if(!sensorInitQ)
		return -1.0;
	return(sensor.getCalibratedR());
}"|>,
"GetCalibratedS"-><|
"ArgumentTypes"->{},
"ReturnType"->Real,
"Code"->"
float GetCalibratedS(void) {
	if(!sensorInitQ)
		return -1.0;
	return(sensor.getCalibratedS());
}"|>,

"TakeMeasurements"-><|
"ArgumentTypes"->{},
"ReturnType"->Integer,
"Code"->"
int TakeMeasurements(void) {
	if(!sensorInitQ)
		return 0;

	int red = 2;
	pinMode(red, INPUT);
	int redState = digitalRead(red);
	while (redState == 1)
		redState = digitalRead(red);
	sensor.takeMeasurements();
	return 1;
}"
|>|>}];

Measurements

bananaD=Dataset[Map[Association,
Map[Apply[Rule,MapAt[ToExpression,StringSplit[#,{"[","]"}],2]]&,banana,{2}]]]

Response

Plot[bananaResp,{λ,350,750},PlotStyle->styles,Evaluated->True,PlotRange->All,AxesLabel->{"λ",None},PlotLabel->"AS7262 Bananas"]

Plot[Total[bananaResp],{λ,350,750},PlotStyle->Darker[Yellow],Evaluated->True,PlotRange->All,AxesLabel->{"λ",None},PlotLabel->"AS7262 Bananas"]

Classifier

cf=Classify[<|"Banana"->bananaTrain,"Apple"->appleTrain|>,ValidationSet-><|"Banana"->bananaTest,"Apple"->appleTest|>]

cf[bananaTest]

Hyperspectra

https://doi.org/10.1364/JOSAA.35.00B256

Specim VNIR HS-CL-30-V8E-OEM -> 376.20 nm to 821.62 nm at ~1nm λ resolution

Autoencoder

Data amplification

Train autoencoder using  AS7262 measurements matched to Gießen measurements.

400 Channels network adjusted with ambient data

ambientdata400=Import[FileNameJoin[{NotebookDirectory[],"light.csv"}]];
Import[FileNameJoin[{NotebookDirectory[],"samples.mx"}]]

Cutting the channels (leave 70 to 251)

wls=Transpose[ambientdata400]//First;

sensorChannel={450,500,550,570,600,650};

goodchannel400=wls[[#]]&/@Table[x,{x,70,251}]

k[x_]:=x[[70;;251]]

ambientValues400=ambientdata400[[All,2]];
goodambient400=ambientValues400[[#]]&/@Table[x,{x,70,251}];

adjustedodata=k/@oData;
adjustedbdata=k/@bData;
adjustedgdata=k/@gData;
adjustedadata=k/@aData;

Normalize the data

f[x_]:=x[[#]]/Max[x[[#]]]&/@Table[y,{y,1,Length[x]}]

oTrueColor400=Map[#/goodambient400&, f[adjustedodata]];
bTrueColor400=Map[#/goodambient400&, f[adjustedbdata]];
gTrueColor400=Map[#/goodambient400&, f[adjustedgdata]];
aTrueColor400=Map[#/goodambient400&, f[adjustedadata]];

goTrueColor400=Map[GaussianFilter[#,15]&,oTrueColor400];
gbTrueColor400=Map[GaussianFilter[#,15]&,bTrueColor400];
ggTrueColor400=Map[GaussianFilter[#,15]&,gTrueColor400];
gaTrueColor400=Map[GaussianFilter[#,15]&,aTrueColor400];

Dimensions[goTrueColor400]

Network Training

net=NetChain[{LinearLayer[],SoftmaxLayer["Output"->NetDecoder[{"Class",{"banana","orange","apple","green apple"}}]]}]

allData400=Join[
bRules400=Thread[Rule[gbTrueColor400,"banana"]],
aRules400=Thread[Rule[gaTrueColor400,"apple"]],
oRules400=Thread[Rule[goTrueColor400,"orange"]],
gRules400=Thread[Rule[ggTrueColor400,"green apple"]]];

n=1000;
sample=RandomSample[allData400,n];
{train,test}=TakeDrop[sample,Round[.8 n]];

trained=NetTrain[net,train,TimeGoal->10]

Test Network with its own data

NetMeasurements[trained,allData400,"Accuracy"]

NetMeasurements[trained,allData400,"ConfusionMatrixPlot"]

Create a new network to associate 6 channel data with the fruit class

newData6=Join[
newbRules6=Thread[Rule[net2[pbData6],"banana"]],
newaRules6=Thread[Rule[net2[paData6],"apple"]],
newoRules6=Thread[Rule[net2[poData6],"orange"]],
newgRules6=Thread[Rule[net2[pgData6],"green apple"]]];

n=1000;
sample2=RandomChoice[newData6,n];
{train2,test2}=TakeDrop[sample2,Round[.8 n]];

net3=NetChain[{LinearLayer[],SoftmaxLayer["Output"->NetDecoder[{"Class",{"banana","orange","apple","green apple"}}]]}]

newtrained=NetTrain[net3,train2]

NetMeasurements[newtrained,newData6,"Accuracy"]

NetMeasurements[newtrained,newData6,"ConfusionMatrixPlot"]

Results

98 % accuracy!

All together now

Summary

A 15 week class for upper level neuroscience and psychology students without programming experience

Small teams -> 2019 -> one team

Built simple photometer, spectrophotometer for acquisition

Learned basic Arduino, electronics, and procedural programming concepts

Used Mathematica Classifier and ML to do data analysis and modeling

Did end-to-end acquisition to prediction

Didn’t teach students to:

Program

Engineer

‘Make’

Taught students to:

Think

RTFM

Collaborate

Solve