Online Experiments with Wolfram Technologies

Brian Rivera
Educational Psychology PhD Student
University of Alabama

Experimental Psychology
Massive Online Experiments (MOEs)
MOE Technology
My Project
Other Applications of Online Deployment

Experimental Psychology

● Measuring behavior since the early 1900s

● Main comparison has been between/within group differences

●The more the merrier

Experimental Psychology

● Always had geographic limitations (proximity to lab, region, etc.)
	-We have inherited this constraint (US undergrads in psychology)

● With increased speed and reliance of technology allows for the closing of this gap

Massive Online Experiments (MOEs) 
The Good

● Experiments using thousands or tens of thousands of participants

● Short and easy to carry out

● More diverse participant pool (able to cross demographic and geographic boundaries)

● More statistically powerful comparisons

Massive Online Experiments (MOEs) 
More Good

● Shorter experiments (6000 trials =  10 subjects X 600 trials or 100 subjects X 60 trials)

● Higher power (more likely to find a significant effect)

● Not a nicety but the way cognitive science should be done (Laura Germine)

Massive Online Experiments (MOEs) 
The Bad

● Hard to control for a lot of variables (luminance, size of screen, noise, etc.)

● Technical demands: hosting, traffic, coding, data storage, etc.

● Knowledge: Not many know there are such options for data collection

What tools are there?

jspsych

● Pro:
-Has a small community users
-Some publications 

● Con:
-Requires JavaScript & HTML
-Its a client-side library
-One man’s show Josh de Leeuv @ Vassar College

Amazon Mechanical Turk 
(Online Labor Market)

● Pro:
-Can guarantee the number of participant
-Faster
-Many publications

● Con:
-Not designed for experimental psychology
-Hard to counterbalance (trials, groups, etc. )
-Completion difficulties, dropouts, biases
-Uses 3 Languages (HTML, CSS, & JavaScript)
-Not representative (Mostly US & India) (yet better than undergraduate population)

Wolfram Language

● Pro:
-Easy to use!
-Can be done in the same language (experiment, data collection, analysis, etc)
-Enjoys a large community
-Not many people doing it

● Con:
-Not many people doing it

Disclaimer: One man’s show

● I learned WL earlier this year and was able to learn and adapt it to fit this career goal (carrying out experiments)

● I would hope to see many (and better) online experiments

The Project

(MouseoverTemplate["https://wolfr.am/w1b1dA1f", "https://wolfr.am/w1b1dA1f", ])

The Project

● The main skeleton

The Code

Hyperlink@URLShorten@CloudDeploy[
Panel[DynamicModule[
{
bin="vZA0S3eZ",
fcolor=Black,
bcolor=LightGray,
fsize=200,
fsize2=40,
isize=100,
a,
b,
choice,
pressed=1,
counter=1,
ntrials=150,
nbaselinetials=5,
baselinecounter=1,
stimuliPresTime,
baselineStimuli=0
},
Dynamic[Overlay[{
Grid[{
{Spacer[{70,50}]},
{Style["Instructions",FontColor->fcolor,Bold,FontSize->80]},
{Spacer[{70,40}]},
{Style["In the following section, a series of fractions in the form a/b with values ranging from 0 to 1 will be presented one at the time. Please indicate whether the numerical value of the fraction presented is closer to 0 or 1 by pressing either the 0 or 1 buttons at the bottom of the screen.",FontColor->fcolor,FontSize->20]},
{Spacer[{70,20}]},
{Style["There will be 10 initial trials which simply require you to press either the 0 or 1 buttons when a zero or a one appear on the screen. The fraction trials will begin immediately after these calibration trials are completed.",FontColor->fcolor,FontSize->20]},
{Spacer[{70,20}]},
{Style["There will be several trials taking only a few seconds each. The entire session lasts approximately 5 minutes in total.",FontColor->fcolor,FontSize->20]},
{Spacer[{70,20}]},
{Style["Please make sure your browser window is in fullscreen.",FontColor->fcolor,FontSize->20]},
{Spacer[{70,20}]},
{Style["Press the button below when ready",FontColor->fcolor,Bold,FontSize->40]},
{Spacer[{70,20}]},
{Button[Style["Let the trials begin!",Bold,FontSize->fsize2],
DatabinAdd[Databin[bin],<|"ID"->$UserAgentString,"TimeSubmitted"->DateObject[],"Trial"->ntrials|>];
pressed=2;
baselineStimuli=0;
stimuliPresTime=DateObject[],FrameMargins->Large
]
},
{Spacer[{50,30}]}
}],
Which[
TrueQ[baselinecounter<nbaselinetials],
Grid[{
{Spacer[{70,100}]},
{Style[baselineStimuli,FontColor->fcolor,FontSize->fsize,Bold]},
{Spacer[{70,80}]},
{Row
[{Panel[Button[Style["0",FontSize->fsize2],
DatabinAdd[Databin[bin],<|"ID"->$UserAgentString,"Trial"->baselinecounter,"Stimuli"->baselineStimuli,"StimuliPresTime"->stimuliPresTime,"TimeSubmitted"->DateObject[],"Response"->"0"|>];
baselineStimuli=1;
pressed= 2;
baselinecounter++;
stimuliPresTime=DateObject[],FrameMargins->20,Appearance->"Frameless"
]],Spacer[{70,30}],
Panel[Button[Style["1",FontSize->fsize2],
DatabinAdd[Databin[bin],<|"ID"->$UserAgentString,"Trial"->baselinecounter,"Stimuli"->baselineStimuli,"StimuliPresTime"->stimuliPresTime,"TimeSubmitted"->DateObject[],"Response"->"1"|>];
baselineStimuli=0;
pressed= 2;
baselinecounter++;
stimuliPresTime=DateObject[],FrameMargins->20,Appearance->"Frameless"
]]
}]},
{Spacer[{50,30}]}
}],
TrueQ[nbaselinetials==baselinecounter],
Grid[{
{Spacer[{70,100}]},
{Style[baselineStimuli,FontColor->fcolor,FontSize->fsize,Bold]},
{Spacer[{70,80}]},
{Row[{
Panel[Button[Style["0",FontSize->fsize2],
DatabinAdd[Databin[bin],<|"ID"->$UserAgentString,"Trial"->baselinecounter,"Stimuli"->baselineStimuli,"StimuliPresTime"->stimuliPresTime,"TimeSubmitted"->DateObject[],"Response"->"0"|>];
b=RandomInteger[{1,100}];
a=RandomInteger[{1,b-1}];
pressed= 3;
stimuliPresTime=DateObject[],FrameMargins->20,Appearance->"Frameless"
]],Spacer[{70,30}],
Panel[Button[Style["1",FontSize->fsize2],
DatabinAdd[Databin[bin],<|"ID"->$UserAgentString,"Trial"->baselinecounter,"Stimuli"->baselineStimuli,"StimuliPresTime"->stimuliPresTime,"TimeSubmitted"->DateObject[],"Response"->"1"|>];
b=RandomInteger[{1,100}];
a=RandomInteger[{1,b-1}];
pressed=3;
stimuliPresTime=DateObject[],FrameMargins->20,Appearance->"Frameless"
]]
}]},
{Spacer[{50,30}]}
}]
]
,
Which[
TrueQ[counter<ntrials],
Grid[{
{Spacer[{70,50}]},
{Style[(a/b),FontColor->fcolor,FontSize->fsize,Bold]},
{Spacer[{70,30}]},
{Row[{
Panel[Button[Style["0",FontSize->fsize2],
DatabinAdd[Databin[bin],<|"ID"->$UserAgentString,"Trial"->counter,"StimuliPresTime"->stimuliPresTime,"TimeSubmitted"->DateObject[],
"Numerator"->a,"Denominator"->b,"Response"->"0"|>];
b=RandomInteger[{1,100}];
a=RandomInteger[{1,b-1}];
pressed= 3;
counter++;
stimuliPresTime=DateObject[],FrameMargins->20,Appearance->"Frameless"
]],Spacer[{70,30}],
Panel[Button[Style["1",FontSize->fsize2],
DatabinAdd[Databin[bin],<|"ID"->$UserAgentString,"Trial"->counter,"StimuliPresTime"->stimuliPresTime,"TimeSubmitted"->DateObject[],
"Numerator"->a,"Denominator"->b,"Response"->"1"|>];
b=RandomInteger[{1,100}];
a=RandomInteger[{1,b-1}];
pressed= 3;
counter++;
stimuliPresTime=DateObject[],FrameMargins->20,Appearance->"Frameless"
]]
}]},
{Spacer[{50,30}]}
}],
TrueQ[counter==ntrials],
Grid[{
{Spacer[{70,50}]},
{Style[(a/b),FontColor->fcolor,FontSize->fsize,Bold]},
{Spacer[{70,30}]},
{Row[{
Panel[Button[Style["0",FontSize->fsize2],
DatabinAdd[Databin[bin],<|"ID"->$UserAgentString,"Trial"->counter,"StimuliPresTime"->stimuliPresTime,"TimeSubmitted"->DateObject[],
"Numerator"->a,"Denominator"->b,"Response"->"0"|>];
pressed= 4,FrameMargins->20,Appearance->"Frameless"
]],Spacer[{70,30}],
Panel[Button[Style["1",FontSize->fsize2],
DatabinAdd[Databin[bin],<|"ID"->$UserAgentString,"Trial"->counter,"StimuliPresTime"->stimuliPresTime,"TimeSubmitted"->DateObject[],
"Numerator"->a,"Denominator"->b,"Response"->"1"|>];
pressed=4,FrameMargins->20,Appearance->"Frameless"
]]
}]
},
{Spacer[{50,30}]}
}]
]
,
Grid[{{Spacer[{70,50}]},
{Style["You are all done!",FontColor->fcolor,
FontSize->150,Bold]},
{Spacer[{70,50}]},
{Style["Thank you",FontColor->fcolor,FontSize->150,Bold]}}],
{Spacer[{70,50}]},
Alignment->{Center}},
{Which[TrueQ[pressed==1],1,
TrueQ[pressed==2],2,
TrueQ[pressed==3],3,
TrueQ[pressed==4],4]},
Which[TrueQ[pressed==1],1,
TrueQ[pressed==2],2,
TrueQ[pressed==3],3,
TrueQ[pressed==4],4],
Alignment->{Center,Center},Background->White]]],
ImageSize->Full,Background->White,Alignment->Center],
Permissions->"Public"
]
Hyperlink@URLShorten@CloudDeploy[
Panel[DynamicModule[
{
bin="vZA0S3eZ",
fcolor=Black,
bcolor=LightGray,
fsize=200,
fsize2=40,
isize=100,
a,
b,
choice,
pressed=1,
counter=1,
ntrials=150,
nbaselinetials=5,
baselinecounter=1,
stimuliPresTime,
baselineStimuli=0
},
Dynamic[Overlay[{
Grid[{
{Spacer[{70,50}]},
{Style["Instructions",FontColor->fcolor,Bold,FontSize->80]},
{Spacer[{70,40}]},
{Style["In the following section, a series of fractions in the form a/b with values ranging from 0 to 1 will be presented one at the time. Please indicate whether the numerical value of the fraction presented is closer to 0 or 1 by pressing either the 0 or 1 buttons at the bottom of the screen.",FontColor->fcolor,FontSize->20]},
{Spacer[{70,20}]},
{Style["There will be 10 initial trials which simply require you to press either the 0 or 1 buttons when a zero or a one appear on the screen. The fraction trials will begin immediately after these calibration trials are completed.",FontColor->fcolor,FontSize->20]},
{Spacer[{70,20}]},
{Style["There will be several trials taking only a few seconds each. The entire session lasts approximately 5 minutes in total.",FontColor->fcolor,FontSize->20]},
{Spacer[{70,20}]},
{Style["Please make sure your browser window is in fullscreen.",FontColor->fcolor,FontSize->20]},
{Spacer[{70,20}]},
{Style["Press the button below when ready",FontColor->fcolor,Bold,FontSize->40]},
{Spacer[{70,20}]},
{Button[Style["Let the trials begin!",Bold,FontSize->fsize2],
DatabinAdd[Databin[bin],<|"ID"->$UserAgentString,"TimeSubmitted"->DateObject[],"Trial"->ntrials|>];
pressed=2;
baselineStimuli=0;
stimuliPresTime=DateObject[],FrameMargins->Large
]
},
{Spacer[{50,30}]}
}],
Which[
TrueQ[baselinecounter<nbaselinetials],
Grid[{
{Spacer[{70,100}]},
{Style[baselineStimuli,FontColor->fcolor,FontSize->fsize,Bold]},
{Spacer[{70,80}]},
{Row
[{Panel[Button[Style["0",FontSize->fsize2],
DatabinAdd[Databin[bin],<|"ID"->$UserAgentString,"Trial"->baselinecounter,"Stimuli"->baselineStimuli,"StimuliPresTime"->stimuliPresTime,"TimeSubmitted"->DateObject[],"Response"->"0"|>];
baselineStimuli=1;
pressed= 2;
baselinecounter++;
stimuliPresTime=DateObject[],FrameMargins->20,Appearance->"Frameless"
]],Spacer[{70,30}],
Panel[Button[Style["1",FontSize->fsize2],
DatabinAdd[Databin[bin],<|"ID"->$UserAgentString,"Trial"->baselinecounter,"Stimuli"->baselineStimuli,"StimuliPresTime"->stimuliPresTime,"TimeSubmitted"->DateObject[],"Response"->"1"|>];
baselineStimuli=0;
pressed= 2;
baselinecounter++;
stimuliPresTime=DateObject[],FrameMargins->20,Appearance->"Frameless"
]]
}]},
{Spacer[{50,30}]}
}],
TrueQ[nbaselinetials==baselinecounter],
Grid[{
{Spacer[{70,100}]},
{Style[baselineStimuli,FontColor->fcolor,FontSize->fsize,Bold]},
{Spacer[{70,80}]},
{Row[{
Panel[Button[Style["0",FontSize->fsize2],
DatabinAdd[Databin[bin],<|"ID"->$UserAgentString,"Trial"->baselinecounter,"Stimuli"->baselineStimuli,"StimuliPresTime"->stimuliPresTime,"TimeSubmitted"->DateObject[],"Response"->"0"|>];
b=RandomInteger[{1,100}];
a=RandomInteger[{1,b-1}];
pressed= 3;
stimuliPresTime=DateObject[],FrameMargins->20,Appearance->"Frameless"
]],Spacer[{70,30}],
Panel[Button[Style["1",FontSize->fsize2],
DatabinAdd[Databin[bin],<|"ID"->$UserAgentString,"Trial"->baselinecounter,"Stimuli"->baselineStimuli,"StimuliPresTime"->stimuliPresTime,"TimeSubmitted"->DateObject[],"Response"->"1"|>];
b=RandomInteger[{1,100}];
a=RandomInteger[{1,b-1}];
pressed=3;
stimuliPresTime=DateObject[],FrameMargins->20,Appearance->"Frameless"
]]
}]},
{Spacer[{50,30}]}
}]
]
,
Which[
TrueQ[counter<ntrials],
Grid[{
{Spacer[{70,50}]},
{Style[(a/b),FontColor->fcolor,FontSize->fsize,Bold]},
{Spacer[{70,30}]},
{Row[{
Panel[Button[Style["0",FontSize->fsize2],
DatabinAdd[Databin[bin],<|"ID"->$UserAgentString,"Trial"->counter,"StimuliPresTime"->stimuliPresTime,"TimeSubmitted"->DateObject[],
"Numerator"->a,"Denominator"->b,"Response"->"0"|>];
b=RandomInteger[{1,100}];
a=RandomInteger[{1,b-1}];
pressed= 3;
counter++;
stimuliPresTime=DateObject[],FrameMargins->20,Appearance->"Frameless"
]],Spacer[{70,30}],
Panel[Button[Style["1",FontSize->fsize2],
DatabinAdd[Databin[bin],<|"ID"->$UserAgentString,"Trial"->counter,"StimuliPresTime"->stimuliPresTime,"TimeSubmitted"->DateObject[],
"Numerator"->a,"Denominator"->b,"Response"->"1"|>];
b=RandomInteger[{1,100}];
a=RandomInteger[{1,b-1}];
pressed= 3;
counter++;
stimuliPresTime=DateObject[],FrameMargins->20,Appearance->"Frameless"
]]
}]},
{Spacer[{50,30}]}
}],
TrueQ[counter==ntrials],
Grid[{
{Spacer[{70,50}]},
{Style[(a/b),FontColor->fcolor,FontSize->fsize,Bold]},
{Spacer[{70,30}]},
{Row[{
Panel[Button[Style["0",FontSize->fsize2],
DatabinAdd[Databin[bin],<|"ID"->$UserAgentString,"Trial"->counter,"StimuliPresTime"->stimuliPresTime,"TimeSubmitted"->DateObject[],
"Numerator"->a,"Denominator"->b,"Response"->"0"|>];
pressed= 4,FrameMargins->20,Appearance->"Frameless"
]],Spacer[{70,30}],
Panel[Button[Style["1",FontSize->fsize2],
DatabinAdd[Databin[bin],<|"ID"->$UserAgentString,"Trial"->counter,"StimuliPresTime"->stimuliPresTime,"TimeSubmitted"->DateObject[],
"Numerator"->a,"Denominator"->b,"Response"->"1"|>];
pressed=4,FrameMargins->20,Appearance->"Frameless"
]]
}]
},
{Spacer[{50,30}]}
}]
]
,
Grid[{{Spacer[{70,50}]},
{Style["You are all done!",FontColor->fcolor,
FontSize->150,Bold]},
{Spacer[{70,50}]},
{Style["Thank you",FontColor->fcolor,FontSize->150,Bold]}}],
{Spacer[{70,50}]},
Alignment->{Center}},
{Which[TrueQ[pressed==1],1,
TrueQ[pressed==2],2,
TrueQ[pressed==3],3,
TrueQ[pressed==4],4]},
Which[TrueQ[pressed==1],1,
TrueQ[pressed==2],2,
TrueQ[pressed==3],3,
TrueQ[pressed==4],4],
Alignment->{Center,Center},Background->White]]],
ImageSize->Full,Background->White,Alignment->Center],
Permissions->"Public"
]

● Full code avaiable in Github

The Data

CreateDatabin["Name"->"SurveyDatabin1"]

Wolfram Data Drop - https://datadrop.wolframcloud.com/

The Data

bin=Databin["vZA0S3eZ"];
surveyData=Dataset[bin];
filteredData=surveyData[Select[#Numerator>0&]]

The Data

Calculate RT

stimuliPresTime = filteredData[All,"StimuliPresTime"];
timeSubmitted =  filteredData[All,"TimeSubmitted"];
DateObjRTs=Table[(timeSubmitted[[i]]-stimuliPresTime[[i]]),{i,Length[stimuliPresTime]}];
RTs=Table[DateObjRTs[[x,1]],{x,Length[DateObjRTs]}];

Calculate Magnitude

numerators= filteredData[All,"Numerator"];
denominators=filteredData[All,"Denominator"];
magnitudes=Table[N[numerators[[i]]/denominators[[i]]],{i,Length[numerators]}];

Plot RT & Magnitude

RTbyMag=Table[{magnitudes[[i]],RTs[[i]]},{i,Length[magnitudes]}];
ListPlot[{RTbyMag},DataRange->{0,1}]

The Data

Calculate Accuracy & Plot

response= filteredData[All,"Response"];
numberResponse=Table[ToExpression[response[i]],{i,1,Length[response]}];
Clear[incorrect];
Clear[correct];
correct={};
Table[Which[
TrueQ[0.5>magnitudes[[i]]],
If[numberResponse[[i]]==0,AppendTo[correct,{magnitudes[[i]],RTs[[i]]}]]
,
TrueQ[0.5<magnitudes[[i]]],
If[numberResponse[[i]]==1,AppendTo[correct,{magnitudes[[i]],RTs[[i]]}]]
],{i,Length[magnitudes]}];
incorrect={};
Table[Which[
TrueQ[0.5>magnitudes[[i]]],
If[numberResponse[[i]]==1,AppendTo[incorrect,{magnitudes[[i]],RTs[[i]]}]]
,
TrueQ[0.5<magnitudes[[i]]],
If[numberResponse[[i]]==0,AppendTo[incorrect,{magnitudes[[i]],RTs[[i]]}]]
],{i,Length[magnitudes]}];
ListPlot[{correct,incorrect},DataRange->{0,1},PlotLegends->{Style["Correct",18,Bold],Style["Incorrect",18,Bold]},PlotStyle->PointSize[.01],AxesLabel->{Style["Magnitude",Large,Bold],Style["ReactionTime(s)",Large,Bold]},Filling->Axis,AxesStyle->Directive[Black,25],ImageSize->900,Ticks->{{.1,.2,.3,.4,.5,.6,.7,.8,.9,1},{1,2,3,4,5,6,7,8}}]

Results
(Legended[, (PointLegend["Correct", "Incorrect"])])

Other applications of online deployment

● Live polls
● Surveys
● Assessments in classroom