How To Train Your Robotic Sea Turtle |
Close Encounters of the mth Kind

Jay Alan Jackson
Rochester Institute of Technology

Overview

1. Background and Motivation

2. Narrative

3. Designing Choreographics (Crafting Paths of Motion)

4. Fashioning Geometrical Objects

5. Constructing Interactive Musical Instruments

6. Acknowledgments

7. Questions

Background and Motivation

This presentation will show how a combination of Mathematica, UnityLink and the Unity3D game engine are being used to produce an interactive music video featuring AIMSPP, an Artificially Intelligent Multi-sensory Signal and Pattern Perceiver embodied as a robotic sea turtle, and six wild yet playful dolphins, nicknamed A Pod Calypso.  

It will highlight a few of the many capabilities of Wolfram Language that can be useful for making a Unity project:

designing 3D choreographics (calculating paths of motion)

fashioning geometric objects

constructing interactive musical instruments

The interactive music video is intended as a “play-along” that is performed by a group of six actors in the roles of scientists and six dolphin impersonators (impersonifiers), staged upon a mixed-reality platform built around a sailboat.


The sci-fi storyline goes like this . . .  



Note:  Additional background and motivation for this project can be found in this 2018 Wolfram Technology Conference  presentation entitled:                                                                  

   The Caribbean Trapezoid: Computational Thinking via Mixed-Reality Sci-Fi     from which this excerpt that explains the principal motivation for this project is taken:

Design a creative computing course for non-majors, taylored to 1st year science majors

prepare students to pursue numerous diverse and newly emerging career options (“pick any field X, there either is now or soon will be a 'computational X' ”)

motivate and engage science students by performing computational experiments and making interactive simulations/visualizations

develop group problem-solving skills, build original works to share with others (performances, live coding exhibitions, esports)

incorporate combinations of exercise, entertainment and enlightenment with educational activities (provide experiences analogous to membership within music ensembles, sports teams, math/computer programming/science clubs, theatrical and dance troupes)

provide practice working with connected devices, capturing/processing sensory data from cameras, microphones, accelerometers, along with the use of machine learning

Narrative

A Pod Calypso has taken refuge within a desolated aquatic zone known as the Caribbean Trapezoid.  
                                                                                                                                                                                

Scientists aboard Riff Raft, an exploratory vessel from the Cetacean Intelligence and Communication Investigation Lab (CICIL), are seeking to make contact and carry out research, but this atypical tropical troupe of marine mammals remains elusive and cautiously distant.  
                                                                                                                                                                         


Observations from far away reveal A Pod Calypso likes to perform expressive and polyrhythmic body movements while emitting exotic percussive sounds in synchrony.  

Consequently, the squad from CICIL now plans to carry out numerous experiments deploying a purposely-trained AIMSPP to conduct novel song & dance routines composed from myriad mixed messages of music, motion, and math.  
                                                                                            
The conjecture is such actions may draw A Pod Calypso nearer, giving rise to close encounters of the mth kind.

Designing 3D Choreographics:  Calculating Paths of Motion

Consider these expressive social activites, that combine music, motion, and even some math:  square dancing, line dancing, hoop dancing, conga lines, drum circles, ... 

However, in all these cases the movement takes place on a flat surface.  Here’s an example of how similar movement can be designed so as to be performed upon a periodic path upon a curved surface.

Clear[t,x, y, z, r, g, fF, gradofF]
x[t_] := 2Cos[t]
y[t_] := Sin[t]
z[t_] := Sin[2t]
r[t_] := {x[t], y[t], z[t]}
gravity[t_] := {0,0,-1}
gradF[f_] := {D[f,x], D[f,y],D[f,z]}
gradofF[t_]:= {-y[t], -x[t], 1} /Norm[{-y[t], -x[t], 1}]

r[t]
r'[t]
r''[t]
gravity[t]
gradF[z -x y]
gradofF[t]

Manipulate[
Show[
ParametricPlot3D[{r[t]},{t,0,2π}] ,
ContourPlot3D[z==x y,{x,-4,4},{y,-4,4},{z,-4, 4}, ContourStyle->Directive[Orange,Opacity[0.1],Specularity[White,30]], Mesh->None (*All*)],
Graphics3D[ { Red, Arrow[{r[t], r[t] + r'[t]}]}],
Graphics3D[{Yellow, Arrow[{r[t], r[t] + gradofF[t]}]}],
Graphics3D[{Green, Arrow[{r[t], r[t] + gravity[t]}]}]
 ],
{t,0,2π}]

In order to get AIMSPP and A Pod Calypso to move along such a path, related code needs to be written in C# and put within the Update() method and attached as a Script Component to a GameObject in Unity.

totalTime += 0.1250 f * Time.deltaTime;  // the scale factor 0.1250 slows down the time to coincide with song tempo
offTime = totalTime - phase; // phase is some multiple of 360 degrees divided by the number of dolphins
parametric = new Vector3 (2.0 f * Mathf.Cos (offTime), Mathf.Sin (2.0 f * offTime), Mathf.Sin (offTime));
transform.position = parametric;
gradient = new Vector3 (-Mathf.Sin (offTime), 1, -2.0 f * Mathf.Cos (offTime));
gradient = Vector3.Normalize (gradient);  // NOTE :  on the surface y = x z  the vector < -z, +1, -x > is the gradient
velocity = new Vector3 (-2.0 f * Mathf.Sin (offTime), 2.0 f * Mathf.Cos (2.0 f * offTime), Mathf.Cos (offTime));
velocity = Vector3.Normalize (velocity);
transform.up = gradient;
transform.right = Vector3.Cross (gradient, velocity);
transform.forward = velocity;

Fashioning Geometrical Objects

The object pictured above is dubbed a “JukeBox”, and it represents a body language of sorts;  27 different orientations that are the combinations of twist, rock and roll (aka yaw, pitch and roll) consisting of angles that can be one of three possible values of -alpha, 0, or +alpha for some alpha less than 90 degrees.  AIMSPP and A Pod Calypso will then carry out these various rotations rhythmically while performing (not the same old) song & dance routines.    

Note that these 27 poses can be directly correlated (as shown above) with the letters of the alphabet, plus a blank (space) character.  And yet, by using the three distinct zones of polygonal shapes on AIMSPP’s back shell (which we’ll refer to as tip, ring and sleeve) an interesting mapping called “AIMSPP code” can be constructed so as to create a more universally recognizable set of 27 distinguishable patterns.  The tip uses the three innermost hexagons, the ring is the middle zone of 12 key panels, and the sleeve is the outer band of two arcs.

AIMSPPcode = {
 ,,,
,,,
,,,
,,,
,,,
,,,
,,,
,,,
,,};

Let’s see how well Wolfram Language can detect the organizational structure of these patterns.

FeatureSpacePlot[AIMSPPcode]

Given the apparent success of the machine recognition of these patterns, we can investigate the feasibility that our six crafty marine mammals could learn them too, especially given the multiple redundancy of simultaneously displaying the graphical patterns + body language + signature sounds.

cfAIMSPPcode = Predict[Table[AIMSPPcode[[i]] -> i, {i,1,27,1}]]

Table[cfAIMSPPcode[AIMSPPcode[[j]]],{j,1,27,1}]

Here’s what the result looks like within Unity:     and when combined with 3 x 3 x 3 = 27 distinctive “linearly independent” percussion patterns, the (still a work-in-progress) “DDR”-ish Unity game looks like this:

Needs["UnityLink`"]

UnityOpen["/Users/jacksonj/Desktop/Dolphin Dance"]

Constructing Interactive Musical Instruments

Let’s now go through some steps to construct an interactive musical instrument for the crew of Riff Raft, which can also utilize AIMSPP’s graphical display, such that A Pod Calypso could potentially play along in a “Guitar Hero” type song & dance routine.

The piano keyboard is very logically designed, from which we can draw a similarly constructed set of 12 keys.

ImageResize[, 340]
ArrayPlot[{{0, 1,0, 1, 0, 0, 1, 0, 1, 0,1, 0}}, Mesh->All]

Let’s replace the squares with hexagons, and bend them around into a circle instead of a line.  We can now see clearly see how to use the graphical panels of AIMSPP to achieve a circular version of the keyboard.  The three TRS zones we used earlier can now provide a combination of the three R, G, B colors with W and K  to yield clearly recognizable patterns that map to the 12 chromatic tones in an octave.

s = .45
Graphics[{{EdgeForm[Black],Join[
Table[Style[RegularPolygon[{2Cos[(2 π /6)i + π/12 + π/2 ],2Sin[(2 π /6)i  + π/12 + π/2 ]},s,6],White],{i,0,2}],
Table[Style[RegularPolygon[{2Cos[(2 π /6)i + π/12 + π/2 ],2Sin[(2 π/6)i  + π/12 + π/2 ]},s,6],Black],{i,3,5}],
Table[Style[RegularPolygon[{2Cos[(2 π /6)i + π/6 + π/12 + π/2],2Sin[(2 π /6)i + π/6 + π/12 + π/2 ]},s,6],Black], {i,0,2}],
Table[Style[RegularPolygon[{2Cos[(2 π /6)i + π/6 + π/12 + π/2],2Sin[(2 π /6)i + π/6 + π/12 + π/2 ]},s,6],White], {i,3,5}]
]},Inset[]}]

A Guitar Hero controller can be modified slightly in order to provide a way to select the six color zones Red, Yellow, Green, Cyan, Blue, Magenta using three middle buttons and then the left or right end button to pick the white or the black key.  Here’s a picture of that (although the colors still need to be repainted).  Observe that with this particular mapping, at most three fingers are needed to pick any of the 12 notes.

Now it is time to come up with a tune to use in an interactive song & dance routine that would capture A Pod Calypso’s attention.

Sound[Join[
{SoundNote["G3",1,"Steeldrums"],
SoundNote["C4",1,"Steeldrums"],
SoundNote["B3",1,"Steeldrums"],
SoundNote["E3",1,"Steeldrums"],
SoundNote["A3",1,"Steeldrums"],
SoundNote["D3",1,"Steeldrums"],
SoundNote["G3",2,"Steeldrums"] },

{SoundNote["G3",1,"Steeldrums"],
SoundNote["C4",1,"Steeldrums"],
SoundNote["B3",1,"Steeldrums"],
SoundNote["E3",1,"Steeldrums"],
SoundNote["A3",1,"Steeldrums"],
SoundNote["D3",1,"Steeldrums"],
SoundNote["G3",2,"Steeldrums"] },

{SoundNote["C4",2,"Steeldrums"],
SoundNote["B3",2,"Steeldrums"],
SoundNote["A3",1,"Steeldrums"],
SoundNote["D3",1,"Steeldrums"],
SoundNote["G3",2,"Steeldrums"] },

{SoundNote["C4",2,"Steeldrums"],
SoundNote["B3",2,"Steeldrums"],
SoundNote["E3",1,"Steeldrums"],
SoundNote["A3",1,"Steeldrums"],
SoundNote["D3",2,"Steeldrums"]},

{SoundNote["G3",1,"Steeldrums"],
SoundNote["C4",1,"Steeldrums"],
SoundNote["B3",1,"Steeldrums"],
SoundNote["E3",1,"Steeldrums"],
SoundNote["A3",1,"Steeldrums"],
SoundNote["D3",1,"Steeldrums"],
SoundNote["G3",2,"Steeldrums"] }
]]

The song structure above actually has some symbolic significance.  It has a form consisting of 10 call-and-response phrases, intended to signify a “shaking hands” gesture between humans (note that inside dolphin’s flippers there are five phalanges, finger-like bones).  The chord structure is based on the cycle of 5ths too, a very natural progression based on the (next to the octave) most harmonious intervals.  There is also a little riff that is a musical quote from a motif in the film “Close Encounters of the 3rd Kind”.  Here’s a handy chart that can be used to play the tune:

Let’s add one more twist to the keyboard, to make it an object that could be positioned in the center of the path around which AIMSPP and A Pod Calypso will be grooving.  This will be a structure called a “polar pan piano helix” shown below, extended over four octaves.

To build this we’ll first need to use Wolfram Language to provide 12 sectors of an annular region, as such:

a = Annulus[{0,0},{1/2,1},{0,Pi/6}]
bf = Annulus[{0,0},{1/2,1},{Pi/6, Pi/3}]
 b= Annulus[{0,0},{1/2,1},{2(Pi/6), 3(Pi/6)}]
c = Annulus[{0,0},{1/2,1},{3(Pi/6), 4(Pi/6)}]
cs = Annulus[{0,0},{1/2,1},{4(Pi/6), 5(Pi/6)}]
d = Annulus[{0,0},{1/2,1},{5(Pi/6), 6(Pi/6)}]
ds = Annulus[{0,0},{1/2,1},{6(Pi/6), 7(Pi/6)}]
e = Annulus[{0,0},{1/2,1},{7(Pi/6), 8(Pi/6)}]
es = Annulus[{0,0},{1/2,1},{8(Pi/6), 9(Pi/6)}]
gf = Annulus[{0,0},{1/2,1},{9(Pi/6), 10(Pi/6)}]
g = Annulus[{0,0},{1/2,1},{10(Pi/6), 11(Pi/6)}]
af = Annulus[{0,0},{1/2,1},{11(Pi/6), 12(Pi/6)}]

With the use of UnityLink,  we can create 12 GameObjects, add to each an Audio Source component and add an Audio Clip, set the proper positions of the Transform Component, configure the color properties of the Renderer Components, and add Script Components to allow foWe can then replicate to add additional octaves as shown here

Now we can finally take a look at the (nearly) finished Unity Project:

Needs["UnityLink`"]

UnityOpen["/Users/jacksonj/Desktop/NeoTerra"]

Acknowledgements

“I think the most important question facing humanity is, ‘Is the universe a friendly place?’  This is the first and most basic question all people must answer for themselves.”   
- Albert Einstein
 
 “Within a special part of the universe known as showtimespace, the existence of music and dance proves that the answer to Einstein’s most important question is ‘yes’.  Constructing a proof for the general case will rely as much on the cultivation of kindness as it will upon knowledge.  
 - AIMSPP
 
“When I write, I feel like an armless and legless man with a crayon in his mouth.” - Kurt Vonnegut

My heartfelt thanks to the following RIT students in the School of Interactive Games & Media 

for creating the animations in Unity with rigged models made in Maya:

Enan Munzar (AIMSPP)

Monique Beauchamp (A Pod Calypso)

for spending their time in conversation with me, sharing thoughts and providing helpful feedback:

Ashwin Ganapathiraju

Elizabeth Smith

And finally, thanks to Stephen Wolfram all of the wonderful and forward-thinking folks at Wolfram Research, for providing me with the means and this opportunity to dabble within a “new kind of field”:

Computational X, where X equals “Comedy”

Questions ?

Contact:  jaj@igm.rit.edu


Thank you for coming!