State of Graphics in the Cloud

Matthew Adams <matthewa@wolfram.com>
John Pacey <jpacey@wolfram.com>

Background: Wolfram Language Graphics

Grid[{
{
Plot[{Sin[x],0.5Cos[x]},{x,0,4Pi},PlotLabel->Plot,ImageSize->Medium],
ParametricPlot3D[{{4+(3+Cos[v]) Sin[u],4+(3+Cos[v]) Cos[u],4+Sin[v]},{8+(3+Cos[v]) Cos[u],3+Sin[v],4+(3+Cos[v]) Sin[u]}},{u,0,2Pi},{v,0,2Pi},PlotStyle->{Red,Green},PlotLabel->ParametricPlot3D,ImageSize->Medium]
},
{
ContourPlot[Cos[x]+Cos[y],{x,0,4Pi},{y,0,4Pi},PlotLabel->ContourPlot,ImageSize->Medium],
BarChart[{{1,2,3},{1,3,2}},PlotLabel->BarChart,ImageSize->Medium]
}
}]

Background: Wolfram Language Graphics

Graphics[{
Red,Rectangle[],
Green,Polygon[{{1.3,0},{2,1},{2.7,0}}],
Blue,Disk[{3.5,0.5},0.55]
}]

Graphics3D[{
Red,Cuboid[{-1.3,0,0}],
Green,Pyramid[{{0,0,0},{1,0,0},{1,1,0},{0,1,0},{0.5,0.5,1}}],
Blue,Sphere[{1.7,0.5,0.5},0.6]
}]

Background: Wolfram Language Graphics

grid[c_]:={Table[Line[{{n,0},{3n,1}},VertexColors->{c,GrayLevel[1,0]}],{n,-5,5,.1}],Table[{Blend[{c,GrayLevel[1,0]},n],Line[{{-1,n},{1,n}}]},{n,0,1,.05}]};
neonLines=Graphics[{
AbsoluteThickness[5],grid[Hue[0.82,1,0.5]],
AbsoluteThickness[3],grid[Hue[0.82,1,1]],
AbsoluteThickness[1],grid[White]
},AspectRatio->1,PlotRange->{{-1,1},{0,1}},Background->Black,ImageSize->Medium];

mesh=With[{n=30},Graphics[GeometricTransformation[Line[{{#,0},{0,n-#}}]&/@Table[x,{x,1,n}],RotationTransform/@(Pi{0,.5,1,1.5})],ImageSize->Medium]];

glowCircles=Graphics[{FaceForm[{Hue[0.2],Opacity[0.3]}],EdgeForm[{Thick,Hue[0.2],Opacity[0.5]}],Table[Disk[{0,0},n/10],{n,10}]},Background->Black,ImageSize->Medium];


f[n_]:={n*Cos[n],n*Sin[n]};
c[n_]:=Directive[ColorData["SolarColors"][1-n],Opacity[1.4-n]];
spiral=With[{step=0.1,max=15Pi},Graphics[{
PointSize[Large],
Point[
Table[f[n],{n,0,max,step}],
VertexColors->Table[c[n/max],{n,0,max,step}]
]
}
,Background->Black
,PlotRangePadding->Scaled[0.1]
,ImageSize->Medium
]];

Grid[{{mesh,glowCircles},{spiral,neonLines}}]

Challenge: How to implement Cloud graphics?

bubble[pos_,radii_,angle_]:={
RGBColor[.5,.7,.95],Disk[pos+.04radii*{Cos[angle],Sin[angle]},radii],
GrayLevel[0.95],Disk[pos,radii]
};
bubble[pos_,radii_]:=bubble[pos,radii,0]
Graphics[{
{GrayLevel[0.95],Disk[{1.8,0},0.3]},
bubble[{1.5,1},{1,1},0.4Pi],
bubble[{1,-.5},{1,0.7},3Pi/2],
bubble[{3,0},{1,1},1.8Pi],
bubble[{2.6,0.85},{1,0.9},0.2Pi],
bubble[{2.2,-0.65},{0.65,0.5},1.6Pi],
bubble[{0.5,0.7},{1,0.8},0.8Pi],
bubble[{0.1,0},{1.2,0.9},1.1Pi]
},ImageSize->500]

Under the hood: Graphics in Desktop

Helper Data Graphics

Server-side Desktop FrontEnd renders graphic into either a bitmap or JSON “helper data”

Helper Data Graphics

Enabled us to quickly bring Wolfram graphics to the Cloud

Continues to be very effective for many use cases

Has some limitations:

Dynamics

Complexity

FrontEnd-dependent interactions (resize)

Speed

Larger Story: Snappy Manipulate in Cloud

Challenge: going back and forth to the server for each update can be slow

DynamicModule[{ωList={}},Column@{Manipulate[ωList=Append[ωList,ω];
Plot[Sin[ω*x],{x,0,2 Pi}],{ω,1,10},TrackedSymbols:>{ω}],Dynamic[ωList]}]

Larger Story: Snappy Manipulate in Cloud

We have to eliminate all those trips to the server

Partial Solution: Compile the Manipulate

Problem: Still need the server-side FrontEnd for helper-data graphics

Native Graphics

No server-side FrontEnd

Graphic is rendered natively client-side (in browser)

Native Graphics: Challenges

Browser inconsistencies/limitations

Power/flexibility of Wolfram Language graphics spec

Perfectly matching Desktop

Graphics[...] has been in Wolfram Language since 1988

Native Graphics: Benefits

Dynamic

DynamicModule[{hue=0,endY=1},Row@{
Column@{
Slider[Dynamic[hue]],
Slider[Dynamic[endY]]
},
Graphics[{
Point[RandomReal[{0,1},{500,2}]],
Hue[Dynamic[hue]],Thick,
Line[{{0,0},Dynamic[{1,endY}]}]}
,ImageSize->Small]
}]

Full, accurate image dimension and positioning calculations

(* Helper Data *)
Graphics[{Disk[{0,0},Offset[100]]},ImageSize->Medium,Axes->False]

(* Native *)
Graphics[{Disk[{0,0},Offset[100]]},ImageSize->Medium]

Module[{r=.1},
Graphics[{
Red, Disk[Offset[{80, 0}, {0, .5}],r],
Blue, Disk[{1, -.5},r],
RGBColor[0,0,0],Disk[{-1,1},r],Disk[{-1,-1},r]}
,ImageSize->{225,Automatic}]]

Full integration with the rest of the Cloud Notebook

Particularly with Inset, which we’re currently implementing

Pinkbox Errors

Graphics[Disk["foo"]]

Fewer moving parts means fewer bugs

Native Graphics

Support is being added incrementally

Currently, the foundation, most of the directives, and several primitives are implemented

Current primary goal is to support Plot/ListPlot/ListLinePlot

Sneak peak/experimental: function to check whether a graphic is supported

CloudSystem`CloudGraphics`CheckNativeSupport@Graphics[Disk[]]

pts={{0,0},{1,1},{2,-1},{3,0},{4,-2},{5,1}};
Graphics[{BSplineCurve[pts],Green,Line[pts],Red,Point[pts]}]

CloudSystem`CloudGraphics`CheckNativeSupport@%

Experimental Cloud Graphics Features

Zoom/Pan

Graphics[{Red,Disk[{0,0},1,{0,Pi/2}],Green,Disk[{0,0},1,{Pi/2,Pi}],Blue,Disk[{0,0},1,{Pi,3Pi/2}],Yellow,Disk[{0,0},1,{3Pi/2,2Pi}]},Method->{"InteractiveControls"->{"Zoomable"->True,"Toolbar"->True}}]

Plot[Sin[10x]+0.1Sin[1000x+0.2],{x,0,2Pi},Method->"InteractiveControls"->{"Zoomable"->True,"Toolbar"->True}]

Note that this is only a Graphics-level zoom

Interactive Graphics

JSONString[y_] := ExportString[y, "JSON","Compact"->True];

Typeset`MakeBoxes[Annotation[x_, {"JSONMetadata", y_}], fmt_, head_] :=
With[{g = Typeset`MakeBoxes[x, fmt, head],
s = "JSONMetadata"[JSONString[y]]}, Typeset`Hold[TagBox[g, s]]];

Framed[Graphics[{
AbsoluteThickness[2.5],
Hue[0.8,1,0.8],
Annotation[
Line[Table[{x,0.5Sin[3x]},{x,0,2Pi,2Pi/300}]],
{"JSONMetadata",{"InteractiveGraphics"->
{"version"->"1.1",

"TrackMousePosition"->{"x"->True,"y"->False},

"effects"->{{"type"->"HighlightPoint","ratio"->3}}
}}}
]
},
PlotRangePadding->0.5,
ImageSize->500
],FrameMargins->0]

annotatedData=Annotation[Accumulate[RandomReal[{-0.8,1},50]],{"JSONMetadata",{"InteractiveGraphics"->
{"version"->"1.1",

"TrackMousePosition"->{"x"->True,"y"->False},

"effects"->{
{"type"->"Droplines","linePlacement"->{"x"->"All","y"->"All"}}
}
}}}];
Row[{
ListLinePlot[annotatedData,"DefaultGraphicsInteraction"->None,ImageSize->300],
ListPlot[annotatedData,"DefaultGraphicsInteraction"->None,ImageSize->300]
}]

Graphics[Annotation[{
PointSize[0.01],
Red,Point[RandomPoint[Disk[],500]],
Blue,Point[RandomPoint[Disk[{1.2,0}],500]]
},
{"JSONMetadata", {"InteractiveGraphics" ->
{"version"->"1.1",
"TrackMousePosition"->{"x"->True,"y"->True},
"OnlyNearestPrimitive"->True,
"effects"->{{"type"->"Highlight","ratio"->1.4}}
}
}}]]

Web Images

Style[Dynamic[RawBoxes[FEPrivate`ImportImage[URL["https://writings.stephenwolfram.com/data/uploads/2019/02/05-treadmill-r22.gif"],"CloudPixelRatio"->2]]],Selectable->False]

3D Graphics in Cloud

3D Graphics History

Was our first foray into implementing graphics purely on the client at the very beginning stages of the Wolfram Cloud (2011)

Learned a lot from this and the Native Graphics implementation is based on the lessons learned

Details

Hybrid approach that initially renders a static image and transitions to a client side WebGL implementation during interaction. At the end of interaction we render an updated static image.

Limitations

WebGL implementation is not complete, general approach is to render as much of the 3D scene as possible.

Not all primitives are supported and most options are assumed to have default values

Still rely on the server side FrontEnd for generating images

Software rasterization caps performance

A lot of effort to keep up to date with desktop FrontEnd bug fixes and feature changes.

Examples

Graphics3D[{Blue,Cylinder[],Red,Sphere[{0,0,2}],Black,Thick,Dashed,Line[{{-2,0,2},{2,0,2},{0,0,4},{-2,0,2}}],Yellow,Polygon[{{-3,-3,-2},{-3,3,-2},{3,3,-2},{3,-3,-2}}],Green,Opacity[.3],Cuboid[{-2,-2,-2},{2,2,-1}]}]

Plot3D[Sin[x+y^2],{x,-3,3},{y,-2,2}]

Plot3D[Sqrt[1-x^2-y^2],{x,-1,1},{y,-1,1},Mesh->8,ColorFunction->Hue,MeshShading->{{Yellow,Orange},{Pink,Red}}]

ParametricPlot3D[ {Sin[u] Sin[v]+0.05 Cos[20v], Cos[u]Sin[v]+0.05 Cos[20 u], Cos[v]}, {u, -π, π}, {v, -π, π},MaxRecursion->4,PlotStyle->{Orange,Specularity[White,10]},Axes->None,Mesh->None]

Development

Compiled WASM library (longer term)

A lot of work and modernization of the general code base has been done on the desktop side. One advantage of this is the ability to compile to JavaScript

Actively being worked on in coordination with the desktop FrontEnd team

Interest in standalone 3D viewers (shorter term)

Existing JavaScript frameworks that can render various 3D file formats

Does not render identically to Graphics3D in a desktop notebook

Useful for W|A and other Wolfram JavaScript libraries

Conclusion

State of Graphics in the Cloud

Native Graphics well underway

Performance/Behavior Improvements

Fast Manipulate on the horizon

Helper Data Graphics continue to be improved

Focus on performance improvements

Experimental Features In Progress

Zoom/Pan

Interactive Graphics

Web Images

3D Graphics

New implementations are in the long term plan

WASM compiled library

Pure client side implementation on top of Native Graphics

Experimenting with standalone web viewers for use within the cloud and in other Wolfram web technologies

Thanks!

Cloud Notebooks Team:

Jan Pöschko

Konstantin Sushenko

Máté Csigi

Emre Hoş

Desktop Team Contributors:

Yuzhu Lu

Rob Raguet-Schofield

Brett Champion

James Lane

Chris Carlson

Ian Hojnicki

John Fultz

Questions?

Bonus Native Graphic:

cloudLogo[center_,radius_]:=With[{
innerStarShortScale=1/E,innerStarLongScale=1/GoldenRatio,
outerStarShortScale=0.674,outerStarLongScale=1
},{
AbsoluteThickness[radius*12],

(* Central spikes *)
Line[Table[{center,center+innerStarShortScale*radius*{Cos[θ],Sin[θ]}},{θ,Pi/2,2Pi+Pi/2,2Pi/5}]],

(* Inner star *)
Line[Table[
center+If[EvenQ[n],innerStarShortScale,innerStarLongScale]*radius*{Cos[n*2Pi/10+Pi/2],Sin[n*2Pi/10+Pi/2]}
,{n,0,10,1}]],

(* Outer star *)
Line[Table[
center+If[EvenQ[n],outerStarShortScale,outerStarLongScale]*radius*{Cos[n*2Pi/20],Sin[n*2Pi/20]}
,{n,0,20,1}]],

(* Peak-to-peak struts *)
CapForm["Butt"],
Line[Table[
{
center+innerStarLongScale*radius*{Cos[n*2Pi/5+3Pi/10],Sin[n*2Pi/5+3Pi/10]},
center+outerStarLongScale*radius*{Cos[n*2Pi/5+3Pi/10],Sin[n*2Pi/5+3Pi/10]}
}
,{n,0,5,1}]],

(* Trough-to-trough struts *)
Line[Table[
{
center+outerStarShortScale*radius*{Cos[n*2Pi/5+Pi/2-Pi/10],Sin[n*2Pi/5+Pi/2-Pi/10]},
center+innerStarShortScale*radius*{Cos[n*2Pi/5+Pi/2],Sin[n*2Pi/5+Pi/2]},
center+outerStarShortScale*radius*{Cos[n*2Pi/5+Pi/2+Pi/10],Sin[n*2Pi/5+Pi/2+Pi/10]}
}
,{n,0,5,1}]]
}];
cloudLogo[center_]:=cloudLogo[center,1];
cloudLogo[]:=cloudLogo[{0,0}];
With[{(*use a Taylor series to approximate Sin and Cos,since they are not supported in the client-side evaluator yet*)sinSeries=N[Normal[Series[Sin[#],{#,0,9}]]],
period=N[2Pi],threePeriod=N[3*2*Pi],halfPeriod=N[Pi]},
DynamicModule[{t=0,sin=Function[sinSeries]},
Column@{
Animator[Dynamic[t],{0,1},AppearanceElements->{"PlayPauseButton"},AnimationRunning->False],
Graphics[{
(* Data *)
{
Hue[0.6,0.5],Thickness[0.04],
Dashing[{0.01,0.02},Dynamic[-t]],
Line[{{1,0.7},{3,0.7}}],
Dashing[{0.01,0.02},Dynamic[t]],
Line[{{1,0.3},{3,0.3}}]
},

(* Monitor Body *)
GrayLevel[0.3],
Rectangle[{0,0},{4/3,1},RoundingRadius->0.1],
Rectangle[{4/3/2-0.1,-0.2},{4/3/2+0.1,0}],
Disk[{4/3/2,-0.21},{0.3,0.15},{0,Pi}],

(* Monitor Screen *)
White,Rectangle[{0,0}+{0.1,0.1},{4/3,1}-{0.1,0.1}],

(* Monitor Screen Content *)
{
(* Input cell *)
Black,
Line[#-{0.05,0.05}&/@{{0.2,0.8},{0.7,0.8}}],
Line[#-{0.05,0.05}&/@{{0.2,0.75},{0.6,0.75}}],

(* Top bar *)
{
RGBColor[{203,47,30}/255],Rectangle[{0.1,0.85},{0.15,0.9}],
GrayLevel[51/255],Rectangle[{0.15,0.85},{4/3-0.1,0.9}],
GrayLevel[229/255],Rectangle[{0.1,0.8},{4/3-0.1,0.85}]
},

(* Brackets *)
{
GrayLevel[179/255],
Line[#+{0.1,-0.05}&/@{{1.03,0.81},{1.05,0.81},{1.05,0.73},{1.03,0.73}}],
Line[#+{0.1,-0.05}&/@{{1.07,0.81},{1.09,0.81},{1.09,0.39},{1.07,0.39}}],
Line[#+{0.1,-0.16}&/@{{1.03,0.81},{1.05,0.81},{1.05,0.5},{1.03,0.5}}]
},

(* Plot *)
{
Thickness[0.004],Hue[0.4,1,0.6],
Dashing[{0.5,1},Dynamic[0.5*(1-t)]],
Line[Table[{x,2Sin[x]}/15+{0.15,0.5},{x,0,2Pi,0.01}]]
}
},

(* Cloud *)
{
{EdgeForm[{Thickness[0.007],Black}],White,Disk[{3.5,0.4},Dynamic[0.7416407864998737-0.02*sin[Mod[threePeriod*t,period]-halfPeriod]]]},
{RGBColor[{225,18,11}/255],cloudLogo[{3.5,0.4},1/GoldenRatio]}
}
},ImageSize->{576,236},PlotRange->{{0,4.4},{-0.5,1.3}}]},UnsavedVariables->{t}]]