State of Graphics in the Cloud

Matthew Adams <matthewa@wolfram.com>

bubble[pos_,radii_,angle_]:={
RGBColor[.5,.7,.95],Disk[pos+.04radii*{Cos[angle],Sin[angle]},radii],
GrayLevel[0.95],Disk[pos,radii]
};
bubble[pos_,radii_]:=bubble[pos,radii,0]
Graphics[{
{GrayLevel[0.95],Disk[{1.8,0},0.3]},
bubble[{1.5,1},{1,1},0.4Pi],
bubble[{1,-.5},{1,0.7},3Pi/2],
bubble[{3,0},{1,1},1.8Pi],
bubble[{2.6,0.85},{1,0.9},0.2Pi],
bubble[{2.2,-0.65},{0.65,0.5},1.6Pi],
bubble[{0.5,0.7},{1,0.8},0.8Pi],
bubble[{0.1,0},{1.2,0.9},1.1Pi],
Inset[Graphics[{
Red,Rectangle[],
Green,Polygon[{{1.3,0},{2,1},{2.7,0}}],
Blue,Disk[{3.5,0.5},0.55]
},ImageSize->250]]
},ImageSize->500]

Abstract

This talk will dive into the state of graphics in the Cloud - primarily focusing on the ongoing transition to our next-generation 2D graphics rendering system: Native Graphics.

Questions

What?

What is Native Graphics?

Why?

Why are we building Native Graphics?

How?

How does Native Graphics work?

When?

When are specific graphics features going to be supported?

What?

What is Native Graphics?

What is Native Graphics?

Next-generation graphics rendering system

Total reimplementation of Cloud Graphics

Incrementally replacing old rendering system; both are running today

End users don’t need to opt-in

Why?

Why are we building Native Graphics?

Why are we building Native Graphics?

styleTableHeader[text_]:=Style[text,Bold,RGBColor[{225,18,11}/255]];
Style[TextGrid[{
{
Column[styleTableHeader/@{"Old Graphics System","\"Helper-Data\""}],
Column[styleTableHeader/@{"New Graphics System","\"Native\""}]
},
{"Does not support all Wolfram Language Graphics features, but good enough for most use cases","Provides complete graphics support - matches Desktop Mathematica"},
{"Slower","Faster"},
{"More moving parts (bugs more likely)","Fewer moving parts (bugs less likely)"},
{"Requires network for every render","Opens possibility of much faster Manipulate"},
{"Can render any graphic","Supports a smaller (but growing) subset of graphics"}
},Dividers->Center,Spacings->{2,2}],FontSize->18]

Some Specific Improvements

Demo Setup

(* Hacky function to force graphics to render using helper-data that would otherwise be rendered natively. This depends on the option CoordinatesToolOptions not yet being supported natively. *)
forceHelperData[Graphics[args___,opts:OptionsPattern[]]]:=Graphics[args,opts,CoordinatesToolOptions ->Automatic]

Dynamic inside Graphics

DynamicModule[{radius=1},
Column[{
Slider[Dynamic@radius,{1,10}],
forceHelperData@Graphics[{Circle[{0,0},10],Disk[{0,0},Dynamic@radius]}]
}]
]

DynamicModule[{radius=1},
Column[{
Slider[Dynamic@radius,{1,10}],
Graphics[{Circle[{0,0},10],Disk[{0,0},Dynamic@radius]}]
}]
]

Side Note: Dynamic *outside* Helper-Data Graphics *is* supported

DynamicModule[{radius=1},
Column[{
Slider[Dynamic@radius,{1,10}],
Dynamic@forceHelperData@Graphics[{Circle[{0,0},10],Disk[{0,0},radius]}]
}]
]

Correct behavior during manual resize

Also no delay/snap-back after the resize is finished

forceHelperData@Graphics[{Disk[{0,0},Offset[100]]},ImageSize->Medium]

Graphics[{Disk[{0,0},Offset[100]]},ImageSize->Medium]

This can get complicated...

Module[{r=.1},
forceHelperData@Graphics[{
Red, Disk[Offset[{80, 0}, {0, .5}],r],
Blue, Disk[{1, -.5},r],
RGBColor[0,0,0],Disk[{-1,1},r],Disk[{-1,-1},r]}
,ImageSize->{225,Automatic}]]

Module[{r=.1},
Graphics[{
Red, Disk[Offset[{80, 0}, {0, .5}],r],
Blue, Disk[{1, -.5},r],
RGBColor[0,0,0],Disk[{-1,1},r],Disk[{-1,-1},r]}
,ImageSize->{225,Automatic}]]

No jitter

{forceHelperData@Graphics[{Circle[{0.4,0},0.1],Inset[0123456789,{0,0}]},ImageSize->Automatic]}

{Graphics[{Circle[{0.4,0},0.1],Inset[0123456789,{0,0}]},ImageSize->Automatic]}

Performance

~35% faster for the example below (depends on a lot of factors - particularly the network connection)

Potential for much more significant speed increases with compiled Manipulate

data=<|"native"->{{0.,0.55},{0.5654,0.6428},{1.2238,0.5761},{1.8133,0.4866},{2.3129,0.4871},{2.8142,0.485},{3.3138,0.5204},{3.8479,0.6032},{4.4652,0.4836},{4.9656,0.5532},{5.533,0.4802},{6.0276,0.5728},{6.6147,0.5049},{7.1348,0.6002},{7.7495,0.5024},{8.2669,0.4834},{8.7652,0.4843},{9.2625,0.5221},{9.7979,0.471},{10.2834,0.4674}},"helper data"->{{0.,0.7902},{0.7936,0.8095},{1.606,0.9177},{2.5269,0.8199},{3.3496,0.8129},{4.1655,0.8083},{4.9766,0.8171},{5.7964,0.8095},{6.6087,0.8162},{7.4276,0.8098},{8.256,0.8185},{9.0772,0.7826},{9.8627,0.8332},{10.7279,0.7977},{11.528,0.7607},{12.2913,0.7569},{13.0511,0.8203},{13.8743,0.7682},{14.6451,0.765},{15.4128,0.7967}}|>;
ListPlot[data,PlotRange->{All,{0,All}},FrameLabel->{Style["Slider move time (seconds)",16],Style["Time (seconds) from slider move\n until Manipulate DOM update",16]},ImageSize->Large,PlotLabel->Style["Time to update Manipulate with 1000 random circles\n",16,Italic],Frame->True]

Integration with other parts of the notebook

Relative Sizing

DynamicModule[{width=200},
Column[{
Slider[Dynamic[width],{100,500}],
Framed[Column@{
forceHelperData@Graphics[Inset[{Style["hello",FontSize->Scaled[0.2]]}],ImageSize->Full],
Graphics[Inset[{Style["hello",FontSize->Scaled[0.2]]}],ImageSize->Full]
},
ImageSize->Dynamic[width]
]
}]
]

Magnification

Style[forceHelperData@Graphics[Disk[]],Magnification->1/2]

Style[Graphics[Disk[]],Magnification->1/2]

Error Handling

forceHelperData@Graphics[Disk["foo"]]

Graphics[Disk["foo"]]

How?

How are we building Native Graphics?

Background: Anatomy of a Graphic

Background: Anatomy of a graphic

Graphics[]

Graphics[
Line[{{0,0},{1,1}}]
]

Table[{x,Sin[x]},{x,0,2Pi}]//MatrixForm

Graphics[
Line[Table[{x,Sin[x]},{x,0,2Pi}]]
]

Graphics[
Line[Table[{x,Sin[x]},{x,0,2Pi,0.01}]]
]

Graphics[{
RGBColor[0.368417,0.506779,0.709798],
AbsoluteThickness[1.6],
Line[Table[{x,Sin[x]},{x,0,2Pi,0.01}]]
}]

Graphics[{
RGBColor[0.368417,0.506779,0.709798],
AbsoluteThickness[1.6],
Line[Table[{x,Sin[x]},{x,0,2Pi,0.01}]]
}
,Axes->True
]

Graphics[{
RGBColor[0.368417,0.506779,0.709798],
AbsoluteThickness[1.6],
Line[Table[{x,Sin[x]},{x,0,2Pi,0.01}]]
}
,Axes->True
,AspectRatio->1/GoldenRatio
]

Plot[Sin[x],{x,0,2Pi}]

So how do these graphics get drawn?

Need two pieces:

downArrow[]:=Graphics[{Thick,Arrowheads[Large],Arrow[{{0,1},{0,0}}]},PlotRange->{{-0.5,0.1},{0,1}},ImageSize->Tiny];
blackBox[]:=Graphics[Rectangle[],ImageSize->Tiny];
Column[{
Framed[Style["Plot[Sin[x],{x,0,2Pi}]",18]],
downArrow[],
blackBox[],
Row[{downArrow[],Style["<Precise Drawable Graphics Description>",18]}],
blackBox[],
downArrow[],
Framed@Plot[Sin[x],{x,0,2Pi},ImageSize->Small]
}]

So how do these graphics get drawn?

Need two pieces:

downArrow[]:=Graphics[{Thick,Arrowheads[Large],Arrow[{{0,1},{0,0}}]},PlotRange->{{-0.5,0.1},{0,1}},ImageSize->Tiny];
blackBox[]:=blackBox[""];
blackBox[label_]:=Graphics[{Rectangle[],Inset[Style[label,18,White]]},ImageSize->Tiny];
Column[{
Framed[Style["Plot[Sin[x],{x,0,2Pi}]",18]],
downArrow[],
blackBox["Kernel"],
Row[{downArrow[],Style["<Wolfram Box Language>",18]}],
blackBox["FrontEnd"],
downArrow[],
Framed@Plot[Sin[x],{x,0,2Pi},ImageSize->Small]
}]

Under the hood: Graphics in Desktop

Helper Data Graphics

Native Graphics

Native Graphics

When?

When are specific graphics features going to be supported?

Coming Soon to Native Graphics

Headliners

Inset

DynamicModule[{pos={0,0},opos={0,0}},
Row@{
Column@{
Row@{Column@{Style["pos",18],Button["Reset",(pos={0,0})&]},Slider2D[Dynamic[pos],{-1,1}]},Row@{Column@{Style["opos",18],Button["Reset",(opos={0,0})&]},Slider2D[Dynamic[opos],{-1,1}]}
},
Graphics[{
Line[{{{0,-1},{0,1}},{{-1,0},{1,0}}}],
Inset[
Graphics[{PointSize[0.1],Hue[0],PointBox[Dynamic[opos]]},PlotRange->{{-1,1},{-1,1}},Background->Black,ImageSize->Tiny],
Dynamic[pos],
Dynamic[opos]
]
},ImageSize->Medium,PlotRange->{{-1,1},{-1,1}}]
}
]

AxisObject

More Primitives

Polygon (without holes)

FilledCurve and JoinedCurve

More Interactivity

Locator

More Options

ImageMargins

ImagePadding

PlotRangeClipping

Already Supported

TextGrid[{{"Primitives","Directives","Options","Coordinates","External","Other"},
{
Column@{"CircleBox","DiskBox","RectangleBox","PointBox","LineBox","BezierCurveBox*"},
Column@{"Directive","FaceForm","EdgeForm","Thickness","AbsoluteThickness","PointSize","AbsolutePointSize","Opacity","RGBColor","GrayLevel","Hue","CMYKColor","LABColor","LCHColor","LUVColor","XYZColor","CapForm","JoinForm","Dashing","AbsoluteDashing"},
Column@{"AspectRatio","Background","Epilog","ImageSize","ImageSizeRaw","Method","PlotRange","PlotRangePadding","PreserveImageOptions","Prolog"},
Column@{"Scaled","ImageScaled","Offset"},
Column@{"Antialiasing","ImageSizeMultipliers","Magnification"},
Column@{"Dynamic","CompressedData","List","NCache"}
}},Frame->All,Alignment->Top,ItemSize->Full]

Conclusion

State of Graphics in the Cloud

Native Graphics rendering system progressing

Inset, AxisObject, Plot, and more coming soon!

Continuing to support and improve reliability of the helper-data rendering system

Questions - Answered

What?

What is Native Graphics? Full Wolfram Language graphics support in-browser.

Why?

Why are we building Native Graphics? It provides a number of performance and functionality improvements.

How?

How does Native Graphics work? It is the FrontEnd graphics processing code, re-implemented for web browsers.

When?

When are specific graphics features going to be supported? We’re incrementally rolling out new supported features. Maybe there’s one you particularly want to see?

Who?

Cloud Notebooks Team:

Máté Csigi

John Pacey

Emre Hoş

Konstantin Sushenko

Franco Peña-Campos

Desktop Team Contributors:

James Lane

Yuzhu Lu

Rob Raguet-Schofield

Brett Champion

Questions?

bubble[pos_,radii_,angle_]:={
RGBColor[.5,.7,.95],Disk[pos+.04radii*{Cos[angle],Sin[angle]},radii],
GrayLevel[0.95],Disk[pos,radii]
};
bubble[pos_,radii_]:=bubble[pos,radii,0]
Graphics[{
{GrayLevel[0.95],Disk[{1.8,0},0.3]},
bubble[{1.5,1},{1,1},0.4Pi],
bubble[{1,-.5},{1,0.7},3Pi/2],
bubble[{3,0},{1,1},1.8Pi],
bubble[{2.6,0.85},{1,0.9},0.2Pi],
bubble[{2.2,-0.65},{0.65,0.5},1.6Pi],
bubble[{0.5,0.7},{1,0.8},0.8Pi],
bubble[{0.1,0},{1.2,0.9},1.1Pi],
Inset[Graphics[{
Red,Rectangle[],
Green,Polygon[{{1.3,0},{2,1},{2.7,0}}],
Blue,Disk[{3.5,0.5},0.55]
},ImageSize->250]]
},ImageSize->500]

Bonus Slides

Zoom/Pan

Graphics[{Red,Disk[{0,0},1,{0,Pi/2}],Green,Disk[{0,0},1,{Pi/2,Pi}],Blue,Disk[{0,0},1,{Pi,3Pi/2}],Yellow,Disk[{0,0},1,{3Pi/2,2Pi}]},Method->"InteractiveControls"->{"Zoomable"->True,"Toolbar"->True},ImageSize->Medium]

Plot[Sin[10x]+0.1Sin[1000x+0.2],{x,0,2Pi},Method->"InteractiveControls"->{"Zoomable"->True,"Toolbar"->True}]

Note that this is only a Graphics-level zoom

Interactive Graphics

JSONString[y_] := ExportString[y, "JSON","Compact"->True];

Typeset`MakeBoxes[Annotation[x_, {"JSONMetadata", y_}], fmt_, head_] :=
With[{g = Typeset`MakeBoxes[x, fmt, head],
s = "JSONMetadata"[JSONString[y]]}, Typeset`Hold[TagBox[g, s]]];

Framed[Graphics[{
AbsoluteThickness[2.5],
Hue[0.8,1,0.8],
Annotation[
Line[Table[{x,0.5Sin[3x]},{x,0,2Pi,2Pi/300}]],
{"JSONMetadata",{"InteractiveGraphics"->
{"version"->"1.1",

"TrackMousePosition"->{"x"->True,"y"->False},

"effects"->{{"type"->"HighlightPoint","ratio"->3}}
}}}
]
},
PlotRangePadding->0.5,
ImageSize->500
],FrameMargins->0]

annotatedData=Annotation[Accumulate[RandomReal[{-0.8,1},50]],{"JSONMetadata",{"InteractiveGraphics"->
{"version"->"1.1",

"TrackMousePosition"->{"x"->True,"y"->False},

"effects"->{
{"type"->"Droplines","linePlacement"->{"x"->"All","y"->"All"}}
}
}}}];
Row[{
ListLinePlot[annotatedData,"DefaultGraphicsInteraction"->None,ImageSize->300],
ListPlot[annotatedData,"DefaultGraphicsInteraction"->None,ImageSize->300]
}]

Graphics[Annotation[{
PointSize[0.01],
Red,Point[RandomPoint[Disk[],500]],
Blue,Point[RandomPoint[Disk[{1.2,0}],500]]
},
{"JSONMetadata", {"InteractiveGraphics" ->
{"version"->"1.1",
"TrackMousePosition"->{"x"->True,"y"->True},
"OnlyNearestPrimitive"->True,
"effects"->{{"type"->"Highlight","ratio"->1.4}}
}
}}]]

GIF Trick

(*Cloud-Only GIF*)
Style[Dynamic[RawBoxes[FEPrivate`ImportImage[URL["https://content.wolfram.com/uploads/sites/43/2019/02/05-treadmill-r22.gif"],"CloudPixelRatio"->2]]],Selectable->False]

Goal: Snappy Manipulate in Cloud

Challenge: going back and forth to the server for each update can be slow

DynamicModule[{ωList={}},Column@{Manipulate[ωList=Append[ωList,ω];
Plot[Sin[ω*x],{x,0,2 Pi}],{ω,1,10},TrackedSymbols:>{ω}],Dynamic[ωList]}]