Customized Rendering with GLSL

Yuzhu Lu, Wolfram Research

#WolframTechConf

ABSTRACT

Mathematica graphics has support for using existing OpenGL Shading Language (GLSL) files to customize 2D/3D graphics rendering through SurfaceAppearance. Many graphics features are built on top of it. Users can also use it to have customized rendering and more advanced applications.

What is SurfaceAppearance

Graphics features implemented with SurfaceAppearance

How SurfaceAppearance works and simple example

SurfaceAppearance related updates

Multiple pass rendering

Post processing for 3D Graphics

Applications

What is SurfaceAppearance

SurfaceAppearance is an internal function allow user to use GLSL files to program GPU for customized Graphics and Graphics3D rendering

SurfaceAppearance can allow associated GLSL programs (Vertex Shader and Fragment Shader) access resolved primitive vertex data, option values and directives, etc

Graphics features implemented with SurfaceAppearance

ToonShading, GoochShading, StippleShading, HalftoneShading, HatchShading, HatchFilling, PatternFilling, LinearGradientFilling, RadialGradientFilling, ConicGradientFilling, MaterialShading, SolidRegion, DropShadowing, Blurring, Haloing

GraphicsGrid[{{Graphics3D[{ToonShading[],Sphere[]},Lighting->"Accent",PlotLabel->ToonShading],Graphics3D[{GoochShading[],Sphere[]},Lighting->"Accent",PlotLabel->GoochShading],Graphics3D[{StippleShading[],Sphere[]},Lighting->"Accent",PlotLabel->StippleShading],Graphics3D[{HalftoneShading[],Sphere[]},Lighting->"Accent",PlotLabel->HalftoneShading]},{Graphics3D[{HatchShading[],Sphere[]},Lighting->"Accent",PlotLabel->HatchShading],Graphics3D[{MaterialShading["Iron"],Sphere[]},Lighting->"ThreePoint",PlotLabel->MaterialShading],CSGRegion["Difference",{Ball[],Cuboid[{0,-1,0}]},PlotLabel->CSGRegion],Graphics3D[{PhongShading[],Sphere[]},Lighting->"Accent",PlotLabel->PhongShading]},{Graphics3D[{GouraudShading[],Sphere[]},Lighting->"Accent",PlotLabel->GouraudShading],Graphics3D[{FlatShading[],Sphere[]},Lighting->"Accent",PlotLabel->FlatShading],Graphics[{HatchFilling[],Disk[]},PlotLabel->HatchFilling],Graphics[{PatternFilling["DiamondBox",{20,20}],Disk[]},PlotLabel->PatternFilling]},{Graphics[{LinearGradientFilling[{Red,Blue}],Disk[]},PlotLabel->LinearGradientFilling],Graphics[{RadialGradientFilling[{Red,Blue}],Disk[]},PlotLabel->RadialGradientFilling],Graphics[{ConicGradientFilling[{Red,Blue}],Disk[]},PlotLabel->ConicGradientFilling],Graphics[{DropShadowing[],Disk[]},PlotLabel->DropShadowing]},{Graphics[{Blurring[],Disk[]},PlotLabel->Blurring],GeoGraphics[{GeoStyling["OutlineMap", 
      Haloing[{Hue[0.26950361308113985`, 0.2155963274060958, 0.63], 
     White}, 1, 10], RGBColor[
   0.7411765055418048, 0.8549020847953749, 0.6705883350216774]], 
    Polygon[LinguisticAssistant]},PlotLabel->Haloing]}},ImageSize->1200]

How SurfaceAppearance works and simple example

\[LeftAssociation]
"Name"->"name",
"Attributes"->\[LeftAssociation]"attrib1"->attribute type,...\[RightAssociation],"Parameters"->\[LeftAssociation]"parameterName1"->{nameUsedInShader,val,parameter type},...\[RightAssociation],"ShaderPrograms"->\[LeftAssociation]"GLFragmentProgram"->GLSL Fragment Shader,"GLVertexProgram"->GLSL Vertex Shader\[RightAssociation]
\[RightAssociation]

SurfaceAppearanceDocumentationFile=FileNameJoin[{NotebookDirectory[],"SurfaceAppearance.nb"}];
NotebookOpen[SurfaceAppearanceDocumentationFile]

Export["vShaderSimple.glsl",ImportByteArray[FileAsset[EvaluationNotebook[],"vShaderSimple"]["Bytes"],"Text"],"Text"]
Export["fShaderSimple.glsl",ImportByteArray[FileAsset[EvaluationNotebook[],"fShaderSimple"]["Bytes"],"Text"],"Text"]

SetDirectory[NotebookDirectory[]];
shaderGrayLevel=\[LeftAssociation]
"Name"->"GrayLevelTextureMapping",
"Attributes"->\[LeftAssociation]"a_position"->"ATTRIB_VERTEX","a_texcoord"->"ATTRIB_TEXTURECOORD"\[RightAssociation],"Parameters"->\[LeftAssociation]"Matrix"->{"mvp_matrix","TRANSFORMMATRIX","MATRIX4"}\[RightAssociation],"ShaderPrograms"->\[LeftAssociation]"GLFragmentProgram"->"fShaderSimple.glsl","GLVertexProgram"->"vShaderSimple.glsl"\[RightAssociation]
\[RightAssociation];

Dataset[shaderGrayLevel]

Import[#]&/@{"vShaderSimple.glsl","fShaderSimple.glsl"}

Add the shader definition to the SurfaceAppearance library

If added multiple times, the newer one will override the old one

FE`Evaluate@Evaluate[FEPrivate`AddSurfaceAppearanceDefinition@shaderGrayLevel];

Check if added successfully

FE`Evaluate[FEPrivate`ListSurfaceAppearanceDefinitions["GrayLevelTextureMapping"]]

Look up in the SurfaceAppearance library by name and use the shader if available

Graphics[{SurfaceAppearance["GrayLevelTextureMapping"],Texture[],Polygon[{{0,0},{1,0},{1/2,1}}]},ImageSize->300]

SurfaceAppearance related updates

Improved Haloing/DropShadowing rendering and Antialiasing quality for Text, Line and Arrow

Graphics[{#,Text@Style[Sqrt[Math],10]}]&/@{Haloing[Red],DropShadowing[Red]}

Graphics[{#,Arrowheads[{Medium}],Arrow[{{20,9.5},{10.5,9.5}}]}]&/@{Haloing[Red],DropShadowing[Red]}

SurfaceAppearance based XXXShading now renders much faster with transparency (depth peeling technology)

Graphics3D[{Green,Opacity[0.6],Sphere[],MaterialShading[{"Iron",Red}],Sphere[{0.5,0.5,0.5}]},Lighting->"Neutral"]

RegionPlot3D[1<=x^2+y^2+z^2<=2&&x+y+z>0,{x,-2,2},{y,-2,2},{z,-2,2},PlotPoints->15,Mesh->None,PlotStyle->MaterialShading[{"Satin",RGBColor[0,1,1,0.5]}],Lighting->"ThreePoint"]

More predefined values for SurfaceAppearance

"DARK_MODE"
"DEPTH_PEELING_VIEWPORT"
"DEPTH_PEELING_DEPTH_TEXTURE"
"DPI_MAG_SCALE"
"ANTI_ALIASING"
"VALID_VERTEX_COLOR"
"CONTAINS_VERTEX_COLOR_INTERPOLATION"
"VALID_TEXTURE"
"ROTATION"
"IMAGE_EFFECT_BACKGROUND_COLOR"

More utility functions to convert GLSL to other shading languages like SPRIV, MSL and HLSL

Import@"fShaderSimple.glsl"

targetShaders = {"spirv","GLSL","GLSL ES","HLSL","HLSL 51","MSL", "MSL IOS"};

Column[(Grid[{{#,FE`Evaluate@FEPrivate`ConvertGLSLShader["fShaderSimple.glsl","frag","OpenGL",#],#}},Frame->All,FrameStyle->Directive[Blue]]&/@targetShaders),Frame->All,FrameStyle->Directive[Red,Thick]]

Import@"vShaderSimple.glsl"

Column[(Grid[{{#,FE`Evaluate@FEPrivate`ConvertGLSLShader["vShaderSimple.glsl","vert","OpenGL",#],#}},Frame->All,FrameStyle->Directive[Blue]]&/@targetShaders),Frame->All,FrameStyle->Directive[Red,Thick]]

All SurfaceAppearance based features now renders natively with D3D11 on Windows (Experimental)

1. Better rendering quality on Windows
2. All SurfaceAppearance based features now works on Windows without OpenGL support

Multiple pass rendering

SurfaceAppearance image effect functionality can be nested with Inset

Graphics[{DropShadowing[{-8,-8}],Inset@Graphics[{Blurring[],Inset@Button["Hello",Print["I am blur"]]}]},ImageSize->Small]

More passes of rendering through Inset

Export["fShaderTV.glsl",ImportByteArray[FileAsset[EvaluationNotebook[],"fShaderTV"]["Bytes"],"Text"],"Text"]

SetDirectory[NotebookDirectory[]];
badTVShader=\[LeftAssociation]
"Name"->"BadTV",
"Attributes"->\[LeftAssociation]"a_position"->"ATTRIB_VERTEX","a_texcoord"->"ATTRIB_TEXTURECOORD"\[RightAssociation],"Parameters"->\[LeftAssociation]"Matrix"->{"mvp_matrix","TRANSFORMMATRIX","MATRIX4"},"IsImageEffectShader"->{"u_isImageEffectShader",True,"BOOL"},"ImageEffectShowOriginal"->{"u_imageEffectShowOriginal",False,"BOOL"}\[RightAssociation],"ShaderPrograms"->\[LeftAssociation]"GLFragmentProgram"->"fShaderTV.glsl","GLVertexProgram"->"vShaderSimple.glsl"\[RightAssociation]
\[RightAssociation];

FE`Evaluate@Evaluate[FEPrivate`AddSurfaceAppearanceDefinition@badTVShader];
g2d={Thick,Green,Rectangle[{0,-1},{2,1}],Red,Disk[],Blue,Circle[{2,0}],Yellow,Polygon[{{2,0},{4,1},{4,-1}}],Purple,Arrowheads[Large],Arrow[{{4,3/2},{0,3/2},{0,0}}],Black,Dashed,Line[{{-1,0},{4,0}}]};
size={260,260};

Graphics[{SurfaceAppearance["BadTV"],Inset@Graphics[{Haloing[White,3],Opacity[0.2],g2d},ImageSize->size]},ImageSize->size]

Post processing for 3D Graphics

Use image effect/post processing functionality through inset

g3d=Graphics3D[{Blue,Cylinder[],Red,Sphere[{0,0,2}],Black,Thick,Dashed,Line[{{-2,0,2},{2,0,2},{0,0,4},{-2,0,2}}],Yellow,Polygon[{{-3,-3,-2},{-3,3,-2},{3,3,-2},{3,-3,-2}}],Green,Opacity[.3],Cuboid[{-2,-2,-2},{2,2,-1}]},ImageSize->size];

Deploy[Graphics[{#,Inset[InputField[g3d,ImageSize->{Scaled[1],Scaled[1]},Appearance->None]]},ImageSize->size]]&/@{DropShadowing[Lighter[Lighter[Blue]]],Blurring[10]}

multiple image effect/post processing can apply through nested inset

Deploy[Graphics[{#,Inset@Graphics[{Blurring[0],Inset[InputField[g3d,ImageSize->{Scaled[1],Scaled[1]},Appearance->None]]},ImageSize->size]},ImageSize->size]]&/@{DropShadowing[{-8,-8}],SurfaceAppearance["BadTV"]}

Deploy[Graphics[{#,Inset@Graphics[{Blurring[10],Inset[InputField[g3d,ImageSize->{Scaled[1],Scaled[1]},Appearance->None]]},ImageSize->{size}]},ImageSize->size]]&/@{DropShadowing[{-8,-8}],SurfaceAppearance["BadTV"]}

Apple video effects  to Animated RasterBox

SetDirectory[NotebookDirectory[]];
URLDownload["http://exampledata.wolfram.com/cars.avi","cars.avi"];

ApplyEffects[img_]:=RawBoxes[ToBoxes[img]/.RasterBox[arg_,opts___]:>{SurfaceAppearance[Dynamic@videoEffect],RasterBox[arg,opts]}];
video=ApplyEffects[Import["cars.avi","Animation",ControlPlacement->Top]];

PopupMenu[Dynamic[videoEffect],Prepend[{"EdgeDetection","EdgeDetection2","BadTV","VideoBlur","Lens"},None]]

video

FE`Evaluate[FEPrivate`ListSurfaceAppearanceDefinitions["BadTV"]]

Applications

Customized rendering

Reuse GLSL resource and combine it with Mathematica functionality

Move computing to hardware directly:  Reduce CPU cost

No huge data transfer through MathLink: Reduce data transfer load and memory consumption

Easier to add new features( following GLSL 330 standard )