ExternalEvaluate Workshop

Todd Gayley & Riccardo Di Virgilio, Wolfram Reasearch

(RowDefault[, (Spacer1), , (Spacer1), ])  #WolframTechConf

Download this notebook →

Evaluators →

List of all evaluators we will be working with today

All other evaluators →

Shell challenges

When using Shell in ExternalEvaluate you can work with the program output as well as the exit code of any program you are running.

Shell Challenge (easy)

Create a function to clone a git repo to a destination if it doesn’t exists, pull all changes if it does exists and move to a particular branch.

cloneRepo[
"https://github.com/hpaluch/rust-hello-world", 
"path/to/destination", 
"master"
]

Hint

You can check if a folder is a git repo if it contains a folder named “.git”, or you can simply run the clone command and have it fail if the destination already exists.

Solution

cloneRepo[url_, dest_, branch_:"master"] := 
	ExternalEvaluate[
		"Shell", Flatten @ {
			If[
				! FileExistsQ[FileNameJoin[{dest, ".git"}]],
				"git clone " <> url <> " " <> dest,
				{}
			],
			"cd " <> dest,
			"git checkout "<> branch,
			"git pull --all"
		}
	]
	
cloneRepo[
	"https://github.com/hpaluch/rust-hello-world", 
	"~/Downloads/myClonePath", 
	"master"
]

Shell Challenge (advanced)

Use Shell to automate the download of git repository, build the program from source and run the program. 

You can pick a repo of your choice or use one of the following:

You can assume git, and all building tools (such as rustc, cargo, etc.. ) are already present on the machine.

Solution

ExternalEvaluate[
"Shell", {
 "echo 🧹 cleanup the environment",
"rm -rf ~/Downloads/rust-hello-world",
 "echo 🧑‍🔬 download latest code",
"cd ~/Downloads",
"git clone https://github.com/hpaluch/rust-hello-world.git", 
"cd    ~/Downloads/rust-hello-world",
"echo 👷‍♀️ build the program",
"cargo update",
"cargo build",
"echo 🚀 run the program",
"cargo run"
}]

Shell Challenge (hard)

Define a createPythonEnv function that is using Shell under the hood to automate the creation of a python virtual environment and installation of a requirement file using pip. 

The user can input a python target, requirement file and an output directory.

Initialization

requirements = Export[
FileNameJoin[{NotebookDirectory[], "requirements.txt"}], 
"django==4.0.1",
 "Text"
]

target  = ExternalEvaluate[{"Shell", "ReturnType" -> "StandardOutput"}, "which python3"]

outputFolder = FileNameJoin[{NotebookDirectory[], "venv"}]

createPythonEnv[target, requirements, outputFolder] := "…"

Solution

ClearAll[createPythonEnv]
createPythonEnv[target_, requirements_, outputFolder_] := ExternalEvaluate[
"Shell", {
 "echo 🧹 cleanup the environment",
"rm -rf <* outputFolder *>",
 "echo 🏘 creating venv",
"<* target *> -m venv <* outputFolder *>",
 "echo 🍾 installing requirements",
"<* outputFolder *>/bin/python -m pip install -r <* requirements *>"
}]

createPythonEnv[target, requirements, outputFolder]

Python challenges

When using Python in ExternalEvaluate you can use the whole power of wolframclient.

Python Challenge (easy)

Define the range function that is internally using Python. The function should be equivalent to:

range[n1_Integer, n2_Integer, step_Integer:1] :=
	Range[n1, n2, step]
range[1, 10]

Hint

You need to ExternalEvaluate syntax to pass arguments to a defined function

ExternalEvaluate[lang, <|"Command" -> ..., "Arguments" -> ...|>]

Solution

range[args___] := 
ExternalEvaluate["Python", "range" -> {args}]
range[10]
range[1, 10, 2]

Python Challenge (hard)

Create a  function that returns AST for a string of python code.
You can use python evaluator and the AST module. 
The function needs to return a WL expression representing the python code.

getPythonAST["lambda a: [i + 2 for i in a]"]

Hint

you can use ast2json to convert from AST to expressions

Solution

getPythonAST[code_] := 
Last @ ExternalEvaluate["Python", {
"import ast, ast2json",
"lambda code: ast2json.ast2json(ast.parse(code, mode='eval'))" -> code
}]
getPythonAST["lambda a: [i + 2 for i in a]"]

Python Challenge (hard)

Use pygments to create a function that is doing syntax highlighting of a block of code.

The function should take the name of the language as 1st argument and a string of code as second argument

formatCode["python", "map(lambda a: a+2, [r for i in range(10)])"]

Hint 1

you can use pygments.lexers.get_lexer_by_name to resolve the language name

Hint 2

you can use the image formatter to keep things or write a custom formatter.

Solution 1 (easy)

This solution is imported the highlighted text using Image

formatCode[lang_, code_] :=
Replace[
ExternalEvaluate["Python", {
"from pygments.lexers import get_lexer_by_name",
"from pygments import highlight",
"from pygments.formatters import ImageFormatter as F",
"lambda lang, code: highlight(code, get_lexer_by_name(lang), F())" -> {lang, code}
}
],
{___, b_ByteArray} :> ImportByteArray[b, "Image"]
]
formatCode["python", "map(lambda a: a+2, [r for i in range(10)])"]

Solution 2 (harder)

This solution is creating a custom to highlight

getColorForType[expr_] := getColorForType[expr] = RandomColor[]
formatTokens[tokens_]:= Row @ Apply[
Style[#2, getColorForType[#1]]&,
tokens,
{1}
]

formatCode[lang_, code_] :=
Replace[
ExternalEvaluate["Python", {
"from pygments.lexers import get_lexer_by_name",
"from pygments import highlight",
"from pygments.formatter import Formatter",
"from wolframclient.serializers import export",
"F = type('WLFormatter', (Formatter, ), dict(
	format = lambda self, tokensource, outfile: outfile.write(export(tokensource).decode('utf-8'))
))",
"lambda lang, code: highlight(code, get_lexer_by_name(lang), F())" -> {lang, code}
}
],
{___, b_String} :> formatTokens[ImportString[b, "WL"]]
]
formatCode["python", "map(lambda a: a+2, [r for i in range(10)])"]
formatCode["sql", "select a as b from users where b = 2 limit 10"]

NodeJS challanges

NodeJS Challenge (easy)

Define the range function that is internally using Python. The function should be equivalent to:

range[n1_Integer, n2_Integer, step_Integer:1] :=
	Range[n1, n2, step]
range[1, 10]

Hint

You need to ExternalEvaluate syntax to pass arguments to a defined function

ExternalEvaluate[lang, <|"Command" -> ..., "Arguments" -> ...|>]

Solution

range[n1_, n2_, step_] := 
ExternalEvaluate["NodeJS", <|
"Command" ->"function range(n1,n2,step=1) {
		if (! n2 && ! n2 == 0) {
			return range(0,n1-1,step)
		} 
		const arr=[];
		for (let i=n1;i<=n2;i+=step) {
			arr.push(i)
		} 
		return arr
	}",
"Arguments" -> {n1, n2, step}
|>]

range[1, 10]
range[1, 10, 2]

NodeJS Challenge (advanced)

Use prettier to reformat a string of javascript code

reformatJavascriptCode["foo(reallyLongArg(), omgSoManyParameters(), IShouldRefactorThis(), isThereSeriouslyAnotherOne());"]

Hint

use prettier.format to reformat the code

Solution

reformatJavascriptCode[code_]:=
Last @ ExternalEvaluate["NodeJS", {
"const prettier = require('prettier')",
<|
"Command"-> "prettier.format",
"Arguments" -> code
|>
}]

reformatJavascriptCode["foo(reallyLongArg(), omgSoManyParameters(), IShouldRefactorThis(), isThereSeriouslyAnotherOne());"]

SQL challenges

SQL allows you to perform queries to any database in this challanges we will be constructing functions to explore and visualize a database

SQL Challenge (easy)

Define a function that takes the path of an SQLite database and returns a dataset with all tables.

getDatabaseTables["path/to/sqlite"]

Initialization

db = URLDownload["https://github.com/lerocha/chinook-database/raw/master/ChinookDatabase/DataSources/Chinook_Sqlite.sqlite"]

ExternalEvaluate[{"SQL", "Target" -> db, "ReturnType" -> "Dataset"}, "select 2 as foo"]

Solution

getDatabaseTables[db_] := 
	ExternalEvaluate[{"SQL", db}, "
SELECT name FROM sqlite_master WHERE type = 'table'"]

getDatabaseTables[db]

SQL Challenge (advanced)

Define a function that takes the path of an SQLite database and returns a dataset with all tables, columns and types of the database.

getDatabaseInformation["path/to/sqlite"]

Initialization

db = URLDownload["https://github.com/lerocha/chinook-database/raw/master/ChinookDatabase/DataSources/Chinook_Sqlite.sqlite"]

ExternalEvaluate[{"SQL", "Target" -> db, "ReturnType" -> "Dataset"}, "select 2 as foo"]

Solution

getDatabaseInformation[db_] := 
	ExternalEvaluate[{"SQL", db}, "
SELECT t1.name as table_name, t2.name as column_name, t2.type as type_name
FROM (SELECT name FROM sqlite_master WHERE type = 'table') as t1 INNER JOIN pragma_table_info(t1.name) as t2
ORDER BY table_name"]

getDatabaseInformation[Once @ URLDownload["https://github.com/lerocha/chinook-database/raw/master/ChinookDatabase/DataSources/Chinook_Sqlite.sqlite"]]

SQL Challenge (hard)

Define a function that takes the path of an SQLite database (or another type of database if you have one to connect) and is returning a Graph showing relations between tables and table columns.

createDatabaseGraph["path/to/sqlite"]

Initialization

db = URLDownload["https://github.com/lerocha/chinook-database/raw/master/ChinookDatabase/DataSources/Chinook_Sqlite.sqlite"]

ExternalEvaluate[{"SQL", "Target" -> db, "ReturnType" -> "NamedColumns"}, "SELECT name FROM sqlite_master WHERE type='table'"]

Solution

createDatabaseGraph[db_] := 
	Module[
		{session, tables, fks, cols},

		session = StartExternalSession[{"SQL", "Target" -> db, "ReturnType" -> "NamedRows"}];

		tables = ExternalEvaluate[session, "SELECT name FROM sqlite_master WHERE type='table'"][[All, "name"]];

		fks = Flatten @ Map[
			Map[Append["source" -> #], ExternalEvaluate[session, "select * from pragma_foreign_key_list('"<> # <>"') as p"]] &,
			tables
		];
		

		DeleteObject[session];
		



		Graph[
			tables, 
			Rule @@@ Lookup[fks, {"source", "table"}], 
			VertexLabels->"Name",
			EdgeLabels-> Normal @ AssociationThread[Rule[Rule @@@ Lookup[fks, {"source", "table"}], Lookup[fks, "from"]]]
		]


]

createDatabaseGraph[Once @ URLDownload["https://github.com/lerocha/chinook-database/raw/master/ChinookDatabase/DataSources/Chinook_Sqlite.sqlite"]]

Java Challenges

When you use Java in ExternalEvaluate functionality, whether through functions like ExternalEvaluate and ExternalFunction, or via a Java cell created with ⇧+>, you interact with the same JVM that is used by standard J/Link calls. You can use ExternalEvaluate and J/Link interchangeably to accomplish what you want with Java. These challenges require you to interoperate between the J/Link and ExternalEvaluate worlds.

Java Challenge (easy)

Question

You probably already know how to call Java methods from the Wolfram Language using J/Link. Here is a trivial example calling the static method Math.abs():

Needs["JLink`"]

LoadJavaClass["java.lang.Math"]

Math`abs[-1]

Calling existing Java methods directly from the Wolfram Language is incredibly convenient, but in some cases it can be a performance bottleneck. Let’s say you wanted to call Math.abs() on a huge list of numbers (ignoring the fact that the Wolfram Language has a perfectly good Abs[] function):

data=Range[-50000,50000];

Math`abs/@data; // AbsoluteTiming

It takes 15 seconds on my machine because it requires about 100,000 separate calls into Java. What you want is a Java abs() method that takes an array of ints and returns an array, so that only one call into Java is required.

The challenge is to define a Java method that takes an int[] and returns an int[] where every element of the original array has had abs() applied to it. Call this method from the WolframLanguage and see the speed improvement. The entire challenge must be done within a Mathematica notebook (no writing a .java file and compiling it).

Hint

Create a Java cell (start a new cell with ⇧+> and set it to Java style from the dropdown on the left). Type in the appropriate Java method definition and evaluate it.

Solution

Here is a Java method that performs the desired operation in a terse, functional style:

int[] arrayAbs(int[] a) {
   return java.util.stream.IntStream.of(a).map(i -> Math.abs(i)).toArray();
}

Note that when you evaluate a Java cell that defines a method, what is returned is an ExternalFunction expression.

arrayAbs=%

arrayAbs[{-1,2}]

It is thousands of times faster:

arrayAbs[data]; //AbsoluteTiming

Here is a Java implementation of arrayAbs that uses a more traditional Java programming style:

int[] arrayAbs2(int[] a) {
        int[] result = new int[a.length];
        for (int i = 0; i < a.length; i++) {
        	result[i] = Math.abs(a[i]);
        }
        return result;
    }

You can also use ExternalEvaluate directly instead of a Java cell:

arrayAbs2=ExternalEvaluate["Java",
"
      int[] arrayAbs2(int[] a) {
        int[] result = new int[a.length];
        for (int i = 0; i < a.length; i++) {
        	result[i] = Math.abs(a[i]);
        }
        return result;
     }
  "
]

arrayAbs2[{-1,0,1}]

Java Challenge (advanced)

In a Java cell, define a class with a constructor and at least one public method. Then use JavaNew to create an instance of your class and call a method on that instance.

Hint 1

Evaluating a Java cell with a class definition loads the class into the kernel (that is, LoadJavaClass gets automatically called on it).

Hint 2

When you define a class called, say, MyClass, the actual name of the created class is not what you think. Look at the returned JavaClass expression to see the full class name. You can use the true class name in future calls, or just use the returned JavaClass expression directly.

Solution

Here is a simple class definition. When you evaluate a cell containing a class definition, the class gets compiled and loaded into the current session, as if you called LoadJavaClass on it.

public class MyClass {

    public int i;
    
    public MyClass(int i) {
        this.i = i;
    }
    
    public int getValue() {
        return i;
    }
}

To construct an instance, either use the full class name (which will be something like “REPL.$JShell$1$MyClass”), or, better, use the actual JavaClass expression:

obj=JavaNew[%, 42]

To finish the challenge we use J/Link syntax to call a method on the created class:

obj@getValue[]

You can also use an ExternalEvaluate call instead of a Java cell:

myClass=ExternalEvaluate["Java",
"
   public class MyClass {

       public int i;
    
       public MyClass(int i) {
          this.i = i;
       }
    
       public int getValue() {
          return i;
       }
   }
 "
]

JavaNew[myClass, 142]@getValue[]

Julia Challenges

Julia Challenge (easy) Setting up default Julia

Let’s just check we can setup Julia evaluator following the instructions given in the documentation (workflow/ConfigureJuliaForExternalEvaluate). PLEASE ASK FOR HELP if you have any trouble setting up Julia.

j=StartExternalSession["Julia"]

ExternalEvaluate[j,"2+2"]

Julia Challenge (easy) ExternalFunction

You can use ExternalFunction to define a Wolfram Language function that invokes a function in an external language. Use the sqrt function get the square root of a number.

You must have run the StartExternalSession call in the first challenge to define the Julia session referred to as j here.

Solution

juliaSquareRoot = ExternalFunction[j,"sqrt"]

juliaSquareRoot[16]

juliaSquareRoot/@{4,9,16,25,36,49,64,81,100}

Julia Challenge (advanced) Solution of DifferentialEquation using Julia

You may face a situation where you find some code that solves a problem in one language, and you do not want to have to rewrite it in WL.

Take the Julia code from the first example in https://tutorials.sciml.ai/html/models/01-classical_physics.html and execute it in Mathematica. Leave out the two lines at the end that call the Julia plot() function. Re-create their plot of the numerical solution of Carbon-14 half-life using ListLinePlot.

This is a very simple differential equation problem that can be easily handled in WL alone, but think of it as a stand-in for a more complex piece of code that you want to run in your Mathematica session.

Setup

Note that you need Julia 1.8 or later to run this example. You can see the version of Julia you have in the result from StartExternalSession[“Julia”].

The Julia sample code relies on a Julia package called OrdinaryDiffEq, which you might not have installed in your Julia distribution. Here is how to install that package using ExternalEvaluate (this line can take a few minutes):

ExternalEvaluate[j, "using Pkg; Pkg.add(\"OrdinaryDiffEq\")"];

Solution

Here is the Julia code copied directly from the web. The code returns a list of two values, the first of which is a list of time steps and the second is a list of radioactive decay values.

{t,u}= ExternalEvaluate[j,
"using OrdinaryDiffEq
  c1 = 5.730
  u0=1.0
  tspan = (0.0, 1.0)
  radioactivedecay(u,p,t) =-c1*u
  prob = ODEProblem(radioactivedecay,u0,tspan)
  sol = solve(prob,Tsit5())
  [sol.t,sol.u]
"]

ListLinePlot[Transpose[{t,u}],
PlotLabel->"Carbon-14 half-life",
PlotStyle ->Red,
Epilog->Point[Transpose[{t,u}]],
ImageSize->600,
Frame->True,
FrameLabel->{"Time in thousands of years","Percentage left"}
]