What’s new in ExternalEvaluate

Riccardo Di Virgilio, Wolfram Reasearch

(RowDefault[, (Spacer1), , (Spacer1), ])  #WolframTechConf

import torch
from diffusers import StableDiffusionPipeline

pipeline = StableDiffusionPipeline.from_pretrained("CompVis/stable-diffusion-v1-4", use_auth_token="hf_sdfd")
image = pipeline(prompt="a photo of an astronaut riding a horse on mars",height=512, width=512, guidance_scale=7.5, num_inference_steps=40).images[0]

image

What’s new in ExternalEvaluate

New SQL evaluators

New Shell evaluators

Julia improved type support

SQL

We are introducing several new SQL evaluators.

SQL, an high level evaluator that support several native types, compatible with PostgreSQL, MySQL, MicrosoftSQL, SQLite, etc...

SQL

We are introducing several new SQL evaluators.

SQL, an high level evaluator that support several native types, compatible with PostgreSQL, MySQL, MicrosoftSQL, SQLite, etc...

SQL-JDBC,  a Java based evaluator able to connect to databases using DatabaseLink`JDBC connections.

SQL

We are introducing several new SQL evaluators.

SQL, an high level evaluator that support several native types, compatible with PostgreSQL, MySQL, MicrosoftSQL, SQLite, etc...

SQL-JDBC,  a Java based evaluator able to connect to databases using DatabaseLink`JDBC connections.

SQLite, a performance oriented evaluator for SQLite only, to be released.

SQL Example usage

db = URLDownload["https://github.com/lerocha/chinook-database/raw/master/ChinookDatabase/DataSources/Chinook_Sqlite.sqlite"]

RegisterExternalEvaluator["SQL", db]

SELECT name FROM sqlite_master WHERE type='table'

SELECT * FROM Artist LIMIT 10

SELECT 
	Title as  Album, 
	Name as Artist 
FROM 
	Album 
JOIN 
	Artist 
ON 
	Album.ArtistID = Artist.ArtistId 
LIMIT 
	15

SQL Programmatic usage

You can use ExternalEvaluate directly over a database using “Target”.
“ReturnType” is by default a Dataset, but several other shapes can be returned.

ExternalEvaluate[{"SQL", "Target" -> db, "ReturnType" -> "Rows"}, "SELECT * from Artist limit 10"]

ExternalEvaluate[{"SQL", "Target" -> db, "ReturnType" -> "NamedRows"}, "SELECT * from Artist limit 10"]

ExternalEvaluate[{"SQL", "Target" -> db, "ReturnType" -> "Columns"}, "SELECT * from Artist limit 10"]

ExternalEvaluate[{"SQL", "Target" -> db, "ReturnType" -> "NamedColumns"}, "SELECT * from Artist limit 10"]

SQL Support for different databases

Dates

ExternalEvaluate[
{"SQL", "Target" -> "postgresql://localhost:5432/template1", "ReturnType" -> "NamedColumns"}, 
"SELECT day::date FROM generate_series(timestamp '2004-03-07', '2004-03-11', '1 day') day"
]

Dates with timezone

ExternalEvaluate[
{"SQL", "Target" -> "postgresql://localhost:5432/template1", "ReturnType" -> "NamedColumns"}, 
"select NOW()"
]

Interval

ExternalEvaluate[
{"SQL", "Target" -> "postgresql://localhost:5432/template1", "ReturnType" -> "NamedColumns"}, 
"select (date '2017-01-31')::timestamp - (date '2017-01-01')::timestamp as time"
]

Arrays

ExternalEvaluate[
{"SQL", "Target" -> "postgresql://localhost:5432/template1", "ReturnType" -> "Rows"
},
"select ARRAY[n-1, n, n+1], n from generate_series(1, 10) as n"
]

Shell

Execution of commands in real time using your shell

Cmd.exe, Powershell are supported on Windows

ZSH, SH, BASH... are supported on Mac/Linux

date

Shell Programmatic usage

You can use ExternalEvaluate directly over a shell using “Target”.
“ReturnType” is by default an Expression, but several other shapes can be returned.

r = ExternalEvaluate[{"Shell", "Target" -> "/bin/sh"}, "date"]

r[{"StandardOutput", "ExitCode"}]

ExternalEvaluate[{"Shell", "Target" -> "/bin/sh", "ReturnType" -> "StandardOutput"}, "date"]

ExternalEvaluate[{"Shell", "Target" -> "/bin/zsh", "ReturnType" -> "StandardError"}, "rm not/a/path"]

ExternalEvaluate[{"Shell", "Target" -> "/bin/zsh", "ReturnType" -> "ExitCode"}, "rm not/a/path"]

Shell Templated command

ExternalEvaluate[
"Shell", {
 "echo <* First @ $CommandLine *>"
}]

Shell Real time program execution

ExternalEvaluate[
"Shell", {
 "echo 🧹 cleanup the environment",
"rm -rf ~/Downloads/rust.zip",
"rm -rf ~/Downloads/rust-hello-world-master",
 "echo 🧑‍🔬 download latest code",
"curl -Lo ~/Downloads/rust.zip https://github.com/hpaluch/rust-hello-world/archive/refs/heads/master.zip", 
"echo ⚙️ unzip the program",
"unzip ~/Downloads/rust.zip -d ~/Downloads",
"cd    ~/Downloads/rust-hello-world-master",
"echo 👷‍♀️ build the program",
"cargo update",
"cargo build",
"cargo run"
}]

Julia

Julia integration has been improved

Several new types are supported

Support for multidimensional Julia array

Julia improved type support

s = StartExternalSession["Julia"];

ExternalEvaluate[s, "using Dates"]

Receiving DateObject and Quantity

ExternalEvaluate[s, {
"Dates.Year(3)",
"Dates.Millisecond(10)"
}]

ExternalEvaluate[s, {
"Date(2013, 3, 1)",
"DateTime(2017,1,1,10,15,3)",
"DateTime(2017,1,1,10,15,3) - DateTime(2017,1,1,10,15)"
}]

Sending DateObject

juliaToString = ExternalFunction[s, "repr"]

juliaToString[DateObject[]]

juliaToString[DateObject[{2012, 1, 1}]]

Receiving multidimensional Array

ArrayReshape[Range[8], {4, 2}]//MatrixForm

ExternalEvaluate[s, "reshape([(1:8)...], 4, 2)"]//MatrixForm

ExternalEvaluate[s, "map(x -> x*im, reshape([(1:8)...], 4, 2))"]

ExternalEvaluate[s, "map(x -> Dates.Hour(x), reshape([(1:8)...], 4, 2))"]

ExternalEvaluate[s, "map(x -> Date(2017, 1, 1) + Dates.Day(x), reshape([(1:8)...], 4, 2))"]

Sending arrays

juliaToString = ExternalFunction[s, "repr"];

juliaToString[ArrayReshape[Range[8], {2, 4}]]

juliaToString[NumericArray[Range[10],"Real32"]]

juliaToString[NumericArray[Range[10],"UnsignedInteger16"]]

juliaToString[SparseArray[{{1,1}->1,{2,2}->2,{3,3}->3,{1,3}->4}]]

What’s next?

ExternalEvaluate is going to be open-source!

PythonRuntimeDistribution for Mac and Windows.

PacletInstall["PythonRuntimeDistribution", "Site" -> "http://paclet-int.wolfram.com:8080/PacletServerInternal"]

target = PacletObject["PythonRuntimeDistribution"]["AssetLocation", "Target"]

ExternalEvaluate[{"Python", "Target" -> target}, "2+2"]