Advanced Remote Kernel Launching

Roman E. Maeder
Wolfram Research

Advanced Remote Kernel Launching

ABSTRACT

We discuss a new launcher for remote parallel subkernels. It can launch multiple kernels with a single ssh connection, and also supports port forwarding for the kernel WSTP connections, to get around NAT and firewall restrictions and support kernels on cloud compute services, such as Amazon EC2.

Current Remote Kernel Launcher

Needs["SubKernels`RemoteKernels`"]

How it Works

Create a listening WSTP link

run ssh to log into remote machine and launch a kernel

kernel connects back to the listening link

Repeat for each kernel

Limitations

Reverse TCP connections (from remote server to local master) do not traverse firewalls and NAT

No fixed local end point; each kernel requires a separate listening link

LinkHost nuisance (choice of local interface)

One ssh connection per kernel (even to the same remote server)

No standard ssh solution for local Windows machines

Goals of the Redesign

Fast, for larger number of kernels per server

Single outgoing ssh connection

Tunnelling of reverse WSTP connections (optional)

Support for official openssh Windows port from Microsoft

Works also for cloud compute services

“It just works”

If you can log in from a command prompt (terminal)

ssh host wolfram

you can launch kernels, without additional configuration

Single SSH command per remote host

Only one ssh connection is made. It runs a shell script that does a loop to launch as many kernels as are specified for that machine.

Time for some bash magic!

remoteScript="exec &>`mktemp --tmpdir subkernel-XXXXX.log`; t=1; for i in $(seq 1 $1) ; do nohup $2 -mathlink -linkmode Connect -linkprotocol UUIDTCPIP -linkname $3 & sleep $t ; t=0.05 ; done; exit";

This is possible because all kernels can connect to the same socket on the master.

Local Listening Socket

The fine print: which interface to use?

$MachineAddresses
{$MachineName,$MachineDomains}

host=$MachineName<>"."<>First[$MachineDomains]

host=$MachineAddresses[[1]]

A proper socket that can accept any number of connections (to the same address):

socket=WSTP`LinkServer`LinkServerCreate[host]

This is the address we need to connect to (without port forwarding):

address=First[socket]

RunProcess: Bypass the local shell

cmd={"ssh","byblis67.wolfram.com","/bin/sh","-s","2","wolfram",address};

StringRiffle[cmd]

RunProcess[] does not use a shell. It gives more information about the success of the command than does Run[]. We can feed it standard input from a string.

RunProcess[cmd,All,remoteScript]

Collect available links

Once the kernels establish the connection back to the socket, we can collect them.

WSTP`LinkServer`LinksAvailable[socket]

Each link object is a connection to a remote kernel’s $ParentLink (just as before):

kernels=WSTP`LinkServer`GetLinks[socket]

LinkReadyQ/@kernels

LinkRead/@kernels

Interact with kernels

Send a command...

LinkWrite[#,Unevaluated[EvaluatePacket[$ProcessID]]]&/@kernels

... and read results:

LinkRead/@kernels

Close the links which makes the remote kernels exit:

LinkClose/@kernels

Close the listening socket:

WSTP`LinkServer`LinkServerClose[socket]

Windows Openssh

Together with the Powershell, feels almost like linux.

In particular, it supports ssh-agent as a Windows service:
load your ssh keys once, and never be prompted for a passphrase again.
Google knows how to set it up.

Port Forwarding

Create the listening socket on the localhost interface (127.0.0.1).

Tell ssh to create a socket on the remote end (on its localhost interface) and forward all connections to the local listening socket.

cmd={"ssh","-R","11235:localhost:11235","byblis67.wolfram.com","/bin/sh","-s","2","wolfram","11235@localhost"};

Tell the kernels to connect to the socket created on the remote end (11235@localhost).

Those connections are then forwarded through the ssh tunnel to the socket on the local (master) side.

Import[URL["https://res.cloudinary.com/practicaldev/image/fetch/s--6ht2e_RB--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_66%2Cw_880/https://thepracticaldev.s3.amazonaws.com/i/xl0u5wy98adbgna4zr2a.gif"]]

(https://dev.to/samuyi/the-how-to-of-ssh-port-forwarding-1f4e)

Advantages:
no routing requirements other than outgoing ssh access. Great for NAT and reaching cloud compute services.

Possible drawback:
all WSTP links go through a single TCP connection, which might limit bandwidth.

Configuration

This launcher will become another SubKernels` implementation.

Needs["SubKernels`ClusterKernels`"]

Same principle as with existing local and remote kernel launchers:

LaunchKernels[ClusterKernels[{"raspi4b.wri.wolfram.com"},4,RemoteUsername->"pi"]]

ParallelEvaluate[{$Version,$ProcessID}]//TableForm

CloseKernels[];

Parameters (these will change before release!):

Options[LaunchCluster]//Column

Performance

I used an earlier version of this package to launch kernels on a supercomputer, as reported at the European WTC in 2017.

The master kernel issues one ssh command to each node, which runs a shell loop to launch one kernel per core. Each kernel makes one TCP connection back to the master.

With 15 nodes of 24 cores each, this means an attempt to launch 360 Wolfram engine processes, which puts an enormous stress on the file system where the software was installed.

It is not a problem if the software is copied to the flash file system first.

For one of the other file systems, a staggered launch (with a pause of 0.5s between launches on the same node) works well.

In this way it is possible to launch up the 1000 kernels within 30 seconds.

Conclusions

I expect this package to be in the next release.

If you have access to an exotic or large computer/cluster on which you want to try it out, let me  know.

Import[URL["https://d2r55xnwy6nx47.cloudfront.net/uploads/2019/10/Google_Lede1300.jpg"]]