Training on Large Datasets

Timothée Verdier, Wolfram Research

Training Remotly:

Possibly “Harder, ...”

but also, "...Better, Faster, Stronger" !

On demand extention of local capacities:

hyperparameters search

production level training

A typical remote training workflow:

Robust to connexion issues (once started)

Asynchronous (as a consequence)

The Amazon Web Services (AWS) case:

Simple Storage System (S3)

Elastic Cloud Computation (EC2)

Volume storage is more expensive and less mobile than S3 storage...

Training workflow

One (of many) local test:

localScriptPath = "/Users/timotheev/Documents/WolframMathematica/wtc2019training/trainingscript.wls";

The training script in its environement:

SystemOpen[DirectoryName[localScriptPath]]

A remote training recipe:

A bucket containing:

the data files (50,000)

data indexes for:

training data,

validation data,

and available classes

AMI, priviledges configuration, configured AWS CLI

A training script - tested locally

Only S3 is accessible at anytime from your computer

Launching an instance:

Needs["AWSLink`"]
(* 
in Mathematica since 12.0,
without 12.1 "default client",
so add option "Client" -> ec2client, when asked for...
*)
(*requieres to have a configured .aws see below*)
ec2client = AWSClientConnect["ec2"]

(* AMI + launch options *)
baseconfiguration =
	<|"LaunchTemplateName" -> "WL_12.0_p3.2xlarge"|>;

(* Role *)
instancePrivileges =
	<|"Arn" -> "arn:aws:iam::434860378832:instance-profile/WTC2019OnMyBehalf"|>;

(* registered SSH Key *)
accessRegisteredKeyName = "WTC_2019_AWS";
accessRegisteredKeyPath =
	"/Users/timotheev/Documents/WolframMathematica/RemoteTrainings/WTC_2019_AWS.pem";

inst = EC2CreateInstance[
	Automatic,
	"LaunchTemplate" -> baseconfiguration,
	"IamInstanceProfile" -> instancePrivileges,
	"KeyPair" -> accessRegisteredKeyName
	]

inst["PublicDnsName"]

inst["State"]

Dataset@inst

Connecting a running instance :

remotehost = RemoteConnect[
	inst["PublicDnsName"],
	Authentication -> <|
		"Username" -> "ubuntu",
		"SSHKey" -> File@accessRegisteredKeyPath
		|>
	]

Run some code on the distant machine:

RemoteRunProcess[remotehost, "ls"]

RemoteRunProcess[remotehost, "wolframscript -code 2+2"]

Move the script on the distant machine

url = CopyFile[
	File[localScriptPath],
	RemoteFile[
		"scp://"
		<>inst["PublicDnsName"]
		<>":"
		<>"/home/ubuntu/trainingscript.wls",
		Authentication -> <|
			"Username" -> "ubuntu",
			"SSHKey" -> File@accessRegisteredKeyPath
			|>
	]
]

Launch and monitor the script:

(* screen or nohup are use to keep process running after deconnection - you can switch of local machine *)
RemoteRunProcess[remotehost, {"screen","-d","-m","./trainingscript.wls"}]

(* monitor the last lines of the log file *)
RemoteRunProcess[remotehost, {"tail","traininglogs.txt"}]["StandardOutput"]

A quick tour of the script:

SystemOpen[localScriptPath]

In v.s. out of core (batch generator)

Checkpointing

Early-Stopping

Avoid Import (as much as possible)

The return of the results:

downloadpath = "/Users/timotheev/Documents/WolframMathematica/wtc2019trainingRemote/";

(* call AWS CLI - local installation and PATH may differ *)
RunProcess[
		{
		"~/Library/Python/3.7/bin/aws",
		"s3",
		"cp",
			"s3://wtc2019trainingpubliccontent/results/",
			downloadpath,
		"--recursive",
		"--quiet"
		}
		];

SystemOpen[downloadpath]

Next steps:

From data indexes text files to database (more industrial level)

Database to store jobs (see MongoLink`)

Robustifying your script (error handling when things go wrong)

SendMail on training events

Coming soon:

ExternalStorage`=> S3 from WL directly and efficiently

Streams

Setting up an AWS account:

Create an AWS account at: https://aws.amazon.com

Create a user, and a programmatic access key:

Services > IAM > Users > SecurityCredential > Access Key > Create access key

https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html

eventually you need two files on your local machine

~/.aws/config

[default]
region=us-east-1
output=json

~/.aws/credentials

[default]
aws_access_key_id=AKIAIOSFODNN7EXAMPLE
aws_secret_access_key=wJalrXUtnFEMI/K7MDENG/bPxRfiCY...

Use a public AMI to launch  an Instance

install Mathematica, activate, set up your favorite configuration, ...

Create a new AMI from the instance

Create a key pair

Create a bucket, and an instance role allowing the machine S3 connection and read/write to the bucket

Create a policy for the user allowing to assign the role to an instance

Questions ?