Continuous Integration & Continuous Deployment of Paclets

Richard Hennigan, Wolfram Research

(RowDefault[, (Spacer1), , (Spacer1), ])  #WolframTechConf

ABSTRACT

Continuous integration and continuous deployment (CI/CD) is a set of practices in software engineering used to automate many aspects of source control, testing, and deployment of software. This talk will introduce a new set of utilities for the Wolfram Language that are designed to quickly set up CI/CD pipelines for paclets with a focus on integration with GitHub Actions. See how to set up automatic event-driven workflows that can check your paclet for errors, compile code for multiple platforms, run test suites, and even deploy to the new Wolfram Language Paclet Repository all from within GitHub.

PacletInstall["Wolfram/PacletCICD"];
Needs["Wolfram`PacletCICD`"];

Some Assumptions

Wolfram Language paclets

Resource definition notebooks

Wolfram testing framework (VerificationTest, TestReport)

Source control concepts

Commit

Push

Branches

Pull Requests

Merge

CI/CD Workflows

A CI/CD workflow is a set of processes that run automatically upon triggering some type of work on your project.

Some examples of triggers and corresponding workflows for paclets:

When changes are pushed, check the paclet for potential errors

When a pull request is opened, check paclet and run all test files for the corresponding branch

Check and build the paclet on a daily schedule

When merging into the default branch, build the paclet and create a downloadable .paclet file

When merging to a specified release branch, build, create a tagged release, and submit the paclet to the paclet repository

Checking

Using the definition notebook, we can get interactive hints for potential issues:

However, manually opening the notebook and running the check every time you make changes to the code is not always practical.

Wouldn’t it be nice to automate this so that it runs every time we push changes to GitHub?

Wouldn’t it be neat if we didn’t have to open the notebook to see the results?

And wouldn’t it be really cool to run it on GitHub machines instead of locally?

Wouldn’t it be nice to automate this so that it runs every time we push changes to GitHub?

Wouldn’t it be neat if we didn’t have to open the notebook to see the results?

And wouldn’t it be really cool to run it on GitHub machines instead of locally?

Spoiler: https://github.com/rhennigan/SamplePaclet/actions/runs/3286174450

Creating GitHub Workflows

It’s really easy! All you need to do is create a YAML file tells GitHub what to do and push the changes.

Example: https://github.com/WolframResearch/PacletCICD/blob/main/.github/workflows/check-paclet.yml

Okay, maybe I lied…

That wasn’t easy to write.

It was actually quite unpleasant, so I decided to create a set of utilities for generating these files:

PacletInstall["Wolfram/PacletCICD"]

Needs["Wolfram`PacletCICD`"]

How to create a workflow (the easy way)

A local paclet with a corresponding GitHub repo:

paclet=PacletObject[File["Paclets/SamplePaclet"]]

https://github.com/rhennigan/SamplePaclet

A symbolic representation of a workflow that’s equivalent to using the “Check” button in the definition notebook:

wf=Workflow["Check"]

Convert it to the corresponding YAML syntax that GitHub understands and put it in the right place:

WorkflowExport[paclet,wf]

Committing and pushing this to GitHub will now automatically trigger a workflow run.

One-Time Repository Setup

Your GitHub repository will need to be configured with a license entitlement in order to run the wolfram engine in GitHub actions.

PacletCICD has a tutorial page that explains how to do this.

https://github.com/rhennigan/SamplePaclet/settings/secrets/actions

Running the Workflow

Now simply pushing the changes to GitHub will trigger the workflow run:

cd <* paclet["Location"] *>
git add *
git commit -m "Added a check workflow"
git push

Check that the workflow is running:

SystemOpen["https://github.com/rhennigan/SamplePaclet/actions"]

A previous run in the interest of saving time: https://github.com/rhennigan/SamplePaclet/actions/runs/3278510563

Paclet Testing

Here is another premade workflow that runs the test files in a paclet and generates a report:

wf=Workflow["Test"]

Export it and push to GitHub:

WorkflowExport[paclet,wf]

git add *
git commit -m "Added a testing workflow"
git push

Check that the workflow is running:

SystemOpen["https://github.com/rhennigan/SamplePaclet/actions"]

A previous run in the interest of saving time: https://github.com/rhennigan/SamplePaclet/actions/runs/3278527698

Publish to the Paclet Repository

Create a workflow to submit to the paclet repository:

wf=Workflow["PublishPaclet"]

Export it and push to GitHub:

WorkflowExport[paclet,wf]

git add *
git commit -m "Added a publish workflow"
git push

Start the workflow on the actions page:

SystemOpen["https://github.com/rhennigan/SamplePaclet/actions/workflows/PublishPaclet.yml"]

The Building Blocks of a CI/CD Workflow

Workflow

Workflow[]

Contains information about how/when the workflow should run

Specifies one or more WorkflowJob objects

WorkflowJob

WorkflowJob[]

Contains information about what type of machine to run on (OS, container, etc)

Specifies one or more WorkflowStep objects

WorkflowStep

WorkflowStep[]

The computational unit of a workflow

Can represent a shell command, a script file, premade actions, etc.

The Building Blocks of a CI/CD Workflow

wf=Workflow["Compile"]

A workflow runs jobs:

jobs=wf["Jobs"]

These run concurrently when possible:

wf["JobGraph"]

Each workflow job executes steps sequentially:

job=jobs["BuildPaclet"]

job["Steps"]

How does one create these?

WorkflowStep

WorkflowStep[<|"Name"->"PublishPaclet","Uses"->"WolframResearch/submit-paclet"|>]

WorkflowStep[<|"Name"->"Hello","Run"->"echo 'hello world'"|>]

Named presets

WorkflowStep["Checkout"]

WorkflowStep["Build"]

WorkflowStep["UploadBuildArtifacts"]

Shell commands

WorkflowStep[WorkflowEvaluate["echo 'hello world!'"]]

WorkflowStep[WorkflowEvaluate["apt update && apt install build-essential -y"]]

Wolfram Language code

WorkflowStep[WorkflowEvaluate[Print["hello world"]]]

WorkflowStep[WorkflowEvaluate[CopyFile["file1","file2"]]]

Wolfram Language scripts

WorkflowStep[File["Scripts/MyScript.wls"]]

WorkflowJob

WorkflowJob[<|
"Name"->"MyBuildJob",
"Needs"->{"Check"},
"Steps"->{
WorkflowStep["Checkout"],
WorkflowStep[File["Scripts/PrepareForBuild.wls"]],
WorkflowStep["Build"],
WorkflowStep["UploadBuildArtifacts"],
WorkflowStep[WorkflowEvaluate["echo 'awesome!"]]
}
|>
]

Named presets

Many of the same named presets as WorkflowStep:

WorkflowJob["Build"]

This surrounds the step of the same name with additional necessary steps:

%["Steps"]

A list of steps

WorkflowJob[{
"Checkout",
File["Scripts/BuildMX.wls"],
"Build",
"UploadArtifact"
}]

Other specifications

WorkflowJob can also take a File argument:

WorkflowJob[File["Scripts/MyScript.wls"]]

%["Steps"]

Unlike WorkflowStep, it can also take an OperatingSystem option, which controls what type of machine it will run on:

WorkflowJob[File["Scripts/MyScript.wls"],OperatingSystem->"Windows"]

Workflow

Workflow[
<|
"Name"->"Check",
"On"-><|
"Push"->None,
"PullRequest"->True,
"Schedule"->{<|"cron"->"30 12 * * *"|>}
|>,
"Jobs"->{
WorkflowJob[{
"Checkout",
WorkflowStep[File["Scripts/BuildMX.wls"]],
"Build",
"UploadArtifact"
},<|"Name"->"Check"|>],
WorkflowJob[{
"Checkout",
WorkflowStep[File["Scripts/BuildMX.wls"]],
"Test"
},<|"Name"->"Test"|>]
}
|>]

Real example of this workflow: https://github.com/rhennigan/CodeEquivalenceUtilities/actions/runs/3273547364

Advanced Workflow Examples

paclet=PacletObject[File["Paclets/AdvancedSamplePaclet"]]

Compile

$compileWF=Workflow["Compile"]

$compileWF["JobGraph"]

WorkflowExport[paclet,$compileWF]

cd <* paclet["Location"] *>
git add *
git commit -m "Added a compile workflow"
git push

Check that the workflow is running:

SystemOpen["https://github.com/rhennigan/AdvancedSamplePaclet/actions"]

A previous run in the interest of saving time: https://github.com/rhennigan/AdvancedSamplePaclet/actions/runs/3286307458

Compile with Custom Build

Start from the existing compilation workflow and extract the jobs:

compileJobs=$compileWF["Jobs"]

Modify the BuildPaclet job

Extract the steps:

steps=compileJobs["BuildPaclet"]["Steps"]

Insert two new steps that run custom scripts:

newSteps=Flatten@Insert[
steps,
{
WorkflowStep[File["Scripts/BuildMX.wls"]],
WorkflowStep[File["Scripts/SetPacletInfo.wls"]]
},
3
]

Assemble into a new job that depends on the compilation jobs:

customBuildJob=WorkflowJob[
newSteps,
<|
"Name"->"BuildPaclet",
"Needs"->{"Compile-Windows-x86-64","Compile-MacOSX-x86-64","Compile-Linux-x86-64"}
|>
]

Combine jobs into a new workflow

$customCompileWF=Workflow[<|
"Name"->"Compile",
"Jobs"->{
compileJobs["Compile-Windows-x86-64"],
compileJobs["Compile-MacOSX-x86-64"],
compileJobs["Compile-Linux-x86-64"],
customBuildJob
}
|>]

compileYAML=WorkflowExport[paclet,$customCompileWF]

$customCompileWF["JobGraph"]

git add *
git commit -m "Added a custom compile workflow"
git push

Check that the workflow is running:

SystemOpen["https://github.com/rhennigan/AdvancedSamplePaclet/actions"]

A previous run in the interest of saving time: https://github.com/rhennigan/AdvancedSamplePaclet/actions/runs/3286314482

Adding Additional Jobs

Next we’ll start new test jobs on each platform using the built paclet.

This step can be used in a job to download the result of the BuildPaclet step from earlier:

downloadStep=WorkflowStep[\[LeftAssociation]
"Name"->"DownloadBuiltPaclet",
"Uses"->"actions/download-artifact@v3",
"With"->\[LeftAssociation]"name"->"artifact","path"->"build"\[RightAssociation]
\[RightAssociation]]

Generate test jobs using the operating system as an input:

testJob[os_]:=
Module[{meta,job,data},
meta=<|"Name"->"Test-"<>os,"Needs"->"BuildPaclet"|>;
job=WorkflowJob[File["Scripts/TestPaclet.wls"],meta,OperatingSystem->os];
data=job["Data"];
data["steps"]=Insert[data["steps"],downloadStep["Data"],4];
WorkflowJob[data]
];

testJobs=testJob/@{"Windows-x86-64","MacOSX-x86-64","Linux-x86-64"}

More jobs:

check=WorkflowJob["Check"]

submit=WorkflowJob["Submit",<|"Needs"->{"Check","Test-Windows-x86-64","Test-MacOSX-x86-64","Test-Linux-x86-64"}|>]

Combine with the compile jobs:

jobs=Join[Values@$customCompileWF["Jobs"],testJobs,{check,submit}]

Assemble into a workflow:

$compileTestAndPublishWF=Workflow[<|"Name"->"CompileTestAndPublish","On"->"push","Jobs"->jobs|>]

Visualize the job dependency graph:

$compileTestAndPublishWF["JobGraph"]

Delete the previous compilation workflow so we’re not running redundant jobs:

DeleteFile[compileYAML]

Export the new workflow and push changes:

WorkflowExport[paclet,$compileTestAndPublishWF]

git add *
git commit -m "Added a big fancy workflow"
git push

Check that the workflow is running:

SystemOpen["https://github.com/rhennigan/AdvancedSamplePaclet/actions"]

A previous run in the interest of saving time: https://github.com/rhennigan/AdvancedSamplePaclet/actions/runs/3286400254

Cool, but how much does it cost?

https://github.com/rhennigan/AdvancedSamplePaclet/actions/runs/3286400254/usage

ResourceFunction["BirdSay"]["neat!"]

Documentation Tools/Authoring Documentation for Paclets
Brian Van Vertloo
3:15 PM - 3:40 PM CDT on Thursday, October 20

The Structure of Paclets
Connor Gray
9:00 AM - 9:25 AM CDT on Friday, October 21

Advanced Paclet Development
Connor Gray
9:25 AM - 9:50 AM CDT on Friday, October 21

Workshop: Hands-on Paclet Development
10:30 AM - 11:30 AM CDT on Friday, October 21