Paclets and Paclet Development

Todd Gayley
Wolfram Research

Join the Conversation #WolframTechConf

ABSTRACT

Paclets are a way of packaging units of Wolfram system functionality so they can be easily found, distributed, updated, and loaded. Many paclets contain Wolfram Language packages, but a paclet can be as simple as a single documentation notebook, all the way up to a large application with multiple packages, documentation, front end palettes, libraries, etc. Paclets have been used internally in Wolfram products for years to componentize parts of the system, and make it easy to provide updates to users. We are now enhancing the system and documenting it for users. We will also host a public Paclet Repository that accepts submissions from users. This talk will introduce paclets, explain their capabilities, and show how to create and distribute them.

Paclets

The Paclet system was created years ago to allow Wolfram to provide updates to data categories like CountryData, GraphData, CityData, etc.

Paclets

The Paclet system was created years ago to allow Wolfram to provide updates to data categories like CountryData, GraphData, CityData, etc.

From the beginning, it was designed to handle arbitrary content, in particular Wolfram Language packages

Paclets

The Paclet system was created years ago to allow Wolfram to provide updates to data categories like CountryData, GraphData, CityData, etc.

From the beginning, it was designed to handle arbitrary content, in particular Wolfram Language packages

Paclets are heavily used within Wolfram Research

Paclets

The Paclet system was created years ago to allow Wolfram to provide updates to data categories like CountryData, GraphData, CityData, etc.

From the beginning, it was designed to handle arbitrary content, in particular Wolfram Language packages

Paclets are heavily used within Wolfram Research

We are now documenting and exposing the paclet system to users

Paclets

The Paclet system was created years ago to allow Wolfram to provide updates to data categories like CountryData, GraphData, CityData, etc.

From the beginning, it was designed to handle arbitrary content, in particular Wolfram Language packages

Paclets are heavily used within Wolfram Research

We are now documenting and exposing the paclet system to users

We are creating a Paclet Repository that will accept user submissions, hoping to create a vibrant ecosystem of easily-discoverable and accessible paclets

Paclets

The Paclet system was created years ago to allow Wolfram to provide updates to data categories like CountryData, GraphData, CityData, etc.

From the beginning, it was designed to handle arbitrary content, in particular Wolfram Language packages

Paclets are heavily used within Wolfram Research

We are now documenting and exposing the paclet system to users

We are creating a Paclet Repository that will accept user submissions, hoping to create a vibrant ecosystem of easily-discoverable and accessible paclets

If you are creating any kind of add-on content for Wolfram products that does not fit into one of our other Repositories (like a single function for the Function Repository), you should create it in the form of a paclet

Uses of Paclets Within Wolfram Products

Componentize the product and build system

Uses of Paclets Within Wolfram Products

Componentize the product and build system

Push out immediate updates to functionality that changes much more rapidly than Mathematica releases (for example, integration with the Wolfram Cloud)

Uses of Paclets Within Wolfram Products

Componentize the product and build system

Push out immediate updates to functionality that changes much more rapidly than Mathematica releases (for example, integration with the Wolfram Cloud)

Critical bug fixes

Uses of Paclets Within Wolfram Products

Componentize the product and build system

Push out immediate updates to functionality that changes much more rapidly than Mathematica releases (for example, integration with the Wolfram Cloud)

Critical bug fixes

New functionality that doesn’t need to wait for a new Mathematica release

Uses of Paclets Within Wolfram Products

Componentize the product and build system

Push out immediate updates to functionality that changes much more rapidly than Mathematica releases (for example, integration with the Wolfram Cloud)

Critical bug fixes

New functionality that doesn’t need to wait for a new Mathematica release

Documentation updates

Uses of Paclets Within Wolfram Products

Componentize the product and build system

Push out immediate updates to functionality that changes much more rapidly than Mathematica releases (for example, integration with the Wolfram Cloud)

Critical bug fixes

New functionality that doesn’t need to wait for a new Mathematica release

Documentation updates

There are over 200 paclets in the Mathematica 12.1 layout

There are over 2300 paclets on the Wolfram Paclet Server

Paclet System Documentation

I have written a tutorial-style document: https://www.wolframcloud.com/obj/tgayley/Published/PacletDevelopment.nb

Full Wolfram-style documentation coming in-product in 12.1

Introduction to Paclets

Paclets are units of Wolfram functionality, packaged up in a way that allows them to be discovered, installed, updated, and to integrate seamlessly into the Wolfram environment.

Introduction to Paclets

Paclets are units of Wolfram functionality, packaged up in a way that allows them to be discovered, installed, updated, and to integrate seamlessly into the Wolfram environment.

The essential element that makes a paclet is the PacletInfo.wl file, a small, simple file of metadata that describes the paclet, its requirements, and the ways in which it extends the Wolfram environment.

Introduction to Paclets

Paclets are units of Wolfram functionality, packaged up in a way that allows them to be discovered, installed, and updated, and to integrate seamlessly into the Wolfram environment.

The essential element that makes a paclet is the PacletInfo.wl file, a small, simple file of metadata that describes the paclet, its requirements, and the ways in which it extends the Wolfram environment.

A component of the Wolfram Engine called the Paclet Manager is what manages paclets and provides functionality to find, install, update, remove, and manage paclets.

Introduction to Paclets

Paclets are versioned

Paclet Manager always uses the highest version number it finds

side-by-side installation of multiple versions, possibly with different requirements and compatibilities

Introduction to Paclets

Paclets are versioned

Paclet Manager always uses the highest version number it finds

side-by-side installation of multiple versions, possibly with different requirements and compatibilities

Paclets have requirements

declare that they are only compatible with certain versions of the Wolfram Engine (e.g., “11+“), certain $SystemID values (“MacOSX-x86-64”), and certain Product Names (“Mathematica”)

paclets that do not match the current system become invisible and will not be used

Introduction to Paclets

Paclets are versioned

Paclet Manager always uses the highest version number it finds

side-by-side installation of multiple versions, possibly with different requirements and compatibilities

Paclets have requirements

declare that they are only compatible with certain versions of the Wolfram Engine (e.g., “11+“), certain $SystemID values (“MacOSX-x86-64”), and certain Product Names (“Mathematica”)

paclets that do not match the current system become invisible and will not be used

Paclets have extensions

describe the way in which they extend the Wolfram system (“I have a palette”, “I have documentation”, “I have WL code files”, etc.)

Introduction to Paclets

Paclets are versioned

Paclet Manager always uses the highest version number it finds

side-by-side installation of multiple versions, possibly with different requirements and compatibilities

Paclets have requirements

declare that they are only compatible with certain versions of the Wolfram Engine (e.g., “11+“), certain $SystemID values (“MacOSX-x86-64”), and certain Product Names (“Mathematica”)

paclets that do not match the current system become invisible and will not be used

Paclets have extensions

describe the way in which they extend the Wolfram system (“I have a palette”, “I have documentation”, “I have WL code files”, etc.)

Paclets can be disabled and enabled without deleting them

Introduction to Paclets

Basic operation of the Paclet Manager (Fundamental Principle): When looking for a paclet by name, or a resource that paclets might provide, the Paclet Manager finds all the paclets that provide that resource, culls out any that are disabled or do not match the current system, then picks the highest version.

Introduction to Paclets

Basic operation of the Paclet Manager (Fundamental Principle): When looking for a paclet by name, or a resource that paclets might provide, the Paclet Manager finds all the paclets that provide that resource, culls out any that are disabled or do not match the current system, picks the highest version.

This is a smarter lookup than a simple search path like $Path. When looking for a file, the Paclet Manager is consulted first, and $Path is only used if a paclet-based file is not found.

Introduction to Paclets

Basic operation of the Paclet Manager (Fundamental Principle): When looking for a paclet by name, or a resource that paclets might provide, the Paclet Manager finds all the paclets that provide that resource, culls out any that are disabled or do not match the current system, then picks the highest version.

This is a smarter lookup than a simple search path like $Path. When looking for a file, the Paclet Manager is consulted first, and $Path is only used if a paclet-based file is not found.

By design, paclets are mostly invisible to users, beyond installation.

Paclet Changes in Version 12.1

Paclet Manager functions are moving from the PacletManager` context to the System` context.

Old PacletManager` names are still supported.

Syntax of PacletInfo.wl files is slightly changed; old syntax still supported

The Wolfram Language in-kernel representation of a paclet is an expression with head PacletObject; formerly this was PacletManager`Paclet.

The PacletInfo.wl File

It is the presence of a PacletInfo.wl file that makes a collection of files into a paclet

A simple file of metadata in Wolfram Language format

You can also name this file PacletInfo.m

The PacletInfo.wl File

Paclet[
    "Name" -> "FileUtilities",  (* You can include comments *)
    "Version" -> "1.0",
    "Extensions" -> {
        {"Kernel", "Root" -> "Kernel", "Context" -> "FileUtilities`"}
    }
]

The PacletInfo.wl File

Paclet[
    "Name" -> "FileUtilities",  (* You can include comments *)
    "Version" -> "1.0",
    "Extensions" -> {
        {"Kernel", "Root" -> "Kernel", "Context" -> "FileUtilities`"}
    }
]

Directory layout to match this PacletInfo.wl:

FileUtilities/
    PacletInfo.wl
    Kernel/
        FileUtilities.wl

The PacletInfo.wl File

Fields

Paclet[
    "Name" -> "FileUtilities",
    "Version" -> "1.0",
    "WolframVersion" -> "12.*",
    "SystemID" -> {"Windows", "Windows-x86-64"},
    "ProductName" -> "WolframEngine",  (* i.e., "Wolfram Engine for Developers" *)
    "Creator" -> "tgayley@wolfram.com",
    "Description" -> "Some great file utilities",
    "Support" -> "http://somecompany.com/FileUtilities",
    "SomeCustomField" -> 12345678,
    "Extensions" -> {
        {"Kernel", "Root" -> "Kernel", "Context" -> "FileUtilities`"}
    }
]

The PacletInfo.wl File

Extensions

The Extensions section is where you declare the types of resources your paclet provides—the ways in which it “extends” the system.

Extension Types:

Kernel

FrontEnd

Documentation

Path

LibraryLink

JLink

Resource

Custom

Example Paclet

Complex Example Paclet

Paclet[
    "Name" -> "ExamplePaclet",
    "Version" -> "1.0.1234",
    "Description" -> "A really great paclet. A lot of people are 
                      saying it's the best ever.",
    "Extensions" -> {
        {"Kernel", "Root" -> "Kernel", "Context" -> {"MyPackage`", "MyOtherPackage`"}},
        {"Documentation"},
        {"FrontEnd"},
        {"Path"},
        {"Resource", "Resources" -> {{"sun", "Images/sun.jpg"}, {"moon", "Images/moon.jpg"}}}
    }
]

Example Paclet

Complex Example Paclet

This is the file layout of this paclet:

ExamplePaclet/
    PacletInfo.wl
    Kernel/
        MyPackage.wl
    FrontEnd/
        Palettes/
            SuperPalette.nb
    Documentation/
        English/
            Guides/
                SomeGuide.nb
    Resources/
        Images/
            sun.jpg
            moon.jpg

Example Paclet

PackPaclet["/Users/tgayley/Documents/WTC2019/ExamplePaclet"]

PacletInstall[%]

Needs["MyPackage`"]

ExampleFunction[5]

ReadList["ExamplePaclet/hello.txt", String]

Import[PacletResource["ExamplePaclet", "sun"]]

.paclet Files

When deployed, a paclet is a directory containing a PacletInfo.wl file along with one or more content files and subdirectories.

As a convenient form for distributing and installing, paclets can be compressed into a single .paclet file. This is the form in which they are downloaded from a paclet server.

PackPaclet is the function that does this. There is also UnpackPaclet.

Installing and Updating Paclets

PacletInstall is the function that installs a paclet or updates one if a newer version is available.

PacletInstall is fast if no update is available. It (generally) does not contact the paclet server, instead relying on a local cache of server data.

Paclets are installed into $UserBasePacletsDirectory.

Installing and Updating Paclets

Get a paclet from a paclet server, or get an update to an existing paclet:

PacletInstall["PacletName"]

Install from a .paclet file:

PacletInstall["/path/to/SomePaclet.paclet"]

Install from a URL:

PacletInstall["http://www.mycompany.com/paclets/SomePaclet.paclet"]

Install from a .paclet file published to the cloud:

PacletInstall[CloudObject["https://www.wolframcloud.com/obj/tgayley/SomePaclet.paclet"]]

Finding Paclets

PacletFind is the function that finds paclets installed on your system

PacletFind["Interpreter"]

#["Location"]&/@%

Finding Paclets

PacletFind is the function that finds paclets installed on your system

PacletFind["Interpreter"]

Options[PacletFind]

Finding Paclets

PacletFind is the function that finds paclets installed on your system

PacletFind["N*"]

Finding Paclets

Finding Paclets on Servers

PacletFindRemote is the function that finds paclets available on paclet servers

PacletFindRemote["Interpreter"]

Finding Paclets

Finding Paclets on Servers

PacletFindRemote is the function that finds paclets available on paclet servers

PacletFindRemote["Interpreter"]

PacletCheckUpdate["Interpreter"]

Getting Information About Paclets

PacletInformation["Interpreter"]

Getting Information About Paclets

PacletInformation["Interpreter"]

PacletInformation["Interpreter", "Location"]

Getting Information About Paclets

p=First[PacletFind["Interpreter"]];
Information[p]

Paclet Sites

When you call PacletInstall or PacletFindRemote with a paclet name, the PacletManager looks for that paclet in its known set of paclet sites.

Most users have just one such site, the default Wolfram paclet server.

PacletSites[]

Paclet Sites

When you call PacletInstall or PacletFindRemote with a paclet name, the PacletManager looks for that paclet in its known set of paclet sites.

Most users have just one such site, the default Wolfram paclet server.

PacletSites[]

You can use PacletSiteAdd and PacletSiteRemove to manage the set of known paclet sites. The set is persistent from one session to the next, so once you add a site, it remains.

You would call PacletSiteAdd if you or your organization set up your own site; this is easy to do.

Paclet Sites

Setting Up Your Own Paclet Site

If you have paclets that you want to make available to others within your organization, it is easy to set up a paclet site.

No logic is required on the server side. A paclet site can be a simple web server or even a shared directory on the network.

Paclet Sites

Setting Up Your Own Paclet Site

If you have paclets that you want to make available to others within your organization, it is easy to set up a paclet site.

No logic is required on the server side. A paclet site can be a simple web server or even a shared directory on the network.

PacletManager`BuildPacletSiteFiles["/Users/tgayley/Documents/WTC2019/PacletSite"]

This creates the files PacletSite.m and PacletSite.mz (compressed version). That is all that is required when setting up a paclet site--a Paclets directory containing .paclet files and the PacletSite.m and PacletSite.m files built from it.

Paclet Sites

Setting Up Your Own Paclet Site

mySite=PacletSiteAdd["file:/Users/tgayley/Documents/WTC2019/PacletSite"]

PacletSiteUpdate[mySite]

PacletFindRemote["MinecraftLink"]

#["Location"]&/@%

Developing Paclets

Paclets often have a simple layout and can easily be constructed manually using your favorite IDE or editor.

Wolfram Workbench is a great environment for Wolfram Language development, and has built-in tools for paclet development. These tools are a little out of date, but updates are pending.

There is lots of room for developers to create tools and IDE plugins to assist paclet development.

Developing Paclets

Paclets often have a simple layout and can easily be constructed manually using your favorite IDE or editor.

Wolfram Workbench is a great environment for Wolfram Language development, and has built-in tools for paclet development. These tools are a little out of date, but updates are pending.

There is lots of room for developers to create tools and IDE plugins to assist paclet development.

Example ideas:

automatically create a PacletInfo.wl file from a directory

verify the syntax and completeness of a PacletInfo.wl file

automatically create a basic directory layout for a given type of paclet

there is already a function called Pacletize in the Function Repository that creates very simple paclets

Developing Paclets

Authoring and Building Documentation

Creating and building documentation is the most complex part of paclet development.

The documentation that ships with Wolfram products is created in a certain format, with a set of specialized tools, and then it is processed by another specialized tool into the files that you see in the product.

We want paclet authors to be able to create documentation that looks and behaves like built-in documentation, and to do this you will need to use the same specialized pipeline that is used internally, or at least use tools provided by Wolfram that can do the same job.

Developing Paclets

Authoring and Building Documentation

We are working on a paclet with the tools you need to author and build documentation in Wolfram format.

In the meantime, you have two choices:

Use Workbench [demo]

Don’t use the Wolfram documentation format. You can use ordinary notebooks of your own design for your documentation. This is described in the tutorial linked earlier

Developing Paclets

PacletDirectoryAdd

PacletDirectoryAdd tells the Paclet Manager to look for paclets in a non-standard location, and is an extremely useful function for paclet developers.

A typical workflow while developing a paclet is to create the files in the standard paclet layout and then edit them in place. You can call PacletDirectoryAdd[“/my/development/dir”] to let the Paclet Manager load your development version of the paclet. Now you can edit your .wl files and call Get[“MyContext`”] repeatedly to reload them, picking up your newest changes.

You can think of PacletDirectoryAdd as the analog of PrependTo[$Path, “dir”] in the non-paclet world.

PacletDirectoryAdd["/Users/tgayley/Documents/WTC2019/wtc_workspace/ExamplePaclet"]

Paclet Repository

Under development.

We hope to create a thriving collection of paclets providing functionality of all types. A one-stop shop for Wolfram add-ons, similar to package repositories for other languages (like PyPI for Python and NuGet for .NET).