Persistence and Initialization

Roman E. Maeder
Wolfram Research

The Wolfram Language allows values to be set up to be persistent beyond a single session. The persistent values can be stored in a variety of locations, locally or in the cloud. Values can either be retrieved from a single location or can be aggregated from multiple locations.

This presentation gives an overview over new persistence features, caching of results, and their use for initializing sessions in a sophisticated way.

Join the Conversation #WolframTechConf

Overview

Persistence is a unifying access layer on top of various storage technologies, such as local objects, cloud objects, frontend sessions, ...

Lookup of values is along a search path, and values found can be merged in various ways.

Main uses are initializations (for the kernel at first), and a framework for storing preferences for packages or subsystems.

Terms

Persistent value, item, key-value pair: the smallest unit of data.

PersistentValue["myvalue"]=42;

PersistentValue["myvalue"]

Persistent values behave as much as possible like symbols, but offer additional scopes and life times.

sym=32;

sym

Persistent object: a denotation for a persistent value, consisting of the key (name) and location. This is needed just like symbol names (strings), to manipulate those symbol-like things that go away when you touch them.

PersistentObject["myvalue",$PersistenceBase]

"sym"

Persistence location: a technology that allows storing and retrieving key-value pairs (backing store)

PersistenceLocation["Local"]

Initialization: a persistent value that has a special meaning (to the kernel, for example)

InitializationValue["$initVar"]=43;

Preference: a persistent value that controls the behaviour of a package or subsystem, and is typically set via a GUI.

PersistentValue["MyApplication`Package`$var"]

Underlying Storage Technologies

Persistent values can be stored at various locations, which differ by scope and life time.
Storage locations can be symbols, files, cloud objects, or property storage provided by the frontend.

Symbols

Scope: a single kernel session

Lifetime: a single kernel session

sym=42;

sym

NameQ["sym"]

Remove[sym]

Frontend Session

Scope: all kernels running under the same frontend

Lifetime: a single frontend session

SetOptions[$FrontEndSession,"myval"->43];

"myval"/.Options[$FrontEndSession]

Notebook Documents

Scope: a notebook’s evaluation kernel

Lifetime: same as the document

SetOptions[EvaluationNotebook[],"myval"->44];

"myval"/.Options[EvaluationNotebook[]]

Local Objects

Scope: a user’s login domain, or accessible file system

Lifetime: forever

LocalSymbol["sym"]=45;

LocalSymbol["sym"]

FileExistsQ[LocalObject["sym",$LocalSymbolBase]]

Remove[LocalSymbol["sym"]]

Cloud Objects

Scope: a cloud login (Wolfram ID)

Lifetime: forever

CloudSymbol["sym"]=46;

CloudSymbol["sym"]

FileExistsQ[CloudObject["sym",$CloudSymbolBase]]

Persistence Locations

Persistence locations provide a unified access mechanism to any of the available storage technologies.

Kernel Session

Kernel values are stored in the kernel. Their lifetime is the lifetime of the kernel, and they are visible in a kernel only.

FrontEnd Session

FrontEnd session values are stored in the frontend. Their lifetime is the lifetime of the frontend, and their visibility is each kernel launched by that frontend. In particular, they survive kernel restarts.

Notebook

Values in the Notebook location are stored in the evaluation notebook, and thus travel with it wherever it is used.

Local

User values are stored inside $LocalBase as local objects (as files). Their lifetime is forever, and their visibility is a user’s home directory.

Cloud

Cloud values are stored in the Wolfram Cloud (or private cloud). Their lifetime is forever, and visibility is with a user’s cloud login (Wolfram ID).
There can be several cloud locations (think of private cloud and Wolfram cloud).

LocalShared

Site-wide values are stored inside $BaseDirectory, and are used for site-wide settings. The backing store is LocalObject. These values are read-only for users. Site administrators can populate these values.

Installation

Installation defaults stored inside $InstallationDirectory. Read-only. Used for defaults of all standard preference items, such as initializations.

Persistent Values

Examples

A persistent value is defined by assignment.

PersistentValue["mykey"]=42;

Lookup will retrieve any value present.

PersistentValue["mykey"]

Locations and persistence path

Values can be defined in any available location. The default for assignment is

$PersistenceBase

Lookups happen along a search path

$PersistencePath

Note that the cloud persistence location is not on the default search path.

You can set a value at any location

PersistentValue["mykey","KernelSession"]=41;

PersistentValue["mykey","FrontEndSession"]=42;

PersistentValue["mykey"]=43;

Lookup by default returns the first value found.

PersistentValue["mykey"]

You can retrieve all values present and apply a merging function to the list of results.

PersistentValue["mykey",MergingFunction->Identity]

PersistentValue["mykey",MergingFunction->First]

Advanced Merging

Create a number of persistent values with key "lval" at different persistence locations:

PersistentValue["lval","KernelSession"]={11,12};
PersistentValue["lval","FrontEndSession"]={};
PersistentValue["lval","Local"]={1,{2,12}};

Give the list of all values found:

PersistentValue["lval",MergingFunction->Identity]

Join the list values:

PersistentValue["lval",MergingFunction->Apply[Join]]

Flatten all sublists:

PersistentValue["lval",MergingFunction->Flatten]

Even more fancy merging can be done with values that are associations.

Persistent Objects

Persistent values are stored in persistent objects. Here we ask for all objects for the values used above,

PersistentObjects["mykey"]

and remove them:

DeleteObject[%]

Expiration

Define a persistent value that expires in 5 seconds:

PersistentValue["x",PersistenceTime->5]=1;

After 5 seconds, it is no longer used:

PersistentValue["x"]
Pause[5];
PersistentValue["x"]

Kernel Initialization

Variable Initialization

A subclass of persistent values supports variable initialization.

A setting of

InitializationValue[myvar]=1;

will store the given value in a persistent object and then cause the assignment

myvar=InitializationValue[myvar]

to be performed when you say

Initialize[myvar]

myvar

Support for $PersistenceBase and $PersistencePath works the same was as for PersistentValue, as does MergingFunction.

The stored values are inside persistent objects with a particular prefix in their keys

PersistentObjects["System/Session/*"]

Kernel Initialization

When the kernel starts up, it essentially does

Initialize[All]

This initializes all variables (standard system $ variables, and your own) for which there is an initializer defined.

Use Cases

Adding the Cloud Location to $PersistencePath

For performance reasons, the “Cloud” location is not on the default search path.

$PersistencePath

Set up an initialization value to change the persistence path:

InitializationValue[$PersistencePath,"Local"]={"KernelSession","FrontEndSession","Local","LocalShared","Cloud"};

Quit and restart the kernel to check it out:

Quit[];

$PersistencePath

Caching Initializations in a Notebook

Create an init cell:

value=Once[VeryLongComputation[Print["thinking..."]];RandomReal[],"Notebook"]

“once” computed, it will travel with the notebook, wherever you go, or with whomever you share it, and evaluating the initialization will bring the value back without recomputation.

To delete any stored notebook values that may have accumulated:

PersistentObjects[All,"Notebook"]

DeleteObject[%]

A Debugging Session

Put such a cell near the top of your test/devel notebook (no need for an init cell) and evaluate it once per frontend session.

InitializationValue[$Initialization,"FrontEndSession"]=Hold[AppendTo[$Path,"/my/development/path"]];
InitializationValue[$InitializationContexts,"FrontEndSession"]={"MyPackage`"};

while the frontend is running, quit and restart the kernel, and the code above will be redone.

Quit[];

$Path

$ContextPath

?MyPackage`*

Once you quit the frontend, all evidence of your actions will be removed.

Setting a Default Merging Function

Default merging functions can be set up together with default values for initializations. Suitable locations for such values are “Local” or “Cloud”.

InitializationValue[ivar,"Local",MergingFunction->Flatten]={};

Add an initialization value at the front end persistence location:

InitializationValue[ivar,"FrontEndSession"]={ival1};

The resulting initialization value now uses the default merging function:

InitializationValue[ivar]

When the variable is initialized (either manually or at each kernel start), the default merging function is used:

Initialize[ivar];
ivar

System Variables with Custom Merging Functions

System-wide defaults are set up at the “Installation” persistence location.

InitializationObjects[$Assumptions]

First[Flatten[Values[%]]]

%[All]

$Assumptions

Initializing Parallel Subkernels

You might have noticed that the keys for initializers are of the form System/Session/ctx`name. The second part is the value of $EvaluationEnvironment.

$EvaluationEnvironment

You can set an initializer for a different type of kernel (of course, you need to choose a persistence location that is visible and used by that type of kernels):

InitializationValue[$TimeZone,"Local",EvaluationEnvironment->"Subkernel"]=0;

LaunchKernels[2]

$TimeZone

ParallelEvaluate[$TimeZone]

CloseKernels[];

To list (and delete) such values, you can use

InitializationObjects[All,{"Local","Cloud"},EvaluationEnvironment->"Subkernel"]

DeleteObject[Values[%]]

Troubleshooting

Cleaning up

The equivalent of

ClearAll["Global`*"]

Find all removable values, even if the corresponding locations are not on $PersistencePath (leave out “Cloud” if not connected to a network):

InitializationObjects[All,{"KernelSession","FrontEndSession","Local","Notebook","Cloud"}]

InitializationObjects[All,{"KernelSession","FrontEndSession","Local","Notebook"}]

DeleteObject[Values[%]]

Debugging

If all else fails: start the kernel with -noinit, which suppresses all processing described here

For your initial experiments, set

$PersistenceBase="FrontEndSession"

or use

InitializationValue[$Initialization,"FrontEndSession"]=Hold[Get["MyPackage`"]]

Any values defined at the “FrontEndSession” location go away when you quit the frontend.