Automating Content Management Using The Wolfram Technology Stack

Dr. Oliver Grasl, transentis

(RowDefault[, (Spacer1), , (Spacer1), ])  #WolframTechConf

Introduction

There is a strong trend in the market towards API-driven, multi-source, multi-channel content management

Due to the “API first” and “Cloud micro service” trends, automating state-of-the-art, multi-channel content creation and delivery is a sophisticated and involved process:  services from many different providers need to be orchestrated, content has to be read from many different sources and transformed even more different target formats.

We need "glue" to orchestrate these APIs and extract and transform content and load it to multiple channels.

To create a high degree of automation, you need a technology stack that runs on many different target platforms,  that can easily be connected to diverse APIs, provides sophisticated transformation operations for data in both text formats (XML, JSON) and graphics formats (vector and raster images) and that can be developed and tested in an interactive fashion.

The Wolfram Technology Stack turns out to be a perfect fit.

This presentation illustrates how Wolfram Technologies can be used to pull content from Contentful CMS and convert them into PDFs.

Concrete Challenge

Modern content management has gone “headless”

Store content in (multiple) content repositories

Deliver content over multiple channels (website, social media,...)

Completly separate the “input and storage” layer from the presentation layer (“the head”).

Generate output formats "just in time" or "upon change"

Key Insight: This is essentially an ETL challenge

Modern repositories all provide REST or GraphQL APIs and provide data in JSON format.

So getting content from A to B and transforming it on the way is essentially an ETL challenge.

Why the Wolfram Tech Stack is ideally suited to the challenge

Example: Generating PDF Documents from Contentful CMS

Our website at transentis.com is generated from the Contentful CMS using Gatsby.js.

Key Idea : Generate downloadable PDFs from our blog posts

Step 1: Retrieve Data via REST API

Retrieving data from REST APIs is easy using Wolfram Language

<<(FileNameJoin[{NotebookDirectory[],"wtc_tokens.wl"}] )

entryId="2VRfIjqwmMBSAIQ9QgDwtr";(*id of concrete blog post*)

entryRequest=HTTPRequest[URLBuild[{rootURL,"entries",entryId}],<|"Method"->"GET","Query"->{"access_token"->accessToken,"locale"->"en-US"}|>,TimeConstraint->Infinity];

entry=URLExecute[entryRequest,"JSON"]

entry =<<(FileNameJoin[{NotebookDirectory[],"wtc_contenful_example.wl"}] )

entry//Length

entry//Depth

Find the original post Training AI to Play the Beer Game and the PDF generated from it here.

Highlight: Navigating JSON Structures

Representing JSON in Wolfram Language

JSON structures are converted to nested lists of rules in Wolfram Language:

exampleJSON={"tag1"->"value","tag2"->{"tag2.1"->"value2.1","tag2.1"->"value2.2"}}

Extract substructures via the ReplaceAll operator :

"tag1"/.exampleJSON

Wrap this into a  function for easy reuse

jsonPart[json_List,path_List]:=Fold[ReplaceAll[#2,#1]&,json,path];

jsonPart[entry,{"fields","title"}]

jsonPart[entry,{"fields","subtitle"}]

Step 2 : Convert JSON to Abstract Data Types

The most complicated data structures in Contentful JSON are the rich text fields - to create PDFs, we need to parse them and convert them to notebook cells.

Exploring the Contentful rich data fields

jsonPart[entry,{"fields","content"}]

Let' s get the first paragraph:

FirstCase[jsonPart[entry,{"fields","content"}],{_,_,"nodeType"->"paragraph"},{},Infinity]

All nodeTypes in the document:

Cases[jsonPart[entry,{"fields","content"}],{_,_,"nodeType"->nodeType_}->nodeType,Infinity]//Tally//Sort//TableForm

Small challenge: JSON doesn’t guarantee an ordering - so our patterns can become quite verbose

pattern=Alternatives@(Sequence@@Permutations[{_,_,"nodeType"->nodeType_}])

Solution

The solution is to define a symbolic contentful datatype contenfulRichText[nodeType, data, marks, value] and to convert the JSON structures into this format.

This is similar to the Symbolic XML built in to the Wolfram Language.

Having such a data type is also useful for writing concise patterns.

contentfulRichText::usage="Abstract datatype that represents Contentful rich text elements"

jsonToContentfulRichText[element_/;("nodeType"/. element)!="text"]:=contentfulRichText["nodeType"/. element,"data"/. element,jsonToContentfulRichText/@("content"/. element)]

jsonToContentfulRichText[element_/;("nodeType"/. element)=="text"]:=contentfulRichText["text","data"/. element,"value"/. element];

crt=jsonToContentfulRichText[jsonPart[entry,{"fields","content"}]]

Cases[crt,contentfulRichText[nodeType_,_,_]->nodeType,Infinity]//Tally//Sort//TableForm

Highlight : Multiple Dispatch

Using Wolfram language you can create multiple definitions (downvalues) for the same symbol - which one is used in a concrete call depends on the concrete input parameters.

This powerful feature is known as multiple dispatch.

Many languages (e.g. Python) don’t provide this great feature.

Multiple dispatch is extremely useful for writing ETL scripts.

data={"wtc21", "rocks",123,1.5,{"first", "sublist",{2.0,"subsublist"}}}

transform[x_String]:="String";

transform[x_Integer]:="Integer";

transform/@data

transform[x_List]:=transform/@x

transform[x_Real]:=Real

transform/@data

Step 3 : Convert ADTs to Notebook Cells

Using Wolfram Language’s multiple dispatch, it is easy to convert our new ADTs into notebook cells

contentfulRichTextToWl[contentfulRichText["document",data_,content_]]:=contentfulRichTextToWl[#]&/@content;

contentfulRichTextToWl[contentfulRichText["heading-3",data_,content_]]:=TextCell[Row[contentfulRichTextToWl[#]&/@content],"Subsubsection"];
contentfulRichTextToWl[contentfulRichText["text",data_,content_]]:=content;
contentfulRichTextToWl[contentfulRichText["paragraph",data_,content_]]:=TextCell[Row[contentfulRichTextToWl[#]&/@content],"Text"];
contentfulRichTextToWl[contentfulRichText[type_ /;!MemberQ[{"document","heading-3","text","paragraph"},type] ,data_,content_]]:=Nothing;

textCells=contentfulRichTextToWl[crt];

Cases[textCells,TextCell[_,"Subsubsection"]]

Cases[textCells,TextCell[_,"Text"]]//Length

FirstCase[textCells,TextCell[_,"Text"],{}]

Note : Lot’s of details such as images, text styling etc. not considered here.

Step 5 : Generate PDF

Generating a PDF from a notebook is easy because it is built into the Wolfram language.

nb=CreateDocument[textCells,Visible->False,StyleDefinitions->"transentis-blog.nb"]

Export["document.pdf",nb]

Import["document.pdf"]

%//First

Highlight : Typesetting Mathematical Equations

Some of our blog post contain mathematical equations, which we typeset using LaTex/KaTex on our website and using Wolfram Box Notation for our Blog Post PDFs.

typeset[equation_String]:=DisplayForm@FormBox[ImportString[ToString[#,CharacterEncoding->$SystemCharacterEncoding]&@"\("<>equation<>"\)","wl"],TraditionalForm]

typeset@"p(s\\^′,r|s,a)==Pr{S\\_t==s\\^′,R\\_t==r | \\
S\\_\\(t+1\\)==s,A\\_\\(t+1\\)==a}"

typeset@"∑\\+\\(s\\^′∈𝒮\\)∑\\+\\(r\\∈ℛ\\)p(s\\^′,r|s,a)=1,∀s\\∈𝒮,a∈𝒜(s)"

typeset@"\*GridBox[{{\(q\_*(s,a)\),\(==\), \(𝔼[R\_\(t+1\)+γmax\+a q\_* (S\_\(t+1\),a\^′) | S\_t ==s, A\_t ==a ]\)},{\(\),\( == \),\( ∑\_\(s', r\) p(s\^′,r | s,a)[ r+γ max\+\(a\^′\)  q\_* (s\^′,a\^′)]\)}}]"

Deploying Into Production

Finding the right deployment architecture wasn’t easy...

In theory there are many options (Enterprise Private Cloud, Wolfram App Server, Wolfram Web Engine, Wolfram Engine ...), but it should be as light-weight and scalable as possible.

We finally settled on a simple Wolfram Engine sitting behind a web service powered by a Python uWSGI server, all running on Amazon EKS (Kubernetes).

When the web service is triggered, it starts a Wolfram Engine as a background process  - this way we can use on-demand licensing for the engine.

Unfortunately this isn’t working in practice yet, because there are issues with the PDF generator on WolframEngine Docker Container.

Lessons Learned

Implementation Challenges

Generate documents for multiple locales (German, English)

SVG not supported natively by Wolfram Language: the Resource Function SVGImport came to the rescue.

Converting 1000s of SVG Images: Set up a caching solution.

Efficient generation mechanism: Only regenerate posts and images that have changed (“delta generation mechanism”).

Typesetting mathematical formulae.

Technology Stack Challenges

Font Issues (Generating PDFs from Source Sans Pro on a Mac).

PDF Generation not working with pure WolframEngine.

Documentation lacking when at the “edge” of the Wolfram Tech Stack.

Few HOW-TOs on writing production level code.

Lack of architectural blueprints for production settings.