Import/Export & Data Connection Updates

Piotr Wendykier, Wolfram Research

#WolframTechConf

Outline

Markdown

JSONLines

Log Formats

Other Import/Export Updates

Import/Export Changes in 15.0

Updates to Data Connections

Markdown

Import as Notebook

Import["ExampleData/geese.md","NotebookObject"]

Export of Notebooks

nb=Import["ExampleData/document.nb","NotebookObject"]

Export["file.md",nb]

Import["file.md","Text"]//Snippet[#,10]&

Import["file.md","NotebookObject"]

Opening and saving Markdown in the Front End

To open a Markdown file as a Notebook Use File > Open, and choose Markdown Files as the file type

A Notebook can be saved as a Markdown file using File > Save As > Markdown

Generate Markdown fragments from each Cell in a Notebook

For example, you can take a table in a Notebook, then just do Copy As > Markdown to get a Markdown version

JSONLines

New tabular format

JSONLines consists of multiple JSON objects, one per line, representing individual data rows

Import["ExampleData/world_cups.jsonl"]

Export to JSONLines

tabular=Import["ExampleData/USstates.arrow", {"Tabular",1;;3}]

ExportString[tabular,"JSONLines"]

Log Formats

Updated Import of ApacheLog

Faster Import that returns a Tabular by default

Import["ExampleData/apache_error.log", "ApacheLog"]

“DataFormat” option allows to read data from an arbitrary log format

AbsoluteTiming[tabular=Import[FileNameJoin[{NotebookDirectory[], "CloudPlatform.log"}],"ApacheLog", "DataFormat"->"[`Timestamp`] [`User`] [`SessionID`] [`IP`] [`Severity`] [`Thread`] : `Message`"];]

Dimensions[tabular]

CommonLog - new Import format

Import["ExampleData/common20.log", "CommonLog"]

ExtendedLog - new Import format

Import["ExampleData/extended.log", "ExtendedLog"]

Other Import/Export Updates

CSV, TSV, Markdown, JSONLines

New Export option “ExpressionFormattingFunction”

tabular=Import["ExampleData/USstates.arrow", {"Tabular",1;;3}]

ExportString[tabular,"CSV"]

ExportString[tabular,"JSONLines","ExpressionFormattingFunction"->SpokenString]

Parquet, ORC

Efficient partial row Import

file=FileNameJoin[{NotebookDirectory[], "sample.parquet"}];

FileSize[file]

Import all data from the file:

RepeatedTiming[tabular1=Import[file];]

Dimensions[tabular1]

Import the first 100 rows:

RepeatedTiming[tabular2=Import[file, {"Tabular",1;;100}];]

Dimensions[tabular2]

AVIF

New Import format for static and animated images

BDF, EDF

New Import elements: “Tabular” and “TimeSeriesAssociation”

MDB

New Import elements: “Tabular” and “TabularAssociation”

DBF, NDK, TLE, MTP

New Import element: “Tabular”

GPX

New Import element: “TabularAssociation”

ODS

New Import element “SheetCount”

Updated “Summary” element

SXC

New Import options: “SkipLines” and “HeaderLines”

DIF

Export now supports Tabular objects

MTX

Backend update - fixed bugs and up to 10x speed improvements

Import/Export Changes in 15.0

New config formats: TOML & YAML

TOML - Tom’s Obvious, Minimal Language

TOML is used in Rust, Python and Julia to store package manifests

This example imports Cargo.toml file from wolfram-app-discovery-rs project

Import["https://raw.githubusercontent.com/WolframResearch/wolfram-app-discovery-rs/refs/heads/master/Cargo.toml"]//Dataset

YAML - YAML Ain’t Markup Language

YAML is a data serialization language commonly used for configuration files

It is used to define workflows and automation scripts such as CI/CD in GitHub Actions

This example imports Github Actions script from WL-FunctionCompile-CI-Template project

Import["https://raw.githubusercontent.com/WolframResearch/WL-FunctionCompile-CI-Template/refs/heads/master/.github/workflows/compile-wl-functions.yml"]//Dataset

New document formats: IPYNB & VSNB

IPYNB - Jupyter notebooks

IPYNB  is a JSON-based file format used for storing IPython notebooks, also known as Jupyter notebooks

This example imports Jupyter notebook

Import["ExampleData/example.ipynb"]

VSNB - Visual Studio Code notebooks

VSNB  is a JSON-based file format used by Visual Studio Code notebooks for Wolfram Language

This example imports VSNB notebook

Import["ExampleData/example.vsnb"]

Import/Export Options

Use FileFormatProperties to query for Import and Export options

FileFormatProperties["MP3"]//Dataset

Use SetFileFormatProperties to set Import and Export options

SetFileFormatProperties["MP3", "ExportOptions"-><|SampleRate->22050|>]

Import[Export["out.mp3",Import["ExampleData/car.mp3"]]]//AudioSampleRate

“TimeSeries” and “EventSeries”

New Import elements added to 25 file formats

This example imports data from a CSV file as TimeSeries object

ts=Import["https://fred.stlouisfed.org/graph/fredgraph.csv?mode=fred&id=DEXUSEU&fq=Monthly","TimeSeries"]

DateListPlot[ts,PlotLabel->"U.S. Dollars to Euro Spot Exchange Rate"]

Partial access Import via column names

Tabular formats will support partial access via column names

This example imports two columns from a CSV file

Import["ExampleData/TreesOwnedByTheCityOfChampaign.csv", {"Tabular",1;;5, {"Tree Species","Common Name"}}]

Markdown updates

New Import elements: “Hyperlinks”, “Images”, “Tables”, “Title”

AVIF Export

Export to AVIF will be supported

Image metadata updates

Exif will be imported as nested Association with namespaces

Import["ExampleData/coneflower.jpg","Exif"]//Dataset

Bugfixes and performance improvements

Updates to Data Connections

DataConnectionObject

New object introduced in Version 14.2

Allows to retrieve data from a remote storage system

Handles authentication and data retrieval

Directly import the result of SQL queries from various databases

Import@DataConnectionObject[
<|"ConnectionName"->"SQLite","FilePath"->FindFile["ExampleData/ecommerce-database.sqlite"],"Location"->"SELECT * FROM employees"|>]

List of data connections in 14.3

Connections to file systems

AmazonS3

AzureBlobStorage

Dropbox

IPFS

OneDrive

Kaggle

Connections to relational databases

PostgreSQL

MySQL / MariaDB

MicrosoftSQL

Oracle

SQLite

List of new data connections in 15.0

Connections to file systems

AzureFile

AzureTable

Connections to multidimensional databases

Databricks

Apache Pinot

Snowflake

New documentation

ref/DataConnectionObject

Now links to guide/ListingOfConnections

Connections each have a ref/connection/XXXX page e.g. ref/connection/AmazonS3

Possible future updates

Connections to time series databases

Apache Pinot

TimescaleDB

Connections via APIs

Astronomy data

Weather data

Language integration

CopyFile, Get, CreateArchive, and ExtractArchive from a DataConnectionObject