File Formats Any Data Scientists Cares About

Piotr Wendykier, Sean Cheren
Wolfram Research

(RowDefault[, (Spacer1), , (Spacer1), ])  #WolframTechConf

Outline

Columnar data structure

Apache Arrow Library

CSV

Parquet

Arrow Columnar Format

Columnar Data Structure (work in progress)

Also known as data frame, table and structured data

Typically each column represents a variable and each row represents a data point

Each column is usually of a homogeneous type such as number, category, string or date

Overall, table data is also typically heterogeneous since the columns can be different types

Data entries can be missing

Columnar Data Structure (work in progress)

In the Wolfram Language it is tentatively called Tabular

t=Tabular[
Transposed[
{"sex"->{"M","M","M","F","F"},
"size"->{8.1,13.9,16.1,7.2,14.9},"length"->{16.1,29.2,33.8,14.7,30.1}
}]]

TableView[t]

Schema - an association with keys as column labels and domains as the values

t["Schema"]

DataVectorTable - the table of data as a data structure

dataTable = t["DataVectorTable"]

VectorList - the set of DataVector data structures with consistent length

vectorList = dataTable["VectorList"]

Normal /@vectorList

Apache Arrow Library

“A cross-language development platform for in-memory analytics”

Provides building blocks for high performance analytic operations

Supports various tabular file formats including Parquet, CSV and ORC

Implements Arrow’s IPC format

Used by many projects as an analytic engine

Available for C, C++, C#, Go, Java, JavaScript, Julia, MATLAB, Python, R, Ruby, and Rust

File Used in Benchmarks

Reddit Place 2022 Official Canvas History

20GB comma-separated values (CSV) file

4 columns of type string

Number of rows: 160,353,104

File was converted to Parquet and Arrow IPC

All benchmarks are run on the same machine

CSV

Text file format that uses commas to separate values

Stores numbers and strings in plain text

It is a very popular data exchange format

Import benchmarks

Grid[{{"","reading (s)","conversion (s)", "total import time (s)"},{"WL prototype",12.3,17.4,29.7},{"python 3.11 +\npyarrow 13.0",5.9, 166.7,172.6}},Alignment->Left,Spacings->{2,1},Frame->All,Background->{{LightGray,None},{LightGray,None}}]/.s_String:>Style[Text[s],FontSize->40]

Export benchmarks

Grid[{{"","conversion (s)","writing (s)", "total export time (s)"},{"WL prototype",22.6,26.9,49.5},{"python 3.11 +\npyarrow 13.0",72.3, 27,99.3}},Alignment->Left,Spacings->{2,1},Frame->All,Background->{{LightGray,None},{LightGray,None}}]/.s_String:>Style[Text[s],FontSize->40]

Parquet

Columnar storage format created for the Hadoop ecosystem

Designed for efficient data storage and retrieval

Provides efficient data compression and encoding schemes

Import benchmarks

Grid[{{"","reading (s)","conversion (s)", "total import time (s)"},{"WL prototype",23.8,17.4,41.2},{"python 3.11 +\npyarrow 13.0",26.2, 166.7,192.9}},Alignment->Left,Spacings->{2,1},Frame->All,ItemStyle->"Text",Background->{{LightGray,None},{LightGray,None}}]/.s_String:>Style[Text[s],FontSize->40]

Export benchmarks

Grid[{{"","conversion (s)","writing (s)", "total export time (s)"},{"WL prototype",22.6,37.3,59.9},{"python 3.11 +\npyarrow 13.0",72.3, 81.5,153.8}},Alignment->Left,Spacings->{2,1},Frame->All,ItemStyle->"Text",Background->{{LightGray,None},{LightGray,None}}]/.s_String:>Style[Text[s],FontSize->40]

Arrow Columnar Format

Columnar format for flat and hierarchical data

Includes protocol for serialization and Interprocess Communication (IPC)

Supports zero-copy reads for fast access without serialization

Arrow IPC streaming format -  for streaming

Arrow IPC file format - for random access

Arrow IPC Stream Format

Import benchmarks

Grid[{{"","reading (s)","conversion (s)", "total import time (s)"},{"WL prototype",1.9,17.4,19.3},{"python 3.11 +\npyarrow 13.0",7.2, 166.7,173.9}},Alignment->Left,Spacings->{2,1},Frame->All,ItemStyle->"Text",Background->{{LightGray,None},{LightGray,None}}]/.s_String:>Style[Text[s],FontSize->40]

Export benchmarks

Grid[{{"","conversion (s)","writing (s)", "total export time (s)"},{"WL prototype",22.6,20.9,43.5},{"python 3.11 +\npyarrow 13.0",72.3, 20.4,92.7}},Alignment->Left,Spacings->{2,1},Frame->All,ItemStyle->"Text",Background->{{LightGray,None},{LightGray,None}}]/.s_String:>Style[Text[s],FontSize->40]

Arrow IPC File Format

Import benchmarks

Grid[{{"","reading (s)","conversion (s)", "total import time (s)"},{"WL prototype",1.8,17.4,19.2},{"python 3.11 +\npyarrow 13.0",7.2, 166.7,173.9}},Alignment->Left,Spacings->{2,1},Frame->All,ItemStyle->"Text",Background->{{LightGray,None},{LightGray,None}}]/.s_String:>Style[Text[s],FontSize->40]

Export benchmarks

Grid[{{"","conversion (s)","writing (s)", "total export time (s)"},{"WL prototype",22.6,21.1,43.7},{"python 3.11 +\npyarrow 13.0",72.3, 21.1,93.4}},Alignment->Left,Spacings->{2,1},Frame->All,ItemStyle->"Text",Background->{{LightGray,None},{LightGray,None}}]/.s_String:>Style[Text[s],FontSize->40]

Summary

Columnar data structure (Tabular) will be released in 14.1 (or later)

Import and Export will support new columnar file formats

Benchmarks of these formats show much better performance than Python libraries

Related Talk