Import/Export
Updates and Applications

Sean Cheren, WRI
Rafał Chojna, WRI

Join the Conversation #WolframTechConf

Abstract

Importing and Exporting data is a constantly growing field which remains an integral part of working with Wolfram Language. In this talk we will demonstrate some of the key improvements to various formats in upcoming versions. In particular, we will explore updates to DICOM images, archive/compression formats, and email formats. We will also give a sneak peak of some convenient Import/Export utility functions coming in the next version of Wolfram Language.

DICOM

Import a single DICOM file:

Import[file=FileNameJoin[{NotebookDirectory[],"shoulder_joint.dcm"}]]

Import[file,"Elements"]

Import[file,"Summary"]

DICOM Anonymization

DICOM files contain plenty of sensitive information.

Display patient - related metadata:

meta=Import[file,{"MetaInformation","Patient*"}];
KeySort[meta]//Dataset

Anonymization is often required before such files can be used in research papers or studies.

Needs["DICOMTools`"];
DICOMAnonymize=DICOMTools`Private`DicomAnonymize;
DICOMAnonymize[file,"shoulder_joint_anon.dcm"];

Some tags like PatientName are changed:

(newMeta=Import["shoulder_joint_anon.dcm",{"MetaInformation","Patient*"}])//KeySort//Dataset

Some tags are completely removed:

Complement[Keys[meta],Keys[newMeta]]

Some information is still preserved:

Import["shoulder_joint_anon.dcm",{"MetaInformation","PerformedProcedure*"}]//Dataset

Perform full anonymization, erasing more metadata:

DICOMAnonymize[file,"shoulder_joint_anon_full.dcm","AnonymizationLevel"->"Full"];

Import["shoulder_joint_anon_full.dcm",{"MetaInformation","PerformedProcedure*"}]//Dataset

Get the list of tags that are anonymized:

DICOMTools`Private`basicAnonymizationTags//Keys

Additional tags anonymized in “Full” anonymization mode:

Complement[Keys@DICOMTools`Private`fullAnonymizationTags,%]//Short

DICOMDIR

DICOMDIR is a collection of logically related DICOM files structured by patient -> study -> series.

Inspect the structure of a DICOMDIR:

Import["spine_mr/DICOMDIR","StructureGraph"]

Get a short summary showing the number of patients, studies and series included in the DICOMDIR:

Import["spine_mr/DICOMDIR","Summary"]

Get the structured list of DICOM files that are part of the DICOMDIR:

Map[Short[#,10]&,Import["spine_mr/DICOMDIR","FileNames"],{-3,-2}]

Import a single series for a given patient as a list of images:

Import["spine_mr/DICOMDIR",{"ImageList",1,1,2},"ScalingWindow"->"MinMaxPerFrame"]

Metadata can also be retrieved for DICOMDIR:

Import["spine_mr/DICOMDIR",{"MetaInformation"}][[1,1,1,1,2]]//Dataset

Archive/Compression

Existing CreateArchive and ExtractArchive

CreateArchive takes an existing file or directory, and constructs a .zip, .tar, .tar.gz, or a .tar.bz2 file in Directory[] or a specified location.

cleanup[]:=(Quiet[DeleteFile["test.txt"]];Quiet[DeleteFile["test.txt.zip"]];Quiet[DeleteDirectory["testDirectory"]];Quiet[DeleteDirectory["testDirectory/",DeleteContents->True]];Quiet[DeleteDirectory["testDirectory",DeleteContents->True]];Quiet[DeleteFile["testDirectory.zip"]];Quiet[DeleteDirectory["testDirectoryNew",DeleteContents->True]];
DeleteFile/@FileNames["test *"];)
cleanup[]

CreateFile["test.txt"]

CreateArchive["test.txt"]

CreateDirectory["testDirectory"]

CreateFile["testDirectory/foo.txt"]
CreateFile["testDirectory/bar.txt"]

CreateArchive["testDirectory/"]

Import["testDirectory.zip"]

ExtractArchive takes a .zip, .tar, .tar.gz, or a .tar.bz2 file, and extracts it to Directory[] or a specified location.

ExtractArchive["testDirectory.zip","testDirectoryNew"]

CreateArchive

Updates

ZIP64 and large archive support for ZIP

General bugfix in file times, permissions, and symbolic links on Mac/Linux

Option OverwriteTarget added.

CreateArchive["test"]

CreateArchive["test",OverwriteTarget->True]

CreateArchive["test",OverwriteTarget->"KeepBoth"]

Options[CreateArchive]

List/Association syntax to allow multiple file/directory sources, and control over the names of files in the archive:

FileNames["s*",DirectoryName@NotebookFileName[]]

CreateArchive[<|"spine"->"/Users/scheren/ownCloud/ImageSignalProcessing/Talks/WTC/2019/ImportExport/spine_mr","shoulder.dcm"->"/Users/scheren/ownCloud/ImageSignalProcessing/Talks/WTC/2019/ImportExport/shoulder_joint.dcm"|>,OverwriteTarget->True]

Import[%]//Short[#,5]&

Compression level from 0.1 to 1.0:

Performance

ZIP Create

(Legended[, (SwatchLegend[12.1, 12.0])])

TAR Create

(Legended[, (SwatchLegend[12.1, 12.0])])

ExtractArchive

Updates

ZIP64 and large archive support for ZIP

General bugfix in file times, permissions, and symbolic links on Mac/Linux

Introduce support for 7Z and RAR

Introduce option OverwriteTarget

ExtractArchive["test.zip"]

ExtractArchive["test.zip",OverwriteTarget->True]

Performance

ZIP Extract

(Legended[, (SwatchLegend["12.1", "12.0"])])

TAR Extract

(Legended[, (SwatchLegend[12.1, 12.0])])

Import/Export

Uses the same backend as ExtractArchive and CreateArchive

ZIP64 and large archive support for ZIP

Introduce support for 7Z and RAR

EMail

MBOX/EML

Basic Elements

Import["ExampleData/conv.mbox",{"MessageSummary"}]//First//Keys

KeyDrop[Dataset[Import["ExampleData/conv.mbox",{"MessageSummary"}]],{"BccList","CcList"}]

Import["ExampleData/conv.mbox","MessageElements"]//First//Keys

Dataset@Import["ExampleData/conv.mbox",{"MessageElements",1}]

Import["ExampleData/in.mbox","FullMessageElements"]//First//Keys

Import["ExampleData/conv.mbox",{"FullMessageElements",All,{"Subject","BodyPreview"}}]//Dataset

Elements Updates

Import["ExampleData/attach.mbox","Elements"]

Removed (Deprecated 12.1, Removed 12.2)

Import["ExampleData/attach.mbox",{"Attachments",1}]

Map[Short,Import["ExampleData/attach.mbox",{"AttachmentAssociations",1}],{-1}][[1]]

Keys[Import["ExampleData/attach.mbox",{"AttachmentAssociations",1}][[1]]]

Added

Import["ExampleData/attach.mbox",{"AttachmentList",1}]

Import["ExampleData/attach.mbox",{"AttachmentDetails",1}]//Dataset

Import["ExampleData/attach.mbox",{"AttachmentData",1}]//Dataset

Import["ExampleData/attach.mbox",{"AttachmentDecodedData",1}]//Dataset

Attachment Handling

Import["ExampleData/attach.mbox",{"AttachmentList",1},"AttachmentRules"-><|"audio/x-wav"->"Duration","GIF"->(ColorNegate@ImportByteArray[#]&)|>]

Utilities

RegisterFormat

https://reference.wolfram.com/language/tutorial/DevelopingAnImportConverter.html
https://reference.wolfram.com/language/tutorial/DevelopingAnExportConverter.html

Register format myFmt:

myFormatExport[filename_String,expr_, opts:OptionsPattern[]]:=Export[filename,"myFormat!\n"<>expr,"String"];

ImportExport`RegisterExport["myFmt",myFormatExport];

myFormatImport[filename_String, opts:OptionsPattern[]]:=StringDelete[Import[filename,"String"],"myFormat!\n"];

ImportExport`RegisterImport["myFmt",myFormatImport];

Before, the format had to be specified:

Export["test.myext","Hello, world!"]

Export["test.myext","Hello, world!","myFmt"]

FileFormat["test.myext"]

FilePrint["test.myext"]

Import["test.myext"]

Import["test.myext","myFmt"]

ImportString["myFormat!\nHello, world!","myFmt"]

Register an extension for myFmt:

https://resources.wolframcloud.com/FunctionRepository/resources/RegisterFormat

({{"[◼]", "RegisterFormat"}})["myFmt",<|"Extensions"->"myext"|>]

Import, Export, and FileFormat, now infer the extension from the file name:

Export["test.myext","Hello, world!"]

FileFormat["test.myext"]

FilePrint["test.myext"]

Import["test.myext"]

MIMETypes and Extensions

Get the canonical format name using the same methods as Import internally:

ImportExport`FormatName["jpg"]

ImportExport`FormatToMIMETypeList["jpg"]

ImportExport`FormatToMIMEType["jpg"]

ImportExport`FormatToExtensionList["jpg"]

ImportExport`FormatToExtension["jpg"]

All these functions also work for custom registered formats:

ImportExport`FormatToExtensionList["myFmt"]

LibraryLinkUtilities

4:30 PM - 5:00 PM, Wed, Oct 31, 2019
Little Chief

Video Import/Export

1:00 PM - 1:30 PM, Thu, Oct 31, 2019
Little Chief