Time Series Processing

Gosia Konwerska

Wolfram Research
Champaign, IL - October 20, 2016

Join the Conversation #WolframTechConf

The Plan

0. Introduction

1. Direct Arithmetic and Function Application

2. Processing Functions

3. MovingMap Redesigned

4. TimeSeries Everywhere

Introduction (0/4)

Types of serial data in WL

TimeSeries

EventSeries

TemporalData

Introduction (1/4)

TimeSeries

• TimeSeries assumes continuous time observations and allows only one path (could be univariate or multivariate of constant dimension).
• By default the values between the time stamps are being interpolated. 
• The method of interpolation can be set. 
• Default interpolation method is linear interpolation.

(*real values, integer time stamps*)
tsReal=TimeSeries[RandomReal[1,6],{0}]

tsReal["Path"]

tsReal[1/2]

(*integer values, integer time stamps*)
tsInteger=TimeSeries[RandomInteger[10,6],{0}]

tsInteger["Path"]

tsInteger[1/2]

(*integer values, real time stamps*)
tsInteger2=TimeSeries[RandomInteger[10,6],{0.}]

tsInteger2["Path"]

tsInteger2[1/2]

ts3=TimeSeries[tsInteger2,ResamplingMethod->{"Interpolation",InterpolationOrder->0}]

ts3["Path"]==tsInteger2["Path"]

ts3[1/2]

Clear[a];
ts4=TimeSeries[tsReal,ResamplingMethod->{"Constant",a}]

ts4["Path"]==tsReal["Path"]

ts4[1/2]

Introduction (2/4)

EventSeries

EventSeries assumes discrete time events, hence nothing happens in between time stamps. It cannot be reset.

es=EventSeries[{a,b,c},{0}]

es["Path"]

es[1/2]

Introduction (3/4)

TemporalData

• TemporalData assumes continuous time observations and allows more than one path (dimensionality for all paths must be consistent --> slice distribution).
• By default the values between time stamps are being interpolated. 
• The method of interpolation can be set. 
• Default interpolation method is to hold value from the left.

td=TemporalData[{{1,2,3,4},{a,b,c}},{0}]

td["Paths"]

Create TemporalData from existing TimeSeries:

td2=TemporalData[{tsReal,tsInteger,tsInteger2}]

td2["Paths"]

TemporalData returns DataDistribution, not observations, for the time slice.

td[1/2]

Application: simulate a random process and analyze a slice distribution.

sim=RandomFunction[WienerProcess[1,.1],{0,1,.01},10^3]

sim[.3]

Plot[CDF[sim[.3],x],{x,0,.6}]

Sample from data distribution and compare the sample density histogram to the PDF of the theoretical slice distribution.

sample=RandomVariate[sim[.3],10^6];

sliceDist=WienerProcess[1,.1][.3]

Show[Histogram[sample,Automatic,"PDF"],Plot[PDF[sliceDist,x],{x,0,.6}]]

Introduction (4/4)

Options and Properties

The list of options for TimeSeries.

Options[TimeSeries]//Column

To list properties one needs first to define a valid TimeSeries object.

ts=TimeSeries[RandomReal[1,6],{0}];

ts["Properties"]

ts["Properties"]//Length

Options for EventSeries are similar to those of TimeSeries.

Options[EventSeries]//Column

ResamplingMethod option is not available for EventSeries.

Complement[Options[TimeSeries],Options[EventSeries]]

EventSeries properties.

es=EventSeries[RandomReal[1,6],{0}];

es["Properties"]

es["Properties"]//Length

Since there is no ResamplingMethod defined for EventSeries, the “PathFunction” is not available for EventSeries.

Complement[ts["Properties"],es["Properties"]]

The options of TemporalData are identical to the options of TimeSeries.

Options[TemporalData]==Options[TimeSeries]

The properties of TemporalData are more numerous because of the presence of multipath properties.

td=TemporalData[RandomReal[1,6],{0}];

td["Properties"]

td["Properties"]//Length

Direct Arithmetic and Function Application (1/3)

Basic examples

(*WeatherData["KCMI","Temperature",{{2016,9,1},{2016,10,1}}*)

ts1=TimeSeries[];

DateListPlot[ts1]

Find maximal temperature for this period.

Max[ts1]

*Fix* the missing values by replacing them with interpolated values.

ts2=TimeSeries[ts1,MissingDataMethod->{"Interpolation",InterpolationOrder->1}]

The error message indicates that first or last value of the original time series is missing.

{ts1["FirstValue"],ts1["LastValue"]}

Max[ts2]

Add units to values with QuantityArray.

ts3=TimeSeries[QuantityArray[ts2["Values"],"DegreesCelcius"],{ts2["Dates"]}]

Max[ts3]

Convert to Fahrenheit by directly apply UnitConvert to the TimeSeries.

ts4=UnitConvert[ts3,"DegreesFahrenheit"]

DateListPlot[ts4,FrameLabel->Automatic]

Maximal temperature in degrees Fahrenheit.

Max[ts4]

Direct Arithmetic and Function Application (2/3)

Currency conversions

Financial time series in $:

(*TimeSeries@MapAt[Quantity[#,"$"]&,FinancialData["SBUX",{{2016,1,1},{2016,10,1}}],{All,2}]*)

ts=TimeSeries[];

DateListPlot[ts,FrameLabel->Automatic]

Directly apply conversion with the current exchange rate:

ts1=UnitConvert[ts,"Euro"]

Historical rates for similar period of time:

(*1/Quantity[1,"$"]UnitConvert[TimeSeriesWindow[TimeSeries[First@WolframAlpha["USD to EUR",{{"History",1},"ComputableData"}]],{{2016,1,1},{2016,10,1}}],"Euro"]*)

rates=TimeSeries[];

rates["FirstValue"]

Directly multiply by the time series of rates:

ts2=rates*ts

DateListPlot[{ts1,ts2},FrameLabel->Automatic,PlotLegends->{"with current rate","with historial rates"}]

Direct Arithmetic and Function Application (3/3)

Addition

This simple example is to illustrate how two misaligned time series are being combined.

ts1=TimeSeries[{10,5,10},{{0,2,3.5}}];

ts2=TimeSeries[{1,2,3,4},{1}];

NumberLinePlot[{ts1["Times"],ts2["Times"]},PlotStyle->PointSize[.03]]

sum=ts1+ts2

Show[ListPlot[{ts1,ts2,sum}],ListLinePlot[{ts1,ts2,sum},PlotLegends->{"ts1","ts2","sum"}]]

Intersection of the durations.

common=Interval[{Max[ts1["FirstTime"],ts2["FirstTime"]],Min[ts1["LastTime"],ts2["LastTime"]]}]

Union of the time stamps in the common duration.

Union[Flatten[Select[#,IntervalMemberQ[common,#]&]&/@{ts1["Times"],ts2["Times"]}]]

sum["Times"]

Processing Functions (1/2)

TimeSeriesWindow

Speeds of sustained wind and wind gusts recorded on NOAA buoy 42058 on the Caribbean Sea around the time of hurricane Matthew.

GeoGraphics[GeoMarker[GeoPosition@{14.923,-74.918}],GeoRange->{{12,25},{-85,-65}},GeoProjection->"Mercator"]

ts45=TimeSeries[];

DateListPlot[ts45,FrameLabel->Automatic,PlotLegends->{"sustained wind","gusts"}]

Use TimeSeriesWindow to zoom in to the interesting time interval.

ts=TimeSeriesWindow[ts45,{{2016,9,29},Automatic}]

DateListPlot[ts,FrameLabel->Automatic,PlotLegends->{"sustained wind","gusts"}]

Convert the time series to new units:

ts2=UnitConvert[ts,"mph"]

DateListPlot[ts2,FrameLabel->Automatic,PlotLegends->{"sustained wind","gusts"}]

Change units in visualization only:

DateListPlot[ts,FrameLabel->Automatic,PlotLegends->{"sustained wind","gusts"},TargetUnits->"mph"]

Processing Functions (2/2)

TimeSeriesMap

NOTE: direct time series arithmetic does not make TimeSeriesMap obsolete.

ts=TimeSeries[RandomReal[1,{5,3}],{0}]

ts//Normal

Max of time series returns the maximum value across all components.

Max[ts]

Find maximum of each vector value.

TimeSeriesMap[Max,ts]

%//Normal

TimeSeriesThread

TimeSeriesMapThread

TimeSeriesResample

TimeSeriesRescale

TimeSeriesShift

MovingMap Redesigned

Smoothing and Thinning with MovingMap

Hurricane Matthew winds and gusts time series.

ts=TimeSeries[];

ts["ValueDimensions"]

Extract the time series of sustained winds.

wind=ts["PathComponent",1]

DateListPlot[wind,FrameLabel->Automatic]

Compute the 12-hour moving mean.

smooth=MovingMap[Mean,wind,{{12,"Hour"}}] (*values placed on the right of each window*)

The time series has been smoothed.

DateListPlot[{wind,smooth},PlotRange->All,FrameLabel->Automatic,PlotLegends->{"data","smoothing"},ImageSize->Medium]

Compute the 12-hour moving mean in increments of 6 hours.

thin=MovingMap[Mean,wind,{{12,"Hour"},Right,{6,"Hour"}}]

This smooths and thins the time series.

{wind["PathLength"],smooth["PathLength"],thin["PathLength"]}

Compare the results.

DateListPlot[{wind,smooth,thin},PlotLegends->{"data","smoothing","thinning"},ImageSize->Medium]

Compute the 12-hour moving mean in increments of 6 hours, placing the windows starting on midnight.

positionspec={DateObject@{2016,9,16,0,0,0},Automatic,Quantity[6,"Hour"]};

mmRight=MovingMap[Mean,wind,{Quantity[12,"Hours"],Right,positionspec}]
(*values placed on the right of each window*)

mmLeft=MovingMap[Mean,wind,{Quantity[12,"Hours"],Left,positionspec}]
(*values placed on the left of each window*)

Compare the starting time stamps.

{wind["FirstDate"],thin["FirstDate"],mmRight["FirstDate"],mmLeft["FirstDate"]}

Plot the time stamps for three time series.

legend=PointLegend[{"thinning","thinning with fixed windows (Right)","thinning with fixed windows (Left)"},LegendLayout->"ReversedColumn"];

TimelinePlot[{thin["Dates"],mmRight["Dates"],mmLeft["Dates"]},PlotLegends->legend]

TimeSeries Everywhere (1/3)

Databin

Define a Databin.

bin=Databin[];

Values[bin]

Send counts to the Databin.

n=20;
Do[DatabinAdd[bin,<|"observations"->i|>];,{i,1,n}]

Retrieve all time series from the Databin.

TimeSeries[bin]

Retrieve specific time series from the Databin.

ts=TimeSeries[bin]["observations"]

Plot the observations.

DateListStepPlot[ts,Joined->False]

Plot the time stamps.

TimelinePlot[ts["Dates"]]

Clean the Databin.

DatabinRemove[bin,{1,n}]

TimeSeries Everywhere (2/3)

Audio

AudioLocalMeasurements return TimeSeries.

wind=ExampleData[{"Audio","Wind"}]

The loudness of the example audio.

loudness=AudioLocalMeasurements[wind,"Loudness"]

ListLinePlot[loudness,PlotRange->All]

Compute descriptive statistics of the loudness time series.

summary[ts_]:={Min[ts],Mean[ts],Median[ts],Max[ts]};

TableForm[{summary[loudness]},
TableHeadings->{{"loudness"},{"Min","Mean","Median","Max"}}]

Generate audio with TimeSeries.

Control the frequency of the sine wave with a regularly sampled TimeSeries.

n=50;
frequencies=RandomReal[{100,300},n];
reg=TimeSeries[frequencies,{0,Automatic,.3}];

AudioGenerator[{"Sin", reg}]

Control the frequency of the sine wave with an irregularly sampled TimeSeries.

times=Accumulate[RandomReal[{0.1,.5},n]];
irreg=TimeSeries[frequencies,{times}];

AudioGenerator[{"Sin",irreg}]

TimeSeries Everywhere (3/3)

Wolfram Knowledgebase

Sun

distance=Entity["Star", "Sun"][ "StarDistanceTimeSeries"]

distance[Now]

UnitConvert[distance[Now],"au"]

Set TargetUnits to astronomical unit, which should correspond to mean distance between the Sun and the Earth.

DateListPlot[distance,GridLines->{{{Now,Directive[Thick,Red]}},None},TargetUnits->"au"]

The calculated mean and standard deviation.

UnitConvert[Mean[distance],"au"]

UnitConvert[StandardDeviation[distance],"au"]

The force of attraction that the Sun exerts on the Earth.

force=(Quantity[1, "G", Newtonian gravitational constants, "GravitationalConstant"])*(Entity["Sun", Entity["Star","Sun"], "Entity[\"Star\", \"Sun\"]", "star"])["Mass"]*(Entity["Earth", Entity["Planet","Earth"], "Entity[\"Planet\", \"Earth\"]", "planet"])["Mass"]/(distance^2)

The mean force.

UnitConvert[Mean[force],"SI"]

Population

Iceland=Entity["Country","Iceland"][Dated["Population",All]]

Iceland["FirstValue"]

DateListPlot[Iceland,FrameLabel->Automatic]

Fit a linear model to the log of the population values.

vals=QuantityMagnitude/@Iceland["Values"];

sol=LinearModelFit[Log[vals],x,x]

tab=Exp[Table[sol[x],{x,1,Iceland["PathLength"]}]];

Fit sigmoid function to the population values.

sigmoid[x_]:=c/(b+Exp[- a*x])

sol2=FindFit[vals,sigmoid[x],{a,b,c},x]

tab2=Table[sigmoid[x]/.sol2,{x,1,Iceland["PathLength"]}];

DateListPlot[{Iceland,Transpose[{Iceland["Dates"],tab}],Transpose[{Iceland["Dates"],tab2}]},FrameLabel->Automatic,PlotLegends->{"population","exponential model", "sigmoid model"}]

Thank You!

Questions?

gosiak@wolfram.com

Related Talks

Date Visualization

Nirmal Malapaka, Wednesday Oct 19, 5:30pm, Spartan/Golden Gopher

Quantities in Probability and Statistics

Gosia Konwerska, Wednesday Oct 19, 2:00pm, Badger

What’s New in Probability and Statistics

Hsien-Ching Kao, Thursday Oct 20, 8:30am, Spartan/Golden Gopher