Working with Time Series

Gosia Konwerska

Wolfram Research, Inc.


Champaign, 19 October 2018

Join the Conversation #WolframTechConf

ABSTRACT

We explore Wolfram Language functionality for processing temporally ordered data.

Temporally Ordered Data Structures in WL

headings={"Caller","Head","How many paths","at a time stamp returns","interpolate between time stamps","in between time stamps returns"};
tablecont={Style[#,14,Bold,Darker@Red]&/@{TimeSeries,EventSeries,TemporalData},
{TemporalData,TemporalData,TemporalData},
{"single path only","single path only","multiple paths"},{"value","value",DataDistribution},
{"yes", "no","yes"},{"interpolated value",Missing[],"previous value"}};
TableForm[tablecont,TableSpacing->3,TableHeadings->{headings}]

Examples (TimeSeries)

TimeSeries

Useful for continuous time and continuous values, like physical quantities (air temperature, pressure,...).

Daylight

There is a property of the Sun which gives the length of the day for Today:

(Entity["Sun", Entity["Star","Sun"], "Entity[\"Star\", \"Sun\"]", "star"])[(EntityProperty["daily time above horizon", EntityProperty["Star","DailyTimeAboveHorizon"], "EntityProperty[\"Star\", \"DailyTimeAboveHorizon\"]"])]

But the Sun can be Dated, which means we can extract properties on a given day:

Dated[(Entity["Sun", Entity["Star","Sun"], "Entity[\"Star\", \"Sun\"]", "star"]),Yesterday][(EntityProperty["daily time above horizon", EntityProperty["Star","DailyTimeAboveHorizon"], "EntityProperty[\"Star\", \"DailyTimeAboveHorizon\"]"])]

So for the whole year 2018:

year2018=DateRange[DateObject[{2018,1,1}],DateObject[{2018,12,31}],"Day"];

(*ts1 = TimeSeries[Map[Dated[(Entity["Sun", Entity["Star","Sun"], "Entity[\"Star\", \"Sun\"]", "star"]),#][(EntityProperty["daily time above horizon", EntityProperty["Star","DailyTimeAboveHorizon"], "EntityProperty[\"Star\", \"DailyTimeAboveHorizon\"]"])]&,year2018]]*)

ts1=TimeSeries[];

ts1["FirstValue"]

ts1["FirstDate"]

ts1["DatesQ"]

So the time stamps are just integers:

ts1["UnexpandedRawTimes"]

We can change the time stamps to corresponding dates by using TimeSeriesRescale:

ts2=TimeSeriesRescale[ts1,{DateObject[{2018,1,1}],DateObject[{2018,12,31}],"Day"}]

ts2["DatesQ"]

ts2["FirstDate"]

The values are given in mixed units - this can be change to having hours only with UnitConvert:

ts3=UnitConvert[ts2,"Hours"]

ts3["FirstValue"]

Compute the mean:

mean=Mean[ts3]

Plot:

DateListPlot[ts3,Filling->QuantityMagnitude@mean,FrameLabel->Automatic]//Quiet

Stocks - Financial Data

The closing price of a stock (and in fact the price of a stock) does not change between closing and opening of the given stock exchange, that is it does not change between business days. So the natural choice to store the stock prices is TemporalData, but TimeSeries may be more convenient and the resampling can be set to hold value from the left.

ts=TimeSeries[];

DateListPlot[ts]//Quiet

ts["DatePath"][[;;5]]

ts["ResamplingType"]

ts[DateObject[{2008,1,5}]](*interpolated*)

closing=TimeSeries[ts,ResamplingMethod->{"Interpolation","HoldFrom"->Left}]

closing[DateObject[{2008,1,5}]](*same as on the last business day before*)

Another issue is the definition of many business time series - they are not regularly sampled on DAYS but they are regularly sampled on BUSINESS DAYS. Hence we have :

RegularlySampledQ[closing]

closing["MinimumTimeIncrement"]

This can be “fixed” by redefining or resampling the time series (note definition of HolidayCalendar):

closing1=TimeSeries[ts["Values"],{ts["FirstDate"],ts["LastDate"],"BusinessDay"},HolidayCalendar->{"UnitedStates","NYSE"},ResamplingMethod->{"Interpolation",InterpolationOrder->0}]

closing2=TimeSeriesResample[closing,"BusinessDay",HolidayCalendar->{"UnitedStates","NYSE"}]

{closing1["MinimumTimeIncrement"],closing2["MinimumTimeIncrement"]}

{RegularlySampledQ[closing1],RegularlySampledQ[closing2]}

The paths are the same:

ts["Path"]==closing["Path"]==closing1["Path"]==closing2["Path"]

Now we can trust the modeling algorithms:

model1=TimeSeriesModelFit[closing1]

model1//Normal

Compare to working with the original data:

model2=TimeSeriesModelFit[ts]

model2//Normal

model2["TemporalData"]

model2["TemporalData"]["DatePath"][[;;5]](*linear resampling done before estimation*)

Compare forecasts:

forecast1=TimeSeriesForecast[model1,{50}];
forecast2=TimeSeriesForecast[model2,{50}];

dataSelection=TimeSeriesWindow[closing1,{DateObject[{2018,3,1}],Automatic}];

DateListPlot[{dataSelection,forecast1,forecast2},Joined->False,PlotLegends->{"data","forecast for 50 business day","forecast for 50 days"}]//Quiet

Examples (EventSeries)

EventSeries

Useful for discrete values and/or discrete time events, like counts of insurance, birthdays, deaths...

US Presidents

The event series of birthdays of American presidents:

es=EventSeries[];(* Grover Cleveland *)

Plot the dates using TimelinePlot:

TimelinePlot[es["Dates"]]

TimelinePlot[AssociationThread[es["Values"]->es["Dates"]]]

Birthday distribution within a year:

DateHistogram[es,DateReduction->"Year"]

Thanksgivings

When did the first Thanksgiving happen?

(Entity["English colonists celebrate the first Thanksgiving in Americas", Entity["HistoricalEvent","FirstThanksgiving"], "Entity[\"HistoricalEvent\", \"FirstThanksgiving\"]", "historical event"])["Date"]

firstyear=DateValue[(Entity["English colonists celebrate the first Thanksgiving in Americas", Entity["HistoricalEvent","FirstThanksgiving"], "Entity[\"HistoricalEvent\", \"FirstThanksgiving\"]", "historical event"])["Date"],"Year"]

(* es=EventSeries[Range[firstyear,2018,1],{Table[WolframAlpha["thanksgiving date "~~ToString[y],{{"Result",1},"ComputableData"}],{y,firstyear,2018}]}] *)

es=EventSeries[];

es["LastValue"]

TimelinePlot[es]

Weekday distribution histogram:

DateHistogram[es,DateReduction->"Week"]

And since Thanksgiving without a pie is not complete... - application of TimeSeriesMapThread to map a function over time stamps:

weekday=TimeSeriesMapThread[DateValue[#,"DayName"]&,es]

weekday["FirstValue"]

freq=Tally[weekday["Values"]]

PieChart[freq[[All,2]],ChartLabels->(Style[#,White,Bold]&/@freq[[All,1]]),ChartStyle->"DarkRainbow"]

Now plot the weekdays on the time scale:

res1=GatherBy[weekday["DatePath"],Last][[All,All,1]];

res2=SortBy[res1,DateValue[#[[1]],"ISOWeekDay"]&];

legend=DateValue[#,"DayName"]&/@res2[[All,1]];

TimelinePlot[res2,PlotLegends->PointLegend[legend,LegendLayout->"Reversed"]]

So now how many days between Thanksgivings?

diff=MovingMap[DateDifference@@(#Dates)&,es,Quantity[2,"Events"]]

Histogram[diff]

How long till Thanksgiving this year?

es["LastDate"]-Today

Examples (TemporalData)

TemporalData

TemporalData was designed to store simulation paths of random processes:

proc=ARMAProcess[{.3},{.4},1];
sample1=RandomFunction[proc,{0,123}](* single path *)

sample2=RandomFunction[proc,{0,123},5](* five paths *)

With more paths one can study slice distributions:

sample3=RandomFunction[proc,{0,123},10^3]

sliceDist=sample3[10]

Mean[sliceDist]

DiscretePlot[CDF[sliceDist,x],{x,-4.3,4.3,.1}]

Get slice data:

sliceData=sample3["SliceData",10];

sliceData//Head

sliceData//Length (* equal to number of paths *)

Histogram[sliceData,30,PDF]

Notes on Resampling

Resampling method determines the values between the time stamps.

Many models assume regularly sampled time series so the difference between consecutive time stamps is constant. Let’s look at some examples:

ts=TimeSeries[{a,b,c,d},{{1,3,4,4.5}}]

Check if regularly sample with WL function:

RegularlySampledQ[ts]

The information about time stamps can also be accessed through some properties - this is the full list:

ts["DeveloperProperties"]

ts["UniformlySpacedQ"]

ts["UnexpandedRawTimes"](*note the list levels and precision*)

Compare to:

regts=TimeSeries[{a,b,c,d},{0}]

{regts["UnexpandedRawTimes"],regts["Times"]}

Compare default resampling methods:

ts=TimeSeries[{a,b,c,d},{{1,3,4,5}}];

es=EventSeries[{a,b,c,d},{{1,3,4,5}}];

td=TemporalData[{a,b,c,d},{{1,3,4,5}}];

Time series:

ts1=TimeSeriesResample[ts];(*resamples according to the smallest time increment*)

ts1//Normal//Simplify (*linear interpolation*)

Event series:

es1=TimeSeriesResample[es];

es1//Normal(*fills with Missing[]*)

Temporal data:

td1=TimeSeriesResample[td];

td1//Normal(*holds values from the left*)

The consequences - different behaviour in Accumulate, Differences, filters...

{Accumulate[ts]["Values"],Accumulate[ts["Values"]]}

{Accumulate[es]["Values"],Accumulate[es["Values"]]}(*NO RESAMPLING*)

{Accumulate[td]["Values"],Accumulate[td["Values"]]}

{Differences[ts]["Values"],Differences[ts["Values"]]}//Simplify

{Differences[es]["Values"],Differences[es["Values"]]}(*NO RESAMPLING*)

{Differences[td]["Values"],Differences[td["Values"]]}//Simplify

{MeanFilter[ts,2]["Values"],MeanFilter[ts["Values"],2]}//Simplify

{MeanFilter[es,2]["Values"],MeanFilter[es["Values"],2]}//Simplify

{MeanFilter[td,2]["Values"],MeanFilter[td["Values"],2]}//Simplify

Notes on MissingDataMethod

Example of real life data - temperatures:

(* data=WeatherData[Here,"Temperature",{{2018,1,1},{2018,10,1}}] *)
data=TimeSeries[];

How to know there is Missing[] in the values?

1) Plot has holes:

DateListPlot[data]//Quiet

2) compute Mean:

Mean[data]

What can be done? Few ideas (not necessarily making sense):

Replace the _Missing with a constant or a symbol:

TimeSeries[data,MissingDataMethod->π]

Mean[%]

Let Mathematica take care of it:

TimeSeries[data,MissingDataMethod->Automatic]

Mean[%]

The automatic method takes the hint from “ResamplingType”:

data["ResamplingType"]

Input directly and compare the results (should be the same as above):

TimeSeries[data,MissingDataMethod->{"Interpolation",InterpolationOrder->1}]

Mean[%]

Use different interpolation order:

TimeSeries[data,MissingDataMethod->{"Interpolation",InterpolationOrder->2}]

Mean[%]

Replace Missing with the mean of the other values (an application of MovingMap):

f[vals_,tms_]:=With[{mean=Mean[DeleteMissing[vals]]},tms->(vals/._Missing->mean)]

MovingMap[f[#1,#2]&,data,Quantity[data["PathLength"],"Events"]]

Mean[%]

Delete missing values:

dm[vals_,tms_]:=With[{ord=Position[vals,_Missing]},Delete[tms,ord]->Delete[vals,ord]]

MovingMap[dm[#1,#2]&,data,Quantity[data["PathLength"],"Events"]]

Mean[%]

Thank You!

Questions?

gosiak@wolfram.com