Automotive Data Analysis

Andrew Steinacher
Lead Developer, Wolfram|Alpha

Abstract

In an effort to measure my fuel use on a per-trip scale, I connected a microcontroller to my car’s Controller Area Network (CAN) bus, and I found a fire hose of information from simply recording messages sent over the CAN bus. This fire hose of data has given me a large dataset of over 16 GB from thousands of car trips, including time series for engine temperatures, steering angles, fuel flow and more. Using the Wolfram Language, it was easy to build a user interface for interactively analyzing and processing this CAN data, as well as an  EntityStore to more programmatically analyze the data on longer timescales with EntityValue syntax. These tools have allowed me to analyze warmup times and fuel usage trends, as well as waiting times at intersections with some alignment to GPS location data.

Outline

Web Forms to Track Gas Mileage

CAN Bus Sniffing

CANGUI and Individual Trip Analysis

Trip EntityStore and Bulk Trip Analysis

Alignment to GPS Data and Analysis

Future Directions

Web Forms to Track Gas Mileage

Create a Tracker to Analyze Gas Mileage Using Wolfram Tech

GraphicsRow[
{
BarcodeImage["https://blog.wolfram.com/2017/06/22/create-a-tracker-to-analyze-gas-mileage-using-wolfram-tech/","QR",400],

},ImageSize->1000]

Web Forms to Track Gas Mileage

GraphicsGrid[List@{With[{computedMPG=TimeSeries[],actualMPG=TimeSeries[]},DateListPlot[{computedMPG,actualMPG},PlotTheme->"Detailed",(*PlotLegends->Placed[{"Computed","Actual"},Below],*)FrameLabel->{"","mi/gal"},Epilog->{{Blue,Thin,Dashed,Tooltip[Line[{{computedMPG["FirstDate"],QuantityMagnitude[#]},{computedMPG["LastDate"],QuantityMagnitude[#]}}],#]}&[Mean[computedMPG]],{Darker@Orange,Thin,Dashed,Tooltip[Line[{{computedMPG["FirstDate"],QuantityMagnitude[#]},{computedMPG["LastDate"],QuantityMagnitude[#]}}],#]}&[Mean[actualMPG]]},PlotLabel->Row[{"Average lifetime MPG: ",Round[Mean[actualMPG],0.1]}],PlotRange->{Automatic,{15,40}},ImageSize->Medium,Mesh->All,Filling->{1->{2}},LabelStyle->18]],
DateListPlot[TimeSeries[],PlotTheme->"Detailed",FrameLabel->{"","$/gal"},ImageSize->Medium,Mesh->All,LabelStyle->18,PlotLabel->"Price Per Gallon"]

},ImageSize->Full]

CAN Bus Sniffing: CAN Intro

Controller Area Network (Wikipedia)

Bus standard for allowing microcontrollers to communicate without a host

Released in 1986

From http://lumaleds.com/blog/canbus-explained.

CAN Bus Sniffing: Message Structure

About 4 bytes for ID + up to 8 bytes of data

Priority: Lower IDs are prioritized over higher IDs

Stability: Devices on the network acknowledge (ACK) when messages are received

From https://www.csselectronics.com/screen/page/simple-intro-to-can-bus/language/en.

CAN Bus Sniffing: Parts

Microcontroller: Arduino UNO R3 »

CAN Interface: Seeed CAN-BUS Shield »

RTC & Storage: Adafruit Data Logger Shield »

CAN Bus Sniffing: Other Notes

Cables

OBD-II ⇔ DB-9

12V -> USB adapter + cable for power

Speed

RTC square wave ⇔ interrupt pin (for counting seconds since start)

Safety

Disconnect termination resistor (already exists in car's CAN bus)

Enable silent mode (disable ACK)

CAN Bus Sniffing: Data Format

Plaintext: Slow, large (~40 bytes per message)

StringLength["12:31:25,00D0,FF,FF,FF,FF,FF,FF,FF,FF\n"]

Binary: Small, fast (14 bytes per message)

// Store CAN data in a struct for fast binary writing
volatile struct CANMessage {
  unsigned int t;
  unsigned long int id;
  unsigned char data[8];
} m;

CAN Bus Sniffing: Binary Format

binaryFileFormat=Flatten[
{
"UnsignedInteger16",(* Seconds since start *)
"UnsignedInteger32",(* CAN ID (decimal) *)
ConstantArray["UnsignedInteger8",{8}] (* Data *)
}
];

SetDirectory[NotebookDirectory[]];
data=BinaryReadList["19031409.45B",binaryFileFormat];
data//Length

CAN Bus Sniffing: Binary Format

Grid[data[[;;2000;;150]],]

CAN Bus Sniffing: Time Series

ID: 0x360, Byte #4

DateListPlot[idToByteTimeSeries[864][[4]]]

CANGUI »

Faster and easier to explore and plot data

Point-and-click interface (See Lou’s Manipulate talk Thursday @ 4pm in Badger!)

Message Space View

CANGUI[]

Individual Trip Analysis

Longer-term data is difficult to gather, analyze

Plotting multiple quantities at once is hard

GPS data?

Trip EntityStore

Trip files ⇒ Entities

Easy to filter with implicit EntityClass syntax, etc...

Load only core metadata (date/times, file reference)

Extract TimeSeries data based on CANGUI exploration

Easy to align to GPS data, etc...

Trip EntityStore: Demo

SetDirectory["C:\\Users\\andrews\\git\\CANGUI"];
Get["CANGUI/TripEntityStore.wl"];
store=CreateTripEntityStore["DataFiles","PlotChoices",Databin[]];
EntityUnregister/@store[];
EntityRegister[store]

Trip EntityStore: Demo

(Entity["Trip #2 on 2019-03-22", Entity["Trip","2019-03-22:2"], "Entity[\"Trip\", \"2019-03-22:2\"]", "Trip"])[{(EntityProperty["date", EntityProperty["Trip","Date"], "EntityProperty[\"Trip\", \"Date\"]"]),(EntityProperty["duration", EntityProperty["Trip","Duration"], "EntityProperty[\"Trip\", \"Duration\"]"]),(EntityProperty["start position", EntityProperty["Trip","StartPosition"], "EntityProperty[\"Trip\", \"StartPosition\"]"]),(EntityProperty["end position", EntityProperty["Trip","EndPosition"], "EntityProperty[\"Trip\", \"EndPosition\"]"]),(EntityProperty["start time", EntityProperty["Trip","StartTime"], "EntityProperty[\"Trip\", \"StartTime\"]"]),(EntityProperty["end time", EntityProperty["Trip","EndTime"], "EntityProperty[\"Trip\", \"EndTime\"]"]),(EntityProperty["file", EntityProperty["Trip","File"], "EntityProperty[\"Trip\", \"File\"]"]),(EntityProperty["label", EntityProperty["Trip","Label"], "EntityProperty[\"Trip\", \"Label\"]"]),(EntityProperty["trip number", EntityProperty["Trip","TripNumber"], "EntityProperty[\"Trip\", \"TripNumber\"]"])},"PropertyAssociation"]

Trip EntityStore: Demo

accAndBrake=(Entity["Trip #1 on 2019-03-21", Entity["Trip","2019-03-21:1"], "Entity[\"Trip\", \"2019-03-21:1\"]", "Trip"])[,"PropertyAssociation"]

DateListPlot[accAndBrake,PlotRange->Full,Filling->Axis]

Trip Analysis: ∫FuelFlow(t)dt

fuelFlow=(Entity["Trip #2 on 2019-03-14", Entity["Trip","2019-03-14:2"], "Entity[\"Trip\", \"2019-03-14:2\"]", "Trip"])[(EntityProperty["Fuel Flow time series", EntityProperty["Trip","FuelFlowTimeSeries"], "EntityProperty[\"Trip\", \"FuelFlowTimeSeries\"]"])]

timeSeriesTotalIntegral[fuelFlow]

Trip Analysis: ∫FuelFlow(t)dt

EntityValue[
EntityClass["Trip","Date"->Between[{(DateObject["Day: ""Fri 14 Jun 2019", DateObject[{2019,6,14},"Day","Gregorian",-4.`]]),(DateObject["Day: ""Thu 20 Jun 2019", DateObject[{2019,6,20},"Day","Gregorian",-4.`]])}]],
(EntityProperty["Fuel Flow time series", EntityProperty["Trip","FuelFlowTimeSeries"], "EntityProperty[\"Trip\", \"FuelFlowTimeSeries\"]"]),
Map[timeSeriesTotalIntegral]/*Total]

KeyTake[
Select[mpgTrackerData,DayRound[#Timestamp]===(DateObject["Day: ""Thu 20 Jun 2019", DateObject[{2019,6,20},"Day","Gregorian",-4.`]])&],
{"FuelUsed","PreviousTime"}
]

Trip Analysis: ∫FuelFlow(t)dt

Show[comparisonPlot,Plot[Legended[fit,Style[fit,18]],{x,4,14},PlotStyle->Opacity[0.6]]]

Trip Analysis: Fuel Economy

BoxWhiskerChart[
KeyMap[DateValue[DateObject[{2018,#}],"MonthNameShort"]&,monthToMPGDistribution],
ImageSize->Large,PlotTheme->"Detailed",ChartLabels->Automatic,FrameLabel->{None,Quantity[None,"Miles"/"Gallons"]},LabelStyle->18,PlotLabel->"Fuel Economy by Month"
]

GPS Data: Logging with MacroDroid »

Log phone location, speed, accuracy, time every 10 seconds when on Bluetooth

GPS Data: Example Trip

GPSPlot[gpsData,ImageSize->500]

GPS Data: Example Trip

Accuracy is not quite good enough to do lane analysis

GPSPlot[gpsData,ImageSize->800,GeoCenter->GeoPosition[{40.106140,-88.254242}],GeoRange->Quantity[0.75,"Miles"]]

GPS Data: Overview

~70,000 points from Nov 1, 2018 - Oct 1, 2019

Mostly in Champaign, IL, Fort Wayne, IN, and Indianapolis, IN

Manipulate[
DateHistogram[
dates,
PlotTheme->"Detailed",
PlotRange->All,
DateReduction->reduction,
ImageSize->800,
LabelStyle->24
],
{reduction,{None,"Year","Week","Day"}},
LabelStyle->24,
SaveDefinitions->True
]

GPS Data: All

~70,000 points

GPS Data: All (<15 MPH)

~4,500 points

GPS Data: Champaign, IL

~12,000 points in 1/8 × 1/8 mile squares

GPS Data: Champaign, IL (<15 MPH)

~1,100 points in 1/8 × 1/8 mile squares

GPS Data: Fort Wayne, IN

~17,000 points in 1/3 mile hexagons

GPS Data: Fort Wayne, IN (<15 MPH)

~1,300 points in 1/3 mile hexagons

GPS Data: Indianapolis, IN

Left: ~10,000 points in 1/3 mile hexagons; Right: ~800 points in 1/3 mile hexagons (<15 MPH)

GPS Data: Wolfram Parking Lot

parkingSpotsWRI=GeoPosition[];

GeoGraphics[
{Green,PointSize[Medium],Point[GeoPosition[#]]&/@parkingSpotsWRI["Data"]},
GeoBackground->"Satellite",
GeoServer->"DigitalGlobe",
GeoRange->Quantity[450,"Feet"],
GeoCenter->GeoPosition[{40.098020823899546`,-88.24674409250552},"ITRF00"],
ImageSize->Large
]

GPS Data: Wolfram Parking Lot

Future Directions

Driver Detection

Automatic Data Upload

Some aggregated data -> ESP8266/Hotspot -> APIFunction/Databin

More CAN Exploration/Analysis

Livestream data, dynamic feedback

Dead Reckoning/Odometry/Path Analysis

Summary

Web Forms to Track Gas Mileage

CAN Bus Sniffing

CANGUI and Individual Trip Analysis

Trip EntityStore and Bulk Trip Analysis

Alignment to GPS Data and Analysis

Future Directions