A Computational Approach to Daily Life
Utilizing Wolfram Tools Outside of Work

Ben Kickert
SDGCounting / StartingUpGood

(RowDefault[, (Spacer1), , (Spacer1), ])  #WolframTechConf

ABSTRACT

Mathematica and the Wolfram Stack are well known tools within professional settings from academia to the private sector.  Career scientists, mathematicians, and data analysts are accustomed to coding in the Wolfram Language and using the related cloud capabilities in their regular workflows; but how can these tools be leveraged to make life outside the office easier and more efficient?  This presentation will explore ways our team has taken concepts and approaches from our collaborative work and applied them to find solutions needed in daily life.

Practical examples include:

Examining how machine learning code not only helps predict life expectancy in developing countries, but also makes choosing products easier.

Utilizing scheduled tasks and databins to identify newly published reports while also helping manage finances.

Publishing online forms that provide intuitive access to complex assessments and also serving as a simple tool to make daily data updates.

Importing web-based data for cross source analysis while using the same tools to simplify product research.

Tapping into APIs and ServiceConnects for work related functions while also simplifying your own social media presence.

Applying data classification and summarization tools to both work related subjects and personal interests.

By starting with work related examples, our team has developed approaches that have applications within daily life that we utilize regularly and look forward to sharing.

Presentation Overview

About the SDGCounting / StartingUpGood team and our key projects

What is “Computational Living” and what overlaps does it create

Technical overlaps between work skills and personal interests

Exploring personal preferences using data from Untappd, a beer rating app, for visualization and web-based AI tools

Using Wolfram Cloud functionality to track allowances and spending

Utilizing WebSession to collect product data

About the SDGCounting / StartingUpGood Team

SDGCounting and StartingUpGood are two initiatives supported by a small team of data and development professionals.  Both seek to provide positive influence through data analytics, pro-bono partnerships, and advocacy.

SDGCounting focuses on tracking the Sustainable Development Goals with a particular focus on how impact is being tracked and data shared.

StartingUpGood advocates for startups to include social and environmental impact objectives in their business plans and provides support to young businesses as they grow and develop.

The methodologies and tools on display in this presentation were originally developed to support these initiatives and have been adapted for more personal / hobby use.

What is Computational Living?

Computational living is approaching daily questions and problems with the expectation that more informed decisions are possible.

It is a symbiotic relationship between personal and professional life that provides novel insights for both settings.

It is on display in many of the presentations at WTC.

Machine Learning for Development Data and Product Preferences

In previous Wolfram Technology Conferences, our team has demonstrated ways to work with international development data to provide insights and tools for understanding progress.  This has included sessions on public data analytics, combining diverse data sources for machine learning, and working with unstructured data.

The methodologies outlined have allowed our team to better understand how the world is progressing and draw conclusions about current and future trends.  But, those same tools can also help us understand much more nuanced topics like...

...Beer!

Visualizing Beer Preferences and using Machine Learning to help choose a pour

Using Untappd to record data points and create a web deployed predictor

Bringing in the data (originally a CSV file) and create a Dataset

rawdata=CloudGet["https://www.wolframcloud.com/objects/ben.kickert/rawbeer"]

dataset=Dataset[AssociationThread[First[rawdata]->#]&/@Rest[rawdata]]

In order to better track preferences, we need to reconfigure the styles data by splitting styles and sub-styles and creating new columns of data

splittypes=StringTrim/@StringSplit[dataset[[All,"beer_type"]]//Normal,"-"];
maintype=splittypes[[All,1]];
subtype=If[Length[#]==1,"",#[[2]]]&/@splittypes;
mainassoc=AssociationThread["main_type"->#]&/@maintype;
subassoc=AssociationThread["sub_type"->#]&/@subtype;

Since we have lat and long data for locations checked in from, we add those as GeoPositions

venuegeoposition=Quiet[GeoPosition[#]&/@Normal[Values[dataset[[All,{"venue_lat","venue_lng"}]]]]/.GeoPosition[{"",""}]->""];
venuegeo=AssociationThread["venue_geo"->#]&/@venuegeoposition;

Add all that information back into a final dataset and replace blanks with Missing

finaldataset=Dataset[Join[Join[Join[dataset,mainassoc,2],subassoc,2],venuegeo,2]]/.""->Missing["NotAvailable"];

Since we have some geodata, let’s look at where I have checked in beers:

GeoGraphics[GeoMarker[DeleteMissing[finaldataset[[All,"venue_geo"]]//Normal]],GeoRange->"World",ImageSize->Large]

To do the same with brewery locations, it is going to take a bit more work since I only have the city names and not the exact lat/long.  Using the Interpreter[“City”] function, we can find the appropriate Entity and thus location.  In this example, let’s use GeoBubbleChart to better visualize distribution.

locations=DeleteDuplicates[Values[Normal[finaldataset[[All,{"brewery_city","brewery_state","brewery_country"}]]]]];
cityentities=Interpreter["City"][#]&/@locations[[All,1]];
replacelist=Thread[locations->cityentities];
longcityentitylist=Values[Normal[finaldataset[[All,{"brewery_city","brewery_state","brewery_country"}]]]]/.replacelist;
entitycitycounts=Counts[Select[DeleteMissing[longcityentitylist],FailureQ[#]==False&]];
GeoBubbleChart[entitycitycounts,GeoProjection->"Bonne",ImageSize->Large,BubbleSizes->{0.015, 0.12}]

Now let’s start analyzing the data.  Whether dealing with personal or professional projects, these tools can help provide a “sanity check” for evaluating outputs before final processes are established.

Two of the most quantifiable aspects of beer are its alcohol content (ABV) and bitterness (IBU).  I know my general preferences, but are they reflected in my scores?

ListPlot[finaldataset[[All,{"beer_abv","rating_score"}]],ImageSize->Large,PlotLabel->Style["ABV Rating Trends",{Bold,Large}],AxesLabel->{"ABV","Rating"}]

ListPlot[finaldataset[[All,{"beer_ibu","rating_score"}]],ImageSize->Large,PlotLabel->Style["IBU Rating Trends",{Bold,Large}],AxesLabel->{"IBUs","Rating"}]

Additional analysis

Prevalence and preference of various styles

Most common brewery and average scores for each

Quantifying geographic distribution

Timeline trends

Common characteristics of top rated beers

Word trends in tasting notes

Machine Learning

Now let’s step it up a bit and look at some machine learning.  I start by pulling the most essential fields and threading them to their rating to create data to train and test on

aidata=DeleteMissing[Thread[Normal[finaldataset[[All,{"brewery_name","beer_abv","beer_ibu","main_type"}]]]->Normal[finaldataset[[All,"rating_score"]]]]];

trainingdata=RandomSample[aidata,350];

testdata=Complement[aidata,trainingdata];

Since this data is well structured and relatively straight forward, we can use the simple Predict function

predict=Predict[trainingdata,Method->Automatic,PerformanceGoal->"Quality"]

And look at the key stats of that predictor

Information[predict]

And then look at how it did.

PredictorMeasurements[predict,testdata,"ComparisonPlot"]

Now a couple lines of code to check how it works.  You can rerun this line several times to see how it does.

sample=RandomChoice[aidata]
predict[First@sample]

Using the predict function we just developed, we can now put it into a form.

beerform=

And then deploy it to the cloud

CloudDeploy[beerform,"beerpredictor",Permissions->"Public"]

The result is an easy to access tool that can be used when making purchasing decisions.

Scheduled Tasks, Online Forms and Cloud Data Storage for Research and Family Financial Tools

As part of our daily work flow, our team regularly utilizes the Wolfram Cloud and the various functionalities available within it.  These include: Scheduled Tasks, Cloud Deployed Forms, and Databins.  Example of these in practice include:

Automated Twitter tools that utilize scheduled tasks to search for top tweets on particular topics, store these in a Databin, and retweet them if unique.

Online forms that provide easy access to curated data and search functionality.  Currently in use for matching startups to similar organizations and potential funders.

Local COVID-19 data for counties in the US that quantifies trends, provides information on relative risk.

Adapting these tools to manage children’s allowances

Design Requirements

Automatically handle weekly deposits

ScheduledTask

Have a web-based interface for making adjustments and a tool for the kids to track their own status

CloudPublish Form

All data must be persistent and cloud based

Databin

Start by creating a Databin

databin=CreateDatabin[]

Then create a form to input changes

changeform=

checkbalance=

Deploy these to the Cloud so they can be easily accessed

CloudDeploy[changeform,"allowancechange",Permissions->"Public"]
CloudDeploy[checkbalance,"checkallowance",Permissions->"Public"]

Create a Scheduled Task to automatically add $10 per week.

CloudDeploy[ScheduledTask[,"Weekly"],"weeklyallowance"]

Using WebSession to Import and Organize Data

Extracting and formatting data from webpages is one of the most frequently used tools employed by our team.  With a basic knowledge of HTML and CSS Selectors, it is possible to turn diverse individual pages into structured datasets.  We have used this approach for:

Building databases of businesses and investors

Converting poorly formatted data tables into standardized lists

Scraping links for PDF and media links to download  and analys

Importing country data from government websites

Manipulating dynamic websites to display and extract nested information

Due to widely varying hosting and coding approaches, there is not a one-size-fits-all set of code that works, but rather, the basic principles and data cleaning techniques must be used.

On a personal level, WebSession has become my go-to tool for gathering data and I have used it for everything from researching fishing equipment to comparing prices between vendors to analyzing sports statistics

Fishing Rod data

Start a Web Session and open the product page

session=StartWebSession["Chrome"];

WebExecute["OpenPage"->"https://tenbum.com/collections/tenkara-gear?sort_by=price-descending"];

Bring in code to extract tag attributes

Credit: swish from StackExchange

getAttribute[element_WebElementObject,attribute_String]:=getAttribute[$CurrentWebSession,element,attribute]

getAttribute[session_WebSessionObject,element_WebElementObject,attribute_String]:=With[{sessionInfo=session/@{"SessionID","Browser","URL"}},WebUnit`Private`attribute[sessionInfo,element["ElementId"],attribute]]

Pull all the product links so they can be utilized individually

links=getAttribute[#,"href"]&/@WebExecute["LocateElements"->"CSSSelector"->"div.product-card>a"];

Open a sample page to begin testing CSSSelectors

WebExecute["OpenPage"->First@links];

These particular pages are inconsistent, so I want to structure my imports in such a way that I can easily combine fields and create an association from them.  The simplest format in this case is the key followed by the value in a list.

Identity the product

product=product={"Rod",Check[First[WebExecute["ElementText"->WebExecute["LocateElements"->"CSSSelector"->"H1"]]],""]}

Pull in the description

description=description={"Description",Check[StringRiffle[WebExecute["ElementText"->WebExecute["LocateElements"->"CSSSelector"->"div.product-single__description>p"]],"\n"],""]}

{"Description","The Tenryu Furaibo TF39 and TF39TA are tenkara rods made in Japan by a small company that is not yet well known in the US. They are not the world's least expensive tenkara rods. They may be the very best.\nMany tenkara anglers believe a \"zoom\" rod is too much of a compromise, with the rod performing well at only one of the two or three lengths. The Tenryu Furaibo TF39TA was specifically designed to perform equally well at all three lengths. People have asked me why the rod carries a high price. Part of the answer is the exacting design, materials, and manufacturing attention to detail that go into the rod. The result is an exceptional rod, but one which cannot be produced cheaply. You will not find anything like it from any of the Chinese factories that supply rods for the American or European tenkara rod companies.\nThe TF39TA was the first backpack-friendly rod I found that was designed for level lines. It truly is unique. Despite collapsing to a length of only 14 inches (even shorter than the Daiwa Keiryu-X) is not a stiff rod like the Keiryu-X, the Suntech Genryuko or the Sakura Kongo. Nor is it a delicate, thin-walled rod like the Nissin Pocket Mini. It has the action a tenkara angler would expect in a premium tenkara rod with a more standard 22 or 23 inch collapsed length. \nIt is the only rod on the market truly designed for ardent tenkara anglers who backpack, rather than for the ardent backpacker who may want to fish a bit along the trail and who is willing to accept a relatively stiff or relatively delicate rod just because it fits in the backpack. The Tenryu Furaibo TF39TA is a rod for the tenkara angler who wants a great rod that casts well and fishes well - and also fits in the pack. This is the rod, indeed it is THE rod for all the people who have asked me for a rod for the John Muir Trail, or the Pacific Coast Trail or the Appalachian Trial. It just hasn't been available - until now."}

On this particular page, the creator has two ways of listing specific details.  The first is a simple table where we can pull all the TD cells and then partition.

details1=Check[Partition[WebExecute["ElementText"->WebExecute["LocateElements"->"CSSSelector"->"td"]],2],{}]

The second is formatted using either spaces or dashes.  This requires a more in depth approach.

details2=Check[
Select[Select[#,StringLength[#]>0&]&/@(StringTrim[StringSplit[#,{"  ","-"}]]&/@Rest[First[StringSplit[WebExecute["ElementText"->WebExecute["LocateElements"->"CSSSelector"->"div.responsive_col-1"]],"\n"]]]),Length[#]==2&],{}]

This captures all the relevant information, but we still need a way to convert these lists into formatted associations so we create a simple defined function.

detailsAssoc[list_]:=Check[Association[(ToUpperCase[#[[1]]]->#[[2]]&/@list)],{}]

All of this can combined into a single function and all the links can be scraped.

tenkaraScrape[url_]:=

tenkaradata=tenkaraScrape[#]&/@links;

This particular scrape had an unrelated example as the last entry so we can remove that with Most.  Since available fields change for each model, we can use KeyUnion to fill in the gaps and then structure the final Dataset.

Dataset[KeyUnion[Most@tenkaradata]][[All,{"ROD","URL","DESCRIPTION","EXTENDED LENGTH","COLLAPSED LENGTH","WEIGHT","SECTIONS","CARBON/GLASS %","PENNIES","TIP DIAMETER"}]][All,{"URL"->Hyperlink}]

Keys to success with WebSession

Approach every website fresh.  Coding styles and structure vary widely and rarely will the same approach work perfectly on different pages.

Brush up on your HTML and CSS.  Production level coding skills are not required, but deciphering Selectors requires a working understanding of the concepts.

The JavascriptExecute command can be used to load AJAX based pages that load as scrolled.  JS can also be used to reveal hidden elements.

Pay attention to not only the text, but also the attributes.  Many times class or ID names contain relevant data not publically displayed and can be extracted using getAttribute.

It is often easier to import larger strings and manipulate them after scraping than it is to wait for every element to be located and text returned.  Frequently used approaches include:

StringSplit, StringTrim, Partition, Part

Be sure to test your initial approaches on multiple examples (and perhaps even run a larger sample before committing to the entire process to identify issues).

Consider approaches that store each entry in a local file instead of trusting the entire evaluation to complete and provide a final output.

Next Steps

Encourage organic flow between professional tools and personal passions

Engage Wolfram Community to share code and be inspired

Explore Video and PDF Processing

Next Presentation: 4:00p - Creating Computable Data for SDG Metrics from PDFs, Video and Images