Advanced Data Capturing Using WebSession

Ben Kickert, SDGCounting

#WolframTechConf

ABSTRACT

As AI sophistication increases with more parameters and larger token windows, curating high-quality data is crucial. This presentation explores how to acquire web-based data using WebSession / WebExecute, focusing on using selectors effectively, and providing tips for data cleaning and verification.

Introduction

Work with SDGCounting

Advocate for the UN Sustainable Development Goals

Focus on tracking data collection, analysis, reporting and utilization

Interest in the role of AI in this work

United Nations -- Wealth of knowledge that few people read

Multitude of documents and resources exist, but difficult to find specific answers

However, the information is freely available and allowed to be accessed with automation

UN80 Initiative is focusing in improving mandate efficiency (including AI)

Content Overview

Today we will:

Look at specific techniques for using Wolfram tools like WebSession and WebExecute to extract data

Discuss how it can be used to improve AI tools (RAG, code-based analysis, focused queries, etc.) and integrate into other Wolfram workflows

Build on the basic documentation with practical examples

The code in this presentation is straight forward but it does assume a basic understanding of HTML and CSS

WebSession and WebExecute Overview

Utilizes a web driver to execute remote commands in a browser

Originally introduced in  Mathematica  12  (2019) but still listed as “Experimental”

Allows for basic browser operation as well the ability to interact with specific elements from a website making it more versatile than Import (source or plain text)

Ability to also execute JavaScript expands the possibilities

Wolfram Language then allows for efficient batch processing that incorporates these tools

Documentation

Basic Example - UN Web TV

Start a Web Session

StartWebSession[]

Open a page

WebExecute["OpenPage"->"https://webtv.un.org/en/schedule"]

Extract all hyperlinks

links=WebExecute["PageHyperlinks"];
sessionlinks=DeleteDuplicates[
Select[links,StringContainsQ["/asset/"]]
];
Length@sessionlinks
RandomSample[sessionlinks,5]

Locate elements using CSSSelector

h4=WebExecute["LocateElements"->"CSSSelector"->"h4"];
Length@h4
First[h4]

Get the text of those elements

titles=WebExecute["ElementText"->h4];
Length@titles
RandomSample[titles,5]

Including JavaScript - Scroll to the bottom of the page

WebExecute["JavascriptExecute"->"window.scrollTo(0,document.body.scrollHeight);"]

Getting an attribute from an element (First documented in 13.1)

obj=WebExecute["LocateElements"->"CSSSelector"->"h4>a"][[1]];
WebExecute["ElementText"->obj]
WebExecute["ElementAttribute"->{obj,"href"}]

Hypothetical Use Case

Bring in not only titles and links, but also times and descriptions via a morning ScheduledTask

Automate feeding this data into an LLM asking it to check if any sessions are related to your interests

Prioritize coverage based on real-time schedules or earmark for later follow up

Streamlining code with defined functions

In order to simply get the H1 for the Title, it takes 83 characters

First[WebExecute["ElementText"->WebExecute["LocateElements"->"CSSSelector"->"h1"]]]

Create shorthand code to speed up repetitious tasks

CSS - Shortcut for finding CSS Elements

Clear[CSS]
CSS[tag_]:=Quiet[WebExecute["LocateElements"->"CSSSelector"->tag]]/.{$Failed->{""},_Failure->{""}}

CSSText - Shortcut for CSS Element Text

Clear[CSSText]
CSSText[tag_]:=Quiet[If[Head[tag]===WebElementObject||Head[First[tag]]===WebElementObject,WebExecute["ElementText"->tag],WebExecute["ElementText"->WebExecute["LocateElements"->"CSSSelector"->tag]]]]/.{$Failed->{""},_Failure->{""}}

getAttribute - Extract an element from a tag

Clear[getAttribute]
getAttribute[object_,attribute_:"href"]:=WebExecute["ElementAttribute"->{object,attribute}]

jsScroll - Scroll web browser to bottom

Clear[jsScroll]
jsScroll[num_:1,pause_:0]:=Table[
(WebExecute["JavascriptExecute"->"window.scrollTo(0,document.body.scrollHeight);"];
Pause[pause];);,
num];

wsOpen - Shortcut for opening a page in WebSession

Clear[wsOpen]
wsOpen[link_]:=Quiet[WebExecute["OpenPage"->link]];

wsClick - Shortcut for clicking on an Element in WebSession

Clear[wsClick]
wsClick[object_]:=Quiet[WebExecute["ClickElement"->object]]

wsLinks - Shortcut for getting all page links

Clear[wsLinks]
wsLinks[]:=Quiet[WebExecute["PageHyperlinks"]]

Primer on CSS Selectors

One of several ways to locate elements in an webpage

W3Schools Documentation

Familiar for those with experience in web design because often used for formatting

Allows for creative combinations to identify elements -- particularly useful when needing to capture all similar elements on a page

Basic Syntax

h1

Selects all level 1 headings

#elementid

Selects the element with a particular id=“elementid” attribute

div.class

Selects elements within a specific class

Can also be used without a tag

div.class1.class2

Selects elements contained within multiple classes

div > span > p > a

parent / child combinations can be selected using arrows

div p

A space allows you to select all child elements regardless of level

[aria-label = “home”]

Specific attributes within tags can also be selected

Aria Labels can be particular useful, but also can be used for attributes such as style or tab-index

Additional syntax for siblings, nth children, wildcard characters, etc.

Combining these allows for increased specificity

div.main span > p:nth-child(2)  > a[aria-label = “Home”]

Using Inspector in a browser allows you to find these relationships and even copy the specific selector

Putting it into practice - SDG Actions Platform

Start a WebSession and open the page

StartWebSession[];
wsOpen["https://sdgs.un.org/partnerships/browse"];

Extract the links and filter for the partnerships

links=DeleteDuplicates[Select[wsLinks[],StringContainsQ[#,"https://sdgs.un.org/partnerships/"]&&StringContainsQ[#,"/browse"|"/faq"]==False&]];
Length@links
examplelink=

Open an example to explore HTML structure

wsOpen[examplelink];

Extract the elements

title=Check[First[CSSText["h1"]],""]

type=Check[First[CSSText[".views-field-field-entity-types > div"]],""]

start=Quiet@Check[First[StringSplit[First[CSSText[".views-field-field-start-date"]]," ("]],""]
end=Quiet@Check[First[StringSplit[First[CSSText[".views-field-field-date-of-completion"]]," ("]],""]

goals=Check[CSSText[".views-field-field-goals a"],{}]

getAttribute/@CSS[".field-content a[target=\"_blank\"]"]

Clicking Elements

wsClick[First[CSS["#resources-tab"]]];

resources=CSSText[".resources_tab .field__item"]

This process can continue and then all code combined into a single script that cycles through all pages to collect all links, and then extract all elements

Ways we at SDGCounting can use this information

Analyze trends in commitments

Which SDGs are getting the most attention?  How has this changed over the years?

Are commitments still be made?

Duration of commitments

Gather list of additional resources and reports to uncover work we may not have been aware of

Batch process the detailed descriptions to summarize the Actions being taken to further the SDGs using LLMs

When is post-processing a better approach?
Creating a spreadsheet of UN Custodial agencies

Start a Web Session and explore the format

StartWebSession[];
wsOpen["https://unstats.un.org/sdgs/dataContacts/"];

This page contains regularly formatted data, but it is not structured a way that allow for consistent extraction.  Therefore we start by importing all the relevant divisions and then cleaning.

alldiv=CSSText["div.panel.panel-default>div"];
Length@alldiv
Column[alldiv[[;;10]],Frame->All]

Identify the data contact elements

contacts=Select[alldiv,StringContainsQ["Name:"]];
Length@contacts
Length@DeleteDuplicates[contacts]

The remainder will be the indicators

indicators=Complement[alldiv,contacts];
Length@indicators

indicators=Select[alldiv,StringContainsQ[StringTake[#,5],"."]&];
Length@indicators
Length@DeleteDuplicates[indicators]

Since we know the indicators, we can now use Partition and SplitBy to group the indicators and the data contacts.

parts=Partition[SplitBy[alldiv,(MemberQ[indicators,#]&)],2];

parts2={#[[1,1]],#[[2]]}&/@parts;

Then we can combine each indicator with its data contacts using Tuples

parts3=Flatten[Tuples/@parts,1];

The  data from the contact cards can further be processed

individuals=parts3[[All,2]];
RandomSample[individuals,3]

Function to turn the text of the data contact into discrete fields and combine it with the indicator data

Clear@contactsplit
contactsplit[contact_]:=Module[{indicator,goal,lines,name1,name,agency1,agency,title1,title,email,phone,website1,website},Quiet[(
indicator=First[StringSplit[contact[[1]],"\n"]]; (*Split at line break to remove data calendar text*)

goal=ToExpression[First[StringSplit[indicator,"."]]];
lines=StringSplit[contact[[2]],"\n"];

name1=Check[First[Select[lines,StringContainsQ["Name"]]],""];
name=Check[StringTrim[Last[StringSplit[name1,":"]]],""];

agency1=Check[First[Select[lines,StringContainsQ["Agency"]]],""];
agency=Check[StringTrim[Last[StringSplit[agency1,":"]]],""];

title1=Check[First[Select[lines,StringContainsQ["Title"]]],""];
title=Check[StringTrim[Last[StringSplit[title1,":"]]],""];

email=Check[First[Select[lines,StringContainsQ["@"]]],""];

phone=Check[First[Select[lines,ContainsOnly[Characters[#],{"0","1","2","3","4","5","6","7","8","9","(",")",".","-","+"," "}]&]],""];

website1=Check[First[Select[lines,StringContainsQ["Website"]]],""];
website=Check[StringTrim[Last[StringSplit[website1,":"]]],""];

<|"Goal"->goal,"Indicator"->indicator,"Contact Name"->name,"Agency"->agency,"Title"->title,"Email"->email,"Phone"->phone,"Website"->website|>)]]

custodians=Dataset[contactsplit/@parts3]
First@custodians

Ways we can use this information

Identify which agencies hold the most responsibility for official SDG Statistics

Accountability for global development statistics has been at the heart of our work since we began

Given funding challenges, this may allow for better advocacy

Batch process to find social media accounts to better follow UN data related discussions

Final Thoughts

Additional Use Cases

Archive your own content – export posts or pages from self-hosted blogs or legacy sites without built-in export tools.

Build RAG foundations – collect and save web-based text as clean files for retrieval-augmented generation or AI context stores.

Monitor change – track updates, new entries, or version differences across structured or semi-structured sites.

Automate reference building – compile bibliographies, link maps, or institutional directories directly from public web pages.

Operate ethically & legally

Read and follow site Terms of Service – only collect data you’re allowed to access; prefer APIs when they exist.

Check robots.txt and meta-robots – skip disallowed paths and respect crawl delays or access limits.

Throttle your requests – slow, steady collection avoids server strain and keeps you in good standing.

Protect rights and privacy – treat scraped data as copyrighted or sensitive unless clearly public domain.

Document provenance – store URLs, timestamps, and notes so results are reproducible and defensible.

When in doubt, ask – contacting site admins or using official datasets is often the simplest solution.

A note on the future these tools:

The rapid advancement of LLMs will likely bring rapid changes to these approaches

As context windows increase and tools for supplementing knowledge are more integrated, this may not be a necessary process in the near future

Likewise, web content is constantly becoming more guarded due to concerns about how training data is gathered and utilized

Holding your own data will always be a safeguard

What is next for us

Continue to track how UN Data is counted and monitored -- especially important as the 2030 Agenda begins winding down and agencies are facing budget shortfalls

Keen interest in what come next for global development.  We will use tools like these to ensure we have the best record of how the discussions are unfolding.

Be sure to check out Maureen Baehr’s presentation immediately following this one for more discussion on how we got to this point and where we hope to go in the coming years

Thank You!

Q+A Session

Ben Kickert
ben.kickert@gmail.com

Additional Tips and Tricks for Effective WebExecute Use

Extending options by including text field inputs in your work flow

StringCases

Prefer String Manipulation and Partitioning is speed is a concern

PutAppend for long sessions

Ensure checks are in place to avoid failures

Easier to rerun errored imports that try to get it perfect the first time

Iconize to easily store data in notebook after completion

Pause is your friend