StackExchange Exploration
with Entities

Andrew Steinacher
Lead Developer, Wolfram|Alpha

Abstract

The Stack Exchange network of question-and-answer sites provides a large network of knowledge in various subjects, including mathematics, physics, travel and more. Snapshots of these constantly evolving sites are kept and distributed in archives in a uniform, XML-based format, which makes them easy to process. With the power of the Wolfram Language, a tool has been built to convert each of these archives into an EntityStore, which allows for a wide range of detailed analysis on the knowledge of many subjects. Some interesting examples include an analysis of Wolfram Language code on mathematica.stackexchange.com, the theorems and mathematicians mentioned on MathOverflow.net, flight sequences from travel.stackexchange.com and more.

Outline

StackExchange Network Sites

Tools for Exploring SE data

StackExchange Data Explorer

XML Data Dumps

WL EntityStores

Analysis of Specific Sites

MathOverflow.net

Mathematica.StackExchange.com

CodeGolf.StackExchange.com

StackExchange Network Sites

Tour: https://stackexchange.com/tour

Questions, answers

Accepted answers, votes

Question Tags

Users earn reputation points, badges, edit posts, etc...

StackExchange Network Sites

Sites: https://stackexchange.com/sites

StackOverflow: “Q&A for professional and enthusiast programmers”

Mathematics: “Q&A for people studying math at any level and professionals in related fields”

Super User: “Q&A for computer enthusiasts and power users”

Mathematica, MathOverflow.net, Physics, etc...

Pets, Travel, Vegan/Vegetarianism, etc...

Meta sites (community management)

SE Tools: SEDE

StackExchange Data Explorer (SEDE): https://data.stackexchange.com/

Write SQL queries to explore data in a web browser

Queries are time limited

Results are downloadable as CSV files

Tables, basic plotting

https://data.stackexchange.com/mathematica/query/161411/site-activity-and-votegraph #graph

SE Tools: XML Data Dumps

https://archive.org/details/stackexchange

Anonymized dumps of all user-contributed content

XML Files in 7-Zip compressed files

Updated every few months

Schema: https://meta.stackexchange.com/a/2678

WL Entity Stores

Built a WL tool to ingest SE XML archives, create WL EntityStores

PacletDirectoryAdd[NotebookDirectory[]];
Get["StackExchange2EntityStore`"]
SetDirectory[NotebookDirectory[]];
store=StackExchange2EntityStore["crafts.stackexchange.com"];
EntityRegister[store]

WL Entity Stores: Properties

EntityProperties["StackExchange.Crafts:User"]//Shallow

EntityProperties["StackExchange.Crafts:Post"]//Shallow

WL Entity Stores: Users

EntityList["StackExchange.Crafts:User"]//Shallow

WL Entity Stores: Users

(Entity[(RowWithSeparator[(Spacer1), "Tim", (HyperlinkURL["»", https://crafts.stackexchange.com/users/7])]), Entity["StackExchange.Crafts:User","7"], "Entity[\"StackExchange.Crafts:User\", \"7\"]", "crafts:User"])[{(EntityProperty["approximate reputation time series", EntityProperty["StackExchange.Crafts:User","ApproximateReputationTimeSeries"], "EntityProperty[\"StackExchange.Crafts:User\", \"ApproximateReputationTimeSeries\"]"]),(EntityProperty["badges", EntityProperty["StackExchange.Crafts:User","Badges"], "EntityProperty[\"StackExchange.Crafts:User\", \"Badges\"]"]),(EntityProperty["owned posts", EntityProperty["StackExchange.Crafts:User","OwnedPosts"], "EntityProperty[\"StackExchange.Crafts:User\", \"OwnedPosts\"]"]),(EntityProperty["up votes", EntityProperty["StackExchange.Crafts:User","UpVotes"], "EntityProperty[\"StackExchange.Crafts:User\", \"UpVotes\"]"])},"PropertyAssociation"]

WL Entity Stores: Users

See a user’s comments

(Entity[(RowWithSeparator[(Spacer1), "Surb", (HyperlinkURL["»", https://crafts.stackexchange.com/users/71])]), Entity["StackExchange.Crafts:User","71"], "Entity[\"StackExchange.Crafts:User\", \"71\"]", "crafts:User"])["Comments"]//Shallow

WL Entity Stores: Posts

Find posts with a given tag

EntityValue[EntityClass["StackExchange.Crafts:Post","Tags"->(Entity[(RowWithSeparator[(Spacer1), "✓", "watercoloring", (HyperlinkURL["»", https://crafts.stackexchange.com/tags/watercoloring/info])]), Entity["StackExchange.Crafts:Tag","watercoloring"], "Entity[\"StackExchange.Crafts:Tag\", \"watercoloring\"]", "crafts:Tag"])],
"Entities"][[;;5]]

WL Entity Stores: SPARQL

Needs["GraphStore`"]
results=Entity["StackExchange.Crafts:User"]//SPARQLSelect[
SPARQLPropertyPath[
(Entity[(RowWithSeparator[(Spacer1), "Chris H", (HyperlinkURL["»", https://crafts.stackexchange.com/users/319])]), Entity["StackExchange.Crafts:User","319"], "Entity[\"StackExchange.Crafts:User\", \"319\"]", "crafts:User"]),{(EntityProperty["owned posts", EntityProperty["StackExchange.Crafts:User","OwnedPosts"], "EntityProperty[\"StackExchange.Crafts:User\", \"OwnedPosts\"]"]),(EntityProperty["comments", EntityProperty["StackExchange.Crafts:Post","Comments"], "EntityProperty[\"StackExchange.Crafts:Post\", \"Comments\"]"]),(EntityProperty["user", EntityProperty["StackExchange.Crafts:Comment","User"], "EntityProperty[\"StackExchange.Crafts:Comment\", \"User\"]"])},SPARQLVariable["commenter"]
],
"Distinct"->True,"Reduced"->True]//Lookup["commenter"]

Vegetarianism.StackExchange.com

Analyze Dynamics of a Social Network »

(Legended[, (LineLegend["Zanna", "Alexander Rossa", "Attilio", "Alexei", "Nic"])])

Travel.StackExchange.com

Extract Flight Sequences from Text »

MathOverflow.net: Words

The Data Science of MathOverflow »

(Legended[, (LineLegend[x, 10 x])])

MathOverflow.net: TEX Integrals

The Data Science of MathOverflow »

MathOverflow.net: Propositions

The Data Science of MathOverflow »

/.InsetBox[InterpretationBox[x:_String, y_Entity, BaseStyle -> "Entity"], z_List, rest___]:>InsetBox[InterpretationBox[StyleBox[x,FontSize->16], y, BaseStyle -> "Entity"], z, rest]

Mathematica.StackExchange.com

https://github.com/drewsteinacher/MathematicaSE-Analysis

Most used symbols overall:

Mathematica.StackExchange.com

Most used symbols (cleaner):

Mathematica.StackExchange.com

Symbols commonly used by specific users:

Mathematica.StackExchange.com

Association’s adoption TimeSeries after its introduction in V10

Mathematica.StackExchange.com

Association’s adoption TimeSeries after its introduction in V10

CodeGolf.StackExchange.com

https://github.com/drewsteinacher/CodeGolfSEAnalysis

CodeGolf.StackExchange.com

Gather all of the “child” submissions from a “parent” post:

parent=(Entity[(RowWithSeparator[(Spacer1), "Q: [Calvin's Hobbies] Do you want to code a snowman?", (HyperlinkURL["»", https://codegolf.stackexchange.com/q/49671])]), Entity["StackExchange.Codegolf:Post","49671"], "Entity[\"StackExchange.Codegolf:Post\", \"49671\"]", "codegolf:Post"]);

submissions=EntityList@EntityClass["StackExchange.Codegolf:Post",(EntityProperty["parent post", EntityProperty["StackExchange.Codegolf:Post","ParentPost"], "EntityProperty[\"StackExchange.Codegolf:Post\", \"ParentPost\"]"])->parent];
submissionToMetadata=EntityValue[submissions,"CodeGolfMetadata","EntityAssociation"];
submissionToMetadata//Keys//Shallow

CodeGolf.StackExchange.com

Show the submissions sorted by their reported size:

ResourceFunction["NiceGrid"][
{#["ReportedSize"],#["Language"],First[#["CodeSnippets"],Missing["NotFound"]]}&/@SortBy[submissionToMetadata,#ReportedSize&],
Alignment->Left
]

CodeGolf.StackExchange.com

(Entity[(RowWithSeparator[(Spacer1), "A: [Matty] Python 3, 349 336 254 251 bytes\nSo much for doing…", (HyperlinkURL["»", https://codegolf.stackexchange.com/q/49746])]), Entity["StackExchange.Codegolf:Post","49746"], "Entity[\"StackExchange.Codegolf:Post\", \"49746\"]", "codegolf:Post"])["SubmissionCodeSnippets"]

CodeGolf.StackExchange.com

Evaluating it directly in a notebook just requires some easy setup:

l='_===_| ___\n .....|  _\n  /_\| ___\n (_*_)| : |] [|> <|   |>| |\| | : |" "|___|   '.split('|')
l[4:4]=' \  .oO-,._ .oO- /  < / '
def s(a):print(' {}\n{}({}{}{}){}\n{}({}){}\n ({})'.format(*[l[4*m+int(a[int('0421354657'[m])])-1]for m in range(10)]))

s('31112311')

CodeGolf.StackExchange.com

For a given tag, gather the metadata for all submissions (answers) with that tag.
Then, find the ten most commonly submitted languages.

ClearAll[tagToLanguageBreakdown];
tagToLanguageBreakdown[tagConstraint_]:=With[
{
postToTagMetadata=EntityValue[
EntityClass["StackExchange.Codegolf:Post",{"Tags"->tagConstraint,"PostType"->(Entity["answer", Entity["StackExchange:PostType","2"], "Entity[\"StackExchange:PostType\", \"2\"]", "StackExchange:PostType"])}],
"CodeGolfMetadata",
"EntityAssociation"
]
},
Take[ReverseSort@Counts[Lookup[Values[postToTagMetadata],"Language"]],UpTo[10]]
];

CodeGolf.StackExchange.com

Submissions for posts with math-related tags use Wolfram Language quite a bit:

tagToLanguageBreakdown[(Entity[(RowWithSeparator[(Spacer1), "✓", "primes", (HyperlinkURL["»", https://codegolf.stackexchange.com/tags/primes/info])]), Entity["StackExchange.Codegolf:Tag","primes"], "Entity[\"StackExchange.Codegolf:Tag\", \"primes\"]", "codegolf:Tag"])]

tagToLanguageBreakdown[(Entity[(RowWithSeparator[(Spacer1), "✓", "math", (HyperlinkURL["»", https://codegolf.stackexchange.com/tags/math/info])]), Entity["StackExchange.Codegolf:Tag","math"], "Entity[\"StackExchange.Codegolf:Tag\", \"math\"]", "codegolf:Tag"])]

CodeGolf.StackExchange.com

Write a symbolic SPARQL query to extract languages used and their reported submission sizes for a given thread (i.e. “parent post”).

Needs["GraphStore`"]
threadLanguageData=Entity["StackExchange.Codegolf:Post"]//SPARQLSelect[
{
RDFTriple[SPARQLVariable["parentPost"],(EntityProperty["post type", EntityProperty["StackExchange.Codegolf:Post","PostType"], "EntityProperty[\"StackExchange.Codegolf:Post\", \"PostType\"]"]),(Entity["question", Entity["StackExchange:PostType","1"], "Entity[\"StackExchange:PostType\", \"1\"]", "StackExchange:PostType"])],
RDFTriple[SPARQLVariable["submission"],(EntityProperty["parent post", EntityProperty["StackExchange.Codegolf:Post","ParentPost"], "EntityProperty[\"StackExchange.Codegolf:Post\", \"ParentPost\"]"]),SPARQLVariable["parentPost"]],
RDFTriple[SPARQLVariable["submission"],(EntityProperty["submission reported size", EntityProperty["StackExchange.Codegolf:Post","SubmissionReportedSize"], "EntityProperty[\"StackExchange.Codegolf:Post\", \"SubmissionReportedSize\"]"]),SPARQLVariable["reportedSize"]],
RDFTriple[SPARQLVariable["submission"],(EntityProperty["submission programming language", EntityProperty["StackExchange.Codegolf:Post","SubmissionProgrammingLanguage"], "EntityProperty[\"StackExchange.Codegolf:Post\", \"SubmissionProgrammingLanguage\"]"]),SPARQLVariable["language"]]
}->{"parentPost","language","reportedSize","submission"}
];//AbsoluteTiming

threadLanguageData[[;;1]]

CodeGolf.StackExchange.com

Position ranks (with ties) for some common languages:

NiceGrid[
Function[
ranks,
AssociationMap[N[Replace[#[ranks],Except[_?NumericQ]->Missing[]]]&,{Length,Min,Median,Mean,Max}]
]/@languageToRanks
]

CodeGolf.StackExchange.com

Look at the smallest WL winners. Heavy use of built-ins and infix forms:

postToWinningWLSnippet=First[#,Missing["NotFound"]]&/@EntityValue[wlWinningSubmissions,"SubmissionCodeSnippets","EntityAssociation"];
WordCloud[
KeyValueMap[Tooltip[Button[#2,Paste[#1],Appearance->"Frameless"],#1["URL"]]&,TakeSmallestBy[StringTrim/@Select[postToWinningWLSnippet,MatchQ[_String]],StringLength,25]],
ImageSize->Large,
WordSpacings->5,
AspectRatio->9/16
]

CodeGolf.StackExchange.com

Most common non-alphanumeric characters:

WordCloud[
KeySelect[StringMatchQ[LetterCharacter|DigitCharacter]/*Not]@ReverseSort@Counts@Flatten@Characters@StringTrim@Cases[Values[postToWinningWLSnippet],_String],
AspectRatio->9/16,
WordSpacings->3,
ImageSize->Large
]

CodeGolf.StackExchange.com

More analysis in this published cloud notebook: wolfr.am/HyUpWqQe

Submission date/time histograms

Golf Programming Language ML Classifier  (~87% accuracy!)

Submission languages per user

Timelines for users with ~200 unique languages

Historical metadata per post (e.g. reported size/submissions)

Summary

StackExchange Network Sites

Tools for Exploring SE data

StackExchange Data Explorer

XML Data Dumps

WL EntityStores

Analysis of Specific Sites

MathOverflow.net

Mathematica.StackExchange.com

CodeGolf.StackExchange.com