Transcript Analysis of Wolfram Technology Conference Talks

Jesse Friedman, Wolfram Research

The (raw) data

WTC talk video recordings from the Wolfram Research YouTube channel

9 years of WTC (2010-2018)

535 talks

Actually, 535 videos - a very small handful of talks are split across multiple videos:

Select[EntityValue["WolframConferenceTalk","Title"],StringContainsQ["Part "]]

~230 hours

totalTalksDuration=N@UnitConvert[Total@EntityValue["WolframConferenceTalk","VideoInformation"][[All,"Duration"]],"Hours"]

More recorded talks in later years

Counts[EntityValue["WolframConferenceTalk","EventYear"]]

BarChart@%

Creating transcripts (the boring part)

youtube-dl (https://rg3.github.io/youtube-dl/) to download audio as MPEG4 (~12GB)

Google Cloud Speech-to-Text for transcription

ffmpeg to convert to FLAC (~63GB...)

Why not SpeechRecognize?

Speed

Cloud Speech-to-Text is pretty fast

Short[durationVsTranscriptionTime=DeleteMissing[EntityValue["WolframConferenceTalk",{"TranscriptionJobInterval","VideoInformation"},"EntityPropertyAssociation"],1,2]//Query[All,<|
"TranscriptionTime"->(UnitConvert[DateDifference@@#["TranscriptionJobInterval"][[1]],"Minutes"]&),
"Duration"->(N@UnitConvert[#["VideoInformation","Duration"],"Minutes"]&)
|>]]

Short[transcriptionSpeeds=#Duration/#TranscriptionTime&/@durationVsTranscriptionTime]

Mean@transcriptionSpeeds

SpeechRecognize seems fast (~2.5x) on short clips (~30 sec) but slower on longer files - I saw 0.8x on a 12-minute clip

Could just be my machine

totalTalksDuration/0.8

UnitConvert[%,"Days"]

Now-%

Parallelization

Cloud Speech-to-Text inherently runs in parallel

SpeechRecognize... well, you try it.

Transcribing all 535 talks (~230 hours) on Cloud Speech-to-Text took about 8.5 hours wall-clock time, processing 10 talks at a time

So, how good are the transcripts?

Not very.

Stephen Wolfram - Keynote Part 2 (2010):

“Could another site what's the one can learn every every one of these distributions in a sense encapsulate a bunch of knowledge and some particular area whether it's a levy decision, which I have a dog or a normal distribution with a dog or a Landau distribution, which I have to say. I haven't heard over the it might be something from physics. I don't know so that lots of lots of kinds of distribution supported hair. So let’s see other things to show hair wiglets.”

Joel Klein - Livecoding a Web App for the Cloud (2017):

“Hi, my name is Joel Klein going to be doing a taco life coding a web app in the world from cloud.”

Anton Antonov - Object-Oriented Design Patterns (2015):

“Do you see the questions you care for your smart program in Mathematica? Do not use just Mathematica calculator breaking my bottoms yet.”

Why are they so bad?

Not sure, but I suspect audio quality

Metadata

EntityStores[][[1]]

RandomEntity["WolframConferenceTalk"]["Dataset"]

RandomEntity["WolframConferenceSpeaker"]["Dataset"]

EntityClassList["WolframConferenceTalk"]

%[[5]]//EntityList//Shallow

Some quick metadata analysis

Most prolific speakers

mostProlificClass=EntityClass["WolframConferenceSpeaker","Talks"->TakeLargestBy[Length,5]]

EntityList@mostProlificClass

EntityValue[mostProlificClass,"Talks","Dataset"]

Most common speaker first names

WordCloud@EntityValue["WolframConferenceSpeaker",EntityFunction[e,StringSplit[e["Name"]][[1]]]]

Weighted by number of talks

WordCloud@EntityValue["WolframConferenceSpeaker",EntityFunction[e,{StringSplit[e["Name"]][[1]],Length@e["Talks"]}]]

Now on to the transcript analysis...

But first! A cool trick...

The YouTube descriptions look like:

Column[descriptions=EntityValue[{(Entity["What's New in the Neural Network Framework (2018)", Entity["WolframConferenceTalk","q446jlew1Ss"], "Entity[\"WolframConferenceTalk\", \"q446jlew1Ss\"]", "Wolfram Conference Talk"]),(Entity["Beyond the Black Hole Horizon with Mathematica (2017)", Entity["WolframConferenceTalk","bOtvvw1dyN0"], "Entity[\"WolframConferenceTalk\", \"bOtvvw1dyN0\"]", "Wolfram Conference Talk"])},"VideoInformation"][[All,"Description"]],Dividers->All]

We can extract the list of speakers with this:

extractSpeakers=StringCases[("Speaker"|"Speakers")~~": "~~speakerlist:Shortest[__~~EndOfLine]:>speakerlist]/*StringRiffle/*(StringSplit[#,", and"|", &"|","|"&"|";"|" and "]&)/*StringTrim;

extractSpeakers/@descriptions

But what about misspellings?

joseTalkNames=(extractSpeakers/@EntityValue[(Entity["José Martín-García", Entity["WolframConferenceSpeaker","José Martín-García"], "Entity[\"WolframConferenceSpeaker\", \"José Martín-García\"]", Wolfram Conference Speaker])["Talks"],"VideoInformation"][[All,"Description"]])[[2;;,1]]//Union

EditDistance!

EditDistance[u,v] gives the number of one-element deletions, insertions, and substitutions required to transform u to v.

originalSpeakerNames=;

speakerNameGraph=RelationGraph[EditDistance/*LessEqualThan[3],originalSpeakerNames]

speakerNameGraph//ConnectedComponents//Select[Length/*GreaterThan[1]]

OK, transcript analysis for real this time.

Word clouds

Words

jessetalk=(Entity["Jesse Friedman", Entity["WolframConferenceSpeaker","Jesse Friedman"], "Entity[\"WolframConferenceSpeaker\", \"Jesse Friedman\"]", "Wolfram Conference Speaker"])["Talks"][[1]]

jessetalk["Transcript"]//WordCloud

(Entity["Opening Keynote Part 1 (2017)", Entity["WolframConferenceTalk","Hkle4YOr5WE"], "Entity[\"WolframConferenceTalk\", \"Hkle4YOr5WE\"]", "Wolfram Conference Talk"])["Transcript"]//WordCloud

n-grams

2-grams

sw2grams=WordCounts[(Entity["Opening Keynote Part 1 (2017)", Entity["WolframConferenceTalk","Hkle4YOr5WE"], "Entity[\"WolframConferenceTalk\", \"Hkle4YOr5WE\"]", "Wolfram Conference Talk"])["Transcript"],2]

KeyMap[StringRiffle,sw2grams]//WordCloud

sw2gramsNoStopwords=WordCounts[DeleteStopwords@(Entity["Opening Keynote Part 1 (2017)", Entity["WolframConferenceTalk","Hkle4YOr5WE"], "Entity[\"WolframConferenceTalk\", \"Hkle4YOr5WE\"]", "Wolfram Conference Talk"])["Transcript"],2]

KeyMap[StringRiffle,sw2gramsNoStopwords]//WordCloud

3-grams

sw3grams=WordCounts[(Entity["Opening Keynote Part 1 (2017)", Entity["WolframConferenceTalk","Hkle4YOr5WE"], "Entity[\"WolframConferenceTalk\", \"Hkle4YOr5WE\"]", "Wolfram Conference Talk"])["Transcript"],3]

KeyMap[StringRiffle,sw3grams]//WordCloud

sw3gramsNoStopwords=WordCounts[DeleteStopwords@(Entity["Opening Keynote Part 1 (2017)", Entity["WolframConferenceTalk","Hkle4YOr5WE"], "Entity[\"WolframConferenceTalk\", \"Hkle4YOr5WE\"]", "Wolfram Conference Talk"])["Transcript"],3]

Most prolific loquacious speakers

Short[speakerTalkWordCounts=EntityValue[EntityClass["WolframConferenceTalk","Speakers"->Length/*EqualTo[1]],{"Speakers","Words"}]//GroupBy[(#[[1,1]]&)->(Length@#[[2]]&)]]

Histogram[Length/@EntityValue["WolframConferenceTalk","Speakers"]]

TakeLargest[Total/@speakerTalkWordCounts,10]//KeyValueMap[Tooltip[#2,#1]&]//BarChart

Another word cloud!

WordCloud[StringRiffle@EntityValue["WolframConferenceTalk","Transcript"]]

Pane[Column@,{Full,250},Scrollbars->True]

Term trends over time

transcriptsByYear=AssociationMap[EntityValue[#,"Transcript"]&,EntityClassList["WolframConferenceTalk"]];

BarChart[N[
Length@Select[#,StringContainsQ["cloud",IgnoreCase->True]]
/
Length[#]
]&/@transcriptsByYear]

wordTrends[word_]:=AssociationMap[WordFrequency[#,word,IgnoreCase->True]&/@EntityValue[#,"Transcript"]&,EntityClassList["WolframConferenceTalk"]]

cloudTrends=wordTrends["cloud"];

cloudTrends=;

Mean/@cloudTrends//BarChart

**NEW**

**NEW** (StephenOrNot^®)

**NEW** (StephenOrNot^®)

swTalks=(Entity["Stephen Wolfram", Entity["WolframConferenceSpeaker","Stephen Wolfram"], "Entity[\"WolframConferenceSpeaker\", \"Stephen Wolfram\"]", "Wolfram Conference Speaker"])["Talks"]

Shallow[otherTalks=]

Dataset[transcripts=EntityValue[#,"Transcript"]&/*(AssociationThread[{"Training","Test"},TakeDrop[#,Floor[Length@#*.5]]&@#]&)/@<|
True->swTalks,
False->otherTalks
|>]

transcriptCF=Classify[transcripts[[All,"Training"]]]

measurements=ClassifierMeasurements[transcriptCF,transcripts[[All,"Test"]]];
measurements["Accuracy"]

measurements["ConfusionMatrixPlot"]

validationAll=GroupBy[EntityList["WolframConferenceTalk"],(MemberQ[#["Speakers"],(Entity["Stephen Wolfram", Entity["WolframConferenceSpeaker","Stephen Wolfram"], "Entity[\"WolframConferenceSpeaker\", \"Stephen Wolfram\"]", "Wolfram Conference Speaker"])]&)->(EntityValue[#,"Transcript"]&)]

measurementsAll=ClassifierMeasurements[transcriptCF,validationAll]

measurementsAll["Accuracy"]

measurementsAll["ConfusionMatrixPlot"]

measurementsAll["FalsePositiveExamples"]

paragraphs=Flatten@EntityValue[#,"Paragraphs"][[All,All,"Text"]]&/*(AssociationThread[{"Training","Test"},TakeDrop[#,Floor[Length@#*.5]]&@#]&)/@<|
True->swTalks,
False->otherTalks
|>;

paragraphCF=Classify[paragraphs[[All,"Training"]]]

paragraphMeasurements=ClassifierMeasurements[paragraphCF,paragraphs[[All,"Test"]]];
paragraphMeasurements["Accuracy"]

paragraphMeasurements["ConfusionMatrixPlot"]

paragraphValidationAll=Flatten/@GroupBy[EntityList["WolframConferenceTalk"],(MemberQ[#["Speakers"],(Entity["Stephen Wolfram", Entity["WolframConferenceSpeaker","Stephen Wolfram"], "Entity[\"WolframConferenceSpeaker\", \"Stephen Wolfram\"]", "Wolfram Conference Speaker"])]&)->(EntityValue[#,"Paragraphs"][[All,"Text"]]&)]

paragraphMeasurementsAll=ClassifierMeasurements[paragraphCF,paragraphValidationAll];
paragraphMeasurementsAll["Accuracy"]

paragraphMeasurementsAll["ConfusionMatrixPlot"]

sentences=Flatten@TextSentences@EntityValue[#,"Transcript"]&/*(AssociationThread[{"Training","Test"},TakeDrop[#,Floor[Length@#*.5]]&@#]&)/@<|
True->swTalks,
False->otherTalks
|>;

sentenceCF=Classify[sentences[[All,"Training"]]]

ClassifierMeasurements[sentenceCF,sentences[[All,"Test"]], "Accuracy"]

sentenceValidationAll=Flatten/@GroupBy[EntityList["WolframConferenceTalk"],(MemberQ[#["Speakers"],(Entity["Stephen Wolfram", Entity["WolframConferenceSpeaker","Stephen Wolfram"], "Entity[\"WolframConferenceSpeaker\", \"Stephen Wolfram\"]", "Wolfram Conference Speaker"])]&)->(TextSentences@EntityValue[#,"Transcript"]&)]

sentenceMeasurementsAll=ClassifierMeasurements[sentenceCF,sentenceValidationAll];
sentenceMeasurementsAll["Accuracy"]

paragraphMeasurementsAll["ConfusionMatrixPlot"]

One last thing...

Short[streamedSpeechToText=Import["data/streaming-text.txt"]]

WordCloud[streamedSpeechToText]

Thanks!

jessef@wolfram.com

Data will hopefully be published soon on the Wolfram Data Repository, after I fix some issues. Email me if you’d like a preliminary version.

Initialization

SetDirectory[NotebookDirectory[]];
Quiet[EntityUnregister/@{"WolframConferenceSpeaker","WolframConferenceTalk"}];

EntityRegister@Import["data/EntityStore-NoAlternatives.wxf"]