FindTextualAnswer

FIND ANSWERS in unstructured collections of texts

Outline

Scope of FindTextualAnswer

Practical Uses

Under the Hood

Insight into the Deep Learning Approach

Scope of FindTextualAnswer

In this course, you will discover FindTextualAnswer, a new tool for Text Analysis in the Wolfram Language™ (since 11.3).

Field: natural language processing, reading comprehension

Functionality: question answering

Appetizer

FindTextualAnswer answers your questions by quoting some piece of text:

FindTextualAnswer["Paris is the capital and most populous city of France, with a 2015 population of 2,229,621.","How many people live in Paris?"]

It is similar to TextCases in that it also extracts information from a text:

TextCases["Paris is the capital and most populous city of France, with a 2015 population of 2,229,621.","Number"]

But it can handle a wide range of questions in natural language and provide relevant answers:

text="Even thermometers can't keep up with the plunging temperatures in Russia's remote Yakutia region, which hit minus 88.6 degrees Fahrenheit in some areas Tuesday. In Yakutia - a region of 1 million people about 3,300 miles east of Moscow - students routinely go to school even in minus 40 degrees. But school was cancelled Tuesday throughout the region and police ordered parents to keep their children inside.
In the village of Oymyakon, one of the coldest inhabited places on earth, state-owned Russian television showed the mercury falling to the bottom of a thermometer that was only set up to measure down to minus 50 degrees. In 2013, Oymyakon recorded an all-time low of minus 98 Fahrenheit.";

questions={"What is the temperature in Yakutia?",
"Name one of the coldest places on earth",
"When was the lowest temperature recorded in Oymyakon?",
"Where is Yakutia?",
"How many live in Yakutia?",
"How far is Yakutia from Moscow?"};

Thread[questions->FindTextualAnswer[text,questions]]//Column

The technology is different from that used in Wolfram|Alpha®.

Practical Uses

Get Alternatives and Properties of the Answers

FindTextualAnswer can yield several possible answers to a question:

FindTextualAnswer["Paris is the capital and most populous city of France, with a 2015 population of 2,229,621","How many people live in Paris?",2]//Column

FindTextualAnswer can provide the probabilities of those answers being correct and other properties that help you understand the context of each answer:

FindTextualAnswer["Paris is the capital and most populous city of France, with a 2015 population of 2,229,621.","How many people live in Paris?",2,{"Probability","String","Position","HighlightedSentence"}]//TableForm

Output Structure

FindTextualAnswer returns a string or an array whose rank and dimensions are determined by the second, third and fourth arguments:

FindTextualAnswer["Paris is the capital and most populous city of France, with a 2015 population of 2,229,621.",{"How many people live in Paris?","Where is Paris?"}]//InputForm

FindTextualAnswer["Paris is the capital and most populous city of France, with a 2015 population of 2,229,621.",{"How many people live in Paris?","Where is Paris?"},2,"String"]//InputForm

FindTextualAnswer["Paris is the capital and most populous city of France, with a 2015 population of 2,229,621.",{"How many people live in Paris?","Where is Paris?"},2,{"String","Probability","Sentence"}]//InputForm

Practical Uses

Find Answers in a Customized Collection of Texts

Several types of sources can be used, including local files.

Make a custom search, which can be a mix between personal text collection and the web:

contents=Join[TextSearch["ExampleData/Text","constitution","ContentObject"],{File["ExampleData/USConstitution.txt"],ContentObject[WikipediaData["US Constitutional Law"]]}]

Find the answer to your question:

FindTextualAnswer[contents,"which crimes are punished in the US Constitution?",5]

Practical Uses

Find Answers in Wikipedia

Wikipedia is a convenient source of continuously refreshed knowledge that you can use.

Get the pages about a topic:

pageTitles=WikipediaSearch["Sagrada Familia"]

Extract the contents of these pages:

articles=WikipediaData/@pageTitles;

WordCount/@articles

Find possible answers in context to your question:

FindTextualAnswer[articles,"Who built the Sagrada Familia?",5,{"Probability","HighlightedSentence"}]//TableForm

Aggregate several answers occurring in different contexts and plot the big picture:

WordCloud[FindTextualAnswer[articles,"Who built the Sagrada Familia?",20,{"String","Probability"}]]

Note that you can also use the question in WikipediaSearch:

question="Who founded the Who?";
FindTextualAnswer[WikipediaData/@Take[WikipediaSearch["Content"->question],3],question]

Speed Up with GPU

if a compatible GPU is available, the computation can be faster with this device than with a CPU:

FindTextualAnswer[articles,"Who built the Sagrada Familia?"]//AbsoluteTiming

FindTextualAnswer[articles,"Who built the Sagrada Familia?",TargetDevice->"GPU"]//AbsoluteTiming

Under the Hood

FindTextualAnswer combines well-established techniques for information retrieval and state-of-the-art deep learning techniques.

1) Pre-filtering of Text Passages with TFIDF

At first, if a significant number of paragraphs are given to FindTextualAnswer, it first selects the closest ones to the question.

The Term Frequency–Inverse Document Frequency (TFIDF) similarity is used.

The similarity between a question and a passage of text is typically assessed as follows.

Take a corpus of text:

corpus=ContentObject[WikipediaData["Rhinoceros"]]

Build a TFIDF feature extractor from this corpus:

tfidf=FeatureExtraction[TextCases[corpus,"Paragraph"],"TFIDF"];

Compute the distance between a question and some sentences:

question="What are the horns of a rhinoceros made of?"

TakeSmallestBy[TextCases[corpus,"Sentence"],CosineDistance[tfidf@#,tfidf@question]&,3]

Under the Hood

2) Answer Localization and Scoring with Deep Learning

After the TFIDF selection, a limited number of paragraphs are selected, with at most a few hundred words in each.

Each paragraph is fed into a neural network to locate the possible answers and estimate their probability.

This neural network was designed, trained and deployed using the Wolfram Language Neural Networks framework.

The neural network is the following graph of layers:

net=NetModel["Wolfram FindTextualAnswer Net for WL 11.3"]

NetInformation[net,"LayersCount"]

Network inputs: lists of tokens (in the question / the context, respectively), and a list of Boolean vectors to spot words in the texts that occur in the question in some form:

question="What color are elephants?";
context="Elephants have a grey or white skin.";

getTokens=StringSplit[#,{WhitespaceCharacter,x:PunctuationCharacter:>x}]&;
input=<|"Context"->getTokens@context,"Question"->getTokens@question,"WordMatch"->Join[{{0,1,1}},ConstantArray[0,{7,3}]]|>

Network outputs: probabilities for the answer to start and end at each position:

netOutput=net[input]

ListStepPlot[Flatten/@KeyTake[netOutput,{"Start","End"}],
FrameTicks->{MapIndexed[{First@#2+0.5, Rotate[#1,Pi/5]} &, input["Context"]],Automatic},
Filling->Axis,Joined->True,PlotTheme->"Web",PlotStyle->{RGBColor[0.1,0.6,0.1],RGBColor[0.75,0.1,0.1]},PlotRange->{0,1}]

Insight into the Deep Learning Approach

The big picture of the neural network used is the following:

The process encompasses the following steps.

1) Word Embeddings

Word embeddings capture the semantics of each word independently—putting together synonyms and similar concepts:

animals={"Alligator","Bear",};
colors={"Blue","White",};
fruits={"Apple","Apricot",};FeatureSpacePlot[Join[animals,colors,fruits],FeatureExtractor->NetModel["GloVe 300-Dimensional Word Vectors Trained on Wikipedia and Gigaword 5 Data"]]

The word embeddings are shared for the question and the context:

Map[NetExtract[net,#]&,{"word_embedding_context","word_embedding_question"}]

Most of the learned parameters are encompassed in these word embeddings, which were actually pretrained on billions of words:

NetInformation[net,"ArraysTotalElementCount"]

netWithoutEmbedding=Fold[NetDelete,net,{"word_embedding_question","word_embedding_context"}]

NetInformation[netWithoutEmbedding,"ArraysTotalElementCount"]

2) Bidirectional Neural Network Encoding

Recurrent layers capture semantics in context.

The recurrence is bidirectional, using a NetBidirectionalOperator nesting two instances of LongShortTermMemoryLayer:

NetExtract[net,"encode_context"]

The output is an internal representation, with one vector per word that depends on all the words occurring before and after:

questionEncoded=net[input,NetPort["encode_question","Output"]];

ArrayPlot[questionEncoded[[All,1;;20]],FrameTicks->{MapIndexed[{First@#2,#1}&,input["Question"]],None}]

contextEncoded = net[input, NetPort["encode_context", "Output"]];

ArrayPlot[contextEncoded[[All,1;;20]], FrameTicks->{MapIndexed[{First@#2,#1}&,input["Context"]],None}]

3) Two-Way Attention

A two-way attention mechanism enables reinterpreting the question given the text, and the text given the question.

It is based on normalized similarity matrices between the question and the context:

outerProduct=Outer[Dot,questionEncoded,contextEncoded,1];

ticksContext = MapIndexed[{First@#2, Rotate[#1,Pi/6]} &, input["Context"]];
ticksQuestion = MapIndexed[{First@#2, #1} &, input["Question"]];
Labeled[MatrixPlot[SoftmaxLayer[2]@outerProduct,
FrameLabel->{"Question","Context"},FrameTicks->{ticksQuestion,ticksContext},PlotTheme->"Web"],
Style["Question focusing attention on the context",Italic]]

Labeled[MatrixPlot[SoftmaxLayer[1]@outerProduct,
FrameLabel->{"Question","Context"},FrameTicks->{ticksQuestion,ticksContext},PlotTheme->"Web"],
Style["Context focusing attention on the question",Italic]]

The weights are used to recompute another internal representation of the question and the text, like SequenceAttentionLayer in the example:

questionReencoded=SequenceAttentionLayer["Dot"][<|"Input"->contextEncoded,"Query"->questionEncoded|>];

ArrayPlot[questionReencoded[[All,1;;20]],FrameTicks->{MapIndexed[{First@#2,#1}&,input["Question"]],None}]

4) Bidirectional Neural Network Decoding

Recurrent layers then aggregate evidence to build a higher-level representation of the context.

It is a two-layer bidirectional recurrent net with residual connections:

NetTake[net,{"decode/rnn","decode/rnn2/add"}]

The output is again a sequence of vectors with the same length as the number of words in the context:

contextQuestionPairEncoded = net[input, NetPort["decode/rnn2/add", "Output"]];
ArrayPlot[contextQuestionPairEncoded[[All,1;;30]], FrameTicks->{MapIndexed[{First@#2,#1}&,input["Context"]],None}]

5) Pointer Net

A last part points to the possible locations of the start and the end.

Bidirectionality is again used:

NetTake[net,{"decode/rnn2/add",All}]

The output is a pair of sequences of probabilities, one for the starting position and one for the ending position:

ListStepPlot[Flatten/@KeyTake[netOutput,{"Start","End"}],
FrameTicks->{MapIndexed[{First@#2+0.5, Rotate[#1,Pi/5]} &, input["Context"]],Automatic},
Filling->Axis,Joined->True,PlotTheme->"Web",PlotStyle->{RGBColor[0.1,0.6,0.1],RGBColor[0.75,0.1,0.1]},PlotRange->{0,1}]

The set of most probable {start, end} positions is found with a beam search.

6) Post-Processing to Estimate Probabilities

The preceding probability estimates were trained with the assumption that the answer is in the text and is unique. So they are not suited to estimating that the answer is right.

Instead, the probability of the answer being right is based on these intermediate activations of the network:

{startActivations, endActivations} = netOutput /@ {"StartActivation", "EndActivation"}

ListStepPlot[AssociationThread[{"StartActivation (1/2)","StartActivation (2/2)","EndActivation (1/2)","EndActivation (2/2)"}->Flatten[Transpose/@{startActivations,endActivations},1]],FrameTicks->{MapIndexed[{First@#2+0.5, Rotate[#1,Pi/5]} &, input["Context"]],Automatic},
Filling->Axis,Joined->True,PlotTheme->"Web",PlotStyle->Tuples[{{RGBColor[0.1,0.6,0.1],RGBColor[0.75,0.1,0.1]},{Automatic, Dashed}}]]

Some logistic regression is done based on these activations:

scorer=NetModel["Wolfram FindTextualAnswer Scorer Net for WL 11.3"]

positions=<|"grey or white"->{4,6},"grey"->{4,4}, "white"->{6,6}|>;

Map[scorer[Join[startActivations[[First@#]],endActivations[[Last@#]]]]&,positions]

Conclusion

New natural language processing capability based on state-of-the-art neural networks.

Blog post: http://blog.wolfram.com/2018/02/15/new-in-the-wolfram-language-findtextualanswer

The technology is different from that of Wolfram|Alpha (which is based on a carefully curated knowledgebase).

This achievement was possible thanks to the great improvement of the Neural Networks framework in Version 11.3.

More and more similar deep learning–based applications coming soon.