Grid[{{Pane[Column[{Style["Translating Wolfram|Alpha into Other Languages","Title"],Style["","Subtitle"],Style["Meng Lu (陆萌)","Subtitle"],Style["Noriko Yasui (安井紀子)","Subtitle"],Style["Francisco Javier Rodríguez Arias","Subtitle"]},Alignment->Center],Full,Alignment->Center],Pane[animationTitle,Full,Alignment->Center]}},Alignment->{{Center,Center}}]

Join the Conversation #WolframTechConf

Intro: Wolfram|Alpha

Wolfram|Alpha is a unique system for computing answers and providing knowledge — Computational Knowledge Engine

The primary product form is the public website http://www.wolframalpha.com.

Released in May, 2009

It has become one of the most popular Wolfram products.

Goal: “Making the world’s knowledge computable.”

Intro: Computational Knowledge Engine

Natural language querying interface

allows questions/queries to be given in natural language, but so far only in English.

Systematic curated data

many trillions of pieces of data in 1000+ knowledge domains

numerical, textual, binary types of data, but textual data is almost always in English

and sometimes data sources used for constructing the W|A’s knowledgebase are English-only and sometimes US-primary, 
e.g. flights, significantly more granular data for US places such as USStateData

Computational intelligence

domain expertise built into the system as automated methods and algorithms for solving problems and deriving interesting information from data

Dynamic presentation of computational knowledge

generation and formatting of results, summaries, and visualizations

sometimes can have English-language or US oriented designs and conventions

At implementation level, these components can, to different extents, get mixed in the same bodies of source code.

Intro: localized Wolfram|Alpha

All the four components were designed for English speakers, which means that localization needs to tackle all of them.

W|A has an ever expanding coverage of knowledge domains and query answering abilities, but a totally different and orthogonal dimension for W|A to grow is

extending natural language interface and computational knowledge presentation to other (human) languages

broadening both content coverage and userbase to other countries/regions and languages

Goal: W|A: “Making the world’s knowledge computable.”  and Localized W|A: “Making computational knowledge more accessible and useful internationally.”

1.0 release goal: making all math-related features in W|A localized to 日本語 (Japanese), español (Spanish), 中文 (Chinese), 한국어  (Korean), français (French) and русский (Russian).

Intro: localized Wolfram|Alpha example outputs

"Close Example Viewer"

"Open Example Viewer"

Motivation: languages in the world

There are more than 7000 languages in the world, but only 25 have more than 50 million speakers (> 0.6% world population):

EntityClass["Language","TotalSpeakers"->GreaterThan[50000000]]//EntityList

Gathering all Chinese languages, we may compare them by total speakers:

Motivation: localization languages

We are working on Wolfram|Alpha for the following languages:

whose total speaker population is ~2.4×(10^9) people.

Motivation: geographical coverage

Motivation: query statistics

Motivation: traffic distribution

Development workflow: some facts

The team consists of translators, data curators, developers, testers that are language natives. Dozens of people are involved to different degrees.

Localized W|A is built on top of the regular English-language W|A.

All the software components and features are inherited from the regular W|A into the localized W|A.

In a localized W|A, all queries supported by the regular W|A are still supported in English.

Localization happens to selected components and features gradually, with original behaviors as fallback.

Input: localized features can be queried in both the localization language and English. The rest features can still be queried normally in English.

Output: features fully localized will generally show a completely localized output, others may show up as a mixture of localization language and English.

For all this work, we developed content data, new and refactored code and frameworks, various tools and tests, mostly in Wolfram Language.

Development workflow: components to localize

Largest component in terms of quantity of new data points and new lines of code created:

curated data, such as names and descriptions of entities and properties, e.g. PolyhedronData.m:

{"GreatDodecahedron", "Split"} -> {
    "Name" -> "大十二面体"
},
...
"EulerCharacteristic" -> {
    "Description" -> "欧拉示性数"
}

textual data in database localized and then coalesced with the rest of data

linguistic patterns as building blocks for natural language understanding

"AugmentedTruncatedDodecahedron" -> {
    "LinguisticsPatterns" -> {
        FixedOrder["側台", "塔切頭", Synonym["十二", "Number", "ja"], "面体"]
    }
}
...
FVariable["Weight:Wb"] -> {
    "LinguisticsPatterns" -> {
        "物体重量",
        "重量"|"重",
        "体重",
        FixedOrder["重量"|"重",FixedOrder[Optional["一个"|"某个"|"某一"],"物体"|"物质"]],
        "物體重量"
    }
}
...

textual data in database localized and then coalesced with the rest

Source code localization is another large component of work:

grammar rules for matching sentence structures in foreign languages, e.g.

"GrammarRule" -> FixedOrder[
    Optional[WP["equation"]], a : Expr["Math"], Opt[$where], 
    WP["differentiate"], Optional[$endfluff]
] :> InheritedAction["DerivativeRule1"]

source code for constructing presentations (results, summaries, visualizations) that were originally designed and implemented only for English requires generalization and refactoring

Development workflow: source code and textual data in it

The first and an important step of source code localization is extracting static display-purpose strings from code (DisplayText[...]):

SowResult[
	Switch[Sign[e2], 1, DisplayText["Percent increase",14072], -1, DisplayText["Percent decrease",14073]],
	With[{compare = If[Not[MatchQ[e1,_Rational]&&MatchQ[#,Hold[Times[#,Power[#2,-1]]]&[Numerator[e1],Denominator[e1]]]]&&
		MatchQ[#,Hold[_[___]]],
		Row[{HoldForm@@#," = "}],""]&[ReleaseHold[Apply[Hold,Take[heldexpr,1,1],{1}]]]},
	Switch[Sign[e2],  
		1, RowTemplate[DisplayText["`1` is `2`%  larger than `3``4`.",27722], {HoldForm @@ heldexpr == e1 + e2, If[IntegerQ[p], p, N[p, 4]], compare, e1}], 
		-1, RowTemplate[DisplayText["`1` is `2`%  smaller than `3``4`.",27723], {HoldForm @@ heldexpr == e1 + e2, If[IntegerQ[p], p, N[p, 4]], compare, e1}]]],
	1000,
	"MOutput" -> N[100(e2/e1)],
	"MInput" -> Hold[N[100(e2/e1)]],
	"PodIdentifier"->"Percent increase/decrease",
	"Dependencies" -> {"UpdateFrequency" -> "Weekly"}
]

They are collected and registered in a database with curated translations:

Development workflow: translation

Machine translation API (e.g. Google Translate API) has limited use due to high fraction of poor translations, e.g.

divide numbers
— machine translation: 数を分ける
— curated translation: 数を割る

combine several arithmetic operations
— machine translation: いくつかの算術演算を結合する
— curated translation: いくつかの算術演算を組み合わせる

convert an improper fraction to a mixed number
— machine translation: 不適切な小数を混合数に変換する
— curated translation: 仮分数を帯分数に変換する

solve an inhomogeneous equation
— machine translation: 不均一方程式を解く
— curated translation: 非同次方程式を解く

Machine translations are often too literal, and not appropriate within the right technical contexts.

In some cases, machine translations are used as initial draft before manual curation and proofreading.

https://www.wikidata.org and https://www.wikipedia.org are used to translate names of entities (chemical elements, countries, cities, etc.), e.g. ~20% of ~170,000 city names translated to Japanese using this approach.

Various external tools and data sources are used for miscellaneous tasks, such as Chinese word segmentation, which is a step for creating better linguistic patterns for Chinese.

Development workflow: translation and data curation

Mathematica notebooks are used for translation and data curation, e.g.

Challenges: huge amount of data and code to localize

data

~(10^3.5) relational database tables

~(10^3)  database-like source files (.m files storing hierarchies of key-value pairs)

~(10^6) names and descriptions of objects (entities, properties, units, mathematical functions, etc.)

linguistic patterns

~(10^7) pieces of linguistic patterns (for recognizing “tokens”, i.e. semantic constituents in queries)

~(10^4) grammar rules (for parsing/understanding sentences, similar to GrammarRules[...] in WL)

code

~(10^5) source files

~(10^7) lines of source code (mostly in Wolfram Language)

~(10^5) textual pieces embedded in code (words, phrases, incomplete sentence parts for display purposes)

Challenges: data to localize is “heterogeneous”

Content data to localize is scattered in a large number of source files and database tables.

Static data and computational logic are often mixed in source code.

There are different ways to group and slice the large amount of data according to content domains, user visible features, storage locations and formats of data, software components and workflows, teams, etc. Managing the “divide-and-conquer” is non-trivial. There is so far 8000+ issue tracking tickets for 1.0 release for all languages and ~30000 tickets planned.

W|A outputs are dynamically computed and synthesized with many parts and processes rather than, say, some sets of static documents

Source code requires localization, e.g. sentence construction logic originally designed according to English requires re-designing and refactoring.

Some refactoring needs would not be realized until some (partly localized) outputs are reviewed.

Until most textual elements are localized, many results/outputs can stay as mixtures of two languages (localized language + some English), and appear unfinished/unpolished.

Trivial changes in the visible outputs may end up requiring disproportionally significant debugging efforts and complex fixes.

How frequent different textual elements actually show up in outputs is hard to measure, and typically unknown.

Hard to measure overall progress or degree of completeness, because W|A is always growing: impossible to say “all done!”.

Challenges: code localization and refactoring

Codebase is ~10 year old: legacy code, old “temporary” workarounds, localization process partly takes care of housekeeping.

Original implementation is often a mixture of data, computation, and styling. Need to separate static data out from computational logic.

Localized display-purpose strings lives in a separate database, which does not follow the regular English W|A development workflow.

Example: ordinalform[k] refactoring

ordinalform[1], ordinalform[2], ... format integers as ordinal numbers: (1^st),(2^nd)... (n^th)

multiple old implementations, and inconsistent usage needed unification and cleaning up.

10 lines of implementation for English became ~150 lines and many options to support 6 more languages.

Example: template functions used frequently and idiomatically

many (~10%) sentence-constructing code requires being refactored as templates.

needed different kinds of template functions similar to StringTemplate in Wolfram Language.

e.g. 
original:                     Row[{func, " is differentiable at ", var, " == ", val}]
refactored:                 RowTemplate["`1` is differentiable at `2` == `3`", {func,var, val}]
Chinese localization: RowTemplate["`1` 在 `2` == `3` 处可微", {func,var, val}]

W|A is released weekly; content data and source code get hundreds of changes daily. It is a big and persistent challenge to keep things synchronized, consistent, and updated.

Challenges: language-related issues

Various aspects of grammar and conventions of English, Spanish, French, Chinese, Japanese, Korean, and Russian are significantly different.

word variations (grammatical categories): inflections; plural vs. singular; grammatical genders: masculine/feminine/neuter; grammatical cases: nominative/accusative/genitive/dative; classifiers or measure/counter words; definite/indefinite articles, etc.

sentence structures different, orderings of parts differ, partitions of a sentence differ

Writing scripts and character-related technical issues

WL only supports up to Unicode point 0xFFFF (i.e. “plane-0”)
• Bohrium, (107^th) chemical element, Chinese name ‘’ (U+28A0F)
• Tennessine, (117^th) chemical element, Chinese name ‘’ not even in the Unicode character set.

punctuation characters, e.g. decimal separator (period vs. comma) and its parsing ambiguity

double-width and half-width characters used in CJK, characters with diacritics in some languages, adds complexity to query parsing

some built-in functions like ToLowerCase and similar didn’t have proper support for some writing scripts such as Cyrillic.

Challenges: language-related issues

Non-ASCII characters in *.m source files use Wolfram Language’s encoding, e.g.

Greek: π  ⟶  \[Pi]   		 Spanish: ñ   ⟶ \[NTilde]	    French: é  ⟶ \[EAcute]   	           Russian: А  ⟶   \:0410
Japanese: あ  ⟶  \:3042	 Korean: 언 ⟶   \:c5b8		    Chinese: 使用  ⟶ \:4f7f\:7528

~ 7% of all characters in the code repositories are non-ASCII characters encoded this way, which makes it harder to read and edit. The same .m source file viewed in

Wolfram Workbench:

Mathematica front end:

Challenges: language-related issues

“git diff XXX.m” does not display non-ASCII characters in code in readable way, e.g.

Standard diff:

A diff tool written in Wolfram Language:

Status

Planned release: Japanese, Spanish

version 1.0:  most math-related features are localized

about halfway done

some time next year

Other languages in the queue:

Chinese

French, Korean, and Russian

Acknowledgement: thanks to all contributors