Multilingual Translations and Decoding the Sperm Whale Language

Briefly introduction to transformer net models

Introduced in 2017 in the famous paper: “Attention Is All You Need”

A transformer is a deep learning model that adopts the mechanism of attention.

Transformer Networks

Transformers use self-attention to process or generate sequences.

Attention can be seen as a selection operation.

Self-attention means that every element of the sequence attends over every other element.

Transformers use several attention heads in parallel to compute features.

The time to process or generate a sequence using a transformer is quadratic in the length of the sequence.

Transformers led to impressive improvements in natural language processing.

Applications

Natural Language Processing (NLP)

Language models

Text translation

Text summarization

Chat bots

Sentiment analysis ...

Computer Vision (CV)

Transformers have been applied to image processing with results competitive with convolutional neural networks.

Chemistry (protein folding)

AlphaFold 2 predicting the 3D structure of proteins from chains of amino acids

Other applications

DALL·E transformer model from OpenAI can create images from text:

Transformer based models available on the Wolfram Neural Net Repository

BERT (Bidirectional Encoder Representations from Transformers)

RoBERTa

SciBERT

BioBERT

GPT2 (Generative Pre-trained Transformer)

Example of text generation using GPT2

Define a function to predict the next token:

lm=NetModel[{"GPT2 Transformer Trained on WebText Data","Task"->"LanguageModeling"}];
generateSample[languagemodel_][input_String,numTokens_:10,temperature_:1]:=Nest[Function[StringJoin[#,languagemodel[#,{"RandomSample","Temperature"->temperature}]]],input,numTokens];

lm

Generate the next 20 tokens by using it on a piece of text:

generateSample[lm]["Albert Einstein was a German-born theoretical physicist",20]

Toy example of a transformer

Converting dates into different date formats

Creating the training examples with RandomDate function

ResourceFunction["RandomDate"]

dates=[{(DateObject["Year: ""1400", DateObject[{1400},"Year","Gregorian",-5.`]]),2021},15000];

dateFormats={"Date","DateShort","ISOWeekDate","ISOOrdinalDate","LocaleDateShort","LocaleDateLong"};

dateTimeFormats ={"DateTime","DateTimeShort","LocaleDateTimeShort","LocaleDateTimeLong","LocaleDateTimeFull"};

examplesWithoutTimes=Map[DateString[#,RandomChoice[dateFormats]]->DateString[#,"ISODate"]&,dates];

dates2=[{(DateObject["Year: ""1400", DateObject[{1400},"Year","Gregorian",-5.`]]),2021},15000];

examplesWithTimes=Map[DateString[#,RandomChoice[dateTimeFormats]]->DateString[#,"ISODateTime"]&,dates2];

{trainingSet,testSet}=TakeDrop[RandomSample[Join[examplesWithTimes,examplesWithoutTimes]],27500];

RandomSample[trainingSet,1]

Building and training the transformer model

trainingTokens=MapAt[Characters,trainingSet,{All,{1,2}}];
trainingTokens=MapAt[Join[{"Start"},#1,{"End"}]&,trainingTokens,{All,2}];

vocabulary=Union[Flatten[Apply[List,trainingTokens,{1}]]]

We use embeddings of size 128, with 4 heads, and using vectors of size 32 for the keys, queries, and values:

n=128;
nheads=4;
l=32;

First, we need something to embed the sequences into vectors and add positional information:

embedding =
	NetGraph[
		FunctionLayer[
			Module[{emb1, emb2, posembed},
				emb1 = EmbeddingLayer[n][#Input];
				posembed = SequenceIndicesLayer[100][#Input];
				emb2 = EmbeddingLayer[n][posembed];
				emb1 + emb2
			]&
			,
			"Input" -> {"Varying", NetEncoder[{"Class", vocabulary}]}
		]
	] // NetInitialize

Let’s now implement an encoder block:

encoderblock=NetGraph[FunctionLayer[Module[{keys,queries,values,seq,seq2},keys=LinearLayer[{nheads,l}]/@#Input;queries=LinearLayer[{nheads,l}]/@#Input;queries=(queries/(Sqrt[32]));values=LinearLayer[{nheads,l}]/@#Input;seq=AttentionLayer["Dot","MultiHead"->True][keys,values,queries];seq=NetMapOperator[LinearLayer[n]][seq];seq=DropoutLayer[][seq];seq=seq+#Input;seq=NormalizationLayer[2;;All,"Same","Epsilon"->(1/(10^12))][seq];seq2=NetMapOperator[LinearLayer[n 4]][seq];seq2=Ramp[seq2];seq2=NetMapOperator[LinearLayer[n]][seq2];seq2=seq+seq2;NormalizationLayer[2;;All,"Same","Epsilon"->(1/(10^12))][seq2]]&,"Input"->{"Varying",n}]]//NetInitialize

From this we can create the complete encoder (we only use two blocks to train faster):

encoder=NetGraph[<|"emb"->embedding,"block1"->encoderblock,"block2"->encoderblock|>,{"emb"->"block1"->"block2"}]

Let’s now implement a decoder block and visualize its connectivity graph:

decoderblock=FunctionLayer[Module[{keys,queries,values,seq,seq2},keys=NetMapOperator[LinearLayer[{nheads,l}]][#Sequence];queries=NetMapOperator[LinearLayer[{nheads,l}]][#Sequence];queries=(queries/(Sqrt[32]));values=NetMapOperator[LinearLayer[{nheads,l}]][#Sequence];seq=AttentionLayer["Dot","MultiHead"->True,"Mask"->"Causal"][keys,values,queries];seq=NetMapOperator[LinearLayer[n]][seq];seq=DropoutLayer[][seq];seq=seq+#Sequence;seq=NormalizationLayer[2;;All,"Same","Epsilon"->(1/(10^12))][seq];keys=NetMapOperator[LinearLayer[{nheads,l}]][#Input];values=NetMapOperator[LinearLayer[{nheads,l}]][#Input];queries=NetMapOperator[LinearLayer[{nheads,l}]][seq];queries=(queries/(Sqrt[32]));seq2=AttentionLayer["Dot","MultiHead"->True][keys,values,queries];seq2=NetMapOperator[LinearLayer[n]][seq2];seq2=seq+seq2;seq=NormalizationLayer[2;;All,"Same","Epsilon"->(1/(10^12))][seq2];seq2=NetMapOperator[LinearLayer[n 4]][seq];seq2=Ramp[seq2];seq2=NetMapOperator[LinearLayer[n]][seq2];seq2=seq+seq2;NormalizationLayer[2;;All,"Same","Epsilon"->(1/(10^12))][seq2]]&,"Input"->{"Varying",n},"Sequence"->{"Varying",n}]//NetGraph//NetFlatten//NetInitialize;
Information[decoderblock,"SummaryGraphic"]

This block has two inputs, the first one corresponds to the sequence generated, and the second one corresponds to the encoder outputs. Let’s combine two decoder blocks and the embedding to form the complete decoder:

decoder=FunctionLayer[Module[
{emb, block1,block2,lin,proba},
emb=embedding[#Sequence];
block1=decoderblock[emb,#Input];
block2=decoderblock[block1,#Input];
lin=NetMapOperator[LinearLayer[]][block2];
proba=NetMapOperator[SoftmaxLayer[]][lin]
]&,
"Output"-> {"Varying",NetDecoder[{"Class",vocabulary}]}
]//NetGraph//NetInitialize

We can see how the decoder output is injected in each block. Finally, we need to create a training net:

trainingnet=(NetInitialize[NetGraph[#1]]&)[FunctionLayer[Module[{encode,decode,most,rest},most=Most[#Target];rest=Rest[#Target];encode=encoder[#Input];decode=decoder[most,encode];CrossEntropyLossLayer["Index"][decode,rest]]&,"Target"->NetEncoder[{"Class",vocabulary}]]]

results=NetTrain[trainingnet,trainingTokens,All,MaxTrainingRounds->30,ValidationSet->Scaled[0.1]]

To use this network, we first need to extract the encoder:

trainedEncoder=NetTake[results["TrainedNet"],"encode"]

Then, we need to extract the decoder and perform some surgery on it to only predict the next element:

trainedDecoder=NetExtract[results["TrainedNet"],"decode"];
trainedDecoder=NetFlatten[NetChain[{
NetTake[trainedDecoder,"block2"],
SequenceLastLayer[],
NetExtract[trainedDecoder,{"lin","Net"}],
SoftmaxLayer[]
},
"Sequence"->{"Varying",NetEncoder[{"Class",vocabulary}]},
"Output"->NetDecoder[{"Class",vocabulary}]
],1]

Creating the date interpreter function

trainedEncoder=Import["/Users/jofre/WRI related/ML/Neural Networks/NLP/Date-Interpreter/trainedEncoder.wlnet"]

trainedDecoder=Import["/Users/jofre/WRI related/ML/Neural Networks/NLP/Date-Interpreter/trainedDecoder.wlnet"]

dateInterpreter[string_String]:=Module[{code,prediction},
code=trainedEncoder[Characters[string]];
prediction=NestWhile[
Append[#1,trainedDecoder[Association["Sequence"->#1,"Input"->code]]]&,{"Start"},
Last[#1]=!="End"&
];
StringJoin@Rest[Most[prediction]]
];

dateInterpreter["Feb 7, 1893, 7:05:34 AM"]

dateInterpreter["Tuesday February 25, 1603 at 12:56:57 AM"]//AbsoluteTiming

FromDateString["Tuesday February 25, 1603 at 12:56:57 AM"]//AbsoluteTiming

Interpreter["DateTime"]["Tuesday February 25, 1603 at 12:56:57 AM"]//AbsoluteTiming

M2M-100 Multilingual Translation Model Trained on CCMatrix and CCAligned

Overview:

The model translates 100 languages (any language pair is allowed, no intermediate language translation).

uniquePairs[n_Integer]:=n(n-1)/2

uniquePairs[100]

uniqueDirections[n_Integer]:=n^2 -n

uniqueDirections[100]

Trained using a billions of multilingual sentences corpus (combination of CCMatrix and CCAligned datasets).

This model largely outperforms previous ones when directly translating between non-English language pairs and maintains similar performance when English is involved.

Tokenization (Note: files required are local)

spm = File[FileNameJoin[{"/Users/jofre/ownCloud/Translation Project/m2m_100/Huggingface versions/418M-Model", "tokens", "sentencepiece.bpe.model"}]];
enc = NetEncoder[{"SubwordTokens", spm , Method -> {"BPE", "WhitespacePadding" -> Left}}]

enc["Have a nice day!"]

enc["Entrepreneur"]

$languages = StringReplace[Import[FileNameJoin[{"/Users/jofre/ownCloud/Translation Project/m2m_100/Huggingface versions/418M-Model", "tokens", "special_tokens_map.json"}], "RawJSON"][["additional_special_tokens"]], "_" -> ""]

Interpreter["Language"]["af"]

Fast/light configuration (~2.5GB)

Number of heads: 16

Number of encoder blocks: 12

Number of decoder blocks: 12

fastModel=Import["/Users/jofre/ownCloud/Translation Project/m2m_100/Huggingface versions/418M-Model/May/Full-M2M-100-418M.wlnet"]

fastModel[<|"Input"->{"en","Finally, the multilanguage neural network is working well!"},
                     "Out"->{"fr",""}|>]

fastModel[<|"Input"->{"en","Finally, the multilanguage neural network is working well!"},
                     "Out"->{"fr","Enfin"}|>]

fastModel[<|"Input"->{"en","Finally, the multilanguage neural network is working well!"},
                     "Out"->{"fr","Enfin,"}|>]

fastModel[<|"Input"->{"en","Finally, the multilanguage neural network is working well!"},
                     "Out"->{"fr","Enfin, le"}|>]

fastModel[<|"Input"->{"en","Finally, the multilanguage neural network is working well!"},
                     "Out"->{"fr","Enfin, le réseau"}|>]

Creating a TextTranslation function using the model

Loading paclets (Note: internal access only)

PacletDirectoryLoad["/Users/jofre/Sources/Git/neuralnetworks"];

PacletDirectoryLoad["/Users/jofre/Sources/Git/mxnetlink"];

PacletDirectoryLoad["/Users/jofre/Sources/Git/neuralfunctions"];

<<NeuralNetworks`

<<MXNetLink`

<<NeuralFunctions`

Encoder Stack

encoderNetSmall = Import["/Users/jofre/ownCloud/Translation Project/m2m_100/Huggingface versions/418M-Model/m2mEncoderNet.wlnet"]

Decoder Stack

coreNetSmall = Import["/Users/jofre/ownCloud/Translation Project/m2m_100/Huggingface versions/418M-Model/m2mDecoderNetUnfolded.wlnet"]

Vocabulary

$vocab = NetExtract[Import["/Users/jofre/WRI related/ML/WL related/MLTextTranslate/m2mUnfoldedDecoderNet.wlnet"],{"Output","Labels"}]

Custom function using BeamSearch (internal only)

textTranslation[text_String, srcLang_Entity -> tgtLang_Entity, opts:OptionsPattern[
	{"BeamSize" -> 5}]] :=
	Module[{encoderInputs, firstStates},
		encoderInputs = encoderNetSmall[<|"Input" -> {First @ srcLang["Codes"], text}|>];
		firstStates =
			Block[{encoderInput = encoderInputs, coreNetOuptus},
				coreNetOuptus = coreNetSmall[Join[
				<|"Out" -> 3, "Index" -> 1, "Input" -> encoderInputs|>,
				Association @ Table["InState" <> ToString[i] -> {}, {i, 24}]
				]];
				Prepend[List /@ KeyMap[StringReplace["Out" -> "In"], KeyDrop[coreNetOuptus,
					 "Output"]], "Index" -> {2}]
			]; 	
		(* Updater function *) 
		updater[newtokens_List, states_Association] :=
			Block[{encoderInput = ConstantArray[encoderInputs, Length @ newtokens
				], coreNetOuptus},
				coreNetOuptus = coreNetSmall[ <|"Out" -> newtokens, KeyTake[states, "Index"
					], "Input" -> encoderInput, KeyDrop[states, "Index"]|>];
				{(* probabilities *)coreNetOuptus["Output"], Prepend[KeyMap[StringReplace[
					"Out" -> "In"], KeyDrop[coreNetOuptus, "Output"]], KeyTake[states, "Index"
					] + 1]}
			];
		StringTrim @ StringJoin @ Drop[Rest[Part[$vocab, BeamSearch[{updater,
			 firstStates}, "BestSequence", "StartOfSequenceIndex" -> Position[$vocab,
			 "__" <> First @ tgtLang["Codes"] <> "__"][[1, 1]], "EndOfSequenceIndex"
			 -> 3, "BeamSize" -> OptionValue["BeamSize"]]]], -1]
	]

textTranslation["Quelle est la racine carrée principale de -2500",(Entity["French", Entity["Language","French::367gk"], "Entity[\"Language\", \"French::367gk\"]", "language"])->(Entity["English", Entity["Language","English::385w8"], "Entity[\"Language\", \"English::385w8\"]", "language"]),"BeamSize"->1]//AbsoluteTiming

textTranslation["Quelle est la racine carrée principale de -2500",(Entity["French", Entity["Language","French::367gk"], "Entity[\"Language\", \"French::367gk\"]", "language"])->(Entity["English", Entity["Language","English::385w8"], "Entity[\"Language\", \"English::385w8\"]", "language"])]//AbsoluteTiming

Comparison with Microsoft Translation

TextTranslation["Quelle est la racine carrée principale de -2500",(Entity["French", Entity["Language","French::367gk"], "Entity[\"Language\", \"French::367gk\"]", "language"])-> (Entity["English", Entity["Language","English::385w8"], "Entity[\"Language\", \"English::385w8\"]", "language"]),Method-> "Microsoft"]//AbsoluteTiming

Decoding the sperm whale language

Classifying different coda types

Deep Machine Learning Techniques for the Detection and Classification of Sperm Whale Bioacoustics

dataset=SemanticImport["/Users/jofre/WRI related/SS/WSS21/Whale Songs/41598_2019_48909_MOESM2_ESM.xlsx"]

codas=Values@Normal@dataset[All,{"ICI1","ICI2","ICI3","ICI4","ICI5","ICI6","ICI7","ICI8","ICI9"}];

codaTypes=Normal@dataset[All,"CodaType"];

{trainingSet,testSet}=TakeDrop[RandomSample@MapThread[Rule[#1,#2]&,{codas,codaTypes}],8000];

codaTypeClassifier=Classify[trainingSet,PerformanceGoal->"Quality"]

ClassifierMeasurements[codaTypeClassifier,testSet]

Project CETI (Cetacean Translation Initiative)

Automated grammar induction and machine translation without parallel data

References

Cetacean Translation Initiative: a roadmap to deciphering the
communication of sperm whales (2021)

Compound probabilistic context-free grammars for grammar induction (2019)

Phrase-Based & Neural Unsupervised Machine Translation (2018)

Deep Machine Learning Techniques for the Detection and Classification of Sperm Whale Bioacoustics