Custom entities

Toni

Join the Conversation #WolframTechConf

Why entities?

square[area]:=square[sideLength]^2

What are entities?

Definition

Entities are references to objects in the world, physical or abstract: Inventions, inventors, mathematical objects, diseases, drugs, symptoms, events, colors, paintings, ...
Entities have properties that link them to other entities or to values: “parent astronomical body”, “shoe size”, “Lagrangian”, “genres”, “melting point”, “melting point at 200 bars”, ...
Entities can be grouped by the set of properties they share: A “text” has properties “writer”, “publication date”, ..., a “river” has a “length”, “line of coordinates”, ...

That is an entity:

Entity["type","name"]

It has head Entity, a type and an identifier.

Related objects

Examples of entities, properties and classes.

An entity:

Entity["Person","AlbertEinstein::6tb7g"]

A property:

EntityProperty["MathematicalFunction","SeriesRepresentations"]

An entity class:

EntityClass["Element","Insulator"]

A property class:

EntityPropertyClass["PhysicalSystem","QuantumMechanics"]

Built-in

Built-in entity types:

EntityValue[]//Short

All entities of a type:

EntityList["LatticeSystem"]

Properties:

EntityProperties["FoodType"]

Entity classes:

EntityClassList["WritingScript"]

Property classes:

EntityValue["Plant","PropertyClasses"]

Related functions

Does all the work: EntityValue

EntityValue[(Entity["sale of pre-sliced bread is banned", Entity["HistoricalEvent","PreSlicedBreadBanned"], "Entity[\"HistoricalEvent\", \"PreSlicedBreadBanned\"]", "historical event"]),(EntityProperty["date", EntityProperty["HistoricalEvent","Date"], "EntityProperty[\"HistoricalEvent\", \"Date\"]"])]

Many EntityValue calls can be written more shortly.
The entity operator:

(Entity["sale of pre-sliced bread is banned", Entity["HistoricalEvent","PreSlicedBreadBanned"], "Entity[\"HistoricalEvent\", \"PreSlicedBreadBanned\"]", "historical event"])[(EntityProperty["countries involved", EntityProperty["HistoricalEvent","CountriesInvolved"], "EntityProperty[\"HistoricalEvent\", \"CountriesInvolved\"]"])]

The property operator:

[(Entity["United States", Entity["Country","UnitedStates"], "Entity[\"Country\", \"UnitedStates\"]", "country"])]

Explore a domain:

RandomEntity["CurrencyDenomination"]

Access various parts of entities and properties:

EntityTypeName[(Entity["Czech 500 koruny banknote", Entity["CurrencyDenomination",{"CzechRepublic",500,"CzechKoruny","banknote"}], "Entity[\"CurrencyDenomination\", {\"CzechRepublic\", 500, \"CzechKoruny\", \"banknote\"}]", "currency denomination"])]
CanonicalName[(Entity["Czech 500 koruny banknote", Entity["CurrencyDenomination",{"CzechRepublic",500,"CzechKoruny","banknote"}], "Entity[\"CurrencyDenomination\", {\"CzechRepublic\", 500, \"CzechKoruny\", \"banknote\"}]", "currency denomination"])]
CommonName[(Entity["Czech 500 koruny banknote", Entity["CurrencyDenomination",{"CzechRepublic",500,"CzechKoruny","banknote"}], "Entity[\"CurrencyDenomination\", {\"CzechRepublic\", 500, \"CzechKoruny\", \"banknote\"}]", "currency denomination"])]

Outline

- part 1: basics
- part 2: advanced stuff
- part 3: very advanced stuff
- part 4: recommendations

Part 1: Basics

- add entities
- the entity store
- entity classes
- default function
- property classes
- formatting
- endless store

Add entities

No setup needed, just do it:

Entity["myBook","Faust1"]["Author"]="Goethe";

This introduces the entity and property:

EntityList["myBook"]

EntityProperties["myBook"]

The value is there, of course:

(Entity["Faust1", Entity["myBook","Faust1"], "Entity[\"myBook\", \"Faust1\"]", "myBook"])[(EntityProperty["Author", EntityProperty["myBook","Author"], "EntityProperty[\"myBook\", \"Author\"]"])]

Where is it?

$EntityStores

The entity store

Create and register and entity store.

Create

Books an authors:

store=EntityStore["myBook"-><|
"Label" -> "my book",

"Entities"-><|
"Faust1"-><|
"Label"->"Faust. Eine Tragödie.",
"Author"->(Entity["Johann Wolfgang von Goethe", Entity["Person","Goethe::89m93"], "Entity[\"Person\", \"Goethe::89m93\"]", "person"])
|>,
"Faust2"-><|
"Label"->"Faust. Der Tragödie zweiter Teil",
"Author"->(Entity["Johann Wolfgang von Goethe", Entity["Person","Goethe::89m93"], "Entity[\"Person\", \"Goethe::89m93\"]", "person"])
|>,
"PetitPrince"-><|
"Label"->"Le Petit Prince",
"Author"->(Entity["Antoine de Saint\[Hyphen]Exupéry", Entity["Person","AntoineDeSaint-Exupery::9493y"], "Entity[\"Person\", \"AntoineDeSaint-Exupery::9493y\"]", "person"])
|>
|>
|>];

Register

Make it the only active store:

$EntityStores={store};

EntityList["myBook"]

Note the improved gray text?

(Entity["Faust. Eine Tragödie.", Entity["myBook","Faust1"], "Entity[\"myBook\", \"Faust1\"]", "my book"])["Author"]

All my books by Goethe? Implicitly defined entity classes.

EntityClass["myBook","Author"->(Entity["Johann Wolfgang von Goethe", Entity["Person","Goethe::89m93"], "Entity[\"Person\", \"Goethe::89m93\"]", "person"])]

Well, that did not work. Did it?
Entity classes represent a lists of entities. So what are the entities?

EntityList[%]

Entity classes

Named and implicitly defined entity classes.

Fruit store

A fruit store:

$EntityStores={EntityStore["fruit"-><|
"Entities"-><|
"chirimoya"-><|"Image"->,"Volume"->(Quantity[1, "L", liters, "Liters"])(* wild guess *)|>,
"strawberry"-><|"Image"->,"Volume"->(Quantity[60, "mL", milliliters, "Milliliters"])|>,
"raspberry"-><|"Image"->,"Volume"->(Quantity[50, "mL", milliliters, "Milliliters"])|>,
"banana"-><|"Image"->,"Volume"->(Quantity[350, "mL", milliliters, "Milliliters"])|>
|>,

"EntityClasses"-><|
"favorite"-><|"Entities"->{"chirimoya","strawberry","raspberry"}|>,
"small"-><|"Entities"->{"Volume"->LessThan[(Quantity[600, "mL", milliliters, "Milliliters"])]}|>
|>
|>]};

All the named entity classes:

EntityClassList["fruit"]

My favorites:

EntityList[(EntityClass["favorite", EntityClass["fruit","favorite"], "EntityClass[\"fruit\", \"favorite\"]", "fruit"])]

With pictures:

ImageCollage[Rule@@@(EntityClass["favorite", EntityClass["fruit","favorite"], "EntityClass[\"fruit\", \"favorite\"]", "fruit"])[{"Volume","Image"}]]

The members are computed on the fly:

(EntityClass["small", EntityClass["fruit","small"], "EntityClass[\"fruit\", \"small\"]", "fruit"])["Image","EntityAssociation"]

Small

Is a banana small?

MemberQ[
EntityList[EntityClass["fruit","Volume"->LessThan[(Quantity[600, "mL", milliliters, "Milliliters"])]]],
(Entity["banana", Entity["fruit","banana"], "Entity[\"fruit\", \"banana\"]", "fruit"])
]

Fun fact: This is not the most efficient or elegant way to formulate this question in the WL.
Meet the Boolean property:

EntityProperty["fruit","Volume"->LessThan[(Quantity[600, "mL", milliliters, "Milliliters"])]]@(Entity["banana", Entity["fruit","banana"], "Entity[\"fruit\", \"banana\"]", "fruit"])

Or use a dedicated function:

EntityExistsQ[(EntityClass["small", EntityClass["fruit","small"], "EntityClass[\"fruit\", \"small\"]", "fruit"]),(Entity["banana", Entity["fruit","banana"], "Entity[\"fruit\", \"banana\"]", "fruit"])]

Red and yellow fruit

Find red fruit:

EntityClass[
"fruit",
"Image"->(Min[ColorDistance[DominantColors[#],Red]]<0.5&)
]//EntityList

Yellow fruit:

EntityClass[
"fruit",
"Image"->(Min[ColorDistance[DominantColors[#],Yellow]]<0.5&)
]//EntityList

Default function

Wheels.

Explicit values

Start with no wheels:

$EntityStores={EntityStore["wheel"]};

Invent three small wheels, in different sizes:

Entity["wheel","small"][{"Diameter","Radius"}]={(Quantity[1, "m", meters, "Meters"]),(Quantity[0.5, "m", meters, "Meters"])};
Entity["wheel","smaller"][{"Diameter","Radius"}]={(Quantity[0.8, "m", meters, "Meters"]),(Quantity[0.4, "m", meters, "Meters"])};
Entity["wheel","smallest"][{"Diameter","Radius"}]={(Quantity[0.6, "m", meters, "Meters"]),(Quantity[0.2, "m", meters, "Meters"])};

Review all data:

EntityClass["wheel",All]["Dataset"]

Check consistency:

%[All,#[(EntityProperty["Diameter", EntityProperty["wheel","Diameter"], "EntityProperty[\"wheel\", \"Diameter\"]"])]==2#[(EntityProperty["Radius", EntityProperty["wheel","Radius"], "EntityProperty[\"wheel\", \"Radius\"]"])]&]

Built-in consistency?

Derived values

Start again with no wheels:

$EntityStores={EntityStore["wheel"]};

Just specify the diameter:

Entity["wheel","small"]["Diameter"]=(Quantity[1, "m", meters, "Meters"]);
Entity["wheel","smaller"]["Diameter"]=(Quantity[0.8, "m", meters, "Meters"]);
Entity["wheel","smallest"]["Diameter"]=(Quantity[0.6, "m", meters, "Meters"]);

Compute the radius:

EntityProperty["wheel","Radius"]["DefaultFunction"]=Function[entity,entity["Diameter"]/2];

Check again:

EntityClass["wheel",All]["Dataset"]

Property classes

Same idea as entity classes.

Set up

$EntityStores={EntityStore["physicalSystem"-><|
"Entities"-><|
"harmonicOscillator"-><|
"EquationsOfMotion"->x[t]+x''[t],
"HamiltonianOperator"->Function[1/2x[x[#]]+1/2p[p[#]]],
"Lagrangian"->-1/2x[t]^2+1/2m x'[t]^2,
"SchroedingerEquation"->1/2x^2ψ[x]-1/2ψ''[x]==e ψ[x]
|>
|>,

"Properties"-><|
"HamiltonianOperator"-><|"PhysicalQuantity"->(QuantityVariable["Energy", "Energy"])|>,
"Lagrangian"-><|"PhysicalQuantity"->(QuantityVariable["Energy", "Energy"])|>
|>,

"PropertyClasses"-><|
"Classical"-><|
"Label"->"classical description",
"Properties"->{"Lagrangian","EquationsOfMotion"}
|>,
"Quantum"-><|
"Label"->"quantum mechanical description",
"Properties"->{"HamiltonianOperator","SchroedingerEquation"}
|>
|>
|>]};

Use

All property classes:

EntityValue["physicalSystem","PropertyClasses"]

Different descriptions of the same physical system:

(EntityPropertyClass["classical description", EntityPropertyClass["physicalSystem","Classical"], "EntityPropertyClass[\"physicalSystem\", \"Classical\"]"])[(Entity["harmonicOscillator", Entity["physicalSystem","harmonicOscillator"], "Entity[\"physicalSystem\", \"harmonicOscillator\"]", "physicalSystem"]),"PropertyAssociation"]

Different syntax:

(Entity["harmonicOscillator", Entity["physicalSystem","harmonicOscillator"], "Entity[\"physicalSystem\", \"harmonicOscillator\"]", "physicalSystem"])[(EntityPropertyClass["quantum mechanical description", EntityPropertyClass["physicalSystem","Quantum"], "EntityPropertyClass[\"physicalSystem\", \"Quantum\"]"]),"PropertyAssociation"]

Filter properties by physical quantity:

EntityPropertyClass["physicalSystem","PhysicalQuantity"->EqualTo[(QuantityVariable["Energy", "Energy"])]]//EntityProperties

Formatting

Using the formatting function.

Formatting?

Store units only once:

Entity["t","e1"]["Length"]=10;
Entity["t","e2"]["Length"]=12;

EntityProperty["t","Length"]["FormattingFunction"]=Quantity[#,"Meters"]&;

EntityClass["t",All]["Length","EntityAssociation"]

Store entity type only once:

Entity["t","e1"]["Parent"]="e2";

EntityProperty["t","Parent"]["FormattingFunction"]=Entity["t",#]&;

Entity["t","e1"]["Parent"]

Color

Text, randomly colored:

$EntityStores={EntityStore["text"-><|
"Entities"-><|
"a"-><|"p"->"some words"|>,
"b"-><||>
|>,

"Properties"-><|
"p"-><|"FormattingFunction"->Function[value,Style[value,RandomColor[RGBColor[1,_,_]]]]|>
|>
|>]};

Check:

Entity["text","a"]["p"]

Only non-missing values are colored:

Entity["text","b"]["p"]

Function is called once per non-missing values:

EntityValue[{Entity["text","a"],Entity["text","b"]},"p"]

Yawning

- Andrew C. Gallup, Allyson M. Church and Anthony J. Pelegrino: Yawn duration predicts brain weight and cortical neuron number in mammals.
- Gerhard Roth and Ursula Dicke: Evolution of the brain and intelligence.

Data

Mass, brain mass, and average yawn duration for some species:

data=\[LeftAssociation]
"AfricanElephant"->\[LeftAssociation]
"BrainMass"->5712,
"Mass"->Quantity[Interval[{2160,6048}],"Kilograms"],
"Species"->(Entity["African elephants", Entity["Species","Genus:Loxodonta"], "Entity[\"Species\", \"Genus:Loxodonta\"]", "species specification"]),
"YawnDurations"->{7.06}
\[RightAssociation],
"Camel"->\[LeftAssociation]
"BrainMass"->762,
"Mass"->Quantity[Interval[{300,1000}],"Kilograms"],
"Species"->(Entity["Camelus", Entity["Species","Genus:Camelus"], "Entity[\"Species\", \"Genus:Camelus\"]", "species specification"]),
"YawnDurations"->{4.2,3.91,7.18,5.97,3.57}
\[RightAssociation],
"CapuchinMonkey"->\[LeftAssociation]
"BrainMass"->Interval[{35,40}],
"Mass"->Quantity[Interval[{3,9}],"Pounds"],
"Species"->(Entity["capuchin monkey", Entity["Species","Genus:Cebus"], "Entity[\"Species\", \"Genus:Cebus\"]", "species specification"]),
"YawnDurations"->{2.36}
\[RightAssociation],
"Cat"->\[LeftAssociation]
"BrainMass"->25.6,
"Species"->(Entity["domestic cat", Entity["Species","Species:FelisCatus"], "Entity[\"Species\", \"Species:FelisCatus\"]", "species specification"]),
"YawnDurations"->{2.41,1.55,2.43,2.49,1.9,1.56,2.68,1.56,1.52}
\[RightAssociation],
"Chimpanzee"->\[LeftAssociation]
"BrainMass"->440,
"Species"->(Entity["common chimpanzee", Entity["Species","Species:PanTroglodytes"], "Entity[\"Species\", \"Species:PanTroglodytes\"]", "species specification"]),
"YawnDurations"->{3.9,4.2,3.55,4.28,10.83,2.88,5.06,7.02,3.5}
\[RightAssociation],
"Dog"->\[LeftAssociation]
"BrainMass"->72,
"Mass"->Interval[{Quantity[113,"Grams"],Quantity[155.6,"Kilograms"]}],
"Species"->(Entity["dog", Entity["Species","Infraspecies:CanisLupusFamiliaris"], "Entity[\"Species\", \"Infraspecies:CanisLupusFamiliaris\"]", "species specification"]),
"YawnDurations"->{2.11,1.98,2.51,2.28,3.37,4.26,2.65,2.73,1.81,1.78,1.75,1.51}
\[RightAssociation],
"Fox"->\[LeftAssociation]
"BrainMass"->53,
"Species"->(Entity["red fox", Entity["Species","Species:VulpesVulpes"], "Entity[\"Species\", \"Species:VulpesVulpes\"]", "species specification"]),
"YawnDurations"->{1.69,2.51,1.86,1.9,1.39,1.78,1.64,1.55,1.42,1.38}
\[RightAssociation],
"Gibbon"->\[LeftAssociation]
"BrainMass"->Interval[{88,105}],
"Mass"->Quantity[Interval[{4,13}],"Kilograms"],
"Species"->(Entity["gibbons", Entity["Species","Family:Hylobatidae"], "Entity[\"Species\", \"Family:Hylobatidae\"]", "species specification"]),
"YawnDurations"->{4.23,4.17,5.06,3.92,4.36,1.56}
\[RightAssociation],
"Gorilla"->\[LeftAssociation]
"BrainMass"->406,
"Species"->(Entity["Gorilla gorilla", Entity["Species","Species:GorillaGorilla"], "Entity[\"Species\", \"Species:GorillaGorilla\"]", "species specification"]),
"YawnDurations"->{2.06,5.15,3.06,3.51,3.51,5.83,4.08}
\[RightAssociation],
"Hedgehog"->\[LeftAssociation]
"BrainMass"->3.3,
"Species"->(Entity["west european hedgehog", Entity["Species","Species:ErinaceusEuropaeus"], "Entity[\"Species\", \"Species:ErinaceusEuropaeus\"]", "species specification"]),
"YawnDurations"->{2.58,2.67,2.49,2.18,2.71,1.98,2.8,2.41,2.5,3.17,1.92,2.01}
\[RightAssociation],
"Horse"->\[LeftAssociation]
"BrainMass"->532,
"Species"->(Entity["horse (domestic)", Entity["Species","Species:EquusCaballus"], "Entity[\"Species\", \"Species:EquusCaballus\"]", "species specification"]),
"YawnDurations"->{2.71,3.72,7.14,2.6,2.88,2.86,2.51}
\[RightAssociation],
"Human"->\[LeftAssociation]
"BrainMass"->1320,
"Species"->(Entity["human", Entity["Species","Species:HomoSapiens"], "Entity[\"Species\", \"Species:HomoSapiens\"]", "species specification"]),
"YawnDurations"->{6.8,5.55,10.03,4.46,8.89,12.93,5.96,5.25,4.23,4.25,4.68,4.83,5.93,3.48,5.65,3.83,5.05,3.81,4.52,6.6,4.8,4.03,3.31,5.05,2.55,3.85,4.18}
\[RightAssociation],
"Lion"->\[LeftAssociation]
"BrainMass"->240,
"Species"->(Entity["lion", Entity["Species","Species:PantheraLeo"], "Entity[\"Species\", \"Species:PantheraLeo\"]", "species specification"]),
"YawnDurations"->{3.33,3.23,2.43,4.54,2.,2.39,3.6}
\[RightAssociation],
"Marmoset"->\[LeftAssociation]
"BrainMass"->7,
"Mass"->Quantity[Interval[{236,256}],"Grams"],
"Species"->(Entity["marmosets, tamarins", Entity["Species","Family:Callitrichidae"], "Entity[\"Species\", \"Family:Callitrichidae\"]", "species specification"]),
"YawnDurations"->{1.61,1.96,2.36}
\[RightAssociation],
"Mouse"->\[LeftAssociation]
"BrainMass"->0.4,
"Mass"->Quantity[Interval[{18,35}],"Grams"],
"Species"->(Entity["mice", Entity["Species","Genus:Mus"], "Entity[\"Species\", \"Genus:Mus\"]", "species specification"]),
"YawnDurations"->{0.92,0.9,0.75,0.67,0.81,0.52}
\[RightAssociation],
"Opossum"->\[LeftAssociation]
"BrainMass"->6,
"Mass"->Quantity[Interval[{0.8,6.4}],"Kilograms"],
"Species"->(Entity["opossums", Entity["Species","Family:Didelphidae"], "Entity[\"Species\", \"Family:Didelphidae\"]", "species specification"]),
"YawnDurations"->{1.41,1.84,2.51,2.15,2.56,1.78,3.21,1.78,2.73}
\[RightAssociation],
"Rabbit"->\[LeftAssociation]
"BrainMass"->12.1,
"Mass"->Quantity[Interval[{0.4,2}],"Kilograms"],
"Species"->(Entity["rabbits and hares", Entity["Species","Family:Leporidae"], "Entity[\"Species\", \"Family:Leporidae\"]", "species specification"]),
"YawnDurations"->{1.2,1.6,1.13,1.05,1.05,1.5,2.26,1.73,1.71,1.55,1.46,1.71}
\[RightAssociation],
"Rat"->\[LeftAssociation]
"BrainMass"->2,
"Mass"->Quantity[Interval[{3.9,12}],"Ounces"],
"Species"->(Entity["rat", Entity["Species","Genus:Rattus"], "Entity[\"Species\", \"Genus:Rattus\"]", "species specification"]),
"YawnDurations"->{1.21,0.89,1.4,1.06,1.22,1.85,1.36,1.25,1.75,1.21,1.13,1.67,1.1}
\[RightAssociation],
"RhesusMonkey"->\[LeftAssociation]
"BrainMass"->179,
"Species"->(Entity["rhesus monkey", Entity["Species","Species:MacacaMulatta"], "Entity[\"Species\", \"Species:MacacaMulatta\"]", "species specification"]),
"YawnDurations"->{1.61,4.48}
\[RightAssociation],
"Sheep"->\[LeftAssociation]
"BrainMass"->175,
"Species"->(Entity["sheep", Entity["Species","Species:OvisAries"], "Entity[\"Species\", \"Species:OvisAries\"]", "species specification"]),
"YawnDurations"->{1.95,3.76,1.93}
\[RightAssociation],
"Squirrel"->\[LeftAssociation]
"BrainMass"->7.6,
"Mass"->Interval[{Quantity[10,"Grams"],Quantity[8,"Kilograms"]}],
"Species"->(Entity["Sciurus", Entity["Species","Genus:Sciurus"], "Entity[\"Species\", \"Genus:Sciurus\"]", "species specification"]),
"YawnDurations"->{1.75,1.12,1.68,1.76,1.71,3.18,2.36,1.68}
\[RightAssociation],
"SquirrelMonkey"->\[LeftAssociation]
"BrainMass"->22,
"Species"->(Entity["common squirrel monkey", Entity["Species","Species:SaimiriSciureus"], "Entity[\"Species\", \"Species:SaimiriSciureus\"]", "species specification"]),
"YawnDurations"->{2.2}
\[RightAssociation],
"Walrus"->\[LeftAssociation]
"BrainMass"->1130,
"Species"->(Entity["walrus", Entity["Species","Species:OdobenusRosmarus"], "Entity[\"Species\", \"Species:OdobenusRosmarus\"]", "species specification"]),
"YawnDurations"->{5.53,3.91,3.88,5.62}
\[RightAssociation],
"WhiteCapuchin"->\[LeftAssociation]
"BrainMass"->Interval[{26,80}],
"Species"->(Entity["white-headed capuchin", Entity["Species","Species:CebusCapucinus"], "Entity[\"Species\", \"Species:CebusCapucinus\"]", "species specification"]),
"YawnDurations"->{2.28,3.09,2.37}
\[RightAssociation]
\[RightAssociation];

Store

Compute average yawn duration, “clean up” brain mass, specify duration are in seconds.

$EntityStores={EntityStore["species"-><|
"Entities"->data,

"Properties"-><|
"AverageYawnDuration"-><|
"Label"->"average yawn duration",
"DefaultFunction"->(#["YawnDurations"]&)/*Mean
|>,
"BrainMass"-><|
"Label"->"brain mass",
"FormattingFunction"->(Quantity[#,"Grams"]&)@*Replace[i_Interval:>Mean[MinMax[i]]]
|>,
"Mass"-><|
"Label"->"mass",
"DefaultFunction"->EntityPropertySequence["Species","Weight"],
"FormattingFunction"->ReplaceAll[i_Interval:>Mean[MinMax[i]]]
|>,
"YawnDurations"-><|
"Label"->"yawn durations",
"FormattingFunction"->(Quantity[#,"Seconds"]&)
|>
|>
|>]};

Work with the properties

Yawn durations, as Quantity[...] objects:

Entity["species","Squirrel"]["YawnDurations"]

Entity["species","Squirrel"]["AverageYawnDuration"]

Aligned to “Species” entities:

Entity["species","Lion"]["Species"]

Entity["species","Lion"]["Species"]["Image"]

EntityClass["species",All][EntityPropertySequence["Species","Image"]]//ImageCollage

Mass, not available for all built-in “Species”:

EntityClass["species",All][EntityPropertySequence["Species","Weight"]]

Use the “hybrid” property “Mass”: Contains some explicit values, and retrieves the missing values from the corresponding “Species” entities.
Also get rid of the Interval.

EntityClass["species",All]["Mass"]

Insights

Does the brain size depend on the yawn duration?

ListLogLinearPlot[
EntityClass["species",All][{
"BrainMass",
"AverageYawnDuration"
}],
PlotRange->All,
AxesLabel->Automatic
]

Using callout:

ListLogLinearPlot[
EntityClass["species",All][{
"BrainMass",
"AverageYawnDuration"
},"EntityAssociation"]//KeyValueMap[Callout/*Reverse],
PlotRange->All,
AxesLabel->Automatic
]

Or does the yawn duration just depend on the size?

ListLogLinearPlot[
EntityClass["species",All][{
"Mass",
"AverageYawnDuration"
}],
AxesLabel->Automatic
]

Endless store

Using the entity validation function.

Entities

A store of all prime numbers:

$EntityStores={EntityStore["prime"-><|
"EntityValidationFunction"->Interpreter["Integer"]/*PrimeQ
|>]};

Valid entities format nicely:

Entity["prime","x"]

Entity["prime","12"]

Entity["prime","13"]

Computed properties

Explicit values are considered before the “DefaultFunction” is applied.

$EntityStores={EntityStore["prime"-><|
"EntityValidationFunction"->Interpreter["Integer"]/*PrimeQ,

"Properties"-><|
(* All primes are odd... *)
"Odd"-><|"Label"->"odd?","DefaultFunction"->Function[True]|>
|>,

"Entities"-><|
(* ...with one exception: *)
"2"-><|"Odd"->False|>
|>
|>]};

(EntityProperty["odd?", EntityProperty["prime","Odd"], "EntityProperty[\"prime\", \"Odd\"]"])[{(Entity["2", Entity["prime","2"], "Entity[\"prime\", \"2\"]", "prime"]),(Entity["3", Entity["prime","3"], "Entity[\"prime\", \"3\"]", "prime"]),(Entity["5", Entity["prime","5"], "Entity[\"prime\", \"5\"]", "prime"])}]

Part 2: Advanced stuff

- qualifiers
- query preprocessing
- localization
- curation using forms
- dictionary

Qualifiers

Many ways to set up qualifiers.

What are qualifiers?

- “melting point” (property) at “atmospheric pressure” (qualifier)
- “population” (property) in “1900” (qualifier)
- “hours” (property) on “Mondays” (qualifier)

This is a qualified property:

EntityProperty["AnatomicalStructure","Mass",{"StatisticalGroup"->"Female"}]

Examples:

(Entity["heart", Entity["AnatomicalStructure","Heart"], "Entity[\"AnatomicalStructure\", \"Heart\"]", "anatomical structure"])[]

(Entity["heart", Entity["AnatomicalStructure","Heart"], "Entity[\"AnatomicalStructure\", \"Heart\"]", "anatomical structure"])[]

Even shorter notation:

(Entity["Mars", Entity["Planet","Mars"], "Entity[\"Planet\", \"Mars\"]", "planet"])["HelioCoordinates","Date"->Yesterday]

(Entity["Mars", Entity["Planet","Mars"], "Entity[\"Planet\", \"Mars\"]", "planet"])["HelioCoordinates","Date"->Today]

Set up: entity store

Declare everything in the entity store.

$EntityStores={EntityStore["t"-><|
(* explicit values *)
"Entities"-><|
"e1"-><|{"letter","part"->All}->{"a","b","c"}|>,
"e2"-><|{"letter","part"->All}->{"x","y","z"}|>
|>,

(* computed values *)
"Properties"-><|
"letter"-><|"DefaultFunction"->Function[e,Last[e["letter","part"->All]]]|>,
{"letter","part"->_Integer}-><|"DefaultFunction"->Function[{e,q},e["letter","part"->All][[q["part"]]]]|>
|>
|>]};

Last letter:

Entity["t","e2"]["letter"]

Second letter:

Entity["t","e2"]["letter","part"->2]

All letters:

Entity["t","e2"]["letter","part"->All]

Multiple qualifiers? Yes.
{"property",{"qual1"->patt1,"qual2"->patt2,…}}->value
Order of qualifiers does not matter.

Set up: assignments

Use (delayed) assignments.

$EntityStores={EntityStore["t"]};

Explicit values:

Entity["t","e1"]["letter","part"->All]:={"a","b","c"};
Entity["t","e2"]["letter","part"->All]:={"x","y","z"};

Compute unqualified property:

EntityProperty["t","letter"]["DefaultFunction"]=Function[ent,ent["letter","part"->All]//Last];

Compute integer-part qualified property:

EntityProperty["t","letter","part"->_Integer]["DefaultFunction"]=Function[{ent,qual},
ent["letter","part"->All][[qual["part"]]]
];

Same as before:

Entity["t","e2"]["letter"]

Entity["t","e2"]["letter","part"->2]

Entity["t","e2"]["letter","part"->All]

Query preprocessing

How to allow more convenient input and variations in the predicates of implicitly defined entity classes:

EntityClass["book","ISBN"->... (* <- here *)]

Normalization of identifiers

Some nice books:

$EntityStores={EntityStore["book"-><|
"Entities"-><|
"book1"-><|"ISBN"->"1234"|>,
"book2"-><|"ISBN"->"3456"|>,
"book3"-><|"ISBN"->"5678"|>
|>
|>]};

No result because of hyphen:

EntityClass["book","ISBN"->"12-34"]//EntityList

Set up some preprocessing:

EntityProperty["book","ISBN"]["QueryPreprocessingFunction"]=Replace[{
s_String:>StringReplace[s,Except[DigitCharacter]->""],
i_Image:>BarcodeRecognize[i]
}];

Now try again:

EntityClass["book","ISBN"->"12-34"]//EntityList

Fancy code:

EntityClass["book","ISBN"->]//EntityList

Natural language input

Define a grammar

parsePredicate=CloudDeploy[GrammarRules[
{
(* GreaterThan[3] *)
FixedOrder[op:GrammarToken["Predicate"],n:GrammarToken["Number"]]:>op[n],

(* Between[{1, 2}] *)
FixedOrder["between",min:GrammarToken["Number"],"and",max:GrammarToken["Number"]]:>Between[{min,max}]
},
{
"Predicate"->{
FixedOrder["greater",OptionalElement["than"]]->GreaterThan,
FixedOrder["equal",OptionalElement["to"]]->EqualTo
}
}
]];

Test the grammar:

GrammarApply[parsePredicate,"equal to 2"]

GrammarApply[parsePredicate,"greater than 3"]

Set up the entities

$EntityStores={EntityStore["t"-><|
(* data *)
"Entities"-><|
"e1"-><|"p1"->1,"p2"->14|>,
"e2"-><|"p1"->2,"p2"->11|>,
"e3"-><|"p1"->3,"p2"->14|>,
"e4"-><|"p1"->4,"p2"->12|>,
"e5"-><|"p1"->5,"p2"->14|>
|>,

(* properties *)
"Properties"-><|
"p1"-><|"QueryPreprocessingFunction"->Replace[s_String:>GrammarApply[parsePredicate,s]]|>,
"p2"-><|"QueryPreprocessingFunction"->Replace[s_String:>GrammarApply[parsePredicate,s]]|>
|>
|>]};

Test

One property:

EntityClass["t","p1"->"greater than 3"]//EntityList

Two properties:

EntityClass["t",{
"p1"->"between 2 and 7",
"p2"->"equal to 14"
}]["Dataset"]

Predicate selection

Property values that are lists:

$EntityStores={EntityStore["t"-><|
"Entities"-><|
"e1"-><|"p1"->{"a","b","c"}|>,
"e2"-><|"p1"->{"b","c"}|>,
"e3"-><|"p1"->{"a","d"}|>
|>
|>]};

This is empty because there is no value that is equal to “a”:

EntityClass["t","p1"->"a"]//EntityList

Redefine the meaning of a string in the context of this property:

EntityProperty["t","p1"]["QueryPreprocessingFunction"]=Replace[s_String:>ContainsAny[{s}]];

Try again:

EntityClass["t","p1"->"a"]//EntityList

Wolfram Alpha uses this convenient default interpretation for properties that are known to have list values.

EntityClass["Language","WritingScripts"->(Entity["Arabic", Entity["WritingScript","Arabic::3y92v"], "Entity[\"WritingScript\", \"Arabic::3y92v\"]", "writing script"])]//EntityList

Localization

Using qualifiers.

Simple example

$EntityStores={EntityStore["thing"-><|
"Entities"-><|
"tree"-><|
{"Label","Language"->"English"}->"tree",
{"Label","Language"->"German"}->"Baum"
|>,

"towel"-><|
{"Label","Language"->"English"}->"towel",
{"Label","Language"->"German"}->"Handtuch"
|>
|>
|>]};

Language settings are “English”:

EntityList["thing"]

Change them to “German”:

CurrentValue[EvaluationNotebook[],Language]="German";

EntityList["thing"]

(change it back)

CurrentValue[EvaluationNotebook[],Language]="English";

Interesting example

$EntityStores={EntityStore[{
(* things *)
"thing"-><|
"Entities"-><|
"tree"-><|
{"Label","Language"->"English"}->"a tree",
{"Label","Language"->"German"}->"ein Baum",
"color"->Entity["color","green"]
|>,

"towel"-><|
{"Label","Language"->"English"}->"a towel",
{"Label","Language"->"German"}->"ein Handtuch",
"color"->Entity["color","red"]
|>
|>,

"Properties"-><|
"color"-><|
{"Label","Language"->"English"}->"has color",
{"Label","Language"->"German"}->"hat die Farbe"
|>
|>
|>,

(* colors *)
"color"-><|
"Entities"-><|
"green"-><|
{"Label","Language"->"English"}->"green",
{"Label","Language"->"German"}->"Grün"
|>,
"red"-><|
{"Label","Language"->"English"}->"red",
{"Label","Language"->"German"}->"Rot"
|>
|>
|>
}]};

Subject, predicate, object.
In English:

{Entity["thing","tree"],EntityProperty["thing","color"],Entity["thing","tree"]["color"]}

Auf Deutsch (in German):

CurrentValue[EvaluationNotebook[],Language]="German";

{Entity["thing","tree"],EntityProperty["thing","color"],Entity["thing","tree"]["color"]}

CurrentValue[EvaluationNotebook[],Language]="English";

Applications:
- present data in the users preferred language
- natural language processing
- find entities and properties in texts of different language
- generate text in different languages

Curation using forms

- nice user interface
 -> encourage contribution
- validate new data
 -> immediate feedback, faster iteration
- review existing data

Add entities

Require label when adding entities.

store=EntityStore["t"-><|
(* data *)
"Entities"-><|
"e1"-><|"Label"->"some E"|>,
"e2"-><|"Label"->"other E"|>
|>,

(* form *)
"AddEntitiesForm"->FormFunction[
{
"CanonicalName"->"String",
"Label"->"String"
},
(Entity["t",#CanonicalName]["Label"]=#Label)&
]
|>];

The store can’t modify itself. Register it:

$EntityStores={store};

Entity["t"]["AddEntitiesForm"][];

EntityList["t"]

Add data

Workflow.

Initial data

Fungi.

data=<|
"Chanterelle"-><|
"Label"->"chanterelle",
"AlternateLabels"->{"cantharellus cibarius","girolle"},
"Edible"->False,(* wrong *)
"Image"->
|>,
"FlyAgaric"-><|
"Label"->"fly agaric",
"Image"->
|>,
"Porcini"-><|
"Label"->"porcini",
"AlternateLabels"->{"porcino","cep","penny bun","boletus edulis"}
|>
|>;

Store

Data, schema and forms.

store=EntityStore["fungus"-><|
(* data *)
"Entities"->data,

(* schema *)
"Properties"-><|
"Label"-><|
"Label"->"label",
"Interpreter"->"String",
"Required"->True
|>,
"AlternateLabels"-><|
"Label"->"alternate labels",
"Interpreter"->RepeatingElement["String",{0,10}]
|>,
"Edible"-><|
"Label"->"edible?",
"Interpreter"->"Boolean"
|>,
"Image"-><|
"Label"->"image",
"Interpreter"->"Image"
|>
|>,

(* form *)
"AddDataForm"->FormFunction[
{
(* first find an entity to update, looking at label and alternate labels *)
{"Entity"->AutoSubmitting[First[
EntityList[EntityClass["fungus","Label"->#]],
First[
EntityList[EntityClass["fungus","AlternateLabels"->ContainsAny[{#}]]],
Failure[
"InterpretationFailure",
<|"MessageTemplate"->"No entity found."|>
]
]
]&]},

(* then present a form filled with data about this entity *)
(Function[p,
CanonicalName[p]-><|
"Interpreter"->p["Interpreter"],
"Required"->TrueQ[p["Required"]],
"Input"->#Entity[p]
|>
]/@EntityProperties["fungus"])&
},

(* insert valid data *)
KeyValueMap[
Function[{p,v},#Entity[p]=v],
KeyDrop[#,"Entity"]
]&
]
|>];

New data

Register store.

$EntityStores={store};

Review “girolle” (which is actually edible):

Entity["fungus"]["AddDataForm"][];

Add missing data for “fly agaric”:
- alternate labels: {"fly amanita","amanita muscaria"}
- not edible

Entity["fungus"]["AddDataForm"][];

Add data programmatically, using the form for validation:

Entity["fungus"]["AddDataForm"][<|

(* find entity *)
"Entity"->"porcini"
|>][<|

(* add data (if the entity was found) *)
"Label"->"porcini",
"AlternateLabels"->{"porcino","cep","penny bun","boletus edulis"},
"Edible"->"yes",
"Image"->
|>];

Review all data:

EntityClass["fungus",All]["Dataset"]

Dictionary

A dictionary.
Using language tagged labels.

Import utilities

Import Wikidata items with their labels.

http://tools.wmflabs.org/wikidata-exports/rdf/

$file = FileNameJoin[{$HomeDirectory,"Downloads","wikidata-terms.nt"}];

Look at the file:

ReadLine[$file]

Ah, NTriples.
Simple, line based serialization for RDF graphs.

Read line by line, exploiting listability of StringCases[...] and unescape[...] for performance:

getLabels[file_, langs_, lines : Repeated[_Integer, {0, 1}]] := Module[
{res},
res=StringCases[
ReadList[$file,String,lines],
StartOfString~~"<http://www.wikidata.org/entity/"~~
Shortest[q__]~~
"> <http://www.w3.org/2000/01/rdf-schema#label> \""~~
label__~~"\"@"~~lang:langs~~" .":>
{q,label,lang},
1
]//Cases[{x_}:>x];
res[[All,2]]=unescape[res[[All,2]]];
res
]

Undo character escapes:

$unescapeRules = {
(* https://www.w3.org/TR/turtle/#numeric *)
"\\u" ~~ code : RegularExpression["[[:xdigit:]]{4}"] :> FromCharacterCode[FromDigits[code, 16], "Unicode"],
"\\U" ~~ code : RegularExpression["[[:xdigit:]]{8}"] :> FromCharacterCode[FromDigits[code, 16], "Unicode"],

(* https://www.w3.org/TR/turtle/#string *)
"\\t" -> FromCharacterCode[FromDigits["0009", 16], "Unicode"],
"\\b" -> FromCharacterCode[FromDigits["0008", 16], "Unicode"],
"\\n" -> FromCharacterCode[FromDigits["000A", 16], "Unicode"],
"\\r" -> FromCharacterCode[FromDigits["000D", 16], "Unicode"],
"\\f" -> FromCharacterCode[FromDigits["000C", 16], "Unicode"],
"\\\"" -> FromCharacterCode[FromDigits["0022", 16], "Unicode"],
"\\'" -> FromCharacterCode[FromDigits["0027", 16], "Unicode"],
"\\\"" -> FromCharacterCode[FromDigits["0022", 16], "Unicode"]
};
unescape[s_]:=StringReplace[s,$unescapeRules];

Import a few lines; labels in English, German and Spanish:

getLabels[$file,"en"|"de"|"es",1000]

Dictionary store

Make and register the entity store:

$EntityStores={EntityStore["term"-><|
"Entities"->Association@@@GroupBy[

(* only the first few lines *)
getLabels[$file,"en"|"de"|"es",500000],

(* group by item (Q123, ...) *)
First->Function[{"Label","Language"->#[[3]]}->#[[2]]]
]
|>]};//AbsoluteTiming

Display entities in Spanish:

EntityProperty["term","Label"]["DefaultFunction"]=Function[ent,ent["Label","Language"->"es"]];

Take[EntityList["term"],7]

Translate

Translate terms containing “Glück” from German to Spanish:

EntityClass["term",EntityProperty["term","Label","Language"->"de"]->StringContainsQ["Glück"]][{
EntityProperty["term","Label","Language"->"de"],
EntityProperty["term","Label","Language"->"es"]
}]//Grid

Speed

Length[EntityList["term"]]

EntityClass["term",EntityProperty["term","Label","Language"->"de"]->StringContainsQ["Glück"]]//EntityList//Length//Timing

Part 3: Very advanced stuff

- connection to a database
- interoperability
- property sequences, joins, precedence
- predicates and non-predicates
- insertion handler
- batch processing
- caching, part 1
- caching, part 2

Connection to a database

Using batch function, query handler and aggregation handler.

Ant database

A database of ants, with three columns:
- “Entity” (primary key, canonical name of the entities)
- “Age” (in days)
- “Name”

Needs["DatabaseLink`"]

$antDB=OpenSQLConnection[JDBC[
"HSQL(Standalone)",
FileNameJoin[{$UserBaseDirectory,"DatabaseResources","test4179"}]
]];

$entityCount=200000;

(* clear tables *)
SQLDropTable[$antDB,#]&/@SQLTables[$antDB];

(* add columns *)
SQLCreateTable[$antDB,"ant",{
SQLColumn["Entity","DataTypeName"->"LONGVARCHAR","PrimaryKey"->True],
SQLColumn["Age","DataTypeName"->"DOUBLE"],
SQLColumn["Name","DataTypeName"->"LONGVARCHAR"]
}];

(* add data *)
SQLInsert[$antDB,"ant",{"Entity","Age","Name"},
Table[{
"e"<>ToString[i],
RandomReal[1000],
"s"<>ToString[i]
},{i,$entityCount}]
]//Length//AbsoluteTiming

Ant store

Connect an “ant store” to the database.

$EntityStores={EntityStore["ant"-><|
(* entity validation *)
"EntityValidationFunction"->BatchFunction[Function[ents,
With[
{valid=SQLSelect[$antDB,"ant",{"Entity"},SQLMemberQ[ents,SQLColumn["Entity"]]][[All,1]]},
MemberQ[valid,#]&/@ents
]
]],

(* properties *)
"Properties"->{"Age","Name"},

(* type level default function *)
"DefaultFunction"->Function[prop,
(* efficiently process batches *)
BatchFunction[Function[ents,
With[{names=CanonicalName[ents]},
With[{values=<|Rule@@@SQLSelect[
$antDB,
"ant",
{"Entity",CanonicalName[prop]},
SQLMemberQ[names,SQLColumn["Entity"]]
]|>},
values/@names
]
]
]]
],

(* type level query handler*)
"QueryHandler"->Function[prop,
Function[{pred,ents},
SQLSelect[$antDB,"ant",{"Entity"},
And[
Switch[ents,
EntityClass[_,All],True,
_,SQLMemberQ[CanonicalName[ents],SQLColumn["Entity"]]
],
pred[SQLColumn[CanonicalName[prop]]]
]
][[All,1]]//Map[Entity["ant",#]&]
]
]
|>]};

Valid ants:

EntityValue[{Entity["ant","e3"],Entity["ant","xy"]}]

Background check:

(Entity["e3", Entity["ant","e3"], "Entity[\"ant\", \"e3\"]", "ant"])["PropertyAssociation"]

Ant kids:

EntityClass["ant","Age"->LessThan[0.02]]//EntityList

Ants by name:

EntityClass["ant","Name"->"s42"]//EntityList

Speed

Look up data for 2000 ants:

ants=Entity["ant","e"<>ToString[#]]&/@RandomInteger[{1,$entityCount},2000];

EntityValue[ants,"Name"]//Length//AbsoluteTiming

Compare to retrieving values one by one:

EntityValue[#,"Name"]&/@ants//Length//AbsoluteTiming

Multiple conditions:

EntityClass["ant",{
"Age"->LessEqualThan[15],
"Age"->GreaterEqualThan[14]
}]//EntityList//Length//AbsoluteTiming

(Homework: Why is this query tough? What’s happening?)
Better:

EntityClass["ant",{
"Age"->Between[{14,15}]
}]//EntityList//Length//AbsoluteTiming

Predicate support

Predicate not supported? -> return Automatic

Aggregation

Average age of some ants:

EntityValue[ants,"Age"]//Mean//Timing

Equivalently:

EntityValue[ants,"Age",Mean]//Timing

Delegate aggregation to the database:

(EntityProperty["Age", EntityProperty["ant","Age"], "EntityProperty[\"ant\", \"Age\"]"])["AggregationHandler"]=Function[{agg,ents},
Catch[SQLExecute[
$antDB,
"SELECT "<>Replace[agg,{

(* translate aggregation function *)
Mean:>"AVG",
Total:>"SUM",
(* or fall back *)
_:>Throw[Automatic,"agg"]

}]<>"(age) FROM ant WHERE entity IN (`1`)",
{SQLArgument@@CanonicalName[ents]}
]//First//First,"agg"]
];

Faster:

EntityValue[ants,"Age",Mean]//Timing

Fall back:

EntityValue[ants,"Age",Histogram]

Interoperability

Semantic web: RDF export.

Store

A store with two types, strings, numbers, foreign keys and Booleans.

store=EntityStore[{
"t1"-><|
"Entities"-><|
"e1"-><|"p1"->"a","p2"->1,"p3"->Entity["t2","e1"],"p4"->True,"p5"->1.3|>,
"e2"-><|"p1"->"b","p2"->2|>|>
|>,
"t2"-><|
"Entities"-><|
"e1"-><||>
|>
|>
}];

Exporter

A simple exporter.
https://www.w3.org/TR/rdb-direct-mapping/

toTurtleString[store_EntityStore,base_String]:=StringRiffle[
Flatten[{
"@base <"<>base<>">",
Function[type,
KeyValueMap[
Function[{e,values},
StringRiffle[{
(* https://www.w3.org/TR/rdb-direct-mapping/#defn-row_node *)
"<"<>type<>"/CanonicalName="<>e<>">",
StringRiffle[Flatten[{
(* https://www.w3.org/TR/rdb-direct-mapping/#defn-row_type_triple *)
StringRiffle[{
"a",
"<"<>type<>">"
}],
KeyValueMap[
Function[{p,v},
{
If[MatchQ[v,_Entity],
(* https://www.w3.org/TR/rdb-direct-mapping/#defn-reference_triple *)
StringRiffle[{
(* https://www.w3.org/TR/rdb-direct-mapping/#defn-reference_property_IRI *)
"<"<>"#ref-"<>p<>">",
"<"<>EntityTypeName[v]<>"/CanonicalName="<>CanonicalName[v]<>">"
}],
Nothing
],
(* https://www.w3.org/TR/rdb-direct-mapping/#defn-literal_triple *)
StringRiffle[{
"<"<>type<>"#"<>p<>">",
toLiteral[v]
}]
}
],
values
]
}]," ;\n\t"]
},"\n\t"]
],
store[type,"Entities"]
]
]/@store[]
}],
" .\n"
]<>" .";

(* https://www.w3.org/TR/r2rml/#natural-mapping *)
toLiteral[s_String]:="\""<>s<>"\"";
toLiteral[False]:="false";
toLiteral[True]:="true";
toLiteral[n_?NumberQ/;Element[n,Reals]]:=n;
toLiteral[e_Entity]:=toLiteral[CanonicalName[e]];
toLiteral[_]:=$Failed;

Export

toTurtleString[store,"http://www.example.org/"]

Property sequences, joins, precedence

What is a property sequence?

Sequence of properties.

Try this:

(Entity["Germany", Entity["Country","Germany"], "Entity[\"Country\", \"Germany\"]", "country"])["CapitalCity"]["Population"]

At first sight this is a little more inconvenient:

(Entity["Germany", Entity["Country","Germany"], "Entity[\"Country\", \"Germany\"]", "country"])[EntityPropertySequence["CapitalCity","Population"]]

Try doing this for all countries using the first syntax (hint: the value of the first property might be Missing[...], what then is Missing[...][property]?).

EntityValue["Country",EntityPropertySequence["CapitalCity","Population"]]//Short

Now try writing a query involving the population of the capital of all countries.
Just use a property sequence:

EntityClass["Country",EntityPropertySequence["CapitalCity","Population"]->GreaterThan[Quantity[10^7,"People"]]]//EntityList

Or even shorter:

EntityClass["Country","CapitalCity"->"Population"->GreaterThan[Quantity[10^7,"People"]]]//EntityList

Precedence

Set up two stores. The second one has different foreign keys.

store1 = EntityStore[{
"t1" -> <|
"Entities" -> <|
"e1" -> <|"p1" -> Entity["t2", "e1"], "p2" -> 1|>,
"e2" -> <|"p1" -> Entity["t2", "e2"], "p2" -> 2|>
|>
|>,
"t2" -> <|
"Entities" -> <|
"e1" -> <|"p1" -> Entity["t1", "e1"]|>,
"e2" -> <|"p1" -> Entity["t1", "e2"]|>
|>
|>
}];
store2 = EntityStore[{
(* change the mapping from "t2" to "t1" *)
"t2" -> <|
"Entities" -> <|
"e1" -> <|"p1" -> Entity["t1", "e2"]|>,
"e2" -> <|"p1" -> Entity["t1", "e1"]|>
|>
|>
}];
$EntityStores = {store2, store1};

Join along p1 -> p1 -> p2 using directly the first store:

store1[
Entity["t1","e1"],
EntityPropertySequence[
EntityProperty["t1","p1"],
EntityProperty["t2","p1"],
EntityProperty["t1","p2"]
]]

The second store comes fist and overwrites part of the first one:

EntityValue[
Entity["t1","e1"],
EntityPropertySequence[
EntityProperty["t1","p1"],
EntityProperty["t2","p1"],
EntityProperty["t1","p2"]
]]

The same applies when resolving an implicitly defined entity class with respect to an entity store...

EntityList[{store1,EntityClass["t1",EntityPropertySequence[
EntityProperty["t1","p1"],
EntityProperty["t2","p1"],
EntityProperty["t1","p2"]
]->1]}]

... or using all registered types, favoring the declarations for a particular type that come first in $EntityStores.

EntityList[EntityClass["t1",EntityPropertySequence[
EntityProperty["t1","p1"],
EntityProperty["t2","p1"],
EntityProperty["t1","p2"]
]->1]]

More compact syntax:

EntityList[EntityClass["t1",(EntityProperty["p1", EntityProperty["t1","p1"], "EntityProperty[\"t1\", \"p1\"]"])->(EntityProperty["p1", EntityProperty["t2","p1"], "EntityProperty[\"t2\", \"p1\"]"])->(EntityProperty["p2", EntityProperty["t1","p2"], "EntityProperty[\"t1\", \"p2\"]"])->1]]

Predicates and non-predicates

Explanation.

Query normalization

$EntityStores={EntityStore["t"-><|
"Entities"-><|
"e1"-><|"p1"->1|>,
"e2"-><|"p1"->2|>,
"e3"-><|"p1"->3|>,
"e4"-><|"p1"->4|>
|>
|>]};

What does this actually mean?

EntityClass["t","p1"->3]//EntityList

Same as this:

EntityClass["t","p1"->EqualTo[3]]//EntityList

Or this?

EntityClass["t","p1"->1|3]//EntityList

Same as this:

EntityClass["t","p1"->MatchQ[1|3]]//EntityList

But what is this?

EntityClass["t","p1"->TakeLargest[2]]//EntityList

Manual expansion

Reproduce EntityClass["t","p1"->3]:

Select[
EntityList["t"],
EntityProperty["t","p1"]/*EqualTo[3]
]

And EntityClass["t","p1"->1|3]:

Select[
EntityList["t"],
EntityProperty["t","p1"]/*MatchQ[1|3]
]

How about taking the largest? Easy for one of these non-predicates:

EntityList["t"]//TakeLargestBy[EntityProperty["t","p1"],2]

Many Take* operators

$EntityStores={EntityStore["t"-><|
"Entities"-><|
"e1"-><|"p1"->1,"p2"->10|>,
"e2"-><|"p1"->2,"p2"->9|>,
"e3"-><|"p1"->3,"p2"->8|>,
"e4"-><|"p1"->4,"p2"->7|>
|>
|>]};

Intersect:

EntityClass["t",{
"p1"->TakeLargest[3],
"p2"->TakeLargest[3]
}]//EntityList

Intersect after evaluating all predicates:

EntityClass["t",{
"p1"->TakeLargest[2],
"p2"->TakeLargest[2],
"p1"->GreaterThan[1]
}]//EntityList

Why? Make the result independent of the order of conditions.
How to do that manually? Cheat:

Select[
EntityList["t"],
EntityProperty["t",{
"p1"->TakeLargest[2],
"p2"->TakeLargest[2],
"p1"->GreaterThan[1]
}]
]

Without the Boolean property? Homework.

Insertion handler

An entity store backed by an external data store.

Setup

A database:

$db=<|
"e1"-><|"p1"->14|>,
"e2"-><|"p1"->50|>,
"e3"-><||>
|>;

Use data from the database, intercept insertion:

$EntityStores={EntityStore["t"-><|
(* validate entities *)
"EntityValidationFunction"->Function[KeyExistsQ[$db,#]],

"Properties"-><|
"p1"-><|
(* get data *)
"DefaultFunction"->Function[ent,Lookup[$db[CanonicalName[ent]],"p1",Missing["not here"]]],

(* set data *)
"InsertionFunction"->Function[values,
$db=Merge[{
$db,
values//KeyMap[CanonicalName]//Map[<|"p1"->#|>&]
},Last]
]
|>
|>
|>]};

Test

Validation:

Entity["t","X"]["p1"]

Retrieval:

Entity["t","e1"]["p1"]

Entity["t","e3"]["p1"]

Insertion, update:

Entity["t","e1"]["p1"]=15

Insertion, new entity:

Entity["t","new"]["p1"]=99

Check the database:

$db

Batch processing

Batch function, remote API.

Flowers

Infinite flower store:

$EntityStores={EntityStore["flower"-><|
"EntityValidationFunction"->Function[True]
|>]};

No batching

Simple default function:

EntityProperty["flower","p"]["DefaultFunction"]=Function[ent,
Echo[ent];
"x"
];

Entity["flower","tulip"]["p"]

Once per entity:

EntityValue[{Entity["flower","tulip"],Entity["flower","rose"]},"p"]

Batching

Indicate that a function wants a list of things:

EntityProperty["flower","p"]["DefaultFunction"]=BatchFunction[Function[ent,
Echo[ent];
ConstantArray["x",Length[ent]]
]];

Receives a list, always:

Entity["flower","tulip"]["p"]

EntityValue[{Entity["flower","tulip"],Entity["flower","rose"]},"p"]

Elements

Retrieve data from Wikidata. On demand.

Data

Elements and associated Wikidata item ids.

elements={"Actinium"->"Q1121","Aluminum"->"Q663","Americium"->"Q1872","Antimony"->"Q1099","Argon"->"Q696","Arsenic"->"Q871","Astatine"->"Q999","Barium"->"Q1112","Berkelium"->"Q1882","Beryllium"->"Q569","Bismuth"->"Q942","Bohrium"->"Q1249","Boron"->"Q618","Bromine"->"Q879","Cadmium"->"Q1091","Calcium"->"Q706","Californium"->"Q1888","Carbon"->"Q623","Cerium"->"Q1385","Cesium"->"Q1108","Chlorine"->"Q688","Chromium"->"Q725","Cobalt"->"Q740","Copernicium"->"Q1278","Copper"->"Q753","Curium"->"Q1876","Darmstadtium"->"Q1266","Dubnium"->"Q1232","Dysprosium"->"Q1843","Einsteinium"->"Q1892","Erbium"->"Q1849","Europium"->"Q1396","Fermium"->"Q1896","Flerovium"->"Q1302","Fluorine"->"Q650","Francium"->"Q671","Gadolinium"->"Q1832","Gallium"->"Q861","Germanium"->"Q867","Gold"->"Q897","Hafnium"->"Q1119","Hassium"->"Q1252","Helium"->"Q560","Holmium"->"Q1846","Hydrogen"->"Q556","Indium"->"Q1094","Iodine"->"Q1103","Iridium"->"Q877","Iron"->"Q677","Krypton"->"Q888","Lanthanum"->"Q1801","Lawrencium"->"Q1905","Lead"->"Q708","Lithium"->"Q568","Livermorium"->"Q1304","Lutetium"->"Q1857","Magnesium"->"Q660","Manganese"->"Q731","Meitnerium"->"Q1258","Mendelevium"->"Q1898","Mercury"->"Q925","Molybdenum"->"Q1053","Neodymium"->"Q1388","Neon"->"Q654","Neptunium"->"Q1105","Nickel"->"Q744","Niobium"->"Q1046","Nitrogen"->"Q627","Nobelium"->"Q1901","Osmium"->"Q751","Oxygen"->"Q629","Palladium"->"Q1089","Phosphorus"->"Q674","Platinum"->"Q880","Plutonium"->"Q1102","Polonium"->"Q979","Potassium"->"Q703","Praseodymium"->"Q1386","Promethium"->"Q1809","Protactinium"->"Q1109","Radium"->"Q1128","Radon"->"Q1133","Rhenium"->"Q737","Rhodium"->"Q1087","Roentgenium"->"Q1272","Rubidium"->"Q895","Ruthenium"->"Q1086","Rutherfordium"->"Q1226","Samarium"->"Q1819","Scandium"->"Q713","Seaborgium"->"Q1234","Selenium"->"Q876","Silicon"->"Q670","Silver"->"Q1090","Sodium"->"Q658","Strontium"->"Q938","Sulfur"->"Q682","Tantalum"->"Q1123","Technetium"->"Q1054","Tellurium"->"Q1100","Terbium"->"Q1838","Thallium"->"Q932","Thorium"->"Q1115","Thulium"->"Q1853","Tin"->"Q1096","Titanium"->"Q716","Tungsten"->"Q743","Ununoctium"->"Q1307","Ununpentium"->"Q1303","Ununseptium"->"Q1306","Ununtrium"->"Q1301","Uranium"->"Q1098","Vanadium"->"Q722","Xenon"->"Q1106","Ytterbium"->"Q1855","Yttrium"->"Q941","Zinc"->"Q758","Zirconium"->"Q1038"};

Element store

Retrieve “density” in batches.

$EntityStores={EntityStore["element"-><|
"Entities"-><|#-><|"id"->#2|>&@@@elements|>,

"Properties"-><|
(* one property, no units *)
"Density"-><|
"DefaultFunction"->BatchFunction[Function[ents,
URLExecute[
"https://query.wikidata.org/bigdata/namespace/wdq/sparql",
{
"format"->"json",
"query"->StringTemplate[
"SELECT ?e ?d WHERE { VALUES ?e { `` } . ?e wdt:P2054 ?d . }"
][StringRiffle["wd:"<>#&/@EntityValue[ents,"id"]]]
},
"RawJSON"
]//Query["results","bindings",All,{
"e"->"value"/*FileNameTake,
"d"->"value"/*Interpreter["Number"]
}]//GroupBy[#,#e&->(#d&),Mean]&//Lookup[
#,
EntityValue[ents,"id"],
Missing["NotAvailable"]
]&
]]
|>
|>
|>]};

Works for one entity:

(Entity["Silver", Entity["element","Silver"], "Entity[\"element\", \"Silver\"]", "element"])["Density"]//AbsoluteTiming

Or many:

EntityValue[{(Entity["Nickel", Entity["element","Nickel"], "Entity[\"element\", \"Nickel\"]", "element"]),(Entity["Nitrogen", Entity["element","Nitrogen"], "Entity[\"element\", \"Nitrogen\"]", "element"]),(Entity["Copper", Entity["element","Copper"], "Entity[\"element\", \"Copper\"]", "element"])},"Density"]//AbsoluteTiming

Even all:

EntityClass["element",All]["Density"]//AbsoluteTiming

Implicitly defined entity class:

EntityList[EntityClass["element","Density"->LessThan[8]]]//AbsoluteTiming

Caching, part 1

Caching music. Different strategies.

Import

Import data from MusicBrainz.
https://wiki.musicbrainz.org/Development/JSON_Web_Service

musicbrainzImport[type_String,name_String,inc_String : ""]:=URLExecute[
"http://musicbrainz.org/ws/2/"<>type<>"/"<>name,
{"inc"->inc,"fmt"->"json"},
"RawJSON"
];

Check:

musicbrainzImport["artist","a7389752-9e5d-406d-9a02-9141600e4315"]["name"]

Labels

Cache entity labels using Once[…]:

$EntityStores={EntityStore[{
"artist"-><|
"Entities"->{
"af9b9909-23ec-42cc-a386-338d157ca5a4",
"a7389752-9e5d-406d-9a02-9141600e4315",
"d09ac963-025f-4eb6-853c-cc383838ee60",
"f2030ec9-6ca4-405a-b69a-148196b0e4ae"
},

"Properties"-><|
"Label"-><|
"DefaultFunction"->Function[ent,
Once[musicbrainzImport["artist",CanonicalName[ent]]]["name"]
]
|>
|>
|>
}]};

Fast the second time:

EntityList["artist"]

Recordings

Fill the music store whenever data is retrieved.

$EntityStores={EntityStore[{
"artist"-><|
"Entities"->{
"af9b9909-23ec-42cc-a386-338d157ca5a4",
"a7389752-9e5d-406d-9a02-9141600e4315",
"d09ac963-025f-4eb6-853c-cc383838ee60",
"f2030ec9-6ca4-405a-b69a-148196b0e4ae"
},

"Properties"-><|
"Label"-><|
"DefaultFunction"->Function[ent,
(* store and return the label *)
ent["Label"]=musicbrainzImport["artist",CanonicalName[ent]]["name"]
]
|>,
"Recordings"-><|
"DefaultFunction"->Function[ent,With[
{r=musicbrainzImport["artist",CanonicalName[ent],"recordings"]["recordings"]},

(* store all retrieved recordings and labels *)
(Entity["recording",#id]["Label"]=#title)&/@r;

(* return recordings *)
Entity["recording",#id]&/@r
]]
|>
|>
|>,

"recording"-><|
|>
}]}

All artists:

EntityList["artist"]

Some recordings:

(Entity["Roberto Firpo", Entity["artist","af9b9909-23ec-42cc-a386-338d157ca5a4"], "Entity[\"artist\", \"af9b9909-23ec-42cc-a386-338d157ca5a4\"]", "artist"])["Recordings"]

New data is in the store:

$EntityStores//First

Caching, part 2

Cache the result of any function that accepts entities.

Any function

Observe evaluation using Echo:

CachedEntityFunction["x"&/*Echo][(Entity["Rigel Kentaurus A", Entity["Star","RigelKentaurusA"], "Entity[\"Star\", \"RigelKentaurusA\"]", "star"])]

CachedEntityFunction["x"&/*Echo][(Entity["Rigel Kentaurus A", Entity["Star","RigelKentaurusA"], "Entity[\"Star\", \"RigelKentaurusA\"]", "star"])]

Entity property

A constant property:

CachedEntityFunction[(EntityProperty["escape velocity", EntityProperty["Star","EscapeVelocity"], "EntityProperty[\"Star\", \"EscapeVelocity\"]"])/*Echo][(Entity["Rigel Kentaurus A", Entity["Star","RigelKentaurusA"], "Entity[\"Star\", \"RigelKentaurusA\"]", "star"])]

CachedEntityFunction[(EntityProperty["escape velocity", EntityProperty["Star","EscapeVelocity"], "EntityProperty[\"Star\", \"EscapeVelocity\"]"])/*Echo][(Entity["Rigel Kentaurus A", Entity["Star","RigelKentaurusA"], "Entity[\"Star\", \"RigelKentaurusA\"]", "star"])]

Re-check a changing property now and then:

CachedEntityFunction[(EntityProperty["heliocentric XYZ coordinates", EntityProperty["Planet","HelioCoordinates"], "EntityProperty[\"Planet\", \"HelioCoordinates\"]"])/*Echo,UpdateInterval->(Quantity[3, "s", seconds, "Seconds"])][LinguisticAssistant]

CachedEntityFunction[(EntityProperty["heliocentric XYZ coordinates", EntityProperty["Planet","HelioCoordinates"], "EntityProperty[\"Planet\", \"HelioCoordinates\"]"])/*Echo,UpdateInterval->(Quantity[3, "s", seconds, "Seconds"])][LinguisticAssistant]

Failure

Don’t cache failures:

CachedEntityFunction["bad"&/*Echo,"FailurePattern"->$Failed|"bad"][(Entity["Mount Currie", Entity["Mountain","Currie"], "Entity[\"Mountain\", \"Currie\"]", "mountain"])]

CachedEntityFunction["bad"&/*Echo,"FailurePattern"->$Failed|"bad"][(Entity["Mount Currie", Entity["Mountain","Currie"], "Entity[\"Mountain\", \"Currie\"]", "mountain"])]

Default function

Cache the result of an expensive computation:

$EntityStores={EntityStore["number"-><|
"Entities"-><|
"n1"-><|"value"->12|>,
"n2"-><|"value"->2^250-1|>,
"n3"-><|"value"->3^201-1|>,
"n4"-><|"value"->3^207-1|>
|>,
"Properties"-><|
"FactorCount"-><|"DefaultFunction"->CachedEntityFunction[
Function[Total[FactorInteger[#["value"]][[All,2]]]]
]|>
|>
|>]};

First call:

Entity["number","n4"]["FactorCount"]//Timing

Second:

Entity["number","n4"]["FactorCount"]//Timing

Compute only the remaining:

EntityClass["number","FactorCount"->GreaterThan[8]]//EntityList//Timing

EntityClass["number","FactorCount"->GreaterThan[8]]//EntityList//Timing

Other options

“LastUpdate”, “UseFileCache”, “UseMemoryCache”

Part 4: Recommendations

Recommendations for developing a collection of entities.

Use singular type name

For an entity Entity["t","e"] you should be able to say “e is a t”.
For instance for Entity["book","Faust1"] you can say “Faust 1 is a book”.

Use sentence case for labels

This applies to the few languages I know about, including German, Spanish and English.
For a label use the case it would have in the middle of a sentence.
For instance Entity["verb","run"], Entity["color","red"], EntityProperty["location","population"], ... would have labels “run”, “red”, “has population”, ..., respectively while Entity["person","John"] has label “John”.
This is useful for text synthesis that uses these labels.

Share your entities...

... and motivate others to contribute.