Exploring a Giant Legal Citation Network

Seth J. Chandler, University of Houston Law Center
schandler@uh.edu

Join the Conversation #WolframTechConf

ABSTRACT

This presentation uses the Wolfram Language to explore the largest legal citation network ever placed in the public domain: over 3 million American cases (vertices) dating back to the start of the Republic and 30 million citations (edges) among them. The presentation will show how the Wolfram Language is best used to efficiently handle networks of this size. Emphasis will be placed on metrics assessing the power not just of individual cases but of different courts throughout American history. The presentation further examines: (1) the “mortality” of legal opinions broken down by the court from which the opinion emerges; and (2) the extent to which statistics regarding large networks can be approximated by samples of the edges. Also shown are opportunities to relate citation data to other data on the American judicial system. The author has collaborated with free.law such that the datasets will shortly be available as a set of ResourceObjects for other Wolfram Language users to deploy.

The initialization slide

Initialization of the system takes about 10 minutes. This is in part because we are moving about 6 gigabytes of data from disk to memory.

Data

dataDirectory="/Users/sethchandler/Documents/Data";

AbsoluteTiming[opinions=Import[FileNameJoin[{dataDirectory,"casesWithCitations3.wxf"}],"WXF"];]

AbsoluteTiming[preWarrenKeys=Import[FileNameJoin[{dataDirectory,"preWarrenKeys.wxf"}],"WXF"];]

AbsoluteTiming[preWarren=KeyTake[opinions,preWarrenKeys];]

AbsoluteTiming[pre20thKeys=Import[FileNameJoin[{dataDirectory,"pre20thKeys.wxf"}],"WXF"];]

LocalSubmit[KeyTake[opinions,pre20thKeys],
HandlerFunctions-><|"TaskFinished"->((pre20th=#EvaluationResult)&)|>,HandlerFunctionsKeys->{"EvaluationExpression","EvaluationResult"}]

(*AbsoluteTiming[pre20th=KeyTake[opinions,pre20thKeys];]*)

AbsoluteTiming[preCivilWarKeys=Import[FileNameJoin[{dataDirectory,"preCivilWarKeys.wxf"}],"WXF"];]

LocalSubmit[KeyTake[opinions,preCivilWarKeys],
HandlerFunctions-><|"TaskFinished"->((preCivilWar=#EvaluationResult)&)|>,HandlerFunctionsKeys->{"EvaluationExpression","EvaluationResult"}]

(*AbsoluteTiming[preCivilWar=KeyTake[opinions,preCivilWarKeys];]*)

AbsoluteTiming[gr=Import[FileNameJoin[{dataDirectory,"cleanedNetwork.wxf"}],"WXF"];]

]

AbsoluteTiming[courts=Import[FileNameJoin[{dataDirectory,"FreelawCourts"}]];]

LocalSubmit[Query[Association,#id->#&][courts],
HandlerFunctions-><|"TaskFinished"->((courts2=#EvaluationResult)&)|>,HandlerFunctionsKeys->{"EvaluationExpression","EvaluationResult"}]

(*AbsoluteTiming[courts2=Query[Association,#id->#&][courts];]*)

AbsoluteTiming[opinionToCourt=Import[FileNameJoin[{dataDirectory,"opinionToCourt.wxf"}],"WXF"];]

AbsoluteTiming[publishedKeys=Import[FileNameJoin[{dataDirectory,"publishedKeys.wxf"}],"WXF"];]

AbsoluteTiming[publishedEdges=Import[FileNameJoin[{dataDirectory,"publishedEdges.wxf"}],"WXF"];]

Functions

upToFirst[x_]:=If[Length[x]>0,First[x],x]

centralityAssociation[el_,centralityMeasure_:PageRankCentrality]:=AssociationThread[VertexList[el],centralityMeasure[el]];

Why You Should Care

Non lawyers (indeed, non-Americans)

Citation networks abound

Algorithmic analyses of prominence and community structure can prove instructive

A little bit of network theory

Dataset management issues present here will recur in other domains

Use of Query language to interrogate and relate data

Lawyers and other social scientists

See what American judicial opinions are (algorithmically) most important

Discover how importance has evolved over time

Relationship between courts: patterns of deference

Half life of judicial opinions

Previous Work

Supreme Court Network -- 2005

Back in the day, this was hard work! Required XML parsing, Mathematica serving as a hub via JLink with a Java package for network analysis and something called Pajek.

Texas

440, 000 cases and 2.5 million edges

Most important cases in Texas ranked by page rank

A Texas - shaped Word Cloud of an important case on punitive damages in Texas

Supreme Court Decisions

Washington University in St. Louis keeps a detailed repository of every Supreme Court decision since 1946. But it' s hard to use because legal concepts such as “first amendment” or “federalism” or “Illinois” are encoded as integers, requiring constant reference to a lengthy codebook. And dates are not in a form most conductive to date arithmetic. Fixed in my adaptation of the data for the Wolfram Data Repository.

https://resources.wolframcloud.com/DataRepository/resources/Seth%20 J.%20Chandler %3 AUnited-States-Supreme-Court-Decisions-1946-present

Framed[Labeled[Query[GroupBy[#issueAreaS&]/*ReverseSort,Length][Query[Select[#majVotes ==#minVotes+1 && #dateDecision>DateObject[{1988,2,12}] && #decisionDirectionS ==="liberal" && #directionS==="liberal" && #justiceName=="AMKennedy" &],KeyTake[{"caseName","usCite","dateDecision","issueAreaS","lawTypeS"}]]],],Background->ColorData["CoffeeTones"][0.94]]

ResourceObjects available now

Select[ResourceSearch["law"],#["ContributorInformation"]["PublisherID"]==="Seth J. Chandler"&]

The Original Data

Free.law

https://free.law/

https://www.courtlistener.com/api/bulk-info/

Framed[]

Framed[]

The two datasets

The link dataset (just case to case, using integer encodings); only useful in the abstract; no legal meaning

The case dataset : immense because it includes the text of the opinions among other things -- but no explicit information; must be scraped

In theory, I might have attacked free.law with 4 million requests on their server and downloaded the information that would relate the graph to the case information

The Secret

Ask and make a modest contribution to free.law

The good news

They get it

I now have a dataset with basic information that contains the integer case id encodings so that I can now produce a Dataset that has adjacency information with each case.

They are computer savvy

The bad news

There are, unsurprisingly, mistakes in the data

Scrapers not finding citations

Date issues : a lot of examples of Case A citing Case B, even though Case B came 10 years after Case A.

My Data

The case database

Length[opinions]

Dataset[Query[100000;;100005,1;;8][opinions]]

Dataset[Query[100000;;100005,9;;16][opinions]]

Dataset[Query[100000;;100005,17;;-1][opinions]]

caseId keys for quick look up

Basic information

Citation information

Date differencing information

The graph

Through[{VertexCount,EdgeCount}[gr]]

Take[EdgeList[gr],10]

The Citation Graph

Prominence Measures

Degree Centrality

AbsoluteTiming[dc=ReverseSort[centralityAssociation[gr,DegreeCentrality]];]

Dataset[Merge[{Query[(Keys[dc][[1;;20]]),KeyTake[{"caseName","federalCites","stateCites","courtId","dateFiled"}]/*MapAt[upToFirst,{{"federalCites"},{"stateCites"}}]][opinions],Map[Association["pageRank"->#]&,Take[dc,20]]},Merge[First]]]

PageRank Centrality

AbsoluteTiming[prc=ReverseSort[centralityAssociation[gr,PageRankCentrality]];]

pageRankRanking=Dataset[Merge[{Query[(Keys[prc][[1;;20]]),KeyTake[{"caseName","federalCites","stateCites","courtId","dateFiled"}]/*MapAt[upToFirst,{{"federalCites"},{"stateCites"}}]][opinions],Map[Association["pageRank"->#]&,Take[prc,20]]},Merge[First]]]

HITSCentrality and Katz Centrality blow up the kernel \[FreakedSmiley]

The Published Opinion Subset

Might the high salience of mostly criminal cases (and a few on civil procedure) be a result of so-called “unpublished opinions,” which may be particularly likely in criminal cases?

(*pk=AssociationThread[publishedKeys,True];Query[publishedKeys/*Values/*Flatten,Thread[DirectedEdge[ToString[#"_id"],Pick[#cites,Lookup[pk,#cites,False],True]]]&][opinions]*)

(*pk=AssociationThread[publishedKeys,True];Query[publishedKeys/*Values/*Flatten,Thread[DirectedEdge[ToString[#"_id"],Pick[#cites,Lookup[pk,#cites,False],True]]]&][opinions]*)

AbsoluteTiming[publishedPrc=ReverseSort[centralityAssociation[publishedEdges,PageRankCentrality]];]

publishedPageRankRanking=Dataset[Merge[{Query[(Keys[publishedPrc][[1;;20]]),KeyTake[{"caseName","federalCites","stateCites","courtId","dateFiled"}]/*MapAt[upToFirst,{{"federalCites"},{"stateCites"}}]][opinions],Map[Association["pageRank"->#]&,Take[publishedPrc,20]]},Merge[First]]]

Difference in ranking

Grid[Join[{{"Case","Overall Ranking","Published Ranking"}},MapThread[{#1,#2[[1]],#2[[2,1,1]]}&,{Normal@Query[Values,#caseName&][pageRankRanking],With[{kprc=Keys[publishedPrc]},MapIndexed[{#2[[1]],Position[kprc,#1]}&,Keys[Take[prc,20]]]]}]],Dividers->All,Alignment->{Left,Center,Center},ItemSize->{{20,6,6},1}]

Replicating My 2005 Work in a Few Lines of Code

AbsoluteTiming[scotusKeys=Query[ Position["scotus"]/*Flatten/*Map[First],#courtId&][opinions];]

publishedScotusKeys=Intersection[scotusKeys,publishedKeys]

AbsoluteTiming[publishedScotusInternalEdges=With[{psk=AssociationThread[publishedScotusKeys,True]},Query[publishedScotusKeys/*Values/*Flatten,Thread[DirectedEdge[ToString[#"_id"],Pick[#cites,Lookup[psk,#cites,False],True]]]&][opinions]];
]

AbsoluteTiming[publishedScotusInternalPrc=ReverseSort[centralityAssociation[publishedScotusInternalEdges,PageRankCentrality]];]

publishedScotusInternalPageRankRanking=Dataset[Merge[{Query[(Keys[publishedScotusInternalPrc][[1;;20]]),KeyTake[{"caseName","federalCites","stateCites","courtId","dateFiled"}]/*MapAt[upToFirst,{{"federalCites"},{"stateCites"}}]][opinions],Map[Association["pageRank"->#]&,Take[publishedScotusInternalPrc,20]]},Merge[First]]]

A Little Bit of Network Theory

The variation in Page Rank with the number of edges sampled: Code

(*Short[defaultPageRank=AssociationThread[VertexList[gr],0]];
(* sample 10^4 edges*)
AbsoluteTiming[(SeedRandom[10192018];elList4=Map[DeleteDuplicates][RandomChoice[EdgeList[gr], {20,10^4}]];)];
AbsoluteTiming[Monitor[(i=0;pageRankCentralitySampleValues4=Merge[Map[(i++;Merge[{centralityAssociation[#],defaultPageRank},First])&,elList4],Identity]),i];];
AbsoluteTiming[meanStandardDeviationSamplePageRank4=Map[Through[{Mean,StandardDeviation}[#]]&,pageRankCentralitySampleValues4];];
Export[FileNameJoin[{dataDirectory,"meanStandardDeviationSamplePageRank4.wxf"}],meanStandardDeviationSamplePageRank4,"WXF"];
(*sample 10^5 edges*)
AbsoluteTiming[elList5=Map[DeleteDuplicates][RandomChoice[EdgeList[gr], {20,10^5}]];]
AbsoluteTiming[Monitor[(i=0;pageRankCentralitySampleValues5=Merge[Map[(i++;Merge[{centralityAssociation[#],defaultPageRank},First])&,elList5],Identity]),i];];
AbsoluteTiming[meanStandardDeviationSamplePageRank5=Map[Through[{Mean,StandardDeviation}[#]]&,pageRankCentralitySampleValues5];];
Export[FileNameJoin[{dataDirectory,"meanStandardDeviationSamplePageRank5.wxf"}],meanStandardDeviationSamplePageRank5,"WXF"];
(* sample 10^6 edges *)
AbsoluteTiming[elList6=Map[DeleteDuplicates][RandomChoice[EdgeList[gr], {20,10^6}]];];
AbsoluteTiming[Monitor[(i=0;pageRankCentralitySampleValues6=Merge[Map[(i++;Merge[{centralityAssociation[#],defaultPageRank},First])&,elList6],Identity]),i];];
AbsoluteTiming[meanStandardDeviationSamplePageRank6=Map[Through[{Mean,StandardDeviation}[#]]&,pageRankCentralitySampleValues6];];
Export[FileNameJoin[{dataDirectory,"meanStandardDeviationSamplePageRank6.wxf"}],meanStandardDeviationSamplePageRank6,"WXF"];*)

Results

AbsoluteTiming[meanStandardDeviationSamplePageRank4=Import[FileNameJoin[{dataDirectory,"meanStandardDeviationSamplePageRank4.wxf"}],"WXF"];]

AbsoluteTiming[meanStandardDeviationSamplePageRank5=Import[FileNameJoin[{dataDirectory,"meanStandardDeviationSamplePageRank5.wxf"}],"WXF"];]

AbsoluteTiming[meanStandardDeviationSamplePageRank6=Import[FileNameJoin[{dataDirectory,"meanStandardDeviationSamplePageRank6.wxf"}],"WXF"];]

Histogram3D[Map[Values,{meanStandardDeviationSamplePageRank4,meanStandardDeviationSamplePageRank5,meanStandardDeviationSamplePageRank6}],Automatic,{"Log","Count"},ImageSize->500,AxesLabel->{"Mean PageRank","Stdev. Page Rank","(log)Count"},ChartLegends->{"(10^4) edges","(10^5) edges","(10^6) edges"},PlotLabel->"Mean and Standard Deviation Page Ranks of Cases\nBased on Edge Sampling"]

Legal History

At the start of the Warren Court (October 5, 1953)

AbsoluteTiming[preWarrenGraph=Graph[Query[Select[Length[#cites]>0&]/*Values/*Flatten,Thread[DirectedEdge[ToString@#"_id",Pick[#cites,QuantityMagnitude@#citesDateDifferential,x_/;x<=0]]]&][preWarren],GraphLayout->None
]]

AbsoluteTiming[Short[prcPreWarren=ReverseSort[centralityAssociation[preWarrenGraph,PageRankCentrality]]]]

Dataset[Merge[{Query[(Keys[prcPreWarren][[1;;20]]),KeyTake[{"caseName","federalCites","stateCites","courtId","dateFiled"}]/*MapAt[upToFirst,{{"federalCites"},{"stateCites"}}]/*MapAt[StringTake[#,UpTo[30]]&,"caseName"]][preWarren],Map[Association["pageRank"->#]&,Take[prcPreWarren,20]]},Merge[First]]]

AbsoluteTiming[pre20thGraph=Graph[Query[Select[Length[#cites]>0&]/*Values/*Flatten,Thread[DirectedEdge[ToString@#"_id",Pick[#cites,QuantityMagnitude@#citesDateDifferential,x_/;x<=0]]]&][pre20th],GraphLayout->None
]]

AbsoluteTiming[Short[prcPre20th=ReverseSort[centralityAssociation[pre20thGraph,PageRankCentrality]]]]

Dataset[Merge[{Query[(Keys[prcPre20th][[1;;20]]),KeyTake[{"caseName","federalCites","stateCites","courtId","dateFiled"}]/*MapAt[upToFirst,{{"federalCites"},{"stateCites"}}]/*MapAt[StringTake[#,UpTo[30]]&,"caseName"]][pre20th],Map[Association["pageRank"->#]&,Take[prcPre20th,20]]},Merge[First]]]

I think there might be data issues in North Carolina.

Using IGraph/M

http://szhorvat.net/pelican/igraphm-a-mathematica-interface-for-igraph.html

Needs["IGraphM`"];

(*AbsoluteTiming[g=Query[Values/*Flatten/*(Graph[#,GraphLayout->None]&),Thread[DirectedEdge[#opinionId,#cites]]&,FailureAction->None][opinions];]*)

Authority

AbsoluteTiming[authorityScores=ReverseSort@Association@Thread[Rule[VertexList[gr],IGAuthorityScore[gr]]];]

top20Authority=Query[Take[Keys[authorityScores],20],KeyTake[{"courtId","caseName","federalCites","stateCites","dateFiled"}]/*(Association[KeyDrop[#,{"federalCites","stateCites"}],"cite"->DeleteCases[{#federalCites,#stateCites},{}]]&)][opinions];

Core

AbsoluteTiming[corenessScores=ReverseSort@Association@Thread[Rule[VertexList[gr],IGCoreness[gr]]];]

corenessScoreCounts=Counts[Values@corenessScores]

ListLinePlot[Thread[List[Keys[corenessScoreCounts],Accumulate[Values[corenessScoreCounts]]]],ScalingFunctions->"Log",Frame->True,FrameLabel->{"Core Size","(log) Number of Opinions"},PlotStyle->Thick,ImageSize->500,BaseStyle->14]

igKCore=Query[kcoreKeys/*SortBy[#dateFiled&],KeyTake[{"courtId","caseName","federalCites","stateCites","dateFiled"}]/*(Association[KeyDrop[#,{"federalCites","stateCites"}],"cite"->DeleteCases[{#federalCites,#stateCites},{}]]&)][opinions];

Labeled[Dataset[igKCore],Style["The K Core of American Jurisprudence in 2018:\nMississippi Criminal Law, Mostly Race and the Death Penalty",FontFamily->"Arial",FontSize->14],Top
]

Pre - 20th Century

The Court-To-Court Citation Graph

Once Code

AbsoluteTiming[opinionToCourt=Query[All,#courtId&][opinions];]

AbsoluteTiming[courtToCourtEdges=Flatten[KeyValueMap[{k,v}|->Thread[DirectedEdge[Lookup[opinionToCourt,k,Missing[]],Lookup[opinionToCourt,v["cites"],Missing[]]]]][opinions]];]

Last[{668.018673`,Null}]

Export["/Users/sethchandler/Dropbox/Scholarship/GiantCitationNetwork/courtToCourt.wxf",courtToCourtEdges,"WXF"]

Code

AbsoluteTiming[courtToCourtEdges=Import["/Users/sethchandler/Dropbox/Scholarship/GiantCitationNetwork/courtToCourt.wxf","WXF"];]

courtToCourtCounts=Counts[courtToCourtEdges]

weightedCourtToCourt=With[{n=Normal[courtToCourtCounts]},Graph[DirectedEdge@@@(Keys@n),EdgeWeight->Values@n,GraphLayout->None]]

MatrixPlot[weightedAdjacencyCourtToCourt=WeightedAdjacencyMatrix[weightedCourtToCourt]]

Column[courtCommunities=FindGraphCommunities[weightedCourtToCourt]]

asymmetries=weightedAdjacencyCourtToCourt-Transpose[weightedAdjacencyCourtToCourt]

deferenceRelations[n_,gr_:weightedCourtToCourt,m_:asymmetries]:=With[{vl=VertexList[gr]},Map[vl[[#]]&,Position[Normal@m,x_/;x>n,{2}],{2}]]

Framed@CommunityGraphPlot[Graph[DirectedEdge@@@deferenceRelations[25000,weightedCourtToCourt,asymmetries],VertexLabels->"Name"],ImageSize->800,PlotLabel->Style["Patterns of Deference in American Legal Opinions",18]]

Citation Mortality

Concept

In a common law system, legal opinions tend to be “pinged” or “cited” by subsequent legal opinions. These subsequent opinions sometimes massage, clarify, refine or even overturn the opinions they cite. Thus, after some period of time, many legal opinions are no longer cited frequently. Other opinions may have greater longevity, particularly, one would imagine, landmark cases or decisions from jurisdictions in which the law is particularly stable. We can use the data here to examine the comparative mortality of legal opinions by court.

Mortality By Court

AbsoluteTiming[citedByDateDifferentialsByCourt=Query[GroupBy[#courtId&],Select[#citedBy=!={}&]/*Values/*Flatten,Select[QuantityMagnitude@#citedByDateDifferential,#>0&]&][opinions];
]

AbsoluteTiming[Short[citationDistributionByCourt=Monitor[(progress=0;Map[(progress++;activeLength=Length[#];If[Length[#]>500,FindDistribution[RandomChoice[#,Min[Length[#],10000]],TargetFunctions->"Continuous"],Missing["TooFewCitations"]])&,citedByDateDifferentialsByCourt]),{progress,activeLength}]]
]

We can see that courts that lie at the apex of a judicial system (even those subject to theoretical check on occasion by the United States Supreme Court), tend to have legal opinions with the longest half lives. Even prestigious intermediate appellate courts, which are subject to reversal and refinement more frequently, tend to have legal opinions with considerably shorter half lives.

Framed[Labeled[Dataset[KeyMap[k|->Query[k,#shortName&][courts2],ReverseSort[Map[If[MissingQ[#],#,UnitConvert[Quantity[Quantile[#,0.5],"Days"],"Years"]]&,citationDistributionByCourt]]]
],Style["The Half Life of Citations\nTo Cases Decided By This Court\nBy All Courts",FontFamily->"Swiss",FontSize->14],Top],Background->ColorData["CoffeeTones"][0.9]]

We can also look to see what sort of distributions tend to describe the mortality of citations. It turns out that in addition to mixture distributions -- and I wish there were a TargetFunctionExclusions option to FindDistribution -- the most common distributions come from the Exponential and Gamma families.

AbsoluteTiming[distributionDistribution=Query[DeleteMissing/*Counts/*ReverseSort,Head][citationDistributionByCourt]]

The distributions contained in the mixture distributions are most frequently Gamma and LogNormal.

AbsoluteTiming[mixtureComponentsDistribution=Query[Select[Head[#]===MixtureDistribution&]/*Values/*Flatten/*Map[Head]/*Counts/*ReverseSort,#[[2]]&][citationDistributionByCourt]
]

Overall Mortality

We can also consider the overall rate of mortality of legal opinions. I do this by sampling 20,000 cases 10 times and seeing what distributions result.  Some of the distributions are exponential; others are mixtures of gamma distributions and lognormal distributions. It appears that, overall, the half life of a legal opinion is about 9 years.

AbsoluteTiming[flattenedCitesDistribution=Query[Values/*Flatten/*(RandomChoice[#,{10,20000}]&)/*(Map[FindDistribution[#,TargetFunctions->"Continuous"]&])][citedByDateDifferentialsByCourt];
]

flattenedCitesDistribution

Map[UnitConvert[Quantity[Quantile[#,0.5],"Days"],"Years"]&,flattenedCitesDistribution]

Dataset Management Issues

Things I believe may be in Version 12 to help

Structured Arrays to compress data. (Possibly a column orientation to the dataset)

Use of compilation on Associations for speed

Better Mongo and other external Dataset connectivity (I’ve done some preliminary work with MongoLink, but I want to see what other developments occur)

More explicit use of parallelization within Query and Dataset

Further Explorations

Cleaning up the data -- will need to collaborate with free.law

Better citation scraping possibly using recurrent neural networks or similar technology

Automated updating as new opinions are issued or older ones scraped

Integration with WUSTL Dataset

Possible reformulating data as a single graph database (RDF/SPARQL?)