Initialization

This file’s Accession number:: 648740159897

ℙ="D:\\wtc19Packages";

SetDirectory[ℙ];

Quiet@Needs["generalizedConfidenceIntervals`generalConfidenceIntervalsPackageV0R4`"]

?generalConfidenceIntervalsPackageV0R4

Quiet[Needs[
"DTRMFTURPP`DTRMFTURPPackageV1R0`"
]]

?DTRMFTURPPackageV1R0

SetDirectory[𝔾="D:\\Users\\DinoBe\\Desktop\\wtc19PackageDirectory\\getAcronymsProject"];

Get["fullComp"]

ResetDirectory[];

fullComp::usage="List[].  Contains the metric of common acronyms for all combinations of chapters in E:\\Users\\DinoBe\\Desktop\\wtc19PackageDirectory\\getAcronymsProject\\DTR_MFTURP_text_search ; Note that fullComp is a List of List, and should be flattened before being used as input to Histogram[].";

ResetDirectory[];

Promise of Mathematica in DoD Operations Research (OR)

William Lewis

Join the Conversation #WolframTechConf

ABSTRACT

Working operations research (OR) is a very mixed bag.  Academic OR functions have already been applied as vendor supplied packages.  Most of the current work is in data acquisition/estimation, finding the right mathematics, and writing the software (if any).  

The analyst is a generalist, using general knowledge to select mathematical approaches that Mathematica implements.

This presentation gives three examples of DoD OR work and draws some general conclusions about the utility of Mathematica.

Three Grand projects

1: What is the fundamental design objective of Service Oriented Architecture (SOA)?

2: Can you control a stochastic process that has a non-Normal distribution?

3: How can you check voluminous regulations for consistency?

(1^st) OR Example: Service Oriented Architecture (SOA) 
(~6 months work)

Q: What was SOA developed to do?

A: Avoid the “freeze” that makes previous software architectures impossible to alter after a decade or two, and inoperable shortly thereafter.

Q: Is SOA needed?

A: Yes.  We have a generation of corporate resource management software that is becoming unusable.  It should be replaced by SOA.  SOA could implement a market metaphor, and markets persist indefinitely without Big Ball of Mud (BBOM) freeze that eventually makes current architectures inoperable.  “BBOM freeze” referes to the gradual loss of structure under maintenance changes that gradually leaves the software with as little organization as a big ball of mud, at which point any change (including bug fixes) introduces more than one bug.  At that point, the software is “frozen” and mainteance ceases.

Q: Can SOA really deliver?

A: No, not in its present form.

In its present form, SOA is used to build opaque monolithic systems that do not support markets and are subject to BBOM freeze.

However, given a framework and motivated administrators, SOA could implement a market that would resist BBOM freeze.

SOA is intended to support a free market in services and data [1].  Such a market needs frameworks for product definition and for legally enforceable contracts.  It also needs a cadre of administrators familiar with running such a market.  Neither frameworks nor administrators nor plans to develop either exist.  I know of no serious development work, nor, given the generally anti-market mentality of most software companies, is serious development work likely.

In this study Mathematica was used only to determine load characteristics of the Enterprise Service Bus for hierarchical SOA networks.  Conclusion: SOA can be safely configured as a hierarchical network.

1] Search for "OASIS soa-rm soa-ra".

(2^nd) OR Example: Controlling a stochastic process having an arbitrary distribution.
(~ 2 months work)

Control charts and Six Sigma assume the process to be controlled has a Normal distribution.  Can processes that are not so distributed be controlled?

Example: Lognormal[] distribution

Language extension: generalConfidenceIntervals[] calculates confidence intervals for most Mathematica supported distributions.  See: genralizedConfidenceIntervalsPackage.nb in session downloads.

?generalConfidenceIntervals

Region 1: Nearly Normal Lognormal.

Visualization

Manipulate[Plot[PDF[LogNormalDistribution[α,β],x],{x,0.001,10},{PlotLabel->"x vs. PDF[LogNormalDistribution["<>ToString[ α]<>", "<>ToString[N[β,2]]<>"],x]",AxesLabel->{"x","PDF[LND[α,β], x]"},GridLines->{{Mean[LogNormalDistribution[α,β]]}},PlotRange->{{0,6},{0,3}}}],{{α,1},(1./(10^4)),35/10},{{β,(1/10)},(1./(10^4)),5}]

Confidence intervals

With[{𝒟=LogNormalDistribution[1,(1/10)]},With[{mu=Mean[𝒟],ds=Table[Values[generalConfidenceIntervals[(k/10),𝒟]],{k,9}],plotLabels={"{0, x}","{x, ∞}","Symmetric about mean"}},Array[NumberLinePlot[ds[[All,#]],GridLines->{{mu},{}},PlotLabel->plotLabels[[#]]]&,3]]]

Count of points outside the confidence intervals appears to be Poisson[n] distributed, where n is the number of points expected to be outside the CI for the sample size gathered.

Max symmetric CI exists only for the probability interval {0, ~.96}

maxCI2=With[{𝒟=LogNormalDistribution[1,(1/10)]},N[2] (1-CDF[𝒟,Mean[𝒟]])]

And its boundaries are:

N[With[{𝒟=LogNormalDistribution[1,(1/10)]},With[{𝒫=Rationalize[2 (1-CDF[𝒟,Mean[𝒟]]),(1/100000000000)]},generalConfidenceIntervals[𝒫,𝒟]]]["toInfinity"]]

Conclusion: you can control this process.

Region 2: Skewed Normal, long tail

Visualization

Manipulate[Plot[PDF[LogNormalDistribution[α,β],x],{x,0.001,10},{PlotLabel->"x vs. PDF[LogNormalDistribution["<>ToString[ α]<>", "<>ToString[N[β,2]]<>"],x]",AxesLabel->{"x","PDF[LND[α,β], x]"},GridLines->{{Mean[LogNormalDistribution[α,β]]}},PlotRange->{{0,6},{0,3}}}],{{α,1},(1/(10^2)),(200/100),1/100},{{β,(3/10)},(1./(10^2)),5,(1/100)}]

Print["Mean[LogNormalDistribution[1,(3/10)] == ",EngineeringForm[N@Mean[LogNormalDistribution[1,(3/10)]],3]]

Confidence intervals

With[{𝒟=LogNormalDistribution[1,(3/10)]},With[{mu=Mean[𝒟],ds=Table[Values[generalConfidenceIntervals[(k/10),𝒟]],{k,9}],plotLabels={"{0, x}","{x, ∞}","Symmetric about mean"}},Array[NumberLinePlot[ds[[All,#]],GridLines->{{mu},{}},PlotLabel->plotLabels[[#]]]&,3]]]

Max symmetric CI exists only for the probability interval {0, ~.88}

maxCI3=With[{𝒟=LogNormalDistribution[1,(3/10)]},N[2] (1-CDF[𝒟,Mean[𝒟]])]

And its boundaries are:

N[With[{𝒟=LogNormalDistribution[1,(3/10)]},With[{𝒫=Rationalize[2 (1-CDF[𝒟,Mean[𝒟]]),(1/100000000000)]},generalConfidenceIntervals[𝒫,𝒟]]]["toInfinity"]]

Conclusion: you can control this process, but not as closely as the preceding process.

Region 3: Super thick tail

Visualization

Manipulate[Plot[PDF[LogNormalDistribution[α,β],x],{x,0.001,10},{PlotLabel->"x vs. PDF[LogNormalDistribution["<>ToString[ α]<>", "<>ToString[N[β,2]]<>"],x]",AxesLabel->{"x","PDF[LND[α,β], x]"},GridLines->{{Mean[LogNormalDistribution[α,β]]}},PlotRange->{{0,6},{0,3}}}],{{α,3},(1./(10^4)),35/10},{{β,(36/10)},(1/(10^4)),5}]

Print["Mean[LogNormalDistribution[3,36/10] == ",EngineeringForm[N@Mean[LogNormalDistribution[3,36/10]],3]]

Note that half the events are 20 or less, but Mean[PDF[LogNormalDistribution[3,36] ≃13,100.

Plot[CDF[LogNormalDistribution[3,(36/10)],x],{x,0,25},GridLines->Automatic,PlotRange->All,PlotLabel->"x vs. CDF[LogNormalDistribution[3,(36/10)],x]",AxesLabel->{"x","CDF[𝒟,x]"}]

Ultra thick tail.  Note that the mean, about 13.1*(10^3),  doesn’t even show up on the above plot.

Confidence intervals

No symmetric confidence intervals region of interest.
No asymmetric confidence intervals of interest, 
Note that Mean[LogNormalDistribution[3,(36/10)]] ≃ 13,100 (3 places)

With[{𝒟=LogNormalDistribution[3,(36/10)]},With[{mu=Mean[𝒟],ds=Table[Values[generalConfidenceIntervals[(k/10),𝒟]],{k,9}],plotLabels={"{0, x}","{x, ∞}","Symmetric about mean"}},Array[NumberLinePlot[ds[[All,#]],GridLines->{{mu},{}},PlotLabel->plotLabels[[#]],PlotRange->{0,2200}]&,3]]]

Max symmetric CI exists only for the probability interval {0, ~.071}

maxCI1=With[{𝒟=LogNormalDistribution[3,(36/10)]},N[2] (1-CDF[𝒟,Mean[𝒟]])]

And its boundaries are:

N[With[{𝒟=LogNormalDistribution[3,(36/10)]},With[{𝒫=Rationalize[2 (1-CDF[𝒟,Mean[𝒟]]),(1/100000000000)]},generalConfidenceIntervals[𝒫,𝒟]]]["toInfinity"]]

Conclusion: you can control this process, but only very loosely.

(3^rd) OR Example: 
Q: Can regulations be checked for inconsistencies?
(~2 months work)

Use HP Autonomy’s “common jargon” approach
Implementation: Find chapters whose acronym sets are similar.

All chapter titles have the same format.

Acronyms are jargon with a well defined pattern (caps, 4-7 characters, some symbols, no previous meaning (mostly)).

String processing finds chapter titles and acronyms

level1Partition[txt_String]:=Module[{splitString},splitString=StringSplit[txt,{StartOfLine~~StartOfLine~~x:(Repeated[LetterCharacter,{1,2}])?(UpperCaseQ[#1]&)~~"."~~" "..~~WordCharacter?(UpperCaseQ[#1]&)..}:>x,All];Prepend[Table[{splitString[[k]],splitString[[k+1]]},{k,2,Length[splitString]-1,2}],splitString[[1]]]
]

Get all acronyms in each chapter.

SetDirectory[":\\Users\\DinoBe\\Desktop\\wtc19PackageDirectory\\getAcronymsProject\\DTR_MFTURP_text_search"];

ds = makeDataSetDTR[FileNames[]]

ds[20 ;; 22, {Hash[#Tag, "Adler32"], #Acronyms, Snippet[#English, 1]} &]

(CopyTag)

Define a metric for comparing sets

getMetric::notes

Apply the metric to all possible combinations of chapters, using hashes to ID different tags.

similarity = ({#1[[1]], Hash[#1[[2, 1]], "Adler32"], Hash[#1[[2, 2]], "Adler32"]} &) /@ Reverse[Sort[panMetricsList[ds[[20 ;; 22]]]]]; TableForm[similarity, TableHeadings -> {None, {"Similarity", "Hash[tag1]", "Hash[tag2]"}}]

({{"Similarity", "Hash[tag1]", "Hash[tag2]"}, {0.25`3., 447099420, 447164957}, {0, 447164957, 447230494}, {0, 447099420, 447230494}})

A human reads most similar pairs, to find contradictions.

Histogram[Flatten[fullComp], PlotRange -> All, PlotLabel -> "Histogram, metric vs. binCount\nAll chapters", AxesLabel -> {"metric", "count"}]

Conclusion: Comparing only pairs with metric ≥ 0.2 eliminates 97.87% of comparisons, leaving 5821 chapter pairs for a human to check.

At 1/3 hour per comparison, that would take about 300 work days of labor.

Q : Can regulations be checked for errors?

A : Yes, but it would be expensive,  months Mathematica + one person for a year.

Many fields require generalists

Generalists introduce hypotheses and techniques from different fields

Testing hypotheses and techniques is possible only with Mathematica support

Wolfram Language cognition enhancers

LearnDistribution[], Solve[], Simplify[], Plot[].

Successes: 
LearnDistribution[], FindDistribution[], Solve[], Simplify[], Plot[], DataSet[].

Almost: 
FourierTransform[], integration over regions, FindEquationalProof[].

Mathematica is necessary for exploratory OR

That’s all, folks!