Finding Bugs in the Wolfram Language

Good tools make good workers. - French proverb

Brenton Bostick, Wolfram Research

brenton@wolfram.com

Outline

Motivation

Quiz

Lint paclet

CodeAssistance

Other editors

Motivation

Why preemptively find bugs?

More expensive to find "the hard way".

Some (in)famous software failures

Mars Climate Orbiter

imperial / metric mixup -> “spacecraft encountered Mars on a trajectory that brought it too close to the planet”

Ariane 5 Flight 501

64-bit / 16-bit conversion error -> $370m self-destruction fireball

Therac-25

race condition -> radiation overdoses and deaths

Quiz

Can you spot problems in WL code?

Quiz

what's the problem here?

makeButton[label_ func_] := 
  With[{func1 = func}, Button[label, func1[], ImageSize -> Automatic,]
    ]

Quiz

what's problem here?

iFilterRules[a__, b_] :=
  	  	    Block[{res, opts = Flatten[{a}]},
  	  	
  	  	        res = Complement[opts[[All, 1]], b[[All, 1]]];
  	  	        If[res =!= {}, Message[OptionValue::optnf, res, HPC]];
 	  	
  	  	        opts = FilterRules[opts, b];
  	  	        If[opts === {}, Return[b]];
  	  	
  	  	        res = FilterRules[b, Except[Alternatives @@ opts[[All, 1]]]];
  	  	        Join[opts, res]
  	  	
  	  	    ]
	  	  	
  	  	iFilterRules[____] := {}

Quiz

what's problem here?

agree[self_, sub_?TypeSubstitutionQ] :=-
	AllTrue[
	        Intersection[
	           self["varMap"]["values"],
	           sub["varMap"]["values"],
	           SameTest -> sameQ
	        ],
	        TypeUnifiableQ[self["apply", #], sub["apply", #], "TypeEnvironment" -> self["getProperty", "TypeEnvironment", Undefined]]&
	    ]

Quiz

what's the problem here?

EC2InstanceQ[] :=
    Module[{res},
      Switch[$OperatingSystem,
        "Windows",
        res = 
          StreamExecute[
            "!wmic path win32_computersystemproduct get uuid"];
        res = 
          MatchQ[res, 
            str_?StringQ /; StringStartsQ[str, "ec2", IgnoreCase -> True]]
        ,
        "Linux",
        Which[FileExistsQ["/sys/hypervisor/uuid"], 
          res = ReadString["/sys/hypervisor/uuid"];
          res = 
            MatchQ[res, 
              str_?StringQ /; StringStartsQ[str, "ec2", IgnoreCase -> True]],
           FileExistsQ["/sys/class/dmi/id/board_vendor"], 
          res = ReadString["/sys/class/dmi/id/board_vendor"];
          res = 
            MatchQ[res, 
              str_String /; StringMatchQ[str, "Amazon EC2" ~~ __]], True, 
          False]
        ,
        _,
        False]
      ];

Moral of the Story

WL syntax is complicated

There are a lot of operators in WL

Unintuitive precedences

Implicit Times is its own beast

Leads to:

missing commas

missing semicolon

etc.

all being valid syntax

Static Analysis

How to analyze code without evaluating it

Use pattern matcher!

Needs["AST`"]

ast=ParseString["a+1+2+3+4.5"]

process[ast_]:=
Module[{ints,syms},
ints=Cases[ast,LeafNode[Integer,_,_],Infinity];
syms=Cases[ast,LeafNode[Symbol,_,_],Infinity];
Print["found ",Length[ints]," integers!"];
Print["found ",Length[syms]," symbols!"]
]

process[ast]

Static Analysis is difficult for WL

WL is:

Very dynamic

Weakly typed

Symbol creation

symbols can be introduced without declaration

hard to know if missing a Needs[]

implicit Times

who has done this accidentally?

foo[] :=
Module[{res1, res2},
  res1 = func1[]
  res2 = func2[res1]
  res
]

Operator forms

Now valid syntax:

m = Map[f]

res = StringMatchQ["foo"]

Runtime checking is also difficult (but not discussed here)

Returning unevaluated

functions return unevaluated with warnings or errors

Clear[a,b]

Which[a==b,1]

Can lead to accidental propagation:

Clear[a,f,func1]

a=<|"key1"->func1|>;
f=a["key"];
f[]

Building-up unwanted DownValues

forget to use Clear in interactive code

Lint paclet

Needs["Lint`"]

LintString["If[a,b,b]"]

LintStringReport["If[a,b,b]"]

Linting files

codetoolsExamplesDir=NotebookDirectory[];

file=FileNameJoin[{codetoolsExamplesDir,"Collatz","Collatz.m"}];

LintFile[file]

LintFileReport[file,ConfidenceLevel->0,"SeverityExclusions"->{}]

Can also integrate with other editors

Lint`External`$Editor="Sublime Text";

LintFileReport[file,ConfidenceLevel->0,"SeverityExclusions"->{}]

Implicit Times

This is important because it will always be hard to find problems.

Need to display all implicit Times for the user to see.

Needs["Lint`ImplicitTimes`"]

ImplicitTimesString["a b"]

ImplicitTimesStringReport["a b"]

codetoolsExamplesDir=NotebookDirectory[];

file=FileNameJoin[{codetoolsExamplesDir,"Collatz","Collatz.m"}];

LintFile[file]

ImplicitTimesFileReport[file]

Command line

Lint` supports formatted output for command line

Demo CodeAssistance

Latest effort is adding support for reporting inside notebooks

Setup

Needs["CodeAssistance`"]

EnableCodeAssistance[EvaluationNotebook[]]

Example #1 (Extra ,)

Stray commas at the end of arguments mean that there is a Null inserted at the end.

makeButton[label_,func_]:=With[{func1=func},Button[label,func1[],ImageSize->Automatic,]
]

Example #2 (unexpected _)

It is common to accidentally type ____ (i.e., 4 _) when defining a catch-all for a function.

The correct code would be ___ (i.e., 3 _).

Because of implicit Times in WL, ____ is valid syntax.

iFilterRules[a_]:=a

iFilterRules[a__,b_]:=
 Block[{res,opts=Flatten[{a}]},
res=Complement[opts[[All,1]],b[[All,1]]];
If[res=!={},Message[OptionValue::optnf,res,HPC]];
opts=FilterRules[opts,b];
If[opts==={},Return[b]];
res=FilterRules[b,Except[Alternatives@@opts[[All,1]]]];
Join[opts,res]
]

iFilterRules[____]:={}

Example #3 ($OperatingSystem and Linux)

When branching on values of $OperatingSystem, it is common to write Linux instead of Unix.

EC2InstanceQ[]:=
Module[{res},
Switch[$OperatingSystem,
"Windows",
res=StreamExecute["!wmic path win32_computersystemproduct get uuid"];
res=MatchQ[res,str_?StringQ/;StringStartsQ[str,"ec2",IgnoreCase->True]]
,
"Linux",
Which[FileExistsQ["/sys/hypervisor/uuid"],res=ReadString["/sys/hypervisor/uuid"];
res=MatchQ[res,str_?StringQ/;StringStartsQ[str,"ec2",IgnoreCase->True]],FileExistsQ["/sys/class/dmi/id/board_vendor"],res=ReadString["/sys/class/dmi/id/board_vendor"];
res=MatchQ[res,str_String/;StringMatchQ[str,"Amazon EC2"~~__]],True,False]
,
_,
False]
];

Teardown

DisableCodeAssistance[EvaluationNotebook[]]

Demo BracketMatcher

From  Timothée Verdier

Using machine learning to understand where brackets should be added / removed.

Setup

Needs["LanguageModel4Code`"]

EnableBracketMatcher[EvaluationNotebook[]]

Example #1 missing }

No curly after the with:

makeButton[label_,func_]:=With[func1=func},Button[label,func1[],ImageSize->Automatic]
]

Example #2 missing ]

random bracket was removed

iFilterRules[a_:=a

iFilterRules[a__,b_]:=
 Block[{res,opts=Flatten[{a}]},
res=Complement[opts[[All,1]],b[[All,1]]];
If[res=!={},Message[OptionValue::optnf,res,HPC]];
opts=FilterRules[opts,b];
If[opts==={},Return[b]];
res=FilterRules[b,Except[Alternatives@@opts[[All,1]]]];
Join[opts,res]
]

iFilterRules[___]:={}

Example #3 too many parenthesis

x +3*((2+y)^4

Example #4 association

table={<|a->1,b->2|>,<|c->3,d->1,<|a->4,b->3|>};

Teardown

DisableBracketMatcher[EvaluationNotebook[]]

Other editors and IDEs

Language Server Protocol

A Language Server is meant to provide the language-specific smarts and communicate with development tools over a protocol that enables inter-process communication.

LSP is a win for both language providers and tooling vendors!

Sublime Text

<<show>>

Visual Studio Code

<<show>>

Future

Ideas for lint patterns

let me know!

More semantics

More interaction with FE

squiggly lines!

If[a,b,b]

More LSP clients

Thank you!

Links

Mars Climate Orbiter disaster

https://www.bugsnag.com/blog/bug-day-mars-climate-orbiter

What are the most common pitfalls awaiting new users?

https://mathematica.stackexchange.com/questions/18393/what-are-the-most-common-pitfalls-awaiting-new-users

https://community.wolfram.com/groups/-/m/t/1070264

Obtaining Lint paclet

PacletUpdate["Lint","Site"->"http://pacletserver.wolfram.com","UpdateSites"->True]

Obtaining CodeAssistance paclet

PacletUpdate["CodeAssistance","Site"->"http://pacletserver.wolfram.com","UpdateSites"->True]

Obtaining LSPServer paclet

PacletUpdate["LSPServer","Site"->"http://pacletserver.wolfram.com","UpdateSites"->True]

Language Server Protocol

https://microsoft.github.io/language-server-protocol/