What-If Analysis of Placement Test Cutoff Scores

Kari Grafton
Lincoln Land Community College

Join the Conversation #WolframTechConf

ABSTRACT

Wolfram Language’s Manipulate function lends itself well to what-if analysis and produces a simple user interface that anyone can use. This research looks at five years of student developmental course placements based on placement test scores, and how those students performed in the course(s), to see if changes are needed in how the college places students within developmental coursework. The format of the output allows the user to try out a number of different scenarios to see how they’d work.

Project Outline

Problem

Research suggests that placement tests may not be the best (e.g., the most accurate) way to determine a student’s competence level in a given subject.

Research Questions

1. How accurately do Accuplacer placements predict a student’s performance in reading, writing, and math courses?
2. If the accuracy is not high enough, will changes to the placement test cutoff scores improve accuracy?

Impact

Developmental courses are well-documented in the research as being a major factor in students not completing a degree or credential. (Attewell, Lavin, Domina, & Levey, 2006)

(Source of data: Lincoln Land Community College Voluntary Framework of Accountability 2019 report)
Fall 2012 cohort (includes new students starting summer or fall of 2012): 2,370 students
Developmental needed in at least one subject: 1,272 students (54%)
Developmental needed & not successful by end of year 6: 560 students (44% of students with developmental need, 24% of overall cohort)
	Note: A student is considered successful if they have received a degree or certificate, have transferred to another college or university, or are still enrolled 6 years from their start date.
	
The overall body of research on the efficacy of developmental education suggests that at best, it does not hurt students, but at worst, it may decrease the likelihood among at least some students of attaining their postsecondary education goals. (Bailey, Jaggars, & Jenkins., 2015; Bailey, Jeong, & Cho, 2010; Boatman and Long, 2010; Calcagno & Long, 2008; Crisp & Delgado, 2014; Scott-Clayton & Rodriguez, 2015)
My research here suggests that Accuplacer placement tests at the college are only accurately placing students about 2/3 of the time. 
This means that up to 373 of the students in this cohort who placed into developmental and were not successful by the end of 6 years may have been held back because they were placed into developmental-level coursework despite not needing it. (16% of the overall cohort)

Steps

1. Extract and clean data (already had this done from another project)
2. Import into Mathematica and format
3. High-level view of data
4. Predictive analysis
5. Build and test model
6. Deploy model (not done because of the findings)

Step 1: Extract and Clean Data

•Cohort: all students who took a placement test in reading, writing, or math from 2013-2017
•Pulled each student’s maximum test score for each test
•Determined the lowest-level course each student placed into in each area
	◦ For college - level math, where the student could start with multiple classes, selected the one the student actually started with
• For each student who completed or withdrew from the first course in the series, found their grade and marked whether it was successful (A, B, C, P) or unsuccessful (D, F, W, RP, SP)
	◦ Students who dropped the course, are currently taking the course, or have not taken the course yet were removed from the cohort
	◦ Any other grade was flagged for me to manually check
• Assigned a dummyID to each student to de-identify the data
• Cleaned data 
	◦ Corrected or removed data where the it did not fall into the correct range
	◦ Grades of I, AP, TR, and CL were removed
	
• Resulting cohort sizes:
	◦ Reading: 1,875 students
	◦ Writing: 5,415 students
	◦ Math: 4,627 students

Step 2: Import Data to Mathematica and Format

Steps

• Import the file, get the appropriate data for each set
• Separate out the first row to designate as the header
• Separate out the remaining rows to designate as the data
• Format the data as a dataset, associating each header value with the appropriate value in each row, and display it

Code

spreadsheet = FileNameJoin[{NotebookDirectory[], "WhatIf data.xlsx"}];

Reading Data

rawReading = Import[spreadsheet, 
    {"Data", 1, Range[1875], 2 ;; 6}]; 
readingHeader = rawReading[[1]]; 
readingRawData = rawReading[[2 ;; All]]; 
readingDataSet = Dataset[(Map[Association])[
    (Thread[readingHeader -> #1] & ) /@ readingRawData]]

Writing Data

rawWriting = Import[spreadsheet,{"Data", 2, Range[5415], 2;;7}];
writingHeader = rawWriting[[1]];
writingRawData=rawWriting[[2;;]];
writingDataSet=Thread[writingHeader->#]&/@writingRawData//Map[Association]//Dataset

Math Data

rawMath=Import[spreadsheet,{"Data",3,Range[4627],2;;8}];
mathHeader=rawMath[[1]];
mathRawData=rawMath[[2;;]];
mathDataSet=Thread[mathHeader->#]&/@mathRawData//Map[Association]//Dataset

Step 3: High-level View of Data

Start with some descriptive statistics

Distribution of Placement Test Scores

GraphicsGrid[{
(* reading *){BoxWhiskerChart[readingDataSet[All,"Reading"],"Outliers",PlotLabel->Style["Reading",Bold,20],PlotRange->{0,120}]
(* writing *)
,BoxWhiskerChart[writingDataSet[All,"SentenceSkills"],"Outliers",PlotLabel->Style["Sentence Skills",Bold,20],PlotRange->{0,120}]
,BoxWhiskerChart[writingDataSet[All,"WritingSample"],"Outliers",PlotLabel->Style["Writing Sample",Bold,20,Blue],
	PlotRange->{0,8},AxesStyle->Directive[Red]]}
(*math *)
,{BoxWhiskerChart[mathDataSet[All,"Arithmetic"],
	"Outliers",PlotLabel->Style["Arithmetic",Bold,20],
	PlotRange->{0,120}]
,BoxWhiskerChart[mathDataSet[All,"ElementaryAlgebra"],
	"Outliers",PlotLabel->Style["Elementary Algebra",Bold,20],
	PlotRange->{0,120}]
,BoxWhiskerChart[mathDataSet[All,"CollegeMath"],"Outliers",
	PlotLabel->Style["College Math",Bold,20],PlotRange->{0,120}]}
}]

Note: 
• Reading scores appear skewed because if student scored 76 or above, they do not need any class in this category, so they are not included in this data set
• Note that the scale on the writing sample test is different from all the others

Bar Chart of Course Placement

(* orders the bars in the chart correctly *)
levelOrder={"3 levels below","2 levels below","1 level below","college level","college level 1","college level 2","college level 3"};
(* reading *)
BarChart[KeyTake[Counts[readingDataSet[All,"level"]],levelOrder],PlotLabel->Style["Reading Placement",Bold,20],ChartLabels->Automatic]
(* writing *)
BarChart[KeyTake[Counts[writingDataSet[All,"level"]],levelOrder],PlotLabel->Style["Writing Placement",Bold,20],ChartLabels->Automatic]
(* math *)
BarChart[KeyTake[Counts[mathDataSet[All,"level"]],levelOrder],PlotLabel->Style["Math Placement",Bold,20],ChartLabels->Automatic]

Note: from this we see that math college level 2 and college level 3 do not have enough data to perform an analysis

Step 4: Predictive Analysis

Build a model to predict what the result on outcomes is when we increase or decrease the placement score cutoffs.
	Independent variable: test score(s)
	Dependent variable: success in first course of sequence

Code

Reading

(* format dataset as list of rules *)
reading=Riffle[{readingDataSet[All,"Reading"]//Normal,readingDataSet[All,"FirstReadingCourse"]//Normal}~Flatten~{2,1}~Partition~2/.{a_,b_}->{a,b},readingDataSet[All,"outcome"]//Normal]~Partition~2/.{a_,b_}->{a->b}//Flatten;

(* Find the accuracy of each classifier method *)
{#,ClassifierInformation[Classify[reading,"Method"->#],"Accuracy"]}&/@{"DecisionTree","GradientBoostedTrees","LogisticRegression","Markov","NaiveBayes","NearestNeighbors","NeuralNetwork","RandomForest","SupportVectorMachine"}//Dataset

Repeat the following steps several times with the methods with the highest accuracy to see how they perform. 
(Current code gave best results overall.)

(* build model with classifier that gives the best results *)
ClassifierInformation[classifyReading=Classify[reading,Method->"LogisticRegression",PerformanceGoal->"Quality"]]

(* check to make sure output from model makes sense *)
classifyReading[{120,"RDG-098"},"Probabilities"]
classifyReading[{120,"RDG-099"},"Probabilities"]
classifyReading[{Missing[],"RDG-098"},"Probabilities"]
classifyReading[{Missing[],"RDG-099"},"Probabilities"]

Show[{
Plot[classifyReading[{x,"RDG-098"},"Probability"->"successful"],{x,0,120},PlotStyle->Green,PlotRange->All,AxesOrigin->{0,0}]
,Plot[classifyReading[{y,"RDG-098"},"Probability"->"unsuccessful"],{y,0,120},PlotStyle->Red,PlotRange->All, AxesOrigin->{0,0}]
}]

Writing

(* remove rows where WritingSample score is missing *)
filteredWritingDataSet=writingDataSet[Select[#WritingSample≥0&]];
(* format dataset as list of rules *)
writing=
Riffle[
{filteredWritingDataSet[All,"SentenceSkills"]//Normal,filteredWritingDataSet[All,"WritingSample"]//Normal,filteredWritingDataSet[All,"FirstWritingCourse"]//Normal}~Flatten~{2,1}~Partition~3,
filteredWritingDataSet[All,"outcome"]//Normal
]~Partition~2/.{a_,b_}->{a->b}//Flatten

(* Find the accuracy of each classifier method *)
{#,ClassifierInformation[Classify[writing,"Method"->#,FeatureTypes->{"Numerical","Numerical","Nominal"}],"Accuracy"]}&/@{"DecisionTree","GradientBoostedTrees","LogisticRegression","Markov","NaiveBayes","NearestNeighbors","NeuralNetwork","RandomForest","SupportVectorMachine"}//Dataset

Repeat the following steps several times with the methods with the highest accuracy to see how they perform. 
(Current code gave best results overall.)

(* build model with classifier that gives the best results *)
ClassifierInformation[classifyWriting=Classify[writing,Method->"LogisticRegression",PerformanceGoal->"Quality",FeatureTypes->{"Numerical","Numerical","Nominal"}]]

(* check to make sure output from model makes sense *)
classifyWriting[{120,8,"EGL-098"},"Probabilities"]
classifyWriting[{120,8,"EGL-099"},"Probabilities"]
classifyWriting[{120,8,"EGL-101"},"Probabilities"]
classifyWriting[{Missing[],Missing[],"EGL-098"},"Probabilities"]
classifyWriting[{Missing[],Missing[],"EGL-099"},"Probabilities"]
classifyWriting[{Missing[],Missing[],"EGL-101"},"Probabilities"]

Math

(* format dataset as list of rules *)
math=
Riffle[
{mathDataSet[All,"Arithmetic"]//Normal,mathDataSet[All,"ElementaryAlgebra"]//Normal,mathDataSet[All,"CollegeMath"]//Normal,mathDataSet[All,"level"]//Normal}~Flatten~{2,1}~Partition~4,
mathDataSet[All,"outcome"]//Normal
]~Partition~2/.{a_,b_}->{a->b}//Flatten

(* Find the accuracy of each classifier method *)
{#,ClassifierInformation[Classify[math,"Method"->#],"Accuracy"]}&/@{"DecisionTree","GradientBoostedTrees","LogisticRegression","Markov","NaiveBayes","NearestNeighbors","NeuralNetwork","RandomForest","SupportVectorMachine"}//Dataset

(* build model with classifier that gives the best results *)
ClassifierInformation[classifyMath=Classify[math,Method->"LogisticRegression",PerformanceGoal->"Quality"]]

(* check to make sure output from model makes sense *)
classifyMath[{120,120,120,"3 levels below"},"Probabilities"]
classifyMath[{120,120,120,"2 levels below"},"Probabilities"]
classifyMath[{120,120,120,"1 level below"},"Probabilities"]
classifyMath[{120,120,120,"college level 1"},"Probabilities"]
classifyMath[{Missing[],Missing[],Missing[],"3 levels below"},"Probabilities"]
classifyMath[{Missing[],Missing[],Missing[],"2 levels below"},"Probabilities"]
classifyMath[{Missing[],Missing[],Missing[],"1 level below"},"Probabilities"]
classifyMath[{Missing[],Missing[],Missing[],"college level 1"},"Probabilities"]

Step 5: Build and Test Model

(* set a color value and a number format to reuse *)
LLCCBlue=RGBColor[0,57/255,166/255];
myPercentPrecision[x_]:=SetPrecision[Interpreter["PercentFraction"][x],4]

Column[{
Style[""],
Style[" What-if Analysis of Placement Test Score Cutoffs ",Bold,38,FontColor->LLCCBlue],
Style["\n  The cutoff score indicates the highest score that will be placed into the listed class.  \n  The number in the bar indicates the percentage of students who fall into the associated category.  ",16],
Style["  Note: If a red dot appears at either end of a slider bar, the current value is outside the possible range.  \n",16,Italic],
TabView[{
Style["Reading",16,Bold]->Manipulate[
(* write function to calculate average probability in a range *)
RDG098recursive[x_Integer]:=If[x≥0,classifyReading[{x,"RDG-098"},"Probability"->"successful"]+RDG098recursive[x-1],
0];
RDG098success[x_Integer]:=RDG098recursive[x]/(x+1);
RDG099recursive[x_Integer]:=If[x>RDG098cutoff,
classifyReading[{x,"RDG-099"},"Probability"->"successful"]+RDG099recursive[x-1],
,0];
RDG099success[x_Integer]:=(RDG099recursive[x]-Null)/(x-RDG098cutoff);
(* display as stacked bar chart *)
BarChart[{
{Labeled[myPercentPrecision[RDG098success[RDG098cutoff]],myPercentPrecision[RDG098success[RDG098cutoff]],Center,Bold,18],
myPercentPrecision[1-RDG098success[RDG098cutoff]]},
{Labeled[myPercentPrecision[RDG099success[RDG099cutoff]],myPercentPrecision[RDG099success[RDG099cutoff]],Center,Bold,18],
myPercentPrecision[1-RDG099success[RDG099cutoff]]}
},ChartLayout->"Stacked"
,ChartStyle->{{LLCCBlue,LightGray}}
,BarSpacing->1.5
,ChartLabels->{Placed[{Style["RDG-098\n",Bold,22],Style["RDG-099\n",Bold,22]},Above],None}
,ChartLegends->Placed[{Style["Successful",18],Style["Unsuccessful",18]},Right]
(*,ImageSize->1020*)
]
(* style info for sliders *)
,{{RDG098cutoff,59,Style["RDG-098 cutoff score",16]},0,120,1,Appearance->"Labeled",ImageSize->Large,BaseStyle->FontSize->16}
,{{RDG099cutoff,75,Style["RDG-099 cutoff score",16]},RDG098cutoff+1,120,1,Appearance->"Labeled",ImageSize->Large}
]
,Style["Writing",16,Bold]->Text["Not finished yet"] (* will be added once code is optimized *)
,Style["Math",16,Bold]->Text["Not finished yet"] (* will be added once code is optimized *)
}]
},Background->LightGray]

Conclusions

The accuracy is not high enough to be considered acceptable. Is it the model or are the placement tests not as good of an indicator of student performance as we originally thought?

Changes to the cutoff scores do not increase the accuracy to an acceptable level.

The literature suggests that placement testing is not the greatest predictor of student knowledge and performance.

Action Taken

The college switched to using a multiple metrics model, using a combination of ACT/SAT scores, high school GPA, and performance in certain high school courses. 
	See Smith, A. A. (2018) for more information on multiple metrics.

References

Attewell, P. A., Lavin, D. E., Domina, T., & Levey, T. (2006). New Evidence on College Remediation. The Journal of Higher Education, 77(5), 886–924. doi: 10.1353/jhe.2006.0037

Bailey, T., Jeong, D. W., & Cho, S.-W. (2010). Referral, enrollment, and completion in developmental education sequences in community colleges. Economics of Education Review, 29(2), 255–270. doi: 10.1016/j.econedurev.2009.09.002

Bailey, T. R., Jaggars, S. S., & Jenkins, D. (2015). Redesigning Americas community colleges: a clearer path to student success. Cambridge, MA: Harvard University Press.

Boatman, A., & Long, B. T. (2017). Does Remediation Work for All Students? How the Effects of Postsecondary Remedial and Developmental Courses Vary by Level of Academic Preparation. Educational Evaluation and Policy Analysis, 40(1), 29–58. doi: 10.3102/0162373717715708

Calcagno, J. C., & Long, B. T. (2008). The Impact of Postsecondary Remediation Using a Regression Discontinuity Approach: Addressing Endogenous Sorting and Noncompliance (NBER working paper No. 14194). doi: 10.3386/w14194

Crisp, G., & Delgado, C. (2013). The Impact of Developmental Education on Community College Persistence and Vertical Transfer. Community College Review, 42(2), 99–117. doi: 10.1177/0091552113516488

Department of Institutional Research, Lincoln Land Community College (2019). Voluntary Framework of Accountability. Internal Lincoln Land Community College report: unpublished.

Scott-Clayton, J., & Rodriguez, O. (2015). Development, Discouragement, or Diversion? New Evidence on the Effects of College Remediation Policy. Education Finance and Policy, 10(1), 4–45. doi: 10.1162/edfp_a_00150

Smith, A. A. (2018, September 20). New report encourages use of multiple measurements for student placement. Retrieved from https://www.insidehighered.com/news/2018/09/20/new-report-encourages-use-multiple-measurements-student-placement.

Contact Info

Kari Grafton
Lincoln Land Community College
Kari.Grafton@LLCC.edu