CodeSeal:
A System for Teaching the Art of Scientific Narratives

Dr. Kyle Keane
George Varnavides
Prof. W. Craig Carter

http://weller.mit.edu
www.CodeSeal.org

Join the Conversation #WolframTechConf

Who We Are

CODE SEAL

CODE, Science, Engineering And Learning

Code | See | Learn

www.CodeSeal.org

Contents

Authoring Process

Grading Workflow

Assessment

Data Collection

Educational Research

Authoring Process

Crowdsourcing Computational Curriculum

Step 1 of 3: Author content and add markup

→

Step 2 of 3: Upload onto CodeSeal to convert into a module

Step 3 of 3: Assign module to class

Grading Workflow

Offline and Self-paced

Step 1 of 4: Student downloads assignment

Standardized pre-assignment survey and assessment (Coming Soon)

Step 2 of 4: Student completes assignment offline

Extensive data collection about student work (Discussed Later)

Step 3 of 4: Student uploads completed assignment

Standardized post-assignment survey and assessment (Coming Soon)

Step 4 of 4: Generate augmented grading report

There is some exciting stuff on the horizon (Discussed Later)

Assessment

Learning Goals and Outcomes

What we are testing

So far, we’ve been showing small code snippets and calling them “student solutions”

However, writing code is just one component of their work

Students must also construct a full narrative that

can be used by a peer to learn about and expand upon the topics covered

combines logic, scientific concepts, mathematical reasoning, and computer programming

demonstrates a thorough and independent exploration of a topic

We expect students to become professional journal authors

Example solution from Amina

For a canonical ensemble that is classical and discrete, the canonical partition function is defined as

Z=(∑_(n=0)^∞)Exp[-(En/(k T))]

where n is the index for the microstates of the system and En is the total energy of the system in the respective microstate.

In our problem the total energy of the system in the microstate n is

En = n m g h

Therefore, with τ, the normalized temperature the partition function is

Z=(∑_(n=0)^∞)Exp[-((n m g h T)/k)]=(∑_(n=0)^∞)Exp[-(n/tau)]

and can be easily coded as shown below.

partfunctionSingle= Sum[Exp[-n/tau],{n,0,Infinity}] (*Note: it is important to put the zero (our first energy is n=0) in the interval as by default summation in mathematica begins at 1*)

The partition function usefulness is not obvious. One important point is to notice that it depends on microstates energies, that are themselves depending on microscopic quantites (such as mass). Thus, the partition function models the microscopic constituents of a system, and as we will see later, will allow us to calculate thermodynamic properties of the system. Therefore, the partition function links the macroscopic properties to microscopic variables which is an essential point of statistical mechanics.

Beyond standard digital assessment

Standard digital assessment is “multiple choice” questions 
(bad proxy for applied, contextual recall)

Fancy is “short answer” questions
(saying keywords or phrases is not effective communication)

Cutting edge is “automated code verification”
(completely counter-productive to critical thinking)

We require students to

apply concepts in the context of solving real problems

communicating their findings such that their peers could learn from them.

The previous example is uncommonly short.

Students often construct a modern 5 paragraph essay and some become so engaged they write multiple page answers.

Data Collection

What is going on here?

What we are collecting

Students who opt to the contribute their data to our research provide information about:

Every iteration of each piece of computer code

Every key that was pressed when typing in the module

Any submitted solutions to problems

All actions taken in the interface (opening sections, submitting work, viewing solutions)

The time the file is downloaded, and how many downloads

When they began working on the module

Any comments or feedback they gave about the module

Keypresses

What are they typing?

What we are collecting

Yup, every single one

(CopyTag)

When were they typing?

Long-term study habits

When were they typing?

Zomming in

Where were they typing?

We can now understand what Amina was doing and when

(CopyTag)

What were they typing where?

Top keys used when coding

(CopyTag)

Evaluations

What are they coding?

What we are collecting

Yup, every one of them

(CopyTag)

Who’s code were they running when?

We are collecting each piece of code the students writes while developing their solution

What edits were they making?

Each tree represents a piece of code the student evaluated

What were they trying to build?

Link this up to the solution timestamps

There were over 30 iterations of the code to get this solution

When were mistakes happening?

This student has a good grasp of the grammar

Where they using new symbols?

By the end of the module they have used >100 new symbols

E

How was the size of their code evolving?

They seem to be evaluating larger pieces of code throughout the module

Student Feedback

Are they self-identifying as confused?

What data we are collecting

Students are presenting step-by-step solutions and must rate each step to proceed

(CopyTag)

Where were students confused?

3 in third column means they were confused

(CopyTag)

What was happening in that step?

We can look into ppublished module to find the actual cell

{ 102 , 37, 3 }

Show[comparisonPlot,
Plot[{model/.fitOneHalf, model/.fitOneThird},{t,-1,30000},
PlotRange-> All,
PlotStyle->{Directive[
Thickness[0.02],Opacity[0.2],Red],
Directive[
Thickness[0.02],Opacity[0.2],Blue]
},
PlotPoints->200]]

Basic Statistics

What does this grade mean?

Do more keypresses while coding mean a better score?

not necessarily

Do more keypresses while writing mean a better score?

Yes, they sure do. Remember the making students into authors thing?

Advanced Data Analysis

Graph Theory Analysis

Graph communities

Digging into communities

Communities correspond to observable study habbits

For example: 
		"Diligent” students
		“Typical” MIT procrastinators (local time optimizers)

Dendrograms

Machine Learning

Very simple model

Only 34 examples

13 features (a lot! Over-fitting?)

(CopyTag)

Confusion Matrix

50% Accurate (not too shabby for 34 examples)

More importantly, it (probably) doesn’t matter

Used for variable importance - not prediction!

The basic idea

For each feature i, 1 ≤ i ≤ k
	De-associate values of (i^th) column from rest of dataset
	Re-run accuracy tests
	Compare Accuracy Drop
Large Drop -> Important variable

De-associate? How??
	Shuffling
	Missing

Simple feature importance

Simple feature importance

Inter-dependencies?

Pair-wise

Inter-dependencies?

List-wise

Thank You

http://weller.mit.edu
www.CodeSeal.org