CREATING A PROGRAM FOR DATA-INTENSIVE FIELD CAMPAIGNS

Colleen Marciel F. Rosales
O’Neill School of Public and Environmental Affairs, Indiana University
cfrosale@iu.edu

Join the Conversation #WolframTechConf

Outline

-> What is a Field Campaign

-> Computational Needs

-> Why Mathematica

-> Tasks completed using Mathematica

Field Campaigns

-> In atmospheric research, these are “observational studies planned for a specific location and a defined time period during which measurements are conducted from airborne platforms and/or ground sites to study physical and chemical processes in the atmosphere” (Atmospheric Data Science Center, NASA)

-> Usually a multiple-research-group effort

IRRONIC II (2017)

Indiana Radical Reactivity and Ozone Production Intercomparison II

(c) Philip Stevens / psslab.indiana.edu

HOMEChem (2018)

House Observations of Microbial and Environmental Chemistry

(c) Callie Richmond/HOMEChem https://indoorchem.org/projects/homechem/

IRACE (2019)

Indiana Radical and Aerosol Chemistry Experiment

(c) Brandon Boor / @BrandonBoor (Twitter)

LIF-FAGE instrument

-> Based on the fluorescence of the OH (hydroxyl) radical at 308 nm
-> Indirect measurement of other radicals (hydroperoxyl, peroxyl)

Data Collection

Frequency: 1 to 2 points per second

-> Status (Online/Offline)

-> Photon Counts

-> Pressure

-> Voltage

-> Wavelength

-> Flow Rates

etc...

Post-collection data processing

How many operations are required?

30 days 
x 12 hours/day 
x 3600 sec/hr
x 1.5 points/sec 
x 8 parameters/point
x 41 operations/parameter (including sub-operations)

30*12*3600*1.5*8*41

≈ 600 million operations!!

MS Excel and VBA

MS Excel

Handy for quick look of data
Slow for large data sets & calculations
Little or no traceability!

VBA

Helpful for saving output files as individual sheets
Traceable
Program coding is possible
Tedious, long coding (~500 pages of code)
Ran~30 mins for each data set (12 hours data)

Why Mathematica?

Wanted to make data processing faster and traceable

-> Natural - language input, including traditional math notation

-> Complete documentation ("Help") & debugger

-> "Document-based workflow"
	Programming + visualizing + calculations + text all in one window

-> SetDelayed allows defining own functions
	
-> Buttons and fill - in forms!

Tasks completed using Mathematica

-> Consolidating multiple files

-> Matching data from different sources with different formats

-> Creating an "Extractor" program

What can the Extractor do?

Filter irrelevant, blank and/or NaN fields

Position[rawOut, s_String /; StringMatchQ[s,"End"]]

DeleteCases[rawOut,row_/;row=={}]; (* Delete blank rows *)
DeleteCases[rawTable1,{_,_,_,_,_,_,_,_,_,_,_,_,"LOCK",_,_,_,_}];
DeleteCases[rawTable1,{_,_,_,_,_,_,_,_,_,_,_,_,"Test Zero PD",_,_,_,_}];
DeleteCases[rawTable1,{_,_,_,_,_,_,_,_,_,_,_,_,"Test Laser Scatter",_,_,_,_}];

Average online and offline points

Pick[list,pattern]

SplitBy[rawTable,First] (* Split rawTable into blocks/"sheets" according to status *)

N@Mean[x]
N@StandardDeviation[x]

Calculate values only for online points

Which[
test1,result1,
test2,result2
]

If[test1,result1,null]

Erase the first and last points of each block/"sheet"

segregatedDatadropf[x_]:=
If[Length[segregatedData1[[x]]]>1,Drop[segregatedData1[[x]],{1,-1,Length@segregatedData1[[x]]-1}],""];

Data Matching & Interpolation

Position[waterimport,headlabel]

Interpolation[dataseries]

Export and Consolidate Files

AppendTo[s,elem]
Export["extracted.csv",results]

Plot Concentrations

DateListPlot[timeseries]

Extractor Program - Interactive program with a GUI

.out File Extractor for IU LIF-FAGE

About

>> based on VBA_extractor_V9q_rev6 (originally written in VBA for MS Excel by Sebastien Dusanter)
>> for use on .out files directly from the LabView program in the HOx computer
>> written for Wolfram Mathematica 11.3 by Colleen Rosales (cfrosale@iu.edu)
	also works in Mathematica 11.1, but optimum at 11.3

Instructions for Importing Water Values

1. Create a .csv file with a (1) “DateTime” column header and a (2) “water” column header. Only these two quantifiers/keywords are case-sensitive. The program will ignore all other columns.

Make sure that:
	DateTime column is in the format as in the above example (Month (in words), Day, Year, 		Hour:Minutes:Seconds)
	water column is in percent, but ignoring the percent sign.

2. Importing water takes time. Once it is done, you will see the interpolating function as an output:

Extractor

Button["Execute Extractor Program",
CreateDocument[{
Text[Row[{"Extractor for .out files from IU LIF-FAGE"},BaseStyle-> {"Section",Italic},Alignment->Center]],
Row[{Button["Clear global variables before starting a new extraction",Clear@"Global`*", ImageSize->{450,30},BaseStyle->{"Title",15,Bold}]},Alignment-> Center],
Image[,ImageSize-> 40],
Row[{
Button["Set Parameters",
CreateDialog[Grid[
{
{},
{"1. Laser Power and (Subscript[UV, out]) ratio (Save in separate .out files if applying a different power and (Subscript[UV, out]))"},
{"   Laser Power (mW)",InputField[Dynamic[pwin],Number]},
{"   (Subscript[UV, out]) (V)",InputField[Dynamic[uvoutin],Number]},
{"   Comments (ex. time laser power and UVout were taken, dye change, etc)",InputField[Dynamic[timein],String]},
{},
{},
{"2. (Subscript[HO, 2]) Conversion Factors: \!\(\*SubscriptBox[\(C\), \(\*SubscriptBox[\(HO\), \(2\)] -> \\ OH\)]\) = A(Flow NO)+B"},
{"   A (slope)",InputField[Dynamic[facA],Number]},
{"   B (intercept)",InputField[Dynamic[facB],Number]},
{"   NO (sccm) threshold for (Subscript[HO, 2])",InputField[Dynamic[scHO2],Number]},
{"   NO (sccm) threshold for (SuperStar[(Subscript[HO, 2])])",InputField[Dynamic[scHO2st],Number]},
{},
{},
{"3. Sensitivity Factors"},
{"   Sensitivity slope (format: 1E-7)", InputField[Dynamic[slp],String]},
{"   Sensitivity intercept ((Subscript[R, OH])) (format: 1E-7)",InputField[Dynamic[int],String]},
{},
{},
{"4. NO interference"},
{"   Include NO interference (linear relationship) in calculations? (Y/N)", InputField[Dynamic[noYN],String]},
{"   (Subscript[UV, out])/power ratio measured during interference test", InputField[Dynamic[interf],Number]},
{"   Slope, A", InputField[Dynamic[nodep],Number]},
{"   y-intercept, B", InputField[Dynamic[pwdep],Number]},
{},
{},
{"5. File to be processed (.out):      ",FileNameSetter[Dynamic[d],"Open"]},
{}, 
{"6. Import water data? (Y/N)",InputField[Dynamic[waterYN],String]},
{"   If importing water file, select .csv water file:      ",FileNameSetter[Dynamic[w],"Open"]},
{},
{},
{DefaultButton[DialogReturn[cfA=facA;cfB=facB; pw=pwin;uv=uvoutin;timepwuv=timein;interceptE=int;slopeE=slp;nosccmHO2=scHO2;nosccmHO2star=scHO2st;nointerf=interf;nodependence=nodep;
pwdependence=pwdep;
waterYNtest=waterYN;]],
CancelButton[]}
},
Spacings->{1, Automatic},Alignment -> Left],Modal-> True,WindowTitle-> "Input Parameters"],
ImageSize->{250,30},BaseStyle->{"Subtitle",14,Bold},Background-> LightGreen]},
Alignment-> Center],

Row[{
Button["Definitions & Formulas",
CreateDocument[
Column[{
Text[Row[{"Description of additional parameters"},BaseStyle-> {"Section",Italic},Alignment->Center]],
,,
Text[Column[{
Style[Row[{1.,, Power/Subscript[UV, out],,conversion},Alignment-> Center],Bold],
Row[{}],
Row[{"Power = ", Subscript[Pw, initial]/Subscript[UV, out, initial]*Subscript[UV, out]},Alignment-> Center],
Row[{}],
Row[{}],
Style[Row[{"2. ",Subscript[HO, 2]," to OH conversion factors and NO sccm threshold"},Alignment-> Center],Bold],
Row[{Subscript[C,Subscript[HO,2]],"to ", "OH = A (FlowNO) + B"},Alignment-> Center],
Row[{"NO sccm threshold flow rate set points for ", Subscript[HO, 2]," and ",Subscript[HO, 2],"*"},Alignment-> Center],
Row[{}],
Row[{}],
Style[Row[{"3. Sensitivity, ",Subscript[R, OH]},Alignment-> Center],Bold],
Row[{"Corrected sensitivity, (Subscript[R, c])=(Subscript[R, o])-((Subscript[R, o])×(Subscript[A, w])×water%)"}, Alignment-> Center],
Row[{"(Subscript[A, w]) = –\!\(\*FractionBox[\(slope\), \(intercept\\ \((\*SubscriptBox[\(R\), \(o\)])\)\)]\) x",1/100," where (1/100) is conversion for %" },Alignment-> Center],
Row[{}],
Row[{}],
Style[Row[{"4. NO Interference, if applicable"},Alignment-> Center],Bold],
Row[{"Signal = A x Laser Power + B"},Alignment-> Center],
Row[{"     (Linear calibration plot works for a small laser power range)"},Alignment-> Center],
Row[{"A = slope"},Alignment-> Center],
Row[{"B = y-intercept"},Alignment-> Center],
Row[{}],
Row[{}],
Style[Row[{Style["5. Files to be processed (.out)"]},Alignment-> Center],Bold]
},BaseStyle->"Item"]
]
}],WindowTitle->"Definitions and Formulas"
]
]
}],

Image[,ImageSize-> 40],

Row[{Button["Store and review parameters",
Row[{
pw;
uv;
pwconvfac=pw/uv;
timepwuv;
cfA;
cfB;
nosccmHO2;
nosccmHO2star;
interceptE;
slopeE;
intercept=StringReplace[interceptE,"E"-> "×10^"] // ToExpression;
slope=StringReplace[slopeE,"E"-> "×10^"] // ToExpression;
aw=-slope/intercept*(1/100);
rOH=intercept;
noYN;
nointerf;
nodependence;
pwdependence;
d;
relevantfiles=FileNames[d] ;
Length@relevantfiles;
waterYN;
w;
CellPrint[TextCell[pw,CellTags->"parameters"]];
CellPrint[TextCell[uv,CellTags->"parameters"]];
CellPrint[TextCell[pwconvfac,CellTags->"parameters"]];
CellPrint[TextCell[timepwuv,CellTags->"parameters"]];
CellPrint[TextCell[cfA,CellTags->"parameters"]];
CellPrint[TextCell[cfB,CellTags->"parameters"]];
CellPrint[TextCell[nosccmHO2,CellTags->"parameters"]];
CellPrint[TextCell[nosccmHO2star,CellTags->"parameters"]];
CellPrint[TextCell[interceptE,CellTags->"parameters"]];
CellPrint[TextCell[slopeE,CellTags->"parameters"]];
CellPrint[TextCell[intercept,CellTags->"parameters"]];
CellPrint[TextCell[slope,CellTags->"parameters"]];
CellPrint[TextCell[aw,CellTags->"parameters"]];
CellPrint[TextCell[rOH,CellTags->"parameters"]];
CellPrint[TextCell[noYN,CellTags->"parameters"]];
CellPrint[TextCell[nointerf,CellTags->"parameters"]];
CellPrint[TextCell[nodependence,CellTags->"parameters"]];
CellPrint[TextCell[pwdependence,CellTags->"parameters"]];
CellPrint[TextCell[d,CellTags->"parameters"]];
CellPrint[ExpressionCell[relevantfiles,CellTags->"parameters"]] ;
CellPrint[ExpressionCell[Length@relevantfiles,CellTags->"parameters"]];
CellPrint[TextCell[waterYN,CellTags->"parameters"]];
CellPrint[TextCell[w,CellTags->"parameters"]];
paramHeadCol={"Power","UVout","Power conversion factor (mW/UVout)","Time power was measured","HO2 conv.factor A (slope)","HO2 conv.factor B (intercept)","NO sccm threshold HO2","NO sccm threshold HO2*","Sensitivity intercept (ROH)","Sensitivity slope","Aw=-slope/intercept*(1/100)","NO interference (Y/N)?","NO interference slope","NO interference intercept","Directory","Filename","Water imported (Y/N)?"};
inputParamCol={pw,uv,pwconvfac,timepwuv,cfA,cfB,nosccmHO2,nosccmHO2star,interceptE,slopeE,aw,noYN, nodep,pwdep,d,filenamedate,waterYN};
inputParameters=Transpose[{paramHeadCol,inputParamCol}];
}],
ImageSize->{250,30},BaseStyle->{"Subtitle",14,Bold},Background-> LightGreen]
},Alignment-> Center],


Row[{
Button["Clear display",
NotebookLocate["parameters"];
NotebookDelete[],
ImageSize->{120,30},BaseStyle->{"Subtitle",14,Bold}],
" ",
Button["Reset parameters",
ClearAll[pw,uv,pwconvfac,timepwuv,cfA,cfB,nosccmHO2,nosccmHO2star,interceptE,
slopeE,intercept,slope,aw,rOH,nointerf,nodependence,pwdependence,d,relevantfiles,waterYN,w];
ClearAll["relevantfiles"];
NotebookLocate["parameters"];
NotebookDelete[]
,ImageSize->{180,30},BaseStyle->{"Subtitle",14,Bold}]},Alignment-> Center],

Image[,ImageSize-> 40],
Row[{
Button["Import water (only if needed)",
CreateDocument[CellPrint[ExpressionCell[ProgressIndicator[Dynamic[Clock[Infinity]],Indeterminate],CellTags-> "waterprogressbar"]],
CellPrint[TextCell["Water is being imported...",CellTags-> "waterwaitmsg"]];
waterimport=Import[w, "FieldSeparators"-> ","] ;
headlabel="water"; (* case sensitive *)
datelabel="DateTime"; (* case sensitive *)
waterpos=Flatten@Position[waterimport,headlabel] ;(* position where data header 'water' is. {row, column} format. *)
startrowwater=waterpos[[1]]+1;(* water label row+1 *)
startcolwater=waterpos[[2]];(* water label column*)
waterpercent=#[[startcolwater]]&/@waterimport[[startrowwater;;All]];
datepos=Flatten@Position[waterimport,datelabel]; (* position where data header 'date' is. {row, column} format. *)
startrowdate=datepos[[1]]+1;(* date label row+1 *)
startcoldate=datepos[[2]];(* date label column*)
datewater=#[[startcoldate]]&/@waterimport[[startrowdate;;All]];
datewaterunamb[x_]:=DateList[datewater[[x]]]; (*gives the current local date and time in the form {year,month,day,hour,minute,second}.*)
datewaterunambtb=Table[datewaterunamb[i],{i,1,Length@datewater}];
datewaterstring[x_]:=DateString[datewaterunambtb[[x]],{"MonthName"," ","DayShort",", ","Year"," ","Hour12Short",":","Minute",":","Second"," ","AMPM"}];
datewaterstringtb=Table[datewaterstring[i],{i,1,Length@datewater}];
datewatertb=Transpose[{datewaterstringtb,waterpercent}];
absdatewater=AbsoluteTime/@datewaterstringtb;
absdatewatertbduptb=DeleteDuplicates@absdatewater;

(* remove all repeating times, if any - will cause error in interpolation if not removed *)

timewaterpos=Table[i,{i,1,Length@absdatewater}];
 (* original position numbers, length of the water data table *)
timewatermatch[x_]:=Flatten[{Position[absdatewater,absdatewatertbduptb[[x]]]},2] /. {num_} :> num; (* position where the remaining ones match *)
timewatermatchtb=Table[timewatermatch[i],{i,1,Length@absdatewatertbduptb}] ;
timewatermatchdel=DeleteCases[timewatermatchtb,_List] ; (* delete any expression with Head List, aka those inside { }*)
reptimew=Complement[timewaterpos,timewatermatchdel];(* missing positions from timewatermatchtb, i.e. positions that have repeating times *)
reptimewf[x_]:=If[Differences[{reptimew[[x]],reptimew[[x+1]]}]/. {num_} :> num==1,reptimew[[x]],Sequence[]];
reptimewx=Table[reptimewf[i],{i,1,Length@reptimew-1}] /. Null -> Sequence[];
(* extract first position with repeating times *)
timewaterdeldup=Insert[timewatermatchdel,reptimewx,-1] // Flatten // Sort;
(* positions without repeating times *)


(* array without duplicates *)
waterabsdatenodup[x_]:={absdatewater[[x]],waterpercent[[x]]};
waterabsdatenoduptb=waterabsdatenodup/@timewaterdeldup;

waterintfunc=Interpolation[waterabsdatenoduptb];
NotebookLocate["waterprogressbar"],
NotebookDelete[],
NotebookLocate["waterwaitmsg"],
NotebookDelete[],
Print[waterintfunc]] // Quiet
,Method-> "Queued"]}],

Image[,ImageSize-> 40],
Row[{
Button["Extract",
CreateDocument[
{CellPrint[ExpressionCell[ProgressIndicator[Dynamic[Clock[Infinity]],Indeterminate],CellTags-> "progressbar"]],
CellPrint[TextCell["Your file is being processed... Alt+. to abort",CellTags-> "extractwaitmsg"]];
CellPrint[TextCell["And if everything goes well, no messages will appear below...",CellTags-> "extractwaitmsg2"]];
(* from here to next comment next happens on a new window*)
Text[Row[{"Seconds elapsed:"},BaseStyle-> {"Subsubsection",Italic},Alignment->Center]],
runwithabstime=extract@d// AbsoluteTiming // Quiet;
runwithabstime[[1]],
TextCell[Row[{}],CellTags->"resultspreview"],
Text[Row[{"Results Preview (1-100)"},BaseStyle-> {"Subsubsection",Italic},Alignment->Center]],
TableForm[results[[2;;101]],TableHeadings-> {None,{"Ave. DateTime","Ave. Pressure","Ave. NO flow (sccm)","Ave. C3F6 flow (sccm)","Ave. H2O(%)","Ave. UVout (V)", "SD UVout", "Normalized channel A", "Status", "Online signal (cts)", "Offline signal Background (cts)", "Net signal (cts)","Ave. UVout around online (V)","Power (mW)","Corrected Sensitivity", "OH Concentration, mol/cm3","HO2 concentration, mol/cm3","HO2* concentration, mol/cm3"}},
(*new window ends here *)
NotebookLocate["progressbar"],
NotebookDelete[],
NotebookLocate["extractwaitmsg"],
NotebookDelete[],
NotebookLocate["extractwaitmsg2"],
NotebookDelete[],
Print[Style["Extraction done!",Bold,25,"Section",Black,TextAlignment->Center]]]}
,WindowTitle->"Results Preview",WindowMargins->{{Automatic,0},{Automatic,0}}],ImageSize-> {230,50},BaseStyle->{"Subtitle",18,Bold},Method->"Queued",Background->LightGreen]
},Alignment-> Center],

Row[{
Button["Extractor Description",
CreateDocument[
Row[{HoldForm[
Extractor Steps:

1. Import raw .out file, delete blank rows
2. Operate on the rows after “end header”
3. Take the filename and set it as the date for the dataset
4. Extract all relevant columns from the raw .out file
5. Create a main table for columns extracted
6. Delete row if status is blank, scan, : 0s, lock, Test Zero PD, Test Laser Scatter
7. Create a column for chA over UVout called "divchA"
8. Create a dummy column for aveUVout1 (to be replaced by aveUVoutvalue)
9. Append divchA and aveUVout1 to main table
10. Split main table according to status 
11. Erase first and last rows of each status block
12. Get  average and SD of parameters for each data block of online, offline, etc
13. Assign mean average UVout values calculated into each block
14. Calculate normalized chA (= divchA × ave UVout) for each block/split and append to main table
15. Split main table again according to status
16. Get average normalized chA for each block
17. Get start and end time for each block, then get the average of the start and end times
18. Get status of each block
19. Create separate columns for online and offline signals
20. Calculate average UVout around online and use this to convert UVout to power
21. Create column for water data (either imported from a .csv file or embedded in .out file)
22. Calculate corrected sensitivity using water data
23. Calculate OH concentrations 
24. Calculate (Subscript[HO, 2]) and (Subscript[HO, 2])* corrected signals using conversion factors
25. Calculate (Subscript[HO, 2]) and (Subscript[HO, 2])* concentrations
26. Create Results table with OH and (Subscript[HO, 2]) concentrations
	All values are averages of each segregated block
		a. Average of start and end time: avetimestringtb
		b. Average of NO flow: aveflowNOmean
		c. Average of C3F6 flow: aveflowC3F6mean
		d. Average of UVout : aveUVoutmean
		e. SD of UVout: aveUVoutSDtb
		e. Average norm chA (all): avenormchAexnull
		f. Status: firstat
		g. Online counts: onsigtb (=avenormchAexnull if status is online)
		h. Offline counts (Background): offsigtb
		i. Net signal (online - offline): netsigtb (netsigtb for OH, netsigHO2tb for (Subscript[HO, 2]) netsigHO2sttb for (Subscript[HO, 2])*)
		j. Average (Subscript[UV, out ])around online: aveUVoutOLtb
		k. Power: pwtb
		l. Sensitivity corrected for water mixing ratio (%): corrsenstb
		m. OH concentrations, mol (cm^(-3)): concOHtb1
		m. (Subscript[HO, 2]) and (Subscript[HO, 2])* concentrations, mol (cm^(-3)) (respectively): concHO2tb, concHO2sttb
27. Results table preview for the first 100 cells
]
}],WindowTitle->"Extractor Desciption"
]
],
,,
Button["Variable names",
CreateDocument[
Names[$Context<>"*"]//TableForm
,WindowTitle->"Variable Names"]],
, ,
Button["Re-open Results preview",
CreateDocument[
Row[{Text[Row[{"Results Preview (1-100)"},BaseStyle-> {"Subsubsection",Italic},Alignment->Center]],
TableForm[results[[2;;101]],TableHeadings-> {None,{"Ave. DateTime","Ave. Pressure","Ave. NO flow (sccm)","Ave. C3F6 flow (sccm)","Ave. H2O(%)","Ave. UVout (V)", "SD UVout", "Normalized channel A", "Status", "Online signal (cts)", "Offline signal Background (cts)", "Net signal (cts)","Ave. UVout around online (V)","Power (mW)","Corrected Sensitivity", "OH Concentration, mol/cm3","HO2 concentration, mol/cm3","HO2* concentration, mol/cm3"}}]}]
]
],
, ,
Button["Full Results",
CreateDocument[
Row[{Text[Row[{"Results"},BaseStyle-> {"Subsubsection",Italic},Alignment->Center]],
TableForm[results[[2;;]],TableHeadings-> {None,{"Ave. DateTime","Ave. Pressure","Ave. NO flow (sccm)","Ave. C3F6 flow (sccm)","Ave. H2O(%)","Ave. UVout (V)", "SD UVout", "Normalized channel A", "Status", "Online signal (cts)", "Offline signal Background (cts)", "Net signal (cts)","Ave. UVout around online (V)","Power (mW)","Corrected Sensitivity", "OH Concentration, mol/cm3","HO2 concentration, mol/cm3","HO2* concentration, mol/cm3"}}]}]
]
]


}],
Image[,ImageSize-> 40],

Row[{Button["Visualizations",
CreateDocument[Row[{
Button["Plot OH",
timeconcOH=Transpose[{avetimestringtb,concOHtb1}] ;
timeconcOHplot=DateListPlot[timeconcOH,Joined-> False];
CellPrint[ExpressionCell[Show[timeconcOHplot]]]]

,
Button["Plot HO2",
timeconcHO2=Transpose[{avetimestringtb,concHO2tb}] ;
timeconcHO2plot=DateListPlot[timeconcHO2, Joined-> False];
CellPrint[ExpressionCell[Show[timeconcHO2plot]]]]

,
Button["Plot HO2*",
timeconcHO2st=Transpose[{avetimestringtb,concHO2sttb}] ;
timeconcHO2stplot=DateListPlot[timeconcHO2st, Joined -> False];
CellPrint[ExpressionCell[Show[timeconcHO2stplot]]]]}],
WindowTitle->"Visualizations"],
ImageSize-> {230,30},BaseStyle->{"Subtitle",18,Bold},Background-> LightGreen
]},Alignment-> Center],

Row[{
Button["Export OH plot", Export[SystemDialogInput["FileSave",numdate<>"_timeconcOHplot.png"],timeconcOHplot,"png"],Method->"Queued"],,
Button["Export HO2* plot", Export[SystemDialogInput["FileSave",numdate<>"_timeconcHO2stplot.png"],timeconcHO2stplot,"png"],Method->"Queued"],,
Button["Export HO2 plot", Export[SystemDialogInput["FileSave",numdate<>"_timeconcHO2plot.png"],timeconcHO2plot,"png"],Method->"Queued"]}],

Image[,ImageSize-> 40],

Text[Row[{"Export files as csv"},BaseStyle-> {"Item",Bold,18},Alignment->Center]],

Row[{
Button["Export extracted summary",Export[SystemDialogInput["FileSave",numdate<>"_Extracted.csv"],results,"CSV"],Method->"Queued"],
 ,,
Button["Export input parameters",Export[SystemDialogInput["FileSave",numdate<>"_Input Parameters.csv"],inputParameters],Method-> "Queued"],
, ,
Button["Export OH concentrations", Export[SystemDialogInput["FileSave",numdate<>"_concOH.csv"],timeconcOH,"CSV"],Method->"Queued"],
, ,
Button["Export HO2* concentrations", Export[SystemDialogInput["FileSave",numdate<>"_concHO2star.csv"],timeconcHO2st,"CSV"],Method->"Queued"],
 ,,
Button["Export HO2 concentrations",Export[SystemDialogInput["FileSave",numdate<>"_concHO2.csv"],timeconcHO2,"CSV"],Method->"Queued"]
},Alignment-> Center],

dummyfunction[somefile_]:=
Module[{},FileNameTake@somefile],

extract[relevantfile_]:=
(* Import file from file chosen in the initial dialog box *)
Module[{},
rawOut=Import[relevantfile, "Table"];
rawOut=DeleteCases[rawOut,row_/;row=={}]; (* Delete blank rows *)

(* Find the column headers *)
headpos=Position[rawOut, s_String /; StringMatchQ[s,"End"]];
flatpos=Flatten@headpos;
startpos=flatpos[[1]]+1;

(* Take filename and set it as the date for the dataset *)
filedatef[rawdate_]:=FileNameTake[rawdate];
filenamedate=filedatef@relevantfile;
numdate=StringReplace[filenamedate,".out"-> ""];
filedatelong[numform_]:=DateString[numform, {"MonthName"," ","DayShort",", ","Year"}];
filedate=filedatelong@numdate;

(* Pick out rows where it is an online or offline status. Important for column integrity *)
rawOutStat=Pick[rawOut[[startpos;;]],#[[14]]&/@rawOut[[startpos;;]],"LINE"];

(* Extract relevant columns *)
rawTimeSep=Transpose[{#[[1]]&/@rawOutStat,#[[2]]&/@rawOutStat}];
rawTime=Table[rawTimeSep[[i]][[1]]<> " " <> rawTimeSep[[i]][[2]],{i,1,Length@rawOutStat}];
datetime=Table[{filedate} <> " "  <> rawTime[[i]],{i,1,Length@rawTime}];
abstime=AbsoluteTime/@datetime;
rawchA=#[[3]]&/@rawOutStat;
rawref=#[[4]]&/@rawOutStat;
rawpress=#[[5]]&/@rawOutStat;
rawUVin=#[[6]]&/@rawOutStat;
rawUVout=#[[7]]&/@rawOutStat;
rawnm=#[[8]]&/@rawOutStat;
rawflowNOsccm=#[[9]]&/@rawOutStat;
rawflowC3f6sccm=#[[10]]&/@rawOutStat;
rawflowNOTF=#[[11]]&/@rawOutStat;
rawflowC3F6TF=#[[12]]&/@rawOutStat;
rawstat=#[[13]]&/@rawOutStat;
rawh2o=#[[20]]&/@rawOutStat;
rawacq=#[[21]]&/@rawOutStat;
rawUVcal=#[[22]]&/@rawOutStat;
rawNOcycle=#[[23]]&/@rawOutStat;

(* Create a table out of the columns above *)
rawTable1=Transpose[{rawTime, abstime,rawchA,rawref,rawpress,rawUVin,rawUVout,rawnm,rawflowNOsccm,rawflowC3f6sccm,rawflowNOTF,rawflowC3F6TF,rawstat,rawh2o,rawacq,rawUVcal,rawNOcycle}]; 

(* Delete all rows with irrelevant status*)
rawTable1=DeleteCases[rawTable1,{_,_,_,_,_,_,_,_,_,_,_,_,"",_,_,_,_}] ;
rawTable1=DeleteCases[rawTable1,{_,_,_,_,_,_,_,_,_,_,_,_,"SCAN",_,_,_,_}] ;
rawTable1=DeleteCases[rawTable1,{_,_,_,_,_,_,_,_,_,_,_,_,": 0 s",_,_,_,_}] ;
rawTable1=DeleteCases[rawTable1,{_,_,_,_,_,_,_,_,_,_,_,_," : 0 s",_,_,_,_}] ;
rawTable1=DeleteCases[rawTable1,{_,_,_,_,_,_,_,_,_,_,_,_," : 0 S",_,_,_,_}] ;
rawTable1=DeleteCases[rawTable1,{_,_,_,_,_,_,_,_,_,_,_,_," : 0 S ",_,_,_,_}] ;
rawTable1=DeleteCases[rawTable1,{_,_,_,_,_,_,_,_,_,_,_,_,"LOCK",_,_,_,_}];
rawTable1=DeleteCases[rawTable1,{_,_,_,_,_,_,_,_,_,_,_,_,"Test Zero PD",_,_,_,_}];
rawTable1=DeleteCases[rawTable1,{_,_,_,_,_,_,_,_,_,_,_,_,"Test Laser Scatter",_,_,_,_}];
rawTable1=DeleteCases[rawTable1,{_,_,_,_,_,_,_,_,_,_,_,_,"Test laser scatter",_,_,_,_}];

(* Add a position number to rawTable*)
rawTablepos=Table[i,{i,1,Length@rawTable1}];
rawTable1=Insert[rawTable1//  Transpose, rawTablepos,18] // Transpose;

(*Delete all rows with 0.1<rawUVout[[x]]<3 *)
rawTableUVoutpos[x_]:=If[Between[#[[7]]&@rawTable1[[x]],{0.1,3}],#[[18]]&@rawTable1[[x]],Sequence[]];
rawTableUVoutpostb=Table[rawTableUVoutpos[i],{i,First@rawTablepos,Last@rawTablepos}] /. Null :> Sequence[];  (* returns positions of rows where UVout is between 0.1 and 3 *)
rawTablef[x_]:=#[[1;;17]]&@rawTable1[[x]]; 
rawTable=rawTablef/@rawTableUVoutpostb; (* grabs rows where UVout is between {0.1, 3}*) (*drop rawTable1 position numbers *)

(*append new position numbers to rawTable *)
rawTabledelUVpos=Table[i,{i,1,Length@rawTable}]; (* position numbers for rawTable *)
rawTable=Insert[rawTable//  Transpose, rawTabledelUVpos,18] // Transpose;

(* calculate divchA and append to rawTable *)
divchAf[x_]:=(#[[3]]&@rawTable[[x]])÷(#[[7]]&@rawTable[[x]]);
divchA=Table[divchAf[i],{i,1,Length@rawTable}];
rawTable=Insert[rawTable//  Transpose, divchA,18] // Transpose;

(* Split rawTable into blocks/"sheets" according to status *)
segregatedData1=SplitBy[rawTable,Part[#,13]&];
dimseg=Length@segregatedData1; (* displays total number of blocks/splits *)
tally[x_]:=Tally[#[[13]]&/@segregatedData1[[x]] ];
dimsegtally=Table[tally[i],{i,1,dimseg}];


(* Erase the first and last points of each block/"sheet" *)
segregatedDatadropf[x_]:=
If[Length[segregatedData1[[x]]]>1,Drop[segregatedData1[[x]],{1,-1,Length@segregatedData1[[x]]-1}],""];
segregatedDataAllElem=Table[segregatedDatadropf[i],{i,1,dimseg}]; (* rawTable positions maintained *)

(* Check length of each block/"sheet" and drop if 1 element or less *)

segregatedData=Pick[segregatedDataAllElem,Length/@segregatedDataAllElem,x_/;x>1];

(*set new dimseg*)
dimseg=Length@segregatedData;
 
(* Take average values for each segregatedData block *)
avetime[x_]:=N@Mean[#[[2]]&/@segregatedData[[x]]]; (* time *)
avetimemean=Table[avetime[i],{i,1,dimseg}];

aveUVin[x_]:=N@Mean[#[[6]]&/@segregatedData[[x]]]; (* UVin *)
aveUVinmean=Table[aveUVin[i],{i,1,dimseg}];
aveUVinSD[x_]:=N@StandardDeviation[#[[6]]&/@segregatedData[[x]]]; (* SD *)
aveUVinSDtb=Table[aveUVinSD[i],{i,1,dimseg}];

aveUVout[x_]:=N@Mean[#[[7]]&/@segregatedData[[x]]]; (* UVout *)
aveUVoutmean=aveUVoutvalue=Table[aveUVout[i],{i,1,dimseg}];
aveUVoutSD[x_]:=N@StandardDeviation[#[[7]]&/@segregatedData[[x]]];(* SD *)
aveUVoutSDtb=Table[aveUVoutSD[i],{i,1,dimseg}];

aveflowNO[x_]:=N@Mean[#[[9]]&/@segregatedData[[x]]]; (* flow NO *)
aveflowNOmean=Table[aveflowNO[i],{i,1,dimseg}];

aveflowC3F6[x_]:=N@Mean[#[[10]]&/@segregatedData[[x]]]; (* flow C3F6 *)
aveflowC3F6mean=Table[aveflowC3F6[i],{i,1,dimseg}];

aveh2o[x_]:=N@Mean[#[[14]]&/@segregatedData[[x]]]; (* h2o *)
aveh2omean=Table[aveh2o[i],{i,1,dimseg}];

aveacq[x_]:=N@Mean[#[[15]]&/@segregatedData[[x]]]; (* acq rate*)
aveacqmean=Table[aveacq[i],{i,1,dimseg}];

avepress[x_]:=N@Mean[#[[5]]&/@segregatedData[[x]]]; (* pressure*)
avepressmean=Table[avepress[i],{i,1,dimseg}];

(* extract rawTable positions from segregatedData blocks *)
rawTablepossegf[x_]:=#[[19]]&/@segregatedData[[x]];
rawTableposseg=Flatten[Table[rawTablepossegf[i],{i,1,dimseg}]]; (* row numbers in rawTable which were not removed, i.e. points in between the first and last online/offline in each block, and rows where UVout is in between {0.1,3} *)
(* grab rows from rawTable *)
rawTabledropf[x_]:=#[[1;;18]]&@rawTable[[x]];
rawTabledrop=rawTabledropf/@rawTableposseg;
(* rawTable -> UVouts with {0.1,3}
rawTabledrop -> rawTable with the first and last online and offline in each block are removed; rawTable positions removed *)

(*re-segregate rawTabledrop into offline and online sheets; in this block, rawTable position number is eliminated. *)
segregatedData2=SplitBy[rawTabledrop,Part[#,13]&];

(* calculate normchA in each segment *) (* col 18 = divchA *)
normchAf[x_]:=(#[[18]]&/@segregatedData2[[x]])× aveUVoutmean[[x]];
normchA=Table[normchAf[i],{i,1,dimseg}];

(* Get average normchA for each block *)
avenormchAf[x_]:=N[Mean@normchA[[x]]];
avenormchA=Table[avenormchAf[i],{i,1,dimseg}];

(* Get start and end time for each data block *)
upboundsegn[x_]:=First@segregatedData2[[x]] ;
upboundsegntb=Table[upboundsegn[i],{i,1,dimseg}];
lboundsegn[x_]:=Last@segregatedData2[[x]] ;
lboundsegntb=Table[lboundsegn[i],{i,1,dimseg}];
upboundsegnabstime=#[[2]]&/@upboundsegntb; (* absolute start time *)
lboundsegnabstime=#[[2]]&/@lboundsegntb ;(* absolute end time *)
(* Get ave of start and end time and convert to normal time *)
aveabstime[x_]:=N@Mean[{upboundsegnabstime[[x]],lboundsegnabstime[[x]]}];
avetimetb=DateList/@Table[aveabstime[i],{i,1,dimseg}]; (* DateList function converts back to normal time *)
avetimetbstring[x_]:=DateString[avetimetb[[x]],{"MonthName"," ","DayShort",", ","Year"," ","Hour12Short",":","Minute",":","Second"," ","AMPM"}];
avetimestringtb=Table[avetimetbstring[i],{i,1,dimseg}]; (* report to final results table *)

(* Get status of each block *)
stat[x_]:=#[[13]]&/@segregatedData2[[x]];
statustb=Table[stat[i],{i, 1 , dimseg}];
firstat=First/@statustb; (* report to final results table *)

(* Create separate columns for online signals and offline signals *)
segstat[x_]:=#[[13]]&/@segregatedData2[[x]];
segstattb=Table[segstat[i],{i,1,dimseg}];
onsig[x_]:=If[First@segstattb[[x]]=="ON",avenormchA[[x]],""];
onsigtb=Table[onsig[i],{i,1,dimseg}] ;
offsig[x_]:=If[First@segstattb[[x]]=="OFF",avenormchA[[x]],""]; (* background *)
offsigtb=Table[offsig[i],{i,1,dimseg}] ;

(* Count number of points in each block *)
count[x_]:=Count[segstattb[[x]],Except[{}]];
counttb=Table[count[i],{i,1,dimseg}];

(* status per block *)
statblockf[x_]:=First@segstattb[[x]];
statblock=Table[statblockf[i],{i,1,dimseg}];

(* Calculate average UVout around online *)

(* Check if the first sheet is online or offline *)
(* Check if the last sheet is online or offline *)

aveuvoutOL[x_]:=((aveUVoutmean[[x]]×counttb[[x]])+(aveUVoutmean[[x-1]]×counttb[[x-1]])+(aveUVoutmean[[x+1]]×counttb[[x+1]]))/(counttb[[x]]+counttb[[x-1]]+counttb[[x+1]]); (* middle points *)


aveuvoutOLfirstOL[x_]:=((aveUVoutmean[[x]]×counttb[[x]])+(aveUVoutmean[[x+1]]×counttb[[x+1]]))/(counttb[[x]]+counttb[[x+1]]); (* first acquisition is ONLINE or OFFLINE *)
aveuvoutOLlastOL[x_]:=((aveUVoutmean[[x]]×counttb[[x]])+(aveUVoutmean[[x-1]]×counttb[[x-1]]))/(counttb[[x]]+counttb[[x-1]]); (* last acquisition is ONLINE or OFFLINE *)

aveuvoutOLfirst=If[statblock[[1]]== "ON", aveuvoutOLfirstOL@1, 0];
aveuvoutOLlast=If[statblock[[dimseg]]== "ON", aveuvoutOLlastOL@dimseg, 0];
aveuvoutOLelsef[x_]:=If[statblock[[x]]=="ON",aveuvoutOL[x],0];
aveuvoutOLelse=Table[aveuvoutOLelsef[i],{i,2,dimseg-1}];

aveUVoutOLtbx=aveUVoutOLtb=Flatten[{aveuvoutOLfirst,aveuvoutOLelse,aveuvoutOLlast}];

(* to be used for converting to power *)
(* Create column for Power; Use average UVout around online *) 
pwconvx[x_]:=pwconvfac×aveUVoutOLtb[[x]] /. value_ /; value== 0 :> "" ;
pwtb=Table[pwconvx[i],{i,1,dimseg}];
aveUVoutOLtb=aveUVoutOLtb /. value_ /; value== 0 :> "";



(* Calculate net signal for OH *)

(* operate only if status is online *)
netsigelse[x_]:=N@(((onsigtb[[x]])/(aveUVoutmean[[x]]))-((((((offsigtb[[x-1]])/(aveUVoutmean[[x-1]])))+(((offsigtb[[x+1]])/(aveUVoutmean[[x+1]]))))/2)))×((1/3)(aveUVoutmean[[x]]+aveUVoutmean[[x-1]]+aveUVoutmean[[x+1]])); 
netsigfirstOL[x_]:=If[statblock[[1]]== "ON",N@(((onsigtb[[x]])/(aveUVoutmean[[x]]))-(((offsigtb[[x+1]])/(aveUVoutmean[[x+1]]))))×(((aveUVoutmean[[x]]+aveUVoutmean[[x+1]])/2)),netsigelse@x]; 
netsiglastOL[x_]:=If[statblock[[dimseg]]== "ON",N@(((onsigtb[[x]])/(aveUVoutmean[[x]]))-(((offsigtb[[x-1]])/(aveUVoutmean[[x-1]]))))×(((aveUVoutmean[[x]]+aveUVoutmean[[x-1]])/2)),netsigelse@x]; 

netsigOLfirst=If[statblock[[1]]== "ON", netsigfirstOL@1, ""];
netsigOLlast=If[statblock[[dimseg]]== "ON", netsiglastOL@dimseg, netsigelse@dimseg];
netsigOLelsef[x_]:=If[statblock[[x]]=="ON",netsigelse[x],""];
netsigOLelse=Table[netsigOLelsef[i],{i,2,dimseg-1}];

netsigtbx=netsigtb=Flatten[{netsigOLfirst,netsigOLelse,netsigOLlast}];

(* Calculate net signal for HO2 and HO2* *)
(* Calculate net signal minus interference, if desired *)
(* the following is for quantifier "noYN" *)
netsigHO2[x_]:=If[(nosccmHO2star>aveflowNOmean[[x]]>nosccmHO2),netsigtb[[x]]," ", " "];
netsigHO2tb=
Which[
noYN== "N",Table[netsigHO2[i],{i,1,dimseg}],
noYN== "Y",Table[netsigHO2[i]-Sum[nodep×pwtb[[i]],pwdep],{i,1,dimseg}]
];
netsigHO2st[x_]:=If[(aveflowNOmean[[x]]≥nosccmHO2star),netsigtb[[x]]," ", " "];
netsigHO2sttb=
Which[
noYN== "N",Table[netsigHO2st[i],{i,1,dimseg}],
noYN== "Y",Table[netsigHO2st[i]-Sum[nodep×pwtb[[i]],pwdep],{i,1,dimseg}]
];


(* Create column for water data; Use .csv imports if "Y" in dialog box and use water column in .out file if "N" *)
waterf[x_]:=If[waterYN== "Y", waterintfunc[abstime[[x]]], rawh2o[[x]]];
water=Table[waterf[i],{i,1,Length@abstime}];

(* Calculate corrected sensitivity using water data *)
corrsens[x_]:=rOH-(rOH×aw×water[[x]]);
corrsenstb=Table[corrsens[i],{i,1,dimseg}];

(* Calculate OH concentration and create a column *)
concOH[x_]:=Which[
firstat[[x]]=="ON"&&aveflowNOmean[[x]]<1&&aveflowC3F6mean[[x]]<1,((netsigtb[[x]])/(corrsenstb[[x]]×pwtb[[x]])) (* calculate OH only for online signals, flow less than NO and C3F6 threshold *),
firstat[[x]]=="OFF",""
(*aveflowNOmean[[x]]<1,((netsigtb[[x]])/(corrsenstb[[x]]×pwtb[[x]])),
aveflowC3F6mean[[x]]<1,((netsigtb[[x]])/(corrsenstb[[x]]×pwtb[[x]]))*)
];
concOHtb=Table[concOH[i],{i,1,dimseg}];
concOHreal[x_]:=If[Head[concOHtb[[x]]]==Real,concOHtb[[x]]," ", " "];
concOHtb1=Table[concOHreal[i],{i,1,dimseg}];
concOHnoHO2f[x_]:=Which[
concHO2tb[[x]]==" ",concOHtb1[[x]],
concHO2sttb[[x]]==" ",concOHtb1[[x]],
Head[concHO2tb[[x]]]== Real," ",
Head[concHO2sttb[[x]]]== Real," "
];
concOHnoHO2tb=Table[concOHnoHO2f[i],{i,1,dimseg}];

(* Conversion Factors for HO2 and HO2* conversions *)
cf[x_]:=(cfA×aveflowNOmean[[x]])+cfB;
cftb=Table[cf[i],{i,1,dimseg}];

(* Calculate HO2 and HO2* corrected signals using conversion factors *)
corrsigHO2[x_]:=If[Head[netsigHO2tb[[x]]]== Real,((netsigHO2tb[[x]])/(cftb[[x]]))," "," "];
corrsigHO2tb=Table[corrsigHO2[i],{i,1,dimseg}];
corrsigHO2st[x_]:=If[Head[netsigHO2sttb[[x]]]== Real,((netsigHO2sttb[[x]])/(cftb[[x]]))," "," "];
corrsigHO2sttb=Table[corrsigHO2st[i],{i,1,dimseg}];


(* Calculate HO2 and HO2* concentrations *)
concHO2[x_]:=If[Head[corrsigHO2tb[[x]]]==Real,((corrsigHO2tb[[x]])/(corrsenstb[[x]]×pwtb[[x]]))," ", " "];
concHO2tb=Table[concHO2[i],{i,1,dimseg}];
concHO2st[x_]:=If[Head[corrsigHO2sttb[[x]]]==Real,((corrsigHO2sttb[[x]])/(corrsenstb[[x]]×pwtb[[x]]))," "," "];
concHO2sttb=Table[concHO2st[i],{i,1,dimseg}];

(* Create results table *)
reshead={"Ave. DateTime","Ave. Pressure (Torr)","Ave. NO flow (sccm)","Ave. C3F6 flow (sccm)", "Ave. H2O (%)","Ave. UVout (V)", "SD UVout", "Normalized channel A", "Status", "Online signal (cts)", "Offline signal Background (cts)", "Net signal (cts)","Ave. UVout around online (V)","Power (mW)","Corrected Sensitivity", "OH Concentration, mol/cm3","HO2 concentration, mol/cm3","HO2* concentration, mol/cm3"};
results=Transpose[{avetimestringtb,avepressmean,aveflowNOmean,aveflowC3F6mean, aveh2omean, aveUVoutmean, aveUVoutSDtb, avenormchA, firstat, onsigtb, offsigtb, netsigtb,aveUVoutOLtb,pwtb,corrsenstb, concOHnoHO2tb,concHO2tb,concHO2sttb}]  // Insert[reshead,1];
timeconcOH=Transpose[{avetimestringtb,concOHtb1}] ;
timeconcHO2=Transpose[{avetimestringtb,concHO2tb}] ;
timeconcHO2st=Transpose[{avetimestringtb,concHO2sttb}] ;
]

},TextAlignment->Center, WindowTitle-> "Fastest Extractor EVER",WindowSize-> {750,890} ],
Alignment-> Center,ImageSize->{250,40},BaseStyle->{"Chapter",18,Bold,Black},Background-> LightGreen]

Compile result files from extractor

Row[{Button["Execute Results Compiler",
CreateDialog[
Grid[{
{},
{"Choose Files (multiple files may be chosen):", FileNameSetter[Dynamic[resFile],"OpenList"]},
{},
{"Set filename of consolidated file:",InputField[Dynamic[fileName],String]},
{importExport[filesList_]:=Module[{},
Clear@master;
master={};
consolidate/@filesList;
master=Flatten[master,1];
master=SortBy[master,First]; (*this actually does nothing to sort the datetime because datetime is read as a string, and is thus sorted alphabetically/numerically rather than by datetime. *)
Export[SystemDialogInput["FileSave",fileNameSet<>".csv"],master]],
consolidate[file_]:=Module[{},
rawFile=Import[file,"FieldSeparators"-> ","];
AppendTo[master,rawFile];
]},
{DefaultButton[
DialogReturn[
resFile1=resFile;
fileNameSet=fileName;
CreateDialog[
Column[
{TextCell["Chosen Files:","Subsection"],
FileNameTake/@resFile // TableForm,
Button["Consolidate files",
importExport@resFile,
Method-> "Queued"] }
],
WindowSize-> {500,500},
WindowTitle-> "Filenames"
]
]
],
CancelButton[]}
},
Spacings->{1, Automatic},Alignment -> Left],
Modal-> True,
WindowTitle-> "Choose Files",
WindowSize-> {400,160}
]
,
Alignment-> Center,ImageSize->{250,40},BaseStyle->{"Chapter",18,Bold,Black},Background-> GrayLevel[0.84],Method-> "Queued"]
}]

Acknowledgments

-> Indiana University Bloomington Provost’s Travel Award for Women in Science

-> High Performance Systems (Carbonate), supported in part by Lilly Endowment Inc. through Indiana University Pervasive Technology Institute

-> Stevens Lab, O’Neill (SPEA), Indiana University

-> Nicholas Brunk, SICE, Indiana University

Contact/Questions:
Colleen Rosales
cfrosale@iu.edu
colleenrosales.weebly.com

ABSTRACT

Most atmospheric research activities (“field campaigns”) require obtaining high-resolution data for a long period of time (e.g. per-second data for one month). Such campaigns involve collaborations between multiple research groups that have different instruments and could have different output file types and sizes. Thus, data organization could be challenging. In this talk, I will share the structure of field campaigns and how quick and reliable data organization plays a role in ensuring accuracy and traceablility of data. I will also share how I used Mathematica to sort, analyze, and process output files from our instrument. Lastly, I will share how I transformed a Macro originally written in VBA to a GUI-based program entirely written in Mathematica. While both have user-friendly interfaces, writing it in Mathematica greatly reduced processing time from ~30 minutes to ~30 seconds.