#set TITLE = "proc processdata"
#include top

.SH DESCRIPTION
\fBproc processdata\fR may be used to perform various types 
processing on a data set after it is read.  

.SH FEATURES
Various types of processing may be done, including
break processing, accumulation, tabulation and counting, rewriting as percents,
computation of totals, reversing record order, and rotation of row/column matrix.
Only one of these types of processing may be done per invocation of proc processdata, 
and the type of processing is controlled by the \fCaction\fR attribute.
In addition to any one of the actions, data may be further processed in
that fields may be filtered (kept or rejected).
The result may replace the original data set in memory or occupy a new space.
In either case, after proc processdata is finished the result will be 
considered the "current data set" which all plotting procs will use.
The original data set may be returned to later using proc originaldata.

.SH EXAMPLE
A Gallery example where this is used is
.ig >>
<a href="../gallery/stock.htm">
.>>
 stock 
.ig >>
</a>
.>>
where the available data is in reverse chronological order.
Since lineplot must work from left to right, \fBproc processdata\fR
is used to reverse the record order.

.SH VARIABLES THAT ARE SET
.LP
\fBNRECORDS\fR = Number of rows in the data result.
.LP
\fBNFIELDS\fR = Number of fields per row in the data result.
.LP
\fBTOTALS\fR = If totals, percents, or accumulation are being done, 
this variable will be set to hold the field total(s).  If more than
one field is being operated on, this will be a comma-delimited
list of totals; individual totals may be accessed in your script
using something like the following, which would access the first
total in the list:
.ce
\fC#set T = $nmember(1,@TOTALS)\fR
.LP
\fBBREAKFIELD1 .. n\fR = Current contents of break fields, when
\fCaction\fR is \fBbreaks\fR.


.SH MANDATORY ATTRIBUTES
The \fCaction\fR attribute must be specified, unless 
just keeping/rejecting fields.

.SH ATTRIBUTES
.LP
\fBaction\fR \fIa\fR
.IP
The type of processing to perform.
Often the \fCfields\fR attribute is used to indicate which field(s)
are involved.
Legal values for \fIa\fR include: 
.IP
\fBaccumulate\fR  Rewrite field as a cumulative series (accumulation).
The field(s) to operate on must be given in the \fCfields\fR attribute.
For example, the data set on the left would be transformed to the one 
on the right (\fCfields: 2\fR):
.nf
	A21 3			A21 3
	A22 5		-->	A22 8 
	A23 2			A23 10
	A24 1			A24 11
.fi

.IP
\fBbreaks\fR (new in version 1.39).  Break processing.  In data processing
terminology, "break processing" is the act of passing through a sorted
data set and taking some action when a key field or fields change.
Break processing is significantly more efficient than scanning a data 
set multiple times with a select statement.
The data set must be sorted such that key fields are grouped.
The key field(s) must be specified using the \fCfields\fR attribute.
The \fBBREAKFIELD1 .. n\fR variable(s) will be set to the current contents
of the break field(s).  Your script can detect when the entire data set has 
been processed by checking 
the NRECORDS variable (equal to 0), or the BREAKFIELD1 variable ($strlen of 0).
A gallery example that uses this feature is
.ig >>
<a href="../gallery/mouse.htm">
.>>
mouse
.ig >>
</a>
.>>
See MORE ON BREAK PROCESSING below for more information.

.IP
\fBbreakreset\fR  Reset the "current row" to the beginning of the data set,
for the occasional time when more than one pass through a data set will be done
using the \fBbreaks\fR action.


           

.IP
\fBcount\fR  Collapse data by counting the number of instances of a key
field.  Input data must be sorted (or at least grouped) on the key field.
Resulting data set will always have two fields.
One or two fields must be specified using the \fCfields\fR attribute.  
If one field is specified, the result fields will be 1) key field, 2) count.
For example (\fCfields: 1\fR):
.nf
	062698	 		062698 2
	062698 		  -->	062898 1
	062898 			070198 3
	070198 			070498 1
	070198	 
	070198
	070498 
.fi
.IP
If two fields are specified, the result fields will be 1) key field, 2) sum
of the numeric contents found in the second specified field.
For example (\fCfields: 1 2\fR):
.nf
	062698 4		062698 10
	062698 6	  -->	062898 3
	062898 3		070198 9
	070198 2		070498 2
	070198 4
	070198 3
	070498 2
.fi
See also the gallery example
.ig >>
<a href="../gallery/hitcount.htm">
.>>
hitcount
.ig >>
</a>
.>>
(Introduced in version 1.3)

.IP
\fBpercent\fR  Rewrite one or more fields as percentages of its field (column) total.
The field(s) to operate on must be given in the \fCfields\fR attribute.
For example (\fCfields: 1\fR):
.nf
	8		40
	4	   -->	20
	3		15
	5		25
.fi

.IP
\fBreverse\fR  The last record becomes the first one; the
record order is reversed. For example (\fCfields: 2\fR):
.nf
	AXB 34		DIF 14
	BYA 22	   -->	CES 52
	CES 52		BYA 22
	DIF 14		AXB 34 
.fi
.IP
\fBrotate\fR  First row becomes 1st field, 2nd row
becomes 2nd field, and so on.  This may be useful
in that most of the plotting procs work from data fields,
but sometimes data is given (or is more intuitive) in rows.
For example:
.nf
	A 2 4 6 8 10	-->	A B
	B 3 6 9 12 15		2 3
				4 6
				6 9
				8 12
				10 15
.fi
.IP
\fBselect\fR  Select certain records.
Result will be made up of records meeting the
condition given in the \fCselect\fR attribute.

.IP
\fBtotal\fR  Compute field total(s) only and place total(s) into the variable
TOTALS (see above).  This action does not rewrite data.
The field(s) to operate on must be given in the \fCfields\fR attribute.
.br
The decimal format of the total(s) is controlled by the \fCresultformat\fR
attribute.  If total(s) are to be written in presentable notation
(a spacer for thousands, etc.) the \fCresultformat\fR attribute may be
preceded by a \fCn\fR, e.g. \fCn%7.0f\fR.


.LP
\fBfields\fR 
.ig >>
<a href="attributetypes.html#dfield">
.>>
\fI dfield \fR
.ig >>
</a>
.>>
list
.IP
The field(s) to be operated on when doing percents, accumulations, or totals.
.br
Example: \fCfields: 2 5 6 7\fR


.LP
\fBkeepfields\fR
.ig >>
<a href="attributetypes.html#dfield">
.>>
\fI dfield \fR
.ig >>
</a>
.>>
list
.IP
If specified, only the listed fields in the original data set 
will be kept.  The others will be rejected.  
\fCaction\fR may be anything.
.br
Example: \fCkeepfields:  4 5 6\fR

.LP
\fBrejectfields\fR
.ig >>
<a href="attributetypes.html#dfield">
.>>
\fI dfield \fR
.ig >>
</a>
.>>
list
.IP
If specified, the listed fields in the original data set 
will not be kept.  The others will be.  
\fCaction\fR may be anything.
.br
Example: \fCrejectfields:  1 2 3 4\fR


.LP
\fBfieldnames\fR \fInamelist\fR
.IP
If specified, the names given in \fInamelist\fR may be used 
in any plotting proc to identify data fields.  \fInamelist\fR is a
space- or comma- delimited list of names.  Names may include any alphanumeric
characters with a maximum length of 38, and are case-insensitive.
.br
Note that if field names are specified in \fBproc getdata\fR 
and then \fBproc processdata\fR is used to alter the order of fields or 
delete fields, then this \fCfieldnames\fR attribute \fBmust\fR be
used in order to redefine the field names properly.
.br
Example: \fCfieldnames: date group n\fR


.LP
\fBresultformat\fR 
.ig >>
<a href="attributetypes.html#printfspec">
.>>
\fI printf-spec\fR
.ig >>
</a>
.>>
.IP
Controls the decimal format of rewritten percents, accumulations, totals.


.LP
\fBkeepall\fR \fCyes\fR | \fCno\fR
.IP
If \fCyes\fR, original fields are preserved when doing accumulate, percent, or total.
Thus the result will have more fields than the original data set.  Default is \fCno\fR.

.LP
\fBselect\fR 
.ig >>
<a href="condex.html"
.>>
\fI selection-expression \fR
.ig >>
</a>
.>>
.IP
Used when \fCaction\fR is \fCselect\fR in order to specify the selection 
condition.  

.LP
\fBstack\fR \fCyes\fR | \fCno\fR
.IP
Normally the results of the processing take the place of the original data
set in memory.  However, if \fCstack\fR is set to \fCyes\fR, the results become
the current data set and the original data set is preserved (it may
be restored as the 'current' data set by invoking proc originaldata).
This may be useful when working with larger data sets, to avoid re-reading
them from disk.

.LP
\fBshowresults\fR \fCyes\fR | \fCno\fR
.IP
If \fCyes\fR the data are shown after processing, as a diagnostic aid.

.SH MORE ON BREAK PROCESSING
.LP
The \fBbreaks\fR action (new in version 1.39) allows break processing.  
In data processing terminology, "break processing" is the act of passing 
through a sorted data set and taking some action when a key field or fields change.
For example, if we were processing a list of charges ordered by paying budget
number, we could use a break processing strategy to pause and generate
a statement for one budget number, when we reached the point in the
data set where the budget numbers changed.  Then we would continue on.
The \fBbreaks\fR action allows a similar thing to be done with plotting.
Break processing is significantly more efficient than scanning the entire data 
set multiple times with a select statement, especially with larger data sets.
.LP
The data set must be sorted such that key fields are grouped.
The key field(s) must be specified in the \fCfields\fR attribute.
\fBproc processdata\fR is generally called within a #loop.  When \fBproc processdata\fR
finishes, the "current data set" will be the block of data from the previous break
to the current break.  Subsequent invocations of \fBproc processdata\fR will continue
from the previous location in the data set.  The original data set is always assumed
(i.e. it is not necessary to use \fBproc originaldata\fR).  
.LP
Your script can access the current contents of the break field(s) via the
\fBBREAKFIELD1 .. n\fR variable(s).
Your script can detect when the entire data set has been processed by checking 
the NRECORDS variable (equal to 0), or the BREAKFIELD1 variable ($strlen of 0).
The following is an example:
.br
.nf
\0	#loop
\0	   #proc processdata
\0	     action: breaks
\0	     fields: 1 2 3
\0	   #proc endproc
\0
\0	   #if @NRECORDS = 0
\0	     #break
\0	   #endif
\0
\0	   #proc page
\0	   title: Account @BREAKFIELD1
\0
\0	   #proc bars
\0	   ...
\0
\0	 #endloop
.fi
.LP
Limits: up to 5 break fields may be used.
Comparisons for equality are limited to the first 50 characters.

#include bottom
