Data transformations
********************

The pspp procedures examined in this chapter manipulate data and prepare the active dataset for later analyses.  They do not produce output, as a rule.

AGGREGATE
=========

     AGGREGATE
             OUTFILE={*,'FILE_NAME',FILE_HANDLE} [MODE={REPLACE, ADDVARIABLES}]
             /PRESORTED
             /DOCUMENT
             /MISSING=COLUMNWISE
             /BREAK=VAR_LIST
             /DEST_VAR['LABEL']...=AGR_FUNC(SRC_VARS, ARGS...)...

AGGREGATE summarizes groups of cases into single cases. Cases are divided into groups that have the same values for one or more variables called break variables.  Several functions are available for summarizing case contents.

The OUTFILE subcommand is required and must appear first.  Specify a system file or portable file by file name or file handle (see File Handles), or a dataset by its name (see Datasets). The aggregated cases are written to this file.  If * is specified, then the aggregated cases replace the active dataset's data. Use of OUTFILE to write a portable file is a pspp extension.

If OUTFILE=* is given, then the subcommand MODE may also be specified. The mode subcommand has two possible values: ADDVARIABLES or REPLACE. In REPLACE mode, the entire active dataset is replaced by a new dataset which contains just the break variables and the destination varibles. In this mode, the new file will contain as many cases as there are unique combinations of the break variables. In ADDVARIABLES mode, the destination variables will be appended to the existing active dataset. Cases which have identical combinations of values in their break variables, will receive identical values for the destination variables. The number of cases in the active dataset will remain unchanged. Note that if ADDVARIABLES is specified, then the data must be sorted on the break variables.

By default, the active dataset will be sorted based on the break variables before aggregation takes place.  If the active dataset is already sorted or otherwise grouped in terms of the break variables, specify PRESORTED to save time. PRESORTED is assumed if MODE=ADDVARIABLES is used.

Specify DOCUMENT to copy the documents from the active dataset into the aggregate file (see DOCUMENT).  Otherwise, the aggregate file will not contain any documents, even if the aggregate file replaces the active dataset.

Normally, only a single case (for SD and SD., two cases) need be non-missing in each group for the aggregate variable to be non-missing.  Specifying /MISSING=COLUMNWISE inverts this behavior, so that the aggregate variable becomes missing if any aggregated value is missing.

If PRESORTED, DOCUMENT, or MISSING are specified, they must appear between OUTFILE and BREAK.

At least one break variable must be specified on BREAK, a required subcommand.  The values of these variables are used to divide the active dataset into groups to be summarized.  In addition, at least one DEST_VAR must be specified.

One or more sets of aggregation variables must be specified.  Each set comprises a list of aggregation variables, an equals sign (=), the name of an aggregation function (see the list below), and a list of source variables in parentheses.  Some aggregation functions expect additional arguments following the source variable names.

Aggregation variables typically are created with no variable label, value labels, or missing values.  Their default print and write formats depend on the aggregation function used, with details given in the table below.  A variable label for an aggregation variable may be specified just after the variable's name in the aggregation variable list.

Each set must have exactly as many source variables as aggregation variables.  Each aggregation variable receives the results of applying the specified aggregation function to the corresponding source variable.  The MEAN, MEDIAN, SD, and SUM aggregation functions may only be applied to numeric variables.  All the rest may be applied to numeric and string variables.

The available aggregation functions are as follows:

FGT(VAR_NAME, VALUE)
     Fraction of values greater than the specified constant.  The default format is F5.3.

FIN(VAR_NAME, LOW, HIGH)
     Fraction of values within the specified inclusive range of constants. The default format is F5.3.

FLT(VAR_NAME, VALUE)
     Fraction of values less than the specified constant.  The default format is F5.3.

FIRST(VAR_NAME)
     First non-missing value in break group.  The aggregation variable receives the complete dictionary information from the source variable. The sort performed by AGGREGATE (and by SORT CASES) is stable, so that the first case with particular values for the break variables before sorting will also be the first case in that break group after sorting.

FOUT(VAR_NAME, LOW, HIGH)
     Fraction of values strictly outside the specified range of constants. The default format is F5.3.

LAST(VAR_NAME)
     Last non-missing value in break group.  The aggregation variable receives the complete dictionary information from the source variable. The sort performed by AGGREGATE (and by SORT CASES) is stable, so that the last case with particular values for the break variables before sorting will also be the last case in that break group after sorting.

MAX(VAR_NAME)
     Maximum value.  The aggregation variable receives the complete dictionary information from the source variable.

MEAN(VAR_NAME)
     Arithmetic mean.  Limited to numeric values.  The default format is F8.2.

MEDIAN(VAR_NAME)
     The median value.  Limited to numeric values.  The default format is F8.2.

MIN(VAR_NAME)
     Minimum value.  The aggregation variable receives the complete dictionary information from the source variable.

N(VAR_NAME)
     Number of non-missing values.  The default format is F7.0 if weighting is not enabled, F8.2 if it is (see WEIGHT).

N
     Number of cases aggregated to form this group.  The default format is F7.0 if weighting is not enabled, F8.2 if it is (see WEIGHT).

NMISS(VAR_NAME)
     Number of missing values.  The default format is F7.0 if weighting is not enabled, F8.2 if it is (see WEIGHT).

NU(VAR_NAME)
     Number of non-missing values.  Each case is considered to have a weight of 1, regardless of the current weighting variable (see WEIGHT). The default format is F7.0.

NU
     Number of cases aggregated to form this group.  Each case is considered to have a weight of 1, regardless of the current weighting variable. The default format is F7.0.

NUMISS(VAR_NAME)
     Number of missing values.  Each case is considered to have a weight of 1, regardless of the current weighting variable.  The default format is F7.0.

PGT(VAR_NAME, VALUE)
     Percentage between 0 and 100 of values greater than the specified constant.  The default format is F5.1.

PIN(VAR_NAME, LOW, HIGH)
     Percentage of values within the specified inclusive range of constants.  The default format is F5.1.

PLT(VAR_NAME, VALUE)
     Percentage of values less than the specified constant.  The default format is F5.1.

POUT(VAR_NAME, LOW, HIGH)
     Percentage of values strictly outside the specified range of constants.  The default format is F5.1.

SD(VAR_NAME)
     Standard deviation of the mean.  Limited to numeric values.  The default format is F8.2.

SUM(VAR_NAME)
     Sum.  Limited to numeric values.  The default format is F8.2.

Aggregation functions compare string values in terms of internal character codes. On most modern computers, this is  ASCII or a superset thereof.

The aggregation functions listed above exclude all user-missing values from calculations.  To include user-missing values, insert a period (.) at the end of the function name.  (e.g. SUM.). (Be aware that specifying such a function as the last token on a line will cause the period to be interpreted as the end of the command.)

AGGREGATE both ignores and cancels the current SPLIT FILE settings (see SPLIT FILE).

AUTORECODE
==========

     AUTORECODE VARIABLES=SRC_VARS INTO DEST_VARS
             [ /DESCENDING ]
             [ /PRINT ]
             [ /GROUP ]
             [ /BLANK = {VALID, MISSING} ]

The AUTORECODE procedure considers the N values that a variable takes on and maps them onto values 1...N on a new numeric variable.

Subcommand VARIABLES is the only required subcommand and must come first.  Specify VARIABLES, an equals sign (=), a list of source variables, INTO, and a list of target variables.  There must the same number of source and target variables.  The target variables must not already exist.

AUTORECODE ordinarily assigns each increasing non-missing value of a source variable (for a string, this is based on character code comparisons) to consecutive values of its target variable.  For example, the smallest non-missing value of the source variable is recoded to value 1, the next smallest to 2, and so on.  If the source variable has user-missing values, they are recoded to consecutive values just above the non-missing values.  For example, if a source variables has seven distinct non-missing values, then the smallest missing value would be recoded to 8, the next smallest to 9, and so on.

Use DESCENDING to reverse the sort order for non-missing values, so that the largest non-missing value is recoded to 1, the second-largest to 2, and so on.  Even with DESCENDING, user-missing values are still recoded in ascending order just above the non-missing values.

The system-missing value is always recoded into the system-missing variable in target variables.

If a source value has a value label, then that value label is retained for the new value in the target variable.  Otherwise, the source value itself becomes each new value's label.

Variable labels are copied from the source to target variables.

PRINT is currently ignored.

The GROUP subcommand is relevant only if more than one variable is to be recoded.   It causes a single mapping between source and target values to be used, instead of one map per variable.  With GROUP, user-missing values are taken from the first source variable that has any user-missing values.

If /BLANK=MISSING is given, then string variables which contain only whitespace are recoded as SYSMIS.  If /BLANK=VALID is given then they will be allocated a value like any other.  /BLANK is not relevant to numeric values. /BLANK=VALID is the default.

AUTORECODE is a procedure.  It causes the data to be read.

Autorecode Example
------------------

In the file personnel.sav, the variable occupation is a string variable.   Except for data of a purely commentary nature, string variables are generally a bad idea.  One reason is that data entry errors are easily overlooked.  This has happened in personnel.sav; one entry which should read ``Scientist'' has been mistyped as ``Scrientist''.  In autorecode:ex first, this error will be corrected, (One must use care when correcting such data input errors rather than msimply marking them as missing.  For example, if an occupation has been entered ``Barister'', did the person mean ``Barrister'' or did she mean ``Barista''?) then we will use AUTORECODE to create a new numeric variable which takes recoded values of occupation. Finally, we will remove the old variable and rename the new variable to the name of the old variable.

autorecode.sps
Changing a string variable to a numeric variable using AUTORECODE
after correcting a data entry error

Notice in autorecode:res, how  the new variable has been automatically allocated value labels which correspond to the strings of the old variable. This means that in future analyses the descriptive strings are reported instead of the numeric values.

autorecode
The properties of the occupation variable following AUTORECODE

COMPUTE
=======

     COMPUTE VARIABLE = EXPRESSION.

or

     COMPUTE vector(INDEX) = EXPRESSION.

COMPUTE assigns the value of an expression to a target variable.  For each case, the expression is evaluated and its value assigned to the target variable.  Numeric and string variables may be assigned.  When a string expression's width differs from the target variable's width, the string result of the expression is truncated or padded with spaces on the right as necessary.  The expression and variable types must match.

For numeric variables only, the target variable need not already exist.  Numeric variables created by COMPUTE are assigned an F8.2 output format.  String variables must be declared before they can be used as targets for COMPUTE.

The target variable may be specified as an element of a vector (see VECTOR).  In this case, an expression INDEX must be specified in parentheses following the vector name.  The expression INDEX must evaluate to a numeric value that, after rounding down to the nearest integer, is a valid index for the named vector.

Using COMPUTE to assign to a variable specified on LEAVE (see LEAVE) resets the variable's left state.  Therefore, LEAVE should be specified following COMPUTE, not before.

COMPUTE is a transformation.  It does not cause the active dataset to be read.

When COMPUTE is specified following TEMPORARY (see TEMPORARY), the LAG function may not be used (see LAG).

Compute Examples
----------------

The dataset physiology.sav contains the height and weight of persons. For some purposes, neither height nor weight alone is of interest. Epidemiologists are often more interested in the body mass index which can sometimes be used as a predictor for clinical conditions. The body mass index is defined as the weight of the person in kg divided by the square of the person's height in metres. (Since BMI is a quantity with a ratio scale and has units, the term ``index'' is a misnomer, but that is what it is called.)

compute.sps
Computing the body mass index from weight and height

bmi:ex shows how you can use COMPUTE to generate a new variable called bmi and have every case's value calculated from the existing values of weight and height. It also shows how you can add a label to this new variable (see VARIABLE LABELS), so that a more descriptive label appears in subsequent analyses, and this can be seen in the ouput from the DESCRIPTIVES command in bmi:res.

The expression which follows the = sign can be as complicated as necessary. See Expressions for a precise description of the language accepted.

compute
An analysis which includes bmi in its results

COUNT
=====

     COUNT VAR_NAME = VAR... (VALUE...)
         [/VAR_NAME = VAR... (VALUE...)]...

     Each VALUE takes one of the following forms:
             NUMBER
             STRING
             NUM1 THRU NUM2
             MISSING
             SYSMIS
     where NUM1 is a numeric expression or the words LO  or LOWEST
           and NUM2 is a numeric expression  or HI or HIGHEST.

COUNT creates or replaces a numeric target variable that counts the occurrence of a criterion value or set of values over one or more test variables for each case.

The target variable values are always nonnegative integers.  They are never missing.  The target variable is assigned an F8.2 output format. See Input and Output Formats.  Any variables, including string variables, may be test variables.

User-missing values of test variables are treated just like any other values.  They are not treated as system-missing values. User-missing values that are criterion values or inside ranges of criterion values are counted as any other values.  However (for numeric variables), keyword MISSING may be used to refer to all system- and user-missing values.

COUNT target variables are assigned values in the order specified.  In the command COUNT A=A B(1) /B=A B(2)., the following actions occur:

  - The number of occurrences of 1 between A and B is counted.

  - A is assigned this value.

  - The number of occurrences of 1 between B and the new value of A is counted.

  - B is assigned this value.

Despite this ordering, all COUNT criterion variables must exist before the procedure is executed---they may not be created as target variables earlier in the command!  Break such a command into two separate commands.

Count Examples
--------------

In the survey results in dataset hotel.sav a manager wishes to know how many respondents answered with low valued answers to questions v1, v2 and v3.  This can be found using the code in count:ex.  Specifically, this code creates a new variable, and populates it with the number of values in v1--v2 which are 2 or lower.

count.sps
Counting low values to responses v1, v2 and v3

In count:ex the COUNT transformation creates a new variable, low_counts and its values are shown using the LIST command.

In count:res we can see the values of low_counts after the COUNT transformation has completed.  The first value is 1, because there is only one variable amoung v1, v2 and 3 which has a value of 2 or less. The second value is 2, because both v1 and v2 are 2 or less.

count
The values of v1, v2, v3 and low_counts after
the COUNT transformation has run

FLIP
====

     FLIP /VARIABLES=VAR_LIST /NEWNAMES=VAR_NAME.

FLIP transposes rows and columns in the active dataset.  It causes cases to be swapped with variables, and vice versa.

All variables in the transposed active dataset are numeric.  String variables take on the system-missing value in the transposed file.

N subcommands are required.  If specified, the VARIABLES subcommand selects variables to be transformed into cases, and variables not specified are discarded.  If the VARIABLES subcommand is omitted, all variables are selected for transposition.

The variables specified by NEWNAMES, which must be a string variable, is used to give names to the variables created by FLIP.  Only the first 8 characters of the variable are used.  If NEWNAMES is not specified then the default is a variable named CASE_LBL, if it exists. If it does not then the variables created by FLIP are named VAR000 through VAR999, then VAR1000, VAR1001, and so on.

When a NEWNAMES variable is available, the names must be canonicalized before becoming variable names.  Invalid characters are replaced by letter V in the first position, or by _ in subsequent positions.  If the name thus generated is not unique, then numeric extensions are added, starting with 1, until a unique name is found or there are no remaining possibilities.  If the latter occurs then the FLIP operation aborts.

The resultant dictionary contains a CASE_LBL variable, a string variable of width 8, which stores the names of the variables in the dictionary before the transposition.  Variables names longer than 8 characters are truncated.  If FLIP is called again on this dataset, the CASE_LBL variable can be passed to the NEWNAMES subcommand to recreate the original variable names.

FLIP honors N OF CASES (see N OF CASES).  It ignores TEMPORARY (see TEMPORARY), so that ``temporary'' transformations become permanent.

Flip Examples
-------------

In flip:ex, data has been entered using DATA LIST (see DATA LIST) such that the first variable in the dataset is a string variable containing a description of the other data for the case. Clearly this is not a convenient arrangement for performing statistical analyses, so it would have been better to think a little more carefully about how the data should have been arranged. However often the data is provided by some third party source, and you have no control over the form. Fortunately, we can use FLIP to exchange the variables and cases in the active dataset.

flip.sps
Using FLIP to exchange variables and cases in a dataset

As you can see in flip:res before the FLIP command has run there are seven variables (six containing data and one for the heading) and three cases. Afterwards there are four variables (one per case, plus the CASE_LBL variable) and six cases. You can delete the CASE_LBL variable (see DELETE VARIABLES) if you don't need it.

flip
The results of using FLIP to exchange variables and cases in a dataset

IF
==

     IF CONDITION VARIABLE=EXPRESSION.

or

     IF CONDITION vector(INDEX)=EXPRESSION.

The IF transformation conditionally assigns the value of a target expression to a target variable, based on the truth of a test expression.

Specify a boolean-valued expression (see Expressions) to be tested following the IF keyword.  This expression is evaluated for each case. If the value is true, then the value of the expression is computed and assigned to the specified variable.  If the value is false or missing, nothing is done.  Numeric and string variables may be assigned.  When a string expression's width differs from the target variable's width, the string result of the expression is truncated or padded with spaces on the right as necessary.  The expression and variable types must match.

The target variable may be specified as an element of a vector (see VECTOR).  In this case, a vector index expression must be specified in parentheses following the vector name.  The index expression must evaluate to a numeric value that, after rounding down to the nearest integer, is a valid index for the named vector.

Using IF to assign to a variable specified on LEAVE (see LEAVE) resets the variable's left state.  Therefore, LEAVE should be specified following IF, not before.

When IF is specified following TEMPORARY (see TEMPORARY), the LAG function may not be used (see LAG).

RECODE
======

The RECODE command is used to transform existing values into other, user specified values. The general form is:

     RECODE SRC_VARS
             (SRC_VALUE SRC_VALUE ... = DEST_VALUE)
             (SRC_VALUE SRC_VALUE ... = DEST_VALUE)
             (SRC_VALUE SRC_VALUE ... = DEST_VALUE) ...
              [INTO DEST_VARS].

Following the RECODE keyword itself comes SRC_VARS which is a list of variables whose values are to be transformed. These variables may be string variables or they may be numeric. However the list must be homogeneous; you may not mix string variables and numeric variables in the same recoding.

After the list of source variables, there should be one or more mappings. Each mapping is enclosed in parentheses, and contains the source values and a destination value separated by a single =. The source values are used to specify the values in the dataset which need to change, and the destination value specifies the new value to which they should be changed. Each SRC_VALUE may take one of the following forms:

NUMBER
     If the source variables are numeric then SRC_VALUE may be a literal number.

STRING
     If the source variables are string variables then SRC_VALUE may be a literal string (like all strings, enclosed in single or double quotes).

NUM1 THRU NUM2
     This form is valid only when the source variables are numeric. It specifies all values in the range between NUM1 and NUM2, including both endpoints of the range.  By convention, NUM1 should be less than NUM2. Open-ended ranges may be specified using LO or LOWEST for NUM1 or HI or HIGHEST for NUM2.

MISSING
     The literal keyword MISSING matches both system missing and user missing values. It is valid for both numeric and string variables.

SYSMIS
     The literal keyword SYSMIS matches system missing values. It is valid for both numeric variables only.

ELSE
     The ELSE keyword may be used to match any values which are not matched by any other SRC_VALUE appearing in the command. If this keyword appears, it should be used in the last mapping of the command.

After the source variables comes an = and then the DEST_VALUE. The DEST_VALUE may take any of the following forms:

NUMBER
     A literal numeric value to which the source values should be changed. This implies the destination variable must be numeric.

STRING
     A literal string value (enclosed in quotation marks) to which the source values should be changed. This implies the destination variable must be a string variable.

SYSMIS
     The keyword SYSMIS changes the value to the system missing value. This implies the destination variable must be numeric.

COPY
     The special keyword COPY means that the source value should not be modified, but copied directly to the destination value. This is meaningful only if INTO DEST_VARS is specified.

Mappings are considered from left to right. Therefore, if a value is matched by a SRC_VALUE from more than one mapping, the first (leftmost) mapping which matches will be considered. Any subsequent matches will be ignored.

The clause INTO DEST_VARS is optional. The behaviour of the command is slightly different depending on whether it appears or not.

If INTO DEST_VARS does not appear, then values will be recoded ``in place''. This means that the recoded values are written back to the source variables from whence the original values came. In this case, the DEST_VALUE for every mapping must imply a value which has the same type as the SRC_VALUE. For example, if the source value is a string value, it is not permissible for DEST_VALUE to be SYSMIS or another forms which implies a numeric result. It is also not permissible for DEST_VALUE to be  longer than the width of the source variable.

The following example two numeric variables X and Y are recoded in place. Zero is recoded to 99, the values 1 to 10 inclusive are unchanged, values 1000 and higher are recoded to the system-missing value and all other values are changed to 999:

     recode X Y
             (0 = 99)
             (1 THRU 10 = COPY)
             (1000 THRU HIGHEST = SYSMIS)
             (ELSE = 999).

If INTO DEST_VARS is given, then recoded values are written into the variables specified in DEST_VARS, which must therefore contain a list of valid variable names. The number of variables in DEST_VARS must be the same as the number of variables in SRC_VARS and the respective order of the variables in DEST_VARS corresponds to the order of SRC_VARS. That is to say, recoded values whose original value came from the Nth variable in SRC_VARS will be placed into the Nth variable in DEST_VARS. The source variables will be unchanged. If any mapping implies a string as its destination value, then the respective destination variable must already exist, or have been declared using STRING or another transformation. Numeric variables however will be automatically created if they don't already exist. The following example deals with two source variables, A and B which contain string values.  Hence there are two destination variables V1 and V2. Any cases where A or B contain the values apple, pear or pomegranate will result in V1 or V2 being filled with the string fruit whilst cases with tomato, lettuce or carrot will result in vegetable. Any other values will produce the result unknown:

     string V1 (a20).
     string V2 (a20).

     recode A B
             ("apple" "pear" "pomegranate" = "fruit")
             ("tomato" "lettuce" "carrot" = "vegetable")
             (ELSE = "unknown")
             into V1 V2.

There is one very special mapping, not mentioned above. If the source variable is a string variable then a mapping may be specified as (CONVERT). This mapping, if it appears must be the last mapping given and the INTO DEST_VARS clause must also be given and must not refer to a string variable. CONVERT causes a number specified as a string to be converted to a numeric value. For example it will convert the string "3" into the numeric value 3 (note that it will not convert three into 3). If the string cannot be parsed as a number, then the system-missing value is assigned instead. In the following example, cases where the value of X (a string variable) is the empty string, are recoded to 999 and all others are converted to the numeric equivalent of the input value.  The results are placed into the numeric variable Y:

     recode X
            ("" = 999)
             (convert)
             into Y.

It is possible to specify multiple recodings on a single command. Introduce additional recodings with a slash (/) to separate them from the previous recodings:

     recode
             A  (2 = 22) (else = 99)
             /B (1 = 3) into Z
             .

Here we have two recodings. The first affects the source variable A and recodes in-place the value 2 into 22 and all other values to 99. The second recoding copies the values of B into the variable Z, changing any instances of 1 into 3.

SORT CASES
==========

     SORT CASES BY VAR_LIST[({D|A}] [ VAR_LIST[({D|A}] ] ...

SORT CASES sorts the active dataset by the values of one or more variables.

Specify BY and a list of variables to sort by.  By default, variables are sorted in ascending order.  To override sort order, specify (D) or (DOWN) after a list of variables to get descending order, or (A) or (UP) for ascending order.  These apply to all the listed variables up until the preceding (A), (D), (UP) or (DOWN).

The sort algorithms used by SORT CASES are stable.  That is, records that have equal values of the sort variables will have the same relative order before and after sorting.  As a special case, re-sorting an already sorted file will not affect the ordering of cases.

SORT CASES is a procedure.  It causes the data to be read.

SORT CASES attempts to sort the entire active dataset in main memory. If workspace is exhausted, it falls back to a merge sort algorithm that involves creates numerous temporary files.

SORT CASES may not be specified following TEMPORARY.

Sorting Example
---------------