Selecting data for analysis
***************************

This chapter documents PSPP commands that temporarily or permanently select data records from the active file for analysis.

FILTER
======

     FILTER BY var_name.
     FILTER OFF.

FILTER allows a boolean-valued variable to be used to select cases from the data stream for processing.

To set up filtering, specify BY and a variable name.  Keyword BY is optional but recommended.  Cases which have a zero or system- or user-missing value are excluded from analysis, but not deleted from the data stream.  Cases with other values are analyzed. To filter based on a different condition, use transformations such as COMPUTE or RECODE to compute a filter variable of the required form, then specify that variable on FILTER.

FILTER OFF turns off case filtering.

Filtering takes place immediately before cases pass to a procedure for analysis.  Only one filter variable may be active at a time.  Normally, case filtering continues until it is explicitly turned off with FILTER OFF.  However, if FILTER is placed after TEMPORARY, it filters only the next procedure or procedure-like command.

N OF CASES
==========

     N [OF CASES] num_of_cases [ESTIMATED].

Sometimes you may want to disregard cases of your input.  N can do this.  N 100 tells PSPP to disregard all cases after the first 100.

If the value specified for N is greater than the number of cases read in, the value is ignored.

N does not discard cases or prevent them from being read.  It just causes cases beyond the last one specified to be ignored by data analysis commands.

A later N command can increase or decrease the number of cases selected.  (To select all the cases without knowing how many there are, specify a very high number: 100000 or whatever you think is large enough.)

Transformation procedures performed after N is executed do cause cases to be discarded.

SAMPLE, PROCESS IF, and SELECT IF have precedence over N---the same results are obtained by both of the following fragments, given the same random number seeds:

     ...set up, read in data...
     N 100.
     SAMPLE .5.
     ...analyze data...

     ...set up, read in data...
     SAMPLE .5.
     N 100.
     ...analyze data...

Both fragments above first randomly sample approximately half of the cases, then select the first 100 of those sampled.

N with the ESTIMATED keyword gives an estimated number of cases before DATA LIST or another command to read in data.  ESTIMATED never limits the number of cases processed by procedures.  PSPP currently does not make use of case count estimates.

When N is specified after TEMPORARY, it affects only the next procedure (see TEMPORARY).

PROCESS IF
==========

     PROCESS IF expression.

PROCESS IF temporarily eliminates cases from the data stream.  Its effects are active only through the execution of the next procedure or procedure-like command.

Specify a boolean expression (see Expressions).  If the value of the expression is true for a particular case, the case will be analyzed.  If the expression has a false or missing value, then the case will be deleted from the data stream for this procedure only.

Regardless of its placement relative to other commands, PROCESS IF always takes effect immediately before data passes to the procedure. Only one PROCESS IF command may be in effect at any given time.

The effects of PROCESS IF are similar, but not identical, to the effects of executing TEMPORARY, then SELECT IF (see SELECT IF).

The filtering performed by PROCESS IF takes place immediately before cases pass to a procedure for analysis.  Because PROCESS IF affects only a single procedure, its placement relative to TEMPORARY is unimportant.

PROCESS IF is deprecated.  It is included for compatibility with old command files.  New syntax files should use SELECT IF or FILTER instead.

SAMPLE
======

     SAMPLE num1 [FROM num2].

SAMPLE randomly samples a proportion of the cases in the active file.  Unless it follows TEMPORARY, it operates as a transformation, permanently removing cases from the active file.

The proportion to sample can be expressed as a single number between 0 and 1.  If k is the number specified, and N is the number of currently-selected cases in the active file, then after SAMPLE K., approximately k*N cases will be selected.

The proportion to sample can also be specified in the style SAMPLE M FROM N.  With this style, cases are selected as follows:

 1. If N is equal to the number of currently-selected cases in the active file, exactly M cases will be selected.

 2. If N is greater than the number of currently-selected cases in the active file, an equivalent proportion of cases will be selected.

 3. If N is less than the number of currently-selected cases in the active, exactly M cases will be selected from the first N cases in the active file.

SAMPLE and SELECT IF are performed in the order specified by the syntax file.

SAMPLE is always performed before N OF CASES, regardless of ordering in the syntax file (see N OF CASES).

The same values for SAMPLE may result in different samples.  To obtain the same sample, use the SET command to set the random number seed to the same value before each SAMPLE.  Different samples may still result when the file is processed on systems with differing endianness or floating-point formats.  By default, the random number seed is based on the system time.

SELECT IF
=========

     SELECT IF expression.

SELECT IF selects cases for analysis based on the value of a boolean expression.  Cases not selected are permanently eliminated from the active file, unless TEMPORARY is in effect (see TEMPORARY).

Specify a boolean expression (see Expressions).  If the value of the expression is true for a particular case, the case will be analyzed.  If the expression has a false or missing value, then the case will be deleted from the data stream.

Place SELECT IF as early in the command file as possible.  Cases that are deleted early can be processed more efficiently in time and space.

When SELECT IF is specified following TEMPORARY (see TEMPORARY), the LAG function may not be used (see LAG).

SPLIT FILE
==========

     Two possible syntaxes:
             SPLIT FILE BY var_list.
             SPLIT FILE OFF.

SPLIT FILE allows multiple sets of data present in one data file to be analyzed separately using single statistical procedure commands.

Specify a list of variable names to analyze multiple sets of data separately.  Groups of cases having the same values for these variables are analyzed by statistical procedure commands as one group. An independent analysis is carried out for each group of cases, and the variable values for the group are printed along with the analysis.

Specify OFF to disable SPLIT FILE and resume analysis of the entire active file as a single group of data.

When SPLIT FILE is specified after TEMPORARY, it affects only the next procedure (see TEMPORARY).

TEMPORARY
=========

     TEMPORARY.

TEMPORARY is used to make the effects of transformations following its execution temporary.  These transformations will affect only the execution of the next procedure or procedure-like command.  Their effects will not be saved to the active file.

The only specification on TEMPORARY is the command name.

TEMPORARY may not appear within a DO IF or LOOP construct.  It may appear only once between procedures and procedure-like commands.

Scratch variables cannot be used following TEMPORARY.

An example may help to clarify:

     DATA LIST /X 1-2.
     BEGIN DATA.
      2
      4
     10
     15
     20
     24
     END DATA.
     COMPUTE X=X/2.
     TEMPORARY.
     COMPUTE X=X+3.
     DESCRIPTIVES X.
     DESCRIPTIVES X.

The data read by the first DESCRIPTIVES are 4, 5, 8, 10.5, 13, 15.  The data read by the first DESCRIPTIVES are 1, 2, 5, 7.5, 10, 12.

WEIGHT
======

     WEIGHT BY var_name.
     WEIGHT OFF.

WEIGHT assigns cases varying weights, changing the frequency distribution of the active file.  Execution of WEIGHT is delayed until data have been read.

If a variable name is specified, WEIGHT causes the values of that variable to be used as weighting factors for subsequent statistical procedures.  Use of keyword BY is optional but recommended.  Weighting variables must be numeric.  Scratch variables may not be used for weighting (see Scratch Variables).

When OFF is specified, subsequent statistical procedures will weight all cases equally.

A positive integer weighting factor W on a case will yield the same statistical output as would replicating the case W times. A weighting factor of 0 is treated for statistical purposes as if the case did not exist in the input.  Weighting values need not be integers, but negative and system-missing values for the weighting variable are interpreted as weighting factors of 0.  User-missing values are not treated specially.

When WEIGHT is specified after TEMPORARY, it affects only the next procedure (see TEMPORARY).

WEIGHT does not cause cases in the active file to be replicated in memory.