Selecting data for analysis
***************************

This chapter documents pspp commands that temporarily or permanently select data records from the active dataset for analysis.

FILTER
======

     FILTER BY VAR_NAME.
     FILTER OFF.

FILTER allows a boolean-valued variable to be used to select cases from the data stream for processing.

To set up filtering, specify BY and a variable name.  Keyword BY is optional but recommended.  Cases which have a zero or system- or user-missing value are excluded from analysis, but not deleted from the data stream.  Cases with other values are analyzed. To filter based on a different condition, use transformations such as COMPUTE or RECODE to compute a filter variable of the required form, then specify that variable on FILTER.

FILTER OFF turns off case filtering.

Filtering takes place immediately before cases pass to a procedure for analysis.  Only one filter variable may be active at a time.  Normally, case filtering continues until it is explicitly turned off with FILTER OFF.  However, if FILTER is placed after TEMPORARY, it filters only the next procedure or procedure-like command.

N OF CASES
==========

     N [OF CASES] NUM_OF_CASES [ESTIMATED].

N OF CASES limits the number of cases processed by any procedures that follow it in the command stream.  N OF CASES 100, for example, tells pspp to disregard all cases after the first 100.

When N OF CASES is specified after TEMPORARY, it affects only the next procedure (see TEMPORARY).  Otherwise, cases beyond the limit specified are not processed by any later procedure.

If the limit specified on N OF CASES is greater than the number of cases in the active dataset, it has no effect.

When N OF CASES is used along with SAMPLE or SELECT IF, the case limit is applied to the cases obtained after sampling or case selection, regardless of how N OF CASES is placed relative to SAMPLE or SELECT IF in the command file.  Thus, the commands N OF CASES 100 and SAMPLE .5 will both randomly sample approximately half of the active dataset's cases, then select the first 100 of those sampled, regardless of their order in the command file.

N OF CASES with the ESTIMATED keyword gives an estimated number of cases before DATA LIST or another command to read in data.  ESTIMATED never limits the number of cases processed by procedures.  pspp currently does not make use of case count estimates.

SAMPLE
======

     SAMPLE NUM1 [FROM NUM2].

SAMPLE randomly samples a proportion of the cases in the active file.  Unless it follows TEMPORARY, it operates as a transformation, permanently removing cases from the active dataset.

The proportion to sample can be expressed as a single number between 0 and 1.  If K is the number specified, and N is the number of currently-selected cases in the active dataset, then after SAMPLE K., approximately K*N cases will be selected.

The proportion to sample can also be specified in the style SAMPLE M FROM N.  With this style, cases are selected as follows:

 1. If N is equal to the number of currently-selected cases in the active dataset, exactly M cases will be selected.

 2. If N is greater than the number of currently-selected cases in the active dataset, an equivalent proportion of cases will be selected.

 3. If N is less than the number of currently-selected cases in the active, exactly M cases will be selected from the first N cases in the active dataset.

SAMPLE and SELECT IF are performed in the order specified by the syntax file.

SAMPLE is always performed before N OF CASES, regardless of ordering in the syntax file (see N OF CASES).

The same values for SAMPLE may result in different samples.  To obtain the same sample, use the SET command to set the random number seed to the same value before each SAMPLE.  Different samples may still result when the file is processed on systems with differing endianness or floating-point formats.  By default, the random number seed is based on the system time.

SELECT IF
=========

     SELECT IF EXPRESSION.

SELECT IF selects cases for analysis based on the value of EXPRESSION.  Cases not selected are permanently eliminated from the active dataset, unless TEMPORARY is in effect (see TEMPORARY).

Specify a boolean expression (see Expressions).  If the value of the expression is true for a particular case, the case will be analyzed.  If the expression has a false or missing value, then the case will be deleted from the data stream.

Place SELECT IF as early in the command file as possible.  Cases that are deleted early can be processed more efficiently in time and space.

When SELECT IF is specified following TEMPORARY (see TEMPORARY), the LAG function may not be used (see LAG).

SPLIT FILE
==========

     SPLIT FILE [{LAYERED, SEPARATE}] BY VAR_LIST.
     SPLIT FILE OFF.

SPLIT FILE allows multiple sets of data present in one data file to be analyzed separately using single statistical procedure commands.

Specify a list of variable names to analyze multiple sets of data separately.  Groups of adjacent cases having the same values for these variables are analyzed by statistical procedure commands as one group. An independent analysis is carried out for each group of cases, and the variable values for the group are printed along with the analysis.

When a list of variable names is specified, one of the keywords LAYERED or SEPARATE may also be specified.  If provided, either keyword are ignored.

Groups are formed only by adjacent cases.  To create a split using a variable where like values are not adjacent in the working file, you should first sort the data by that variable (see SORT CASES).

Specify OFF to disable SPLIT FILE and resume analysis of the entire active dataset as a single group of data.

When SPLIT FILE is specified after TEMPORARY, it affects only the next procedure (see TEMPORARY).

Example Split
-------------

The file horticulture.sav contains data describing the yield of a number of horticultural specimens which have been subjected to various treatments.   If we wanted to investigate linear statistics of the yeild, one way to do this is using the DESCRIPTIVES (see DESCRIPTIVES). However, it is reasonable to expect the mean to be different depending on the treatment.   So we might want to perform three separate procedures --- one for each treatment. (There are other, possibly better, ways to achieve a similar result using the MEANS or EXAMINE commands.) split:ex shows how this can be done automatically using the SPLIT FILE command.

split.sps
Running DESCRIPTIVES on each value of treatment

In split:res you can see that the table of descriptive statistics appears 3 times --- once for each value of treatment. In this example N, the number of observations are identical in all splits.  This is because that experiment was deliberately designed that way.  However in general one can expect a different N for each split.

split.out
The results of running DESCRIPTIVES with an active split

Unless TEMPORARY was used, after a split has been defined for a dataset it remains active until explicitly disabled.

TEMPORARY
=========

     TEMPORARY.

TEMPORARY is used to make the effects of transformations following its execution temporary.  These transformations will affect only the execution of the next procedure or procedure-like command.  Their effects will not be saved to the active dataset.

The only specification on TEMPORARY is the command name.

TEMPORARY may not appear within a DO IF or LOOP construct.  It may appear only once between procedures and procedure-like commands.

Scratch variables cannot be used following TEMPORARY.

An example may help to clarify:

     DATA LIST /X 1-2.
     BEGIN DATA.
      2
      4
     10
     15
     20
     24
     END DATA.

     COMPUTE X=X/2.

     TEMPORARY.
     COMPUTE X=X+3.

     DESCRIPTIVES X.
     DESCRIPTIVES X.

The data read by the first DESCRIPTIVES are 4, 5, 8, 10.5, 13, 15.  The data read by the first DESCRIPTIVES are 1, 2, 5, 7.5, 10, 12.

WEIGHT
======

     WEIGHT BY VAR_NAME.
     WEIGHT OFF.

WEIGHT assigns cases varying weights, changing the frequency distribution of the active dataset.  Execution of WEIGHT is delayed until data have been read.

If a variable name is specified, WEIGHT causes the values of that variable to be used as weighting factors for subsequent statistical procedures.  Use of keyword BY is optional but recommended.  Weighting variables must be numeric.  Scratch variables may not be used for weighting (see Scratch Variables).

When OFF is specified, subsequent statistical procedures will weight all cases equally.

A positive integer weighting factor W on a case will yield the same statistical output as would replicating the case W times. A weighting factor of 0 is treated for statistical purposes as if the case did not exist in the input.  Weighting values need not be integers, but negative and system-missing values for the weighting variable are interpreted as weighting factors of 0.  User-missing values are not treated specially.

When WEIGHT is specified after TEMPORARY, it affects only the next procedure (see TEMPORARY).

WEIGHT does not cause cases in the active dataset to be replicated in memory.

Example Weights
---------------

One could define a  dataset containing an inventory of stock items. It would be reasonable to use a string variable for a description of the item, and a numeric variable for the number in stock, like in weight:ex.

weight.sps
Setting the weight on the variable quantity

One analysis which most surely would be of interest is the relative amounts or each item in stock. However without setting a weight variable, FREQUENCIES (see FREQUENCIES) will not tell us what we want to know, since there is only one case for each stock item. weight:res shows the difference between the weighted and unweighted frequency tables.

weight
Weighted and unweighted frequency tables of items