1 @node Data Manipulation, Data Selection, Variable Attributes, Top
2 @chapter Data transformations
3 @cindex transformations
5 The PSPP procedures examined in this chapter manipulate data and
6 prepare the active file for later analyses. They do not produce output,
10 * AGGREGATE:: Summarize multiple cases into a single case.
11 * AUTORECODE:: Automatic recoding of variables.
12 * COMPUTE:: Assigning a variable a calculated value.
13 * COUNT:: Counting variables with particular values.
14 * FLIP:: Exchange variables with cases.
15 * IF:: Conditionally assigning a calculated value.
16 * RECODE:: Mapping values from one set to another.
17 * SORT CASES:: Sort the active file.
20 @node AGGREGATE, AUTORECODE, Data Manipulation, Data Manipulation
26 OUTFILE=@{*,'file-name',file_handle@}
31 /dest_var['label']@dots{}=agr_func(src_vars, args@dots{})@dots{}
34 @cmd{AGGREGATE} summarizes groups of cases into single cases.
35 Cases are divided into groups that have the same values for one or more
36 variables called @dfn{break variables}. Several functions are available
37 for summarizing case contents.
39 The OUTFILE subcommand is required and must appear first. Specify a
40 system file, portable file, or scratch file by file name or file
41 handle (@pxref{File Handles}).
42 The aggregated cases are written to this file. If @samp{*} is
43 specified, then the aggregated cases replace the active file. Use of
44 OUTFILE to write a portable file or scratch file is a PSPP extension.
46 By default, the active file will be sorted based on the break variables
47 before aggregation takes place. If the active file is already sorted
48 or otherwise grouped in terms of the break variables, specify
49 PRESORTED to save time.
51 Specify DOCUMENT to copy the documents from the active file into the
52 aggregate file (@pxref{DOCUMENT}). Otherwise, the aggregate file will
53 not contain any documents, even if the aggregate file replaces the
56 Normally, only a single case (for SD and SD., two cases) need be
57 non-missing in each group for the aggregate variable to be
58 non-missing. Specifying /MISSING=COLUMNWISE inverts this behavior, so
59 that the aggregate variable becomes missing if any aggregated value is
62 If PRESORTED, DOCUMENT, or MISSING are specified, they must appear
63 between OUTFILE and BREAK.
65 At least one break variable must be specified on BREAK, a
66 required subcommand. The values of these variables are used to divide
67 the active file into groups to be summarized. In addition, at least
68 one @var{dest_var} must be specified.
70 One or more sets of aggregation variables must be specified. Each set
71 comprises a list of aggregation variables, an equals sign (@samp{=}),
72 the name of an aggregation function (see the list below), and a list
73 of source variables in parentheses. Some aggregation functions expect
74 additional arguments following the source variable names.
76 Aggregation variables typically are created with no variable label,
77 value labels, or missing values. Their default print and write
78 formats depend on the aggregation function used, with details given in
79 the table below. A variable label for an aggregation variable may be
80 specified just after the variable's name in the aggregation variable
83 Each set must have exactly as many source variables as aggregation
84 variables. Each aggregation variable receives the results of applying
85 the specified aggregation function to the corresponding source
86 variable. The MEAN, SD, and SUM aggregation functions may only be
87 applied to numeric variables. All the rest may be applied to numeric
88 and short and long string variables.
90 The available aggregation functions are as follows:
93 @item FGT(var_name, value)
94 Fraction of values greater than the specified constant. The default
97 @item FIN(var_name, low, high)
98 Fraction of values within the specified inclusive range of constants.
99 The default format is F5.3.
101 @item FLT(var_name, value)
102 Fraction of values less than the specified constant. The default
105 @item FIRST(var_name)
106 First non-missing value in break group. The aggregation variable
107 receives the complete dictionary information from the source variable.
108 The sort performed by AGGREGATE (and by SORT CASES) is stable, so that
109 the first case with particular values for the break variables before
110 sorting will also be the first case in that break group after sorting.
112 @item FOUT(var_name, low, high)
113 Fraction of values strictly outside the specified range of constants.
114 The default format is F5.3.
117 Last non-missing value in break group. The aggregation variable
118 receives the complete dictionary information from the source variable.
119 The sort performed by AGGREGATE (and by SORT CASES) is stable, so that
120 the last case with particular values for the break variables before
121 sorting will also be the last case in that break group after sorting.
124 Maximum value. The aggregation variable receives the complete
125 dictionary information from the source variable.
128 Arithmetic mean. Limited to numeric values. The default format is
132 Minimum value. The aggregation variable receives the complete
133 dictionary information from the source variable.
136 Number of non-missing values. The default format is F7.0 if weighting
137 is not enabled, F8.2 if it is (@pxref{WEIGHT}).
140 Number of cases aggregated to form this group. The default format is
141 F7.0 if weighting is not enabled, F8.2 if it is (@pxref{WEIGHT}).
143 @item NMISS(var_name)
144 Number of missing values. The default format is F7.0 if weighting is
145 not enabled, F8.2 if it is (@pxref{WEIGHT}).
148 Number of non-missing values. Each case is considered to have a weight
149 of 1, regardless of the current weighting variable (@pxref{WEIGHT}).
150 The default format is F7.0.
153 Number of cases aggregated to form this group. Each case is considered
154 to have a weight of 1, regardless of the current weighting variable.
155 The default format is F7.0.
157 @item NUMISS(var_name)
158 Number of missing values. Each case is considered to have a weight of
159 1, regardless of the current weighting variable. The default format is F7.0.
161 @item PGT(var_name, value)
162 Percentage between 0 and 100 of values greater than the specified
163 constant. The default format is F5.1.
165 @item PIN(var_name, low, high)
166 Percentage of values within the specified inclusive range of
167 constants. The default format is F5.1.
169 @item PLT(var_name, value)
170 Percentage of values less than the specified constant. The default
173 @item POUT(var_name, low, high)
174 Percentage of values strictly outside the specified range of
175 constants. The default format is F5.1.
178 Standard deviation of the mean. Limited to numeric values. The
179 default format is F8.2.
182 Sum. Limited to numeric values. The default format is F8.2.
185 Aggregation functions compare string values in terms of internal
186 character codes. On most modern computers, this is a form of ASCII.
188 The aggregation functions listed above exclude all user-missing values
189 from calculations. To include user-missing values, insert a period
190 (@samp{.}) at the end of the function name. (e.g.@: @samp{SUM.}).
191 (Be aware that specifying such a function as the last token on a line
192 will cause the period to be interpreted as the end of the command.)
194 @cmd{AGGREGATE} both ignores and cancels the current @cmd{SPLIT FILE}
195 settings (@pxref{SPLIT FILE}).
197 @node AUTORECODE, COMPUTE, AGGREGATE, Data Manipulation
202 AUTORECODE VARIABLES=src_vars INTO dest_vars
207 The @cmd{AUTORECODE} procedure considers the @var{n} values that a variable
208 takes on and maps them onto values 1@dots{}@var{n} on a new numeric
211 Subcommand VARIABLES is the only required subcommand and must come
212 first. Specify VARIABLES, an equals sign (@samp{=}), a list of source
213 variables, INTO, and a list of target variables. There must the same
214 number of source and target variables. The target variables must not
217 By default, increasing values of a source variable (for a string, this
218 is based on character code comparisons) are recoded to increasing values
219 of its target variable. To cause increasing values of a source variable
220 to be recoded to decreasing values of its target variable (@var{n} down
221 to 1), specify DESCENDING.
223 PRINT is currently ignored.
225 @cmd{AUTORECODE} is a procedure. It causes the data to be read.
227 @node COMPUTE, COUNT, AUTORECODE, Data Manipulation
232 COMPUTE variable = expression.
234 COMPUTE vector(index) = expression.
237 @cmd{COMPUTE} assigns the value of an expression to a target
238 variable. For each case, the expression is evaluated and its value
239 assigned to the target variable. Numeric and short and long string
240 variables may be assigned. When a string expression's width differs
241 from the target variable's width, the string result of the expression
242 is truncated or padded with spaces on the right as necessary. The
243 expression and variable types must match.
245 For numeric variables only, the target variable need not already
246 exist. Numeric variables created by @cmd{COMPUTE} are assigned an
247 @code{F8.2} output format. String variables must be declared before
248 they can be used as targets for @cmd{COMPUTE}.
250 The target variable may be specified as an element of a vector
251 (@pxref{VECTOR}). In this case, a vector index expression must be
252 specified in parentheses following the vector name. The index
253 expression must evaluate to a numeric value that, after rounding down
254 to the nearest integer, is a valid index for the named vector.
256 Using @cmd{COMPUTE} to assign to a variable specified on @cmd{LEAVE}
257 (@pxref{LEAVE}) resets the variable's left state. Therefore,
258 @code{LEAVE} should be specified following @cmd{COMPUTE}, not before.
260 @cmd{COMPUTE} is a transformation. It does not cause the active file to be
263 When @cmd{COMPUTE} is specified following @cmd{TEMPORARY}
264 (@pxref{TEMPORARY}), the @cmd{LAG} function may not be used
267 @node COUNT, FLIP, COMPUTE, Data Manipulation
272 COUNT var_name = var@dots{} (value@dots{}).
274 Each value takes one of the following forms:
280 In addition, num1 and num2 can be LO or LOWEST, or HI or HIGHEST,
284 @cmd{COUNT} creates or replaces a numeric @dfn{target} variable that
285 counts the occurrence of a @dfn{criterion} value or set of values over
286 one or more @dfn{test} variables for each case.
288 The target variable values are always nonnegative integers. They are
289 never missing. The target variable is assigned an F8.2 output format.
290 @xref{Input and Output Formats}. Any variables, including long and short
291 string variables, may be test variables.
293 User-missing values of test variables are treated just like any other
294 values. They are @strong{not} treated as system-missing values.
295 User-missing values that are criterion values or inside ranges of
296 criterion values are counted as any other values. However (for numeric
297 variables), keyword MISSING may be used to refer to all system-
298 and user-missing values.
300 @cmd{COUNT} target variables are assigned values in the order
301 specified. In the command @code{COUNT A=A B(1) /B=A B(2).}, the
302 following actions occur:
306 The number of occurrences of 1 between @code{A} and @code{B} is counted.
309 @code{A} is assigned this value.
312 The number of occurrences of 1 between @code{B} and the @strong{new}
313 value of @code{A} is counted.
316 @code{B} is assigned this value.
319 Despite this ordering, all @cmd{COUNT} criterion variables must exist
320 before the procedure is executed---they may not be created as target
321 variables earlier in the command! Break such a command into two
324 The examples below may help to clarify.
328 Assuming @code{Q0}, @code{Q2}, @dots{}, @code{Q9} are numeric variables,
329 the following commands:
333 Count the number of times the value 1 occurs through these variables
334 for each case and assigns the count to variable @code{QCOUNT}.
337 Print out the total number of times the value 1 occurs throughout
338 @emph{all} cases using @cmd{DESCRIPTIVES}. @xref{DESCRIPTIVES}, for
343 COUNT QCOUNT=Q0 TO Q9(1).
344 DESCRIPTIVES QCOUNT /STATISTICS=SUM.
348 Given these same variables, the following commands:
352 Count the number of valid values of these variables for each case and
353 assigns the count to variable @code{QVALID}.
356 Multiplies each value of @code{QVALID} by 10 to obtain a percentage of
357 valid values, using @cmd{COMPUTE}. @xref{COMPUTE}, for details.
360 Print out the percentage of valid values across all cases, using
361 @cmd{DESCRIPTIVES}. @xref{DESCRIPTIVES}, for details.
365 COUNT QVALID=Q0 TO Q9 (LO THRU HI).
366 COMPUTE QVALID=QVALID*10.
367 DESCRIPTIVES QVALID /STATISTICS=MEAN.
371 @node FLIP, IF, COUNT, Data Manipulation
376 FLIP /VARIABLES=var_list /NEWNAMES=var_name.
379 @cmd{FLIP} transposes rows and columns in the active file. It
380 causes cases to be swapped with variables, and vice versa.
382 All variables in the transposed active file are numeric. String
383 variables take on the system-missing value in the transposed file.
385 No subcommands are required. If specified, the VARIABLES subcommand
386 selects variables to be transformed into cases, and variables not
387 specified are discarded. If the VARIABLES subcommand is omitted, all
388 variables are selected for transposition.
390 The variables specified by NEWNAMES, which must be a string variable, is
391 used to give names to the variables created by @cmd{FLIP}. Only the
392 first 8 characters of the variable are used. If
394 specified then the default is a variable named CASE_LBL, if it exists.
395 If it does not then the variables created by FLIP are named VAR000
396 through VAR999, then VAR1000, VAR1001, and so on.
398 When a NEWNAMES variable is available, the names must be canonicalized
399 before becoming variable names. Invalid characters are replaced by
400 letter @samp{V} in the first position, or by @samp{_} in subsequent
401 positions. If the name thus generated is not unique, then numeric
402 extensions are added, starting with 1, until a unique name is found or
403 there are no remaining possibilities. If the latter occurs then the
404 FLIP operation aborts.
406 The resultant dictionary contains a CASE_LBL variable, a string
407 variable of width 8, which stores the names of the variables in the
408 dictionary before the transposition. Variables names longer than 8
409 characters are truncated. If the active file is subsequently
410 transposed using @cmd{FLIP}, this variable can be used to recreate the
411 original variable names.
413 FLIP honors @cmd{N OF CASES} (@pxref{N OF CASES}). It ignores
414 @cmd{TEMPORARY} (@pxref{TEMPORARY}), so that ``temporary''
415 transformations become permanent.
417 @node IF, RECODE, FLIP, Data Manipulation
422 IF condition variable=expression.
424 IF condition vector(index)=expression.
427 The @cmd{IF} transformation conditionally assigns the value of a target
428 expression to a target variable, based on the truth of a test
431 Specify a boolean-valued expression (@pxref{Expressions}) to be tested
432 following the IF keyword. This expression is evaluated for each case.
433 If the value is true, then the value of the expression is computed and
434 assigned to the specified variable. If the value is false or missing,
435 nothing is done. Numeric and short and long string variables may be
436 assigned. When a string expression's width differs from the target
437 variable's width, the string result of the expression is truncated or
438 padded with spaces on the right as necessary. The expression and
439 variable types must match.
441 The target variable may be specified as an element of a vector
442 (@pxref{VECTOR}). In this case, a vector index expression must be
443 specified in parentheses following the vector name. The index
444 expression must evaluate to a numeric value that, after rounding down
445 to the nearest integer, is a valid index for the named vector.
447 Using @cmd{IF} to assign to a variable specified on @cmd{LEAVE}
448 (@pxref{LEAVE}) resets the variable's left state. Therefore,
449 @code{LEAVE} should be specified following @cmd{IF}, not before.
451 When @cmd{IF} is specified following @cmd{TEMPORARY}
452 (@pxref{TEMPORARY}), the @cmd{LAG} function may not be used
455 @node RECODE, SORT CASES, IF, Data Manipulation
460 RECODE var_list (src_value@dots{}=dest_value)@dots{} [INTO var_list].
462 src_value may take the following forms:
469 Open-ended ranges may be specified using LO or LOWEST for num1
470 or HI or HIGHEST for num2.
472 dest_value may take the following forms:
479 @cmd{RECODE} translates data from one range of values to
480 another, via flexible user-specified mappings. Data may be remapped
481 in-place or copied to new variables. Numeric, short string, and long
482 string data can be recoded.
484 Specify the list of source variables, followed by one or more mapping
485 specifications each enclosed in parentheses. If the data is to be
486 copied to new variables, specify INTO, then the list of target
487 variables. String target variables must already have been declared
488 using @cmd{STRING} or another transformation, but numeric target
490 be created on the fly. There must be exactly as many target variables
491 as source variables. Each source variable is remapped into its
492 corresponding target variable.
494 When INTO is not used, the input and output variables must be of the
495 same type. Otherwise, string values can be recoded into numeric values,
496 and vice versa. When this is done and there is no mapping for a
497 particular value, either a value consisting of all spaces or the
498 system-missing value is assigned, depending on variable type.
500 Mappings are considered from left to right. The first src_value that
501 matches the value of the source variable causes the target variable to
502 receive the value indicated by the dest_value. Literal number, string,
503 and range src_value's should be self-explanatory. MISSING as a
504 src_value matches any user- or system-missing value. SYSMIS matches the
505 system missing value only. ELSE is a catch-all that matches anything.
506 It should be the last src_value specified.
508 Numeric and string dest_value's should also be self-explanatory. COPY
509 causes the input values to be copied to the output. This is only value
510 if the source and target variables are of the same type. SYSMIS
511 indicates the system-missing value.
513 If the source variables are strings and the target variables are
514 numeric, then there is one additional mapping available: (CONVERT),
515 which must be the last specified mapping. CONVERT causes a number
516 specified as a string to be converted to a numeric value. If the string
517 cannot be parsed as a number, then the system-missing value is assigned.
519 Multiple recodings can be specified on a single @cmd{RECODE} invocation.
520 Introduce additional recodings with a slash (@samp{/}) to
521 separate them from the previous recodings.
523 @node SORT CASES, , RECODE, Data Manipulation
528 SORT CASES BY var_list.
531 @cmd{SORT CASES} sorts the active file by the values of one or more
534 Specify BY and a list of variables to sort by. By default, variables
535 are sorted in ascending order. To override sort order, specify (D) or
536 (DOWN) after a list of variables to get descending order, or (A) or (UP)
537 for ascending order. These apply to the entire list of variables
540 The sort algorithms used by @cmd{SORT CASES} are stable. That is,
541 records that have equal values of the sort variables will have the
542 same relative order before and after sorting. As a special case,
543 re-sorting an already sorted file will not affect the ordering of
546 @cmd{SORT CASES} is a procedure. It causes the data to be read.
548 @cmd{SORT CASES} attempts to sort the entire active file in main memory.
549 If workspace is exhausted, it falls back to a merge sort algorithm that
550 involves creates numerous temporary files.
552 @cmd{SORT CASES} may not be specified following TEMPORARY.