X-Git-Url: https://pintos-os.org/cgi-bin/gitweb.cgi?a=blobdiff_plain;f=doc%2Fdata-selection.texi;h=c2f3ab31a4ddf0fdd33797a83981acd820e35590;hb=refs%2Fheads%2Fctables10;hp=3ea91c8d7d9d14e016834553fad7d7baac1a0fe2;hpb=a8867fce18005ec258afcb2050063238421b084e;p=pspp diff --git a/doc/data-selection.texi b/doc/data-selection.texi index 3ea91c8d7d..c2f3ab31a4 100644 --- a/doc/data-selection.texi +++ b/doc/data-selection.texi @@ -1,13 +1,21 @@ -@node Data Selection, Conditionals and Looping, Data Manipulation, Top +@c PSPP - a program for statistical analysis. +@c Copyright (C) 2017, 2020 Free Software Foundation, Inc. +@c Permission is granted to copy, distribute and/or modify this document +@c under the terms of the GNU Free Documentation License, Version 1.3 +@c or any later version published by the Free Software Foundation; +@c with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. +@c A copy of the license is included in the section entitled "GNU +@c Free Documentation License". +@c +@node Data Selection @chapter Selecting data for analysis -This chapter documents PSPP commands that temporarily or permanently -select data records from the active file for analysis. +This chapter documents @pspp{} commands that temporarily or permanently +select data records from the active dataset for analysis. @menu * FILTER:: Exclude cases based on a variable. -* N OF CASES:: Limit the size of the active file. -* PROCESS IF:: Temporarily excluding cases. +* N OF CASES:: Limit the size of the active dataset. * SAMPLE:: Select a specified proportion of cases. * SELECT IF:: Permanently delete selected cases. * SPLIT FILE:: Do multiple analyses with one command. @@ -15,19 +23,19 @@ select data records from the active file for analysis. * WEIGHT:: Weight cases by a variable. @end menu -@node FILTER, N OF CASES, Data Selection, Data Selection +@node FILTER @section FILTER @vindex FILTER @display -FILTER BY var_name. +FILTER BY @var{var_name}. FILTER OFF. @end display @cmd{FILTER} allows a boolean-valued variable to be used to select cases from the data stream for processing. -To set up filtering, specify BY and a variable name. Keyword +To set up filtering, specify @subcmd{BY} and a variable name. Keyword BY is optional but recommended. Cases which have a zero or system- or user-missing value are excluded from analysis, but not deleted from the data stream. Cases with other values are analyzed. @@ -41,131 +49,77 @@ filter variable of the required form, then specify that variable on Filtering takes place immediately before cases pass to a procedure for analysis. Only one filter variable may be active at a time. Normally, case filtering continues until it is explicitly turned off with @code{FILTER -OFF}. However, if @cmd{FILTER} is placed after TEMPORARY, it filters only +OFF}. However, if @cmd{FILTER} is placed after @cmd{TEMPORARY}, it filters only the next procedure or procedure-like command. -@node N OF CASES, PROCESS IF, FILTER, Data Selection +@node N OF CASES @section N OF CASES @vindex N OF CASES @display -N [OF CASES] num_of_cases [ESTIMATED]. +N [OF CASES] @var{num_of_cases} [ESTIMATED]. @end display -Sometimes you may want to disregard cases of your input. @cmd{N} can -do this. @code{N 100} tells PSPP to disregard all cases after the -first 100. - -If the value specified for @cmd{N} is greater than the number of cases -read in, the value is ignored. - -@cmd{N} does not discard cases or prevent them from being read. It -just causes cases beyond the last one specified to be ignored by data -analysis commands. - -A later @cmd{N} command can increase or decrease the number of cases -selected. (To select all the cases without knowing how many there are, -specify a very high number: 100000 or whatever you think is large enough.) - -Transformation procedures performed after @cmd{N} is executed -@emph{do} cause cases to be discarded. - -@cmd{SAMPLE}, @cmd{PROCESS IF}, and @cmd{SELECT IF} have -precedence over @cmd{N}---the same results are obtained by both of the -following fragments, given the same random number seeds: - -@example -@i{@dots{}set up, read in data@dots{}} -N 100. -SAMPLE .5. -@i{@dots{}analyze data@dots{}} - -@i{@dots{}set up, read in data@dots{}} -SAMPLE .5. -N 100. -@i{@dots{}analyze data@dots{}} -@end example - -Both fragments above first randomly sample approximately half of the -cases, then select the first 100 of those sampled. - -@cmd{N} with the @code{ESTIMATED} keyword gives an -estimated number of cases before @cmd{DATA LIST} or another command to -read in data. @code{ESTIMATED} never limits the number of cases -processed by procedures. PSPP currently does not make use of -case count estimates. - -When @cmd{N} is specified after @cmd{TEMPORARY}, it affects only -the next procedure (@pxref{TEMPORARY}). - -@node PROCESS IF, SAMPLE, N OF CASES, Data Selection -@section PROCESS IF -@vindex PROCESS IF - -@example -PROCESS IF expression. -@end example - -@cmd{PROCESS IF} temporarily eliminates cases from the -data stream. Its effects are active only through the execution of the -next procedure or procedure-like command. - -Specify a boolean expression (@pxref{Expressions}). If the value of the -expression is true for a particular case, the case will be analyzed. If -the expression has a false or missing value, then the case will be -deleted from the data stream for this procedure only. - -Regardless of its placement relative to other commands, @cmd{PROCESS IF} -always takes effect immediately before data passes to the procedure. -Only one @cmd{PROCESS IF} command may be in effect at any given time. - -The effects of @cmd{PROCESS IF} are similar, but not identical, to the -effects of executing @cmd{TEMPORARY}, then @cmd{SELECT IF} -(@pxref{SELECT IF}). - -The filtering performed by @cmd{PROCESS IF} takes place immediately -before cases pass to a procedure for analysis. Because @cmd{PROCESS -IF} affects only a single procedure, its placement relative to -@cmd{TEMPORARY} is unimportant. - -@cmd{PROCESS IF} is deprecated. It is included for compatibility with -old command files. New syntax files should use @cmd{SELECT IF} or -@cmd{FILTER} instead. - -@node SAMPLE, SELECT IF, PROCESS IF, Data Selection +@cmd{N OF CASES} limits the number of cases processed by any +procedures that follow it in the command stream. @code{N OF CASES +100}, for example, tells @pspp{} to disregard all cases after the first +100. + +When @cmd{N OF CASES} is specified after @cmd{TEMPORARY}, it affects +only the next procedure (@pxref{TEMPORARY}). Otherwise, cases beyond +the limit specified are not processed by any later procedure. + +If the limit specified on @cmd{N OF CASES} is greater than the number +of cases in the active dataset, it has no effect. + +When @cmd{N OF CASES} is used along with @cmd{SAMPLE} or @cmd{SELECT +IF}, the case limit is applied to the cases obtained after sampling or +case selection, regardless of how @cmd{N OF CASES} is placed relative +to @cmd{SAMPLE} or @cmd{SELECT IF} in the command file. Thus, the +commands @code{N OF CASES 100} and @code{SAMPLE .5} both randomly +sample approximately half of the active dataset's cases, then select the +first 100 of those sampled, regardless of their order in the command +file. + +@cmd{N OF CASES} with the @code{ESTIMATED} keyword gives an estimated +number of cases before @cmd{DATA LIST} or another command to read in +data. @code{ESTIMATED} never limits the number of cases processed by +procedures. @pspp{} currently does not make use of case count estimates. + +@node SAMPLE @section SAMPLE @vindex SAMPLE @display -SAMPLE num1 [FROM num2]. +SAMPLE @var{num1} [FROM @var{num2}]. @end display @cmd{SAMPLE} randomly samples a proportion of the cases in the active file. Unless it follows @cmd{TEMPORARY}, it operates as a -transformation, permanently removing cases from the active file. +transformation, permanently removing cases from the active dataset. The proportion to sample can be expressed as a single number between 0 -and 1. If @code{k} is the number specified, and @code{N} is the number -of currently-selected cases in the active file, then after -@code{SAMPLE @var{k}.}, approximately @code{k*N} cases will be +and 1. If @var{k} is the number specified, and @var{N} is the number +of currently-selected cases in the active dataset, then after +@subcmd{SAMPLE @var{k}.}, approximately @var{k}*@var{N} cases are selected. -The proportion to sample can also be specified in the style @code{SAMPLE +The proportion to sample can also be specified in the style @subcmd{SAMPLE @var{m} FROM @var{N}}. With this style, cases are selected as follows: @enumerate @item If @var{N} is equal to the number of currently-selected cases in the -active file, exactly @var{m} cases will be selected. +active dataset, exactly @var{m} cases are selected. @item If @var{N} is greater than the number of currently-selected cases in the -active file, an equivalent proportion of cases will be selected. +active dataset, an equivalent proportion of cases are selected. @item If @var{N} is less than the number of currently-selected cases in the -active, exactly @var{m} cases will be selected @emph{from the first -@var{N} cases in the active file.} +active, exactly @var{m} cases are selected @emph{from the first +@var{N} cases in the active dataset.} @end enumerate @cmd{SAMPLE} and @cmd{SELECT IF} are performed in @@ -181,38 +135,65 @@ samples may still result when the file is processed on systems with differing endianness or floating-point formats. By default, the random number seed is based on the system time. -@node SELECT IF, SPLIT FILE, SAMPLE, Data Selection +@node SELECT IF @section SELECT IF @vindex SELECT IF @display -SELECT IF expression. +SELECT IF @var{expression}. @end display -@cmd{SELECT IF} selects cases for analysis based on the value of a -boolean expression. Cases not selected are permanently eliminated -from the active file, unless @cmd{TEMPORARY} is in effect +@cmd{SELECT IF} selects cases for analysis based on the value of +@var{expression}. Cases not selected are permanently eliminated +from the active dataset, unless @cmd{TEMPORARY} is in effect (@pxref{TEMPORARY}). Specify a boolean expression (@pxref{Expressions}). If the value of the -expression is true for a particular case, the case will be analyzed. If -the expression has a false or missing value, then the case will be +expression is true for a particular case, the case is analyzed. If +the expression has a false or missing value, then the case is deleted from the data stream. Place @cmd{SELECT IF} as early in the command file as possible. Cases that are deleted early can be processed more efficiently in time and space. +Once cases have been deleted from the active dataset using @cmd{SELECT IF} they +cannot be re-instated. +If you want to be able to re-instate cases, then use @cmd{FILTER} (@pxref{FILTER}) +instead. When @cmd{SELECT IF} is specified following @cmd{TEMPORARY} (@pxref{TEMPORARY}), the @cmd{LAG} function may not be used (@pxref{LAG}). -@node SPLIT FILE, TEMPORARY, SELECT IF, Data Selection +@subsection Example Select-If + +A shop steward is interested in the salaries of younger personnel in a firm. +The file @file{personnel.sav} provides the salaries of all the workers and their +dates of birth. The syntax in @ref{select-if:ex} shows how @cmd{SELECT IF} can +be used to limit analysis only to those persons born after December 31, 1999. + +@float Example, select-if:ex +@psppsyntax {select-if.sps} +@caption {Using @cmd{SELECT IF} to select persons born on or after a certain date.} +@end float + +From @ref{select-if:res} one can see that there are 56 persons listed in the dataset, +and 17 of them were born after December 31, 1999. + +@float Result, select-if:res +@psppoutput {select-if} +@caption {Salary descriptives before and after the @cmd{SELECT IF} transformation.} +@end float + +Note that the @file{personnel.sav} file from which the data were read is unaffected. +The transformation affects only the active file. + +@node SPLIT FILE @section SPLIT FILE @vindex SPLIT FILE @display -SPLIT FILE [{LAYERED, SEPARATE}] BY var_list. +SPLIT FILE [@{LAYERED, SEPARATE@}] BY @var{var_list}. SPLIT FILE OFF. @end display @@ -227,20 +208,65 @@ An independent analysis is carried out for each group of cases, and the variable values for the group are printed along with the analysis. When a list of variable names is specified, one of the keywords -LAYERED or SEPARATE may also be specified. If provided, either +@subcmd{LAYERED} or @subcmd{SEPARATE} may also be specified. If provided, either keyword are ignored. -Groups are formed only by @emph{adjacent} cases. To create a split +Groups are formed only by @emph{adjacent} cases. To create a split using a variable where like values are not adjacent in the working file, you should first sort the data by that variable (@pxref{SORT CASES}). -Specify OFF to disable @cmd{SPLIT FILE} and resume analysis of the -entire active file as a single group of data. +Specify @subcmd{OFF} to disable @cmd{SPLIT FILE} and resume analysis of the +entire active dataset as a single group of data. When @cmd{SPLIT FILE} is specified after @cmd{TEMPORARY}, it affects only the next procedure (@pxref{TEMPORARY}). -@node TEMPORARY, WEIGHT, SPLIT FILE, Data Selection +@subsection Example Split + +The file @file{horticulture.sav} contains data describing the @exvar{yield} +of a number of horticultural specimens which have been subjected to +various @exvar{treatment}s. If we wanted to investigate linear statistics +of the @exvar{yeild}, one way to do this is using the @cmd{DESCRIPTIVES} (@pxref{DESCRIPTIVES}). +However, it is reasonable to expect the mean to be different depending +on the @exvar{treatment}. So we might want to perform three separate +procedures --- one for each treatment. +@footnote{There are other, possibly better, ways to achieve a similar result +using the @cmd{MEANS} or @cmd{EXAMINE} commands.} +@ref{split:ex} shows how this can be done automatically using +the @cmd{SPLIT FILE} command. + +@float Example, split:ex +@psppsyntax {split.sps} +@caption {Running @cmd{DESCRIPTIVES} on each value of @exvar{treatment}} +@end float + +In @ref{split:res} you can see that the table of descriptive statistics +appears 3 times --- once for each value of @exvar{treatment}. +In this example @samp{N}, the number of observations are identical in +all splits. This is because that experiment was deliberately designed +that way. However in general one can expect a different @samp{N} for each +split. + +@float Example, split:res +@psppoutput {split} +@caption {The results of running @cmd{DESCRIPTIVES} with an active split} +@end float + +Unless @cmd{TEMPORARY} was used, after a split has been defined for +a dataset it remains active until explicitly disabled. +In the graphical user interface, the active split variable (if any) is +displayed in the status bar (@pxref{split-status-bar:scr}. +If a dataset is saved to a system file (@pxref{SAVE}) whilst a split +is active, the split stastus is stored in the file and will be +automatically loaded when that file is loaded. + +@float Screenshot, split-status-bar:scr +@psppimage {split-status-bar} +@caption {The status bar indicating that the data set is split using the @exvar{treatment} variable} +@end float + + +@node TEMPORARY @section TEMPORARY @vindex TEMPORARY @@ -249,9 +275,9 @@ TEMPORARY. @end display @cmd{TEMPORARY} is used to make the effects of transformations -following its execution temporary. These transformations will +following its execution temporary. These transformations affect only the execution of the next procedure or procedure-like -command. Their effects will not be saved to the active file. +command. Their effects are not be saved to the active dataset. The only specification on @cmd{TEMPORARY} is the command name. @@ -261,52 +287,54 @@ procedure-like commands. Scratch variables cannot be used following @cmd{TEMPORARY}. -An example may help to clarify: - -@example -DATA LIST /X 1-2. -BEGIN DATA. - 2 - 4 -10 -15 -20 -24 -END DATA. -COMPUTE X=X/2. -TEMPORARY. -COMPUTE X=X+3. -DESCRIPTIVES X. -DESCRIPTIVES X. -@end example +@subsection Example Temporary + +In @ref{temporary:ex} there are two @cmd{COMPUTE} transformation. One +of them immediatly follows a @cmd{TEMPORARY} command, and therefore has +effect only for the next procedure, which in this case is the first +@cmd{DESCRIPTIVES} command. -The data read by the first @cmd{DESCRIPTIVES} are 4, 5, 8, -10.5, 13, 15. The data read by the first @cmd{DESCRIPTIVES} are 1, 2, -5, 7.5, 10, 12. +@float Example, temporary:ex +@psppsyntax {temporary.sps} +@caption {Running a @cmd{COMPUTE} transformation after @cmd{TEMPORARY}} +@end float -@node WEIGHT, , TEMPORARY, Data Selection +The data read by the first @cmd{DESCRIPTIVES} procedure are 4, 5, 8, +10.5, 13, 15. The data read by the second @cmd{DESCRIPTIVES} procedure are 1, 2, +5, 7.5, 10, 12. This is because the second @cmd{COMPUTE} transformation +has no effect on the second @cmd{DESCRIPTIVES} procedure. You can check these +figures in @ref{temporary:res}. + +@float Result, temporary:res +@psppoutput {temporary} +@caption {The results of running two consecutive @cmd{DESCRIPTIVES} commands after + a temporary transformation} +@end float + + +@node WEIGHT @section WEIGHT @vindex WEIGHT @display -WEIGHT BY var_name. +WEIGHT BY @var{var_name}. WEIGHT OFF. @end display @cmd{WEIGHT} assigns cases varying weights, -changing the frequency distribution of the active file. Execution of +changing the frequency distribution of the active dataset. Execution of @cmd{WEIGHT} is delayed until data have been read. If a variable name is specified, @cmd{WEIGHT} causes the values of that variable to be used as weighting factors for subsequent statistical -procedures. Use of keyword BY is optional but recommended. Weighting +procedures. Use of keyword @subcmd{BY} is optional but recommended. Weighting variables must be numeric. Scratch variables may not be used for weighting (@pxref{Scratch Variables}). -When OFF is specified, subsequent statistical procedures will weight all +When @subcmd{OFF} is specified, subsequent statistical procedures weight all cases equally. -A positive integer weighting factor @var{w} on a case will yield the +A positive integer weighting factor @var{w} on a case yields the same statistical output as would replicating the case @var{w} times. A weighting factor of 0 is treated for statistical purposes as if the case did not exist in the input. Weighting values need not be @@ -317,6 +345,29 @@ values are not treated specially. When @cmd{WEIGHT} is specified after @cmd{TEMPORARY}, it affects only the next procedure (@pxref{TEMPORARY}). -@cmd{WEIGHT} does not cause cases in the active file to be replicated in -memory. -@setfilename ignored +@cmd{WEIGHT} does not cause cases in the active dataset to be +replicated in memory. + + +@subsection Example Weights + +One could define a dataset containing an inventory of stock items. +It would be reasonable to use a string variable for a description of the +item, and a numeric variable for the number in stock, like in @ref{weight:ex}. + +@float Example, weight:ex +@psppsyntax {weight.sps} +@caption {Setting the weight on the variable @exvar{quantity}} +@end float + +One analysis which most surely would be of interest is +the relative amounts or each item in stock. +However without setting a weight variable, @cmd{FREQUENCIES} +(@pxref{FREQUENCIES}) does not tell us what we want to know, since +there is only one case for each stock item. @ref{weight:res} shows the +difference between the weighted and unweighted frequency tables. + +@float Example, weight:res +@psppoutput {weight} +@caption {Weighted and unweighted frequency tables of @exvar{items}} +@end float