System Files and Portable Files
*******************************

The commands in this chapter read, write, and examine system files and portable files.

APPLY DICTIONARY
================

     APPLY DICTIONARY FROM={'file-name',file_handle}.

APPLY DICTIONARY applies the variable labels, value labels, and missing values taken from a file to corresponding variables in the active file.  In some cases it also updates the weighting variable.

Specify a system file, portable file, or scratch file with a file name string or as a file handle (see File Handles).  The dictionary in the file will be read, but it will not replace the active file dictionary. The file's data will not be read.

Only variables with names that exist in both the active file and the system file are considered.  Variables with the same name but different types (numeric, string) will cause an error message.  Otherwise, the system file variables' attributes will replace those in their matching active file variables, as described below.

If a system file variable has a variable label, then it will replace the active file variable's variable label.  If the system file variable does not have a variable label, then the active file variable's variable label, if any, will be retained.

If the active file variable is numeric or short string, then value labels and missing values, if any, will be copied to the active file variable.  If the system file variable does not have value labels or missing values, then those in the active file variable, if any, will not be disturbed.

Finally, weighting of the active file is updated (see WEIGHT).  If the active file has a weighting variable, and the system file does not, or if the weighting variable in the system file does not exist in the active file, then the active file weighting variable, if any, is retained.  Otherwise, the weighting variable in the system file becomes the active file weighting variable.

APPLY DICTIONARY takes effect immediately.  It does not read the active file.  The system file is not modified.

EXPORT
======

     EXPORT
             /OUTFILE='file-name'
             /UNSELECTED={RETAIN,DELETE}
             /DIGITS=n
             /DROP=var_list
             /KEEP=var_list
             /RENAME=(src_names=target_names)...
             /TYPE={COMM,TAPE}
             /MAP

The EXPORT procedure writes the active file dictionary and data to a specified portable file or scratch file.

By default, cases excluded with FILTER are written to the file.  These can be excluded by specifying DELETE on the UNSELECTED subcommand.  Specifying RETAIN makes the default explicit.

Portable files express real numbers in base 30.  Integers are always expressed to the maximum precision needed to make them exact. Non-integers are, by default, expressed to the machine's maximum natural precision (approximately 15 decimal digits on many machines). If many numbers require this many digits, the portable file may significantly increase in size.  As an alternative, the DIGITS subcommand may be used to specify the number of decimal digits of precision to write.  DIGITS applies only to non-integers.

The OUTFILE subcommand, which is the only required subcommand, specifies the portable file or scratch file to be written as a file name string or a file handle (see File Handles).

DROP, KEEP, and RENAME follow the same format as the SAVE procedure (see SAVE).

The TYPE subcommand specifies the character set for use in the portable file.  Its value is currently not used.

The MAP subcommand is currently ignored.

EXPORT is a procedure.  It causes the active file to be read.

GET
===

     GET
             /FILE={'file-name',file_handle}
             /DROP=var_list
             /KEEP=var_list
             /RENAME=(src_names=target_names)...

GET clears the current dictionary and active file and replaces them with the dictionary and data from a specified file.

The FILE subcommand is the only required subcommand.  Specify the system file, portable file, or scratch file to be read as a string file name or a file handle (see File Handles).

By default, all the variables in a file are read.  The DROP subcommand can be used to specify a list of variables that are not to be read.  By contrast, the KEEP subcommand can be used to specify variable that are to be read, with all other variables not read.

Normally variables in a file retain the names that they were saved under.  Use the RENAME subcommand to change these names.  Specify, within parentheses, a list of variable names followed by an equals sign (=) and the names that they should be renamed to.  Multiple parenthesized groups of variable names can be included on a single RENAME subcommand.  Variables' names may be swapped using a RENAME subcommand of the form /RENAME=(A B=B A).

Alternate syntax for the RENAME subcommand allows the parentheses to be eliminated.  When this is done, only a single variable may be renamed at once.  For instance, /RENAME=A=B.  This alternate syntax is deprecated.

DROP, KEEP, and RENAME are executed in left-to-right order. Each may be present any number of times.  GET never modifies a file on disk.  Only the active file read from the file is affected by these subcommands.

GET does not cause the data to be read, only the dictionary.  The data is read later, when a procedure is executed.

Use of GET to read a portable file or scratch file is a PSPP extension.

GET DATA
========

     GET DATA
             /TYPE={GNM,TXT}
             /FILE={'file-name',file_handle}
             ...additional subcommands depending on TYPE...

The GET DATA command is used to read files and other data sources created by other applications. When this command is executed, the current dictionary and active file are replaced  with variables and data read from the specified source.

The TYPE subcommand is mandatory and must be the first subcommand specified.  It determines the type of the file or source to read. PSPP currently supports the following file types:

GNM
     Spreadsheet files created by Gnumeric (http://gnumeric.org).

TXT
     Textual data files in columnar and delimited formats.

The FILE subcommand is mandatory for all implemented file types. Specify the file to be read as a string file name or (for textual data only) a file handle (see File Handles).

Each supported file type has additional subcommands, explained in separate sections below.

Gnumeric Spreadsheet Files
--------------------------

     GET DATA /TYPE=GNM
             /FILE={'file-name'}
             /SHEET={NAME 'sheet-name', INDEX n}
             /CELLRANGE={RANGE 'range', FULL}
             /READNAMES={ON, OFF}
             /ASSUMEDVARWIDTH=n.

To use GET DATA to read a spreadsheet file created by Gnumeric (http://gnumeric.org), specify TYPE=GNM to indicate the file's format and use FILE to indicate the Gnumeric file to be read.  All other subcommands are optional.

The format of each variable is determined by the format of the spreadsheet cell containing the first datum for the variable. If this cell is of string (text) format, then the width of the variable is determined from the length of the string it contains, unless the ASSUMEDVARWIDTH subcommand is given.

The SHEET subcommand specifies the sheet within the spreadsheet file to read. There are two forms of the SHEET subcommand. In the first form, /SHEET=name SHEET-NAME, the string SHEET-NAME is the name of the sheet to read. In the second form, /SHEET=index IDX, IDX is a integer which is the index of the sheet to read. The first sheet has the index 1. If the SHEET subcommand is omitted, then the command will read the first sheet in the file.

The CELLRANGE subcommand specifies the range of cells within the sheet to read. If the subcommand is given as /CELLRANGE=FULL, then the entire sheet  is read. To read only part of a sheet, use the form /CELLRANGE=range 'TOP-LEFT-CELL:BOTTOM-RIGHT-CELL'. For example, the subcommand /CELLRANGE=range 'C3:P19' reads columns C--P, and rows 3--19 inclusive. If no CELLRANGE subcommand is given, then the entire sheet is read.

If /READNAMES=ON is specified, then the contents of cells of the first row are used as the names of the variables in which to store the data from subsequent rows. If the READNAMES command is omitted, or if /READNAMES=OFF is used, then the variables  receive automatically assigned names.

The ASSUMEDVARWIDTH subcommand specifies the maximum width of string variables read  from the file. If omitted, the default value is determined from the length of the string in the first spreadsheet cell for each variable.

Textual Data Files
------------------

     GET DATA /TYPE=TXT
             /FILE={'file-name',file_handle}
             [/ARRANGEMENT={DELIMITED,FIXED}]
             [/FIRSTCASE={first_case}]
             [/IMPORTCASE={ALL,FIRST max_cases,PERCENT percent}]
             ...additional subcommands depending on ARRANGEMENT...

When TYPE=TXT is specified, GET DATA reads data in a delimited or fixed columnar format, much like DATA LIST (see DATA LIST).  The FILE subcommand must be specified indicate the name or the file handle of the file to be read.

The ARRANGEMENT subcommand determines the file's basic format. DELIMITED, the default setting, specifies that fields in the input data are separated by spaces, tabs, or other user-specified delimiters.  FIXED specifies that fields in the input data appear at particular fixed column positions within records of a case.

By default, cases are read from the input file starting from the first line.  To skip lines at the beginning of an input file, set FIRSTCASE to the number of the first line to read: 2 to skip the first line, 3 to skip the first two lines, and so on.

IMPORTCASE can be used to limit the number of cases read from the input file.  With the default setting, ALL, all cases in the file are read.  Specify FIRST max_cases to read at most max_cases cases from the file.  Use PERCENT percent to read only percent percent, approximately, of the cases contained in the file.  (The percentage is approximate, because there is no way to accurately count the number of cases in the file without reading the entire file.  The number of cases in some kinds of unusual files cannot be estimated; PSPP will read all cases in such files.)

FIRSTCASE and IMPORTCASE may be used with delimited and fixed-format data.  The remaining subcommands, which apply only to one of the two  file arrangements, are described below.

Reading Delimited Data
......................

     GET DATA /TYPE=TXT
             /FILE={'file-name',file_handle}
             [/ARRANGEMENT={DELIMITED,FIXED}]
             [/FIRSTCASE={first_case}]
             [/IMPORTCASE={ALL,FIRST max_cases,PERCENT percent}]

             /DELIMITERS="delimiters"
             [/QUALIFIER="quote"]
             [/DELCASE={LINE,VARIABLES n_variables}]
             /VARIABLES=del_var [del_var]...
     where each del_var takes the form:
             variable format

The GET DATA command with TYPE=TXT and ARRANGEMENT=DELIMITED reads input data from text files in delimited format, where fields are separated by a set of user-specified delimiters.  Its capabilities are similar to those of DATA LIST FREE (see DATA LIST FREE), with a few enhancements.

The required FILE subcommand and optional FIRSTCASE and IMPORTCASE subcommands are described above (see GET DATA /TYPE=TXT).

DELIMITERS, which is required, specifies the set of characters that may separate fields.  Each character in the string specified on DELIMITERS separates one field from the next.  The end of a line also separates fields, regardless of DELIMITERS.  Two consecutive delimiters in the input yield an empty field, as does a delimiter at the end of a line.  A space character as a delimiter is an exception: consecutive spaces do not yield an empty field and neither does any number of spaces at the end of a line.

To use a tab as a delimiter, specify \t at the beginning of the DELIMITERS string.  To use a backslash as a delimiter, specify \\ as the first delimiter or, if a tab should also be a delimiter, immediately following \t.  To read a data file in which each field appears on a separate line, specify the empty string for DELIMITERS.

The optional QUALIFIER subcommand names a character that can be used to quote values within fields in the input.  A field that begins with the specified quote character ends at the next match quote. Intervening delimiters become part of the field, instead of terminating it.

The DELCASE subcommand controls how data may be broken across lines in the data file.  With LINE, the default setting, each line must contain all the data for exactly one case.  For additional flexibility, to allow a single case to be split among lines or multiple cases to be contained on a single line, specify VARIABLES n_variables, where n_variables is the number of variables per case.

The VARIABLES subcommand is required and must be the last subcommand. Specify the name of each variable and its input format (see Input and Output Formats) in the order they should be read from the input file.

Examples
........

On a Unix-like system, the /etc/passwd file has a format similar to this:

     root:$1$nyeSP5gD$pDq/:0:0:,,,:/root:/bin/bash
     blp:$1$BrP/pFg4$g7OG:1000:1000:Ben Pfaff,,,:/home/blp:/bin/bash
     john:$1$JBuq/Fioq$g4A:1001:1001:John Darrington,,,:/home/john:/bin/bash
     jhs:$1$D3li4hPL$88X1:1002:1002:Jason Stover,,,:/home/jhs:/bin/csh

The following syntax reads a file in the format used by /etc/passwd:

     GET DATA /TYPE=TXT /FILE='/etc/passwd' /DELIMITERS=':'
             /VARIABLES=username A20
                        password A40
                        uid F10
                        gid F10
                        gecos A40
                        home A40
                        shell A40.

Consider the following data on used cars:

     model   year    mileage price   type    age
     Civic   2002    29883   15900   Si      2
     Civic   2003    13415   15900   EX      1
     Civic   1992    107000  3800    n/a     12
     Accord  2002    26613   17900   EX      1

The following syntax can be used to read the used car data:

     GET DATA /TYPE=TXT /FILE='cars.data' /DELIMITERS=' ' /FIRSTCASE=2
             /VARIABLES=model A8
                        year F4
                        mileage F6
                        price F5
                        type A4
                        age F2.

Consider the following information on animals in a pet store:

     "Pet Name", "Age", "Color", "Date Received", "Price", "Needs Walking", "Type"
     , (Years), , , (Dollars), ,
     "Rover", 4.5, Brown, "12 Feb 2004", 80, True, "Dog"
     "Charlie", , Gold, "5 Apr 2007", 12.3, False, "Fish"
     "Molly", 2, Black, "12 Dec 2006", 25, False, "Cat"
     "Gilly", , White, "10 Apr 2007", 10, False, "Guinea Pig"

The following syntax can be used to read the pet store data:

     GET DATA /TYPE=TXT /FILE='pets.data' /DELIMITERS=', ' /QUALIFIER='"'
             /FIRSTCASE=3
             /VARIABLES=name A10
                        age F3.1
                        color A5
                        received EDATE10
                        price F5.2
                        needs_walking A5
                        type A10.

Reading Fixed Columnar Data
...........................

     GET DATA /TYPE=TXT
             /FILE={'file-name',file_handle}
             [/ARRANGEMENT={DELIMITED,FIXED}]
             [/FIRSTCASE={first_case}]
             [/IMPORTCASE={ALL,FIRST max_cases,PERCENT percent}]

             [/FIXCASE=n]
             /VARIABLES fixed_var [fixed_var]...
                 [/rec# fixed_var [fixed_var]...]...
     where each fixed_var takes the form:
             variable start-end format

The GET DATA command with TYPE=TXT and ARRANGEMENT=FIXED reads input data from text files in fixed format, where each field is located in particular fixed column positions within records of a case.  Its capabilities are similar to those of DATA LIST FIXED (see DATA LIST FIXED), with a few enhancements.

The required FILE subcommand and optional FIRSTCASE and IMPORTCASE subcommands are described above (see GET DATA /TYPE=TXT).

The optional FIXCASE subcommand may be used to specify the positive integer number of input lines that make up each case.  The default value is 1.

The VARIABLES subcommand, which is required, specifies the positions at which each variable can be found.  For each variable, specify its name, followed by its start and end column separated by - (e.g. 0-9), followed by the input format type (e.g. F).  For this command, columns are numbered starting from 0 at the left column.  Introduce the variables in the second and later lines of a case by a slash followed by the number of the line within the case, e.g. /2 for the second line.

Examples
........

Consider the following data on used cars:

     model   year    mileage price   type    age
     Civic   2002    29883   15900   Si      2
     Civic   2003    13415   15900   EX      1
     Civic   1992    107000  3800    n/a     12
     Accord  2002    26613   17900   EX      1

The following syntax can be used to read the used car data:

     GET DATA /TYPE=TXT /FILE='cars.data' /ARRANGEMENT=FIXED /FIRSTCASE=2
             /VARIABLES=model 0-7 A
                        year 8-15 F
                        mileage 16-23 F
                        price 24-31 F
                        type 32-40 A
                        age 40-47 F.

IMPORT
======

     IMPORT
             /FILE='file-name'
             /TYPE={COMM,TAPE}
             /DROP=var_list
             /KEEP=var_list
             /RENAME=(src_names=target_names)...

The IMPORT transformation clears the active file dictionary and data and replaces them with a dictionary and data from a system, portable file, or scratch file.

The FILE subcommand, which is the only required subcommand, specifies the portable file to be read as a file name string or a file handle (see File Handles).

The TYPE subcommand is currently not used.

DROP, KEEP, and RENAME follow the syntax used by GET (see GET).

IMPORT does not cause the data to be read, only the dictionary.  The data is read later, when a procedure is executed.

Use of IMPORT to read a system file or scratch file is a PSPP extension.

MATCH FILES
===========

     MATCH FILES
             /{FILE,TABLE}={*,'file-name'}
             /RENAME=(src_names=target_names)...
             /IN=var_name

             /BY=var_list
             /DROP=var_list
             /KEEP=var_list
             /FIRST=var_name
             /LAST=var_name
             /MAP

MATCH FILES merges one or more system, portable, or scratch files, optionally including the active file.  Cases with the same values for BY variables are combined into a single case.  Cases with different values are output in order.  Thus, multiple sorted files are combined into a single sorted file based on the value of the BY variables.  The results of the merge become the new active file.

Specify FILE with a system, portable, or scratch file as a file name string or file handle (see File Handles), or with an asterisk (*) to indicate the current active file.  The files specified on FILE are merged together based on the BY variables, or combined case-by-case if BY is not specified.

Specify TABLE with a file to use it as a table lookup file.  Cases in table lookup files are not used up after they've been used once.  This means that data in table lookup files can correspond to any number of cases in FILE files.  Table lookup files correspond to lookup tables in traditional relational database systems. If a table lookup file contains more than one case with a given set of BY variables, only the first case is used.

Any number of FILE and TABLE subcommands may be specified. Ordinarily, at least two FILE subcommands, or one FILE and at least one TABLE, should be specified.  Each instance of FILE or TABLE can be followed by any sequence of RENAME subcommands.  These have the same form and meaning as the corresponding subcommands of GET (see GET), but apply only to variables in the given file.

Each FILE or TABLE may optionally be followed by an IN subcommand, which creates a numeric variable with the specified name and format F1.0.  The IN variable takes value 1 in a case if the given file contributed a row to the merged file, 0 otherwise.  The DROP, KEEP, and RENAME subcommands do not affect IN variables.

When more than one FILE or TABLE contains a variable with a given name, those variables must all have the same type (numeric or string) and, for string variables, the same width.  This rules applies to variable names after renaming with RENAME; thus, RENAME can be used to resolve conflicts.

FILE and TABLE must be specified at the beginning of the command, with any RENAME or IN specifications immediately after the corresponding FILE or TABLE.  These subcommands are followed by BY, DROP, KEEP, FIRST, LAST, and MAP.

The BY subcommand specifies a list of variables that are used to match cases from each of the files.  When TABLE or IN is used, BY is required; otherwise, it is optional.  When BY is specified, all the files named on FILE and TABLE subcommands must be sorted in ascending order of the BY variables.  Variables belonging to files that are not present for the current case are set to the system-missing value for numeric variables or spaces for string variables.

The DROP and KEEP subcommands allow variables to be dropped from or reordered within the new active file.  These subcommands have the same form and meaning as the corresponding subcommands of GET (see GET).  They apply to the new active file as a whole, not to individual input files.  The variable names specified on DROP and KEEP are those after any renaming with RENAME.

The optional FIRST and LAST subcommands name variables that MATCH FILES adds to the active file.  The new variables are numeric with print and write format F1.0.  The value of the FIRST variable is 1 in the first case with a given set of values for the BY variables, and 0 in other cases.  Similarly, the LAST variable is 1 in the last case with a given of BY values, and 0 in other cases.

MATCH FILES may not be specified following TEMPORARY (see TEMPORARY) if the active file is used as an input source.

Use of portable or scratch files on MATCH FILES is a PSPP extension.

SAVE
====

     SAVE
             /OUTFILE={'file-name',file_handle}
             /UNSELECTED={RETAIN,DELETE}
             /{COMPRESSED,UNCOMPRESSED}
             /PERMISSIONS={WRITEABLE,READONLY}
             /DROP=var_list
             /KEEP=var_list
             /VERSION=version
             /RENAME=(src_names=target_names)...
             /NAMES
             /MAP

The SAVE procedure causes the dictionary and data in the active file to be written to a system file or scratch file.

OUTFILE is the only required subcommand.  Specify the system file or scratch file to be written as a string file name or a file handle (see File Handles).

By default, cases excluded with FILTER are written to the system file. These can be excluded by specifying DELETE on the UNSELECTED subcommand.  Specifying RETAIN makes the default explicit.

The COMPRESS and UNCOMPRESS subcommand determine whether the saved system file is compressed.  By default, system files are compressed. This default can be changed with the SET command (see SET).

The PERMISSIONS subcommand specifies permissions for the new system file.  WRITEABLE, the default, creates the file with read and write permission.  READONLY creates the file for read-only access.

By default, all the variables in the active file dictionary are written to the system file.  The DROP subcommand can be used to specify a list of variables not to be written.  In contrast, KEEP specifies variables to be written, with all variables not specified not written.

Normally variables are saved to a system file under the same names they have in the active file.  Use the RENAME subcommand to change these names. Specify, within parentheses, a list of variable names followed by an equals sign (=) and the names that they should be renamed to. Multiple parenthesized groups of variable names can be included on a single RENAME subcommand.  Variables' names may be swapped using a RENAME subcommand of the form /RENAME=(A B=B A).

Alternate syntax for the RENAME subcommand allows the parentheses to be eliminated.  When this is done, only a single variable may be renamed at once.  For instance, /RENAME=A=B.  This alternate syntax is deprecated.

DROP, KEEP, and RENAME are performed in left-to-right order.  They each may be present any number of times.  SAVE never modifies the active file.  DROP, KEEP, and RENAME only affect the system file written to disk.

The VERSION subcommand specifies the version of the file format. Valid versions are 2 and 3.  The default version is 3.  In version 2 system files, variable names longer than 8 bytes will be truncated.  The two versions are otherwise identical.

The NAMES and MAP subcommands are currently ignored.

SAVE causes the data to be read.  It is a procedure.

SYSFILE INFO
============

     SYSFILE INFO FILE='file-name'.

SYSFILE INFO reads the dictionary in a system file and displays the information in its dictionary.

Specify a file name or file handle.  SYSFILE INFO reads that file as a system file and displays information on its dictionary.

SYSFILE INFO does not affect the current active file.

XEXPORT
=======

     XEXPORT
             /OUTFILE='file-name'
             /DIGITS=n
             /DROP=var_list
             /KEEP=var_list
             /RENAME=(src_names=target_names)...
             /TYPE={COMM,TAPE}
             /MAP

The EXPORT transformation writes the active file dictionary and data to a specified portable file.

This transformation is a PSPP extension.

It is similar to the EXPORT procedure, with two differences:

  * XEXPORT is a transformation, not a procedure.  It is executed when the data is read by a procedure or procedure-like command.

  * XEXPORT does not support the UNSELECTED subcommand.

See EXPORT, for more information.

XSAVE
=====

     XSAVE
             /OUTFILE='file-name'
             /{COMPRESSED,UNCOMPRESSED}
             /PERMISSIONS={WRITEABLE,READONLY}
             /DROP=var_list
             /KEEP=var_list
             /VERSION=version
             /RENAME=(src_names=target_names)...
             /NAMES
             /MAP

The XSAVE transformation writes the active file dictionary and data to a system file or scratch file.  It is similar to the SAVE procedure, with two differences:

  * XSAVE is a transformation, not a procedure.  It is executed when the data is read by a procedure or procedure-like command.

  * XSAVE does not support the UNSELECTED subcommand.

See SAVE, for more information.