+@c PSPP - a program for statistical analysis.
+@c Copyright (C) 2017, 2020 Free Software Foundation, Inc.
+@c Permission is granted to copy, distribute and/or modify this document
+@c under the terms of the GNU Free Documentation License, Version 1.3
+@c or any later version published by the Free Software Foundation;
+@c with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts.
+@c A copy of the license is included in the section entitled "GNU
+@c Free Documentation License".
+@c
@node System and Portable File IO
@chapter System and Portable File I/O
* GET DATA:: Read from foreign files.
* IMPORT:: Read from a portable file.
* SAVE:: Write to a system file.
+* SAVE DATA COLLECTION:: Write to a system file and metadata file.
* SAVE TRANSLATE:: Write data in foreign file formats.
* SYSFILE INFO:: Display system file dictionary.
* XEXPORT:: Write to a portable file, as a transformation.
variables in the active dataset. In some cases it also updates the
weighting variable.
-Specify a system file or portable file's name, a data set name
-(@pxref{Datasets}), or a file handle name (@pxref{File Handles}). The
-dictionary in the file will be read, but it will not replace the
-active dataset's dictionary. The file's data will not be read.
+The @subcmd{FROM} clause is mandatory. Use it to specify a system
+file or portable file's name in single quotes, a data set name
+(@pxref{Datasets}), or a file handle name (@pxref{File Handles}).
+The dictionary in the file is be read, but it does not replace the active
+dataset's dictionary. The file's data is not read.
Only variables with names that exist in both the active dataset and the
system file are considered. Variables with the same name but different
-types (numeric, string) will cause an error message. Otherwise, the
-system file variables' attributes will replace those in their matching
+types (numeric, string) cause an error message. Otherwise, the
+system file variables' attributes replace those in their matching
active dataset variables:
@itemize @bullet
@item
-If a system file variable has a variable label, then it will replace
+If a system file variable has a variable label, then it replaces
the variable label of the active dataset variable. If the system
file variable does not have a variable label, then the active dataset
-variable's variable label, if any, will be retained.
+variable's variable label, if any, is retained.
@item
If the system file variable has custom attributes (@pxref{VARIABLE
ATTRIBUTE}), then those attributes replace the active dataset variable's
custom attributes. If the system file variable does not have custom
attributes, then the active dataset variable's custom attributes, if any,
-will be retained.
+is retained.
@item
If the active dataset variable is numeric or short string, then value
-labels and missing values, if any, will be copied to the active dataset
+labels and missing values, if any, are copied to the active dataset
variable. If the system file variable does not have value labels or
-missing values, then those in the active dataset variable, if any, will not
-be disturbed.
+missing values, then those in the active dataset variable, if any, are not
+disturbed.
@end itemize
In addition to properties of variables, some properties of the active
the portable file to be written as a file name string or
a file handle (@pxref{File Handles}).
-@subcmd{DROP}, @subcmd{KEEP}, and @subcmd{RENAME} follow the same format as the
+@subcmd{DROP}, @subcmd{KEEP}, and @subcmd{RENAME} follow the same format as the
@subcmd{SAVE} procedure (@pxref{SAVE}).
The @subcmd{TYPE} subcommand specifies the character set for use in the
@cmd{GET} clears the current dictionary and active dataset and
replaces them with the dictionary and data from a specified file.
-The @subcmd{FILE} subcommand is the only required subcommand.
-Specify the system
-file or portable file to be read as a string file name or
-a file handle (@pxref{File Handles}).
+The @subcmd{FILE} subcommand is the only required subcommand. Specify
+the SPSS system file, SPSS/PC+ system file, or SPSS portable file to
+be read as a string file name or a file handle (@pxref{File Handles}).
By default, all the variables in a file are read. The DROP
subcommand can be used to specify a list of variables that are not to be
once. For instance, @subcmd{/RENAME=@var{A}=@var{B}}. This alternate syntax is
deprecated.
-@subcmd{DROP}, @subcmd{KEEP}, and @subcmd{RENAME} are executed in left-to-right order.
+@subcmd{DROP}, @subcmd{KEEP}, and @subcmd{RENAME} are executed in left-to-right order.
Each may be present any number of times. @cmd{GET} never modifies a
file on disk. Only the active dataset read from the file
is affected by these subcommands.
-@pspp{} tries to automatically detect the encoding of string data in the
-file. Sometimes, however, this does not work well encoding,
-especially for files written by old versions of SPSS or @pspp{}. Specify
-the @subcmd{ENCODING} subcommand with an @acronym{IANA} character set name as its string
-argument to override the default. The @subcmd{ENCODING} subcommand is a @pspp{}
-extension.
+@pspp{} automatically detects the encoding of string data in the file,
+when possible. The character encoding of old SPSS system files cannot
+always be guessed correctly, and SPSS/PC+ system files do not include
+any indication of their encoding. Specify the @subcmd{ENCODING}
+subcommand with an @acronym{IANA} character set name as its string
+argument to override the default. Use @cmd{SYSFILE INFO} to analyze
+the encodings that might be valid for a system file. The
+@subcmd{ENCODING} subcommand is a @pspp{} extension.
@cmd{GET} does not cause the data to be read, only the dictionary. The data
is read later, when a procedure is executed.
Spreadsheet files created by Gnumeric (@url{http://gnumeric.org}).
@item ODS
-Spreadsheet files in OpenDocument format.
+Spreadsheet files in OpenDocument format (@url{http://opendocumentformat.org}).
@item PSQL
Relations from PostgreSQL databases (@url{http://postgresql.org}).
/SHEET=@{NAME '@var{sheet_name}', INDEX @var{n}@}
/CELLRANGE=@{RANGE '@var{range}', FULL@}
/READNAMES=@{ON, OFF@}
- /ASSUMEDVARWIDTH=@var{n}.
+ /ASSUMEDSTRWIDTH=@var{n}.
@end display
@cindex Gnumeric
in OpenDocument format
(@url{http://libreplanet.org/wiki/Group:OpenDocument/Software})
can be read using the @cmd{GET DATA} command.
-Use the @subcmd{TYPE} subcommand to indicate the file's format.
+Use the @subcmd{TYPE} subcommand to indicate the file's format.
/TYPE=GNM indicates Gnumeric files,
/TYPE=ODS indicates OpenDocument.
The @subcmd{FILE} subcommand is mandatory.
-Use it to specify the name file to be read.
+Use it to specify the name file to be read.
All other subcommands are optional.
-The format of each variable is determined by the format of the spreadsheet
+The format of each variable is determined by the format of the spreadsheet
cell containing the first datum for the variable.
If this cell is of string (text) format, then the width of the variable is
-determined from the length of the string it contains, unless the
-@subcmd{ASSUMEDVARWIDTH} subcommand is given.
+determined from the length of the string it contains, unless the
+@subcmd{ASSUMEDSTRWIDTH} subcommand is given.
The @subcmd{SHEET} subcommand specifies the sheet within the spreadsheet file to read.
There are two forms of the @subcmd{SHEET} subcommand.
In the second form, @subcmd{/SHEET=index @var{idx}}, @var{idx} is a
integer which is the index of the sheet to read.
The first sheet has the index 1.
-If the @subcmd{SHEET} subcommand is omitted, then the command will read the
+If the @subcmd{SHEET} subcommand is omitted, then the command reads the
first sheet in the file.
The @subcmd{CELLRANGE} subcommand specifies the range of cells within the sheet to read.
If the subcommand is given as @subcmd{/CELLRANGE=FULL}, then the entire
sheet is read.
-To read only part of a sheet, use the form
+To read only part of a sheet, use the form
@subcmd{/CELLRANGE=range '@var{top_left_cell}:@var{bottom_right_cell}'}.
-For example, the subcommand @subcmd{/CELLRANGE=range 'C3:P19'} reads
+For example, the subcommand @subcmd{/CELLRANGE=range 'C3:P19'} reads
columns C--P, and rows 3--19 inclusive.
If no @subcmd{CELLRANGE} subcommand is given, then the entire sheet is read.
If @subcmd{/READNAMES=OFF} is
used, then the variables receive automatically assigned names.
-The @subcmd{ASSUMEDVARWIDTH} subcommand specifies the maximum width of string
+The @subcmd{ASSUMEDSTRWIDTH} subcommand specifies the maximum width of string
variables read from the file.
-If omitted, the default value is determined from the length of the
+If omitted, the default value is determined from the length of the
string in the first spreadsheet cell for each variable.
GET DATA /TYPE=PSQL
/CONNECT=@{@var{connection info}@}
/SQL=@{@var{query}@}
- [/ASSUMEDVARWIDTH=@var{w}]
+ [/ASSUMEDSTRWIDTH=@var{w}]
[/UNENCRYPTED]
[/BSIZE=@var{n}].
@end display
The PSQL type is used to import data from a postgres database server.
The server may be located locally or remotely.
-Variables are automatically created based on the table column names
+Variables are automatically created based on the table column names
or the names specified in the SQL query.
-Postgres data types of high precision, will loose precision when
+Postgres data types of high precision, loose precision when
imported into @pspp{}.
Not all the postgres data types are able to be represented in @pspp{}.
-If a datum cannot be represented a warning will be issued and that
-datum will be set to SYSMIS.
+If a datum cannot be represented then @cmd{GET DATA} issues a warning
+and that datum is set to SYSMIS.
The @subcmd{CONNECT} subcommand is mandatory.
It is a string specifying the parameters of the database server from
The @subcmd{SQL} subcommand is mandatory.
It must be a valid SQL string to retrieve data from the database.
-The @subcmd{ASSUMEDVARWIDTH} subcommand specifies the maximum width of string
+The @subcmd{ASSUMEDSTRWIDTH} subcommand specifies the maximum width of string
variables read from the database.
-If omitted, the default value is determined from the length of the
+If omitted, the default value is determined from the length of the
string in the first value read for each variable.
The @subcmd{UNENCRYPTED} subcommand allows data to be retrieved over an insecure
connection.
-If the connection is not encrypted, and the @subcmd{UNENCRYPTED} subcommand is
-not given, then an error will occur.
+If the connection is not encrypted, and the @subcmd{UNENCRYPTED} subcommand is
+not given, then an error occurs.
Whether or not the connection is
-encrypted depends upon the underlying psql library and the
+encrypted depends upon the underlying psql library and the
capabilities of the database server.
The @subcmd{BSIZE} subcommand serves only to optimise the speed of data transfer.
The default value is 4096.
If your SQL statement fetches a large number of cases but only a small number of
variables, then the data transfer may be faster if you increase this value.
-Conversely, if the number of variables is large, or if the machine on which
+Conversely, if the number of variables is large, or if the machine on which
@pspp{} is running has only a
-small amount of memory, then a smaller value will be better.
+small amount of memory, then a smaller value is probably better.
The following syntax is an example:
@display
GET DATA /TYPE=TXT
/FILE=@{'@var{file_name}',@var{file_handle}@}
+ [ENCODING='@var{encoding}']
[/ARRANGEMENT=@{DELIMITED,FIXED@}]
[/FIRSTCASE=@{@var{first_case}@}]
- [/IMPORTCASE=@{ALL,FIRST @var{max_cases},PERCENT @var{percent}@}]
+ [/IMPORTCASES=...]
@dots{}additional subcommands depending on ARRANGEMENT@dots{}
@end display
When TYPE=TXT is specified, GET DATA reads data in a delimited or
fixed columnar format, much like DATA LIST (@pxref{DATA LIST}).
-The @subcmd{FILE} subcommand is mandatory. Specify the file to be read as
+The @subcmd{FILE} subcommand is mandatory. Specify the file to be read as
a string file name or (for textual data only) a
file handle (@pxref{File Handles}).
+The @subcmd{ENCODING} subcommand specifies the character encoding of
+the file to be read. @xref{INSERT}, for information on supported
+encodings.
+
The @subcmd{ARRANGEMENT} subcommand determines the file's basic format.
DELIMITED, the default setting, specifies that fields in the input
data are separated by spaces, tabs, or other user-specified
to the number of the first line to read: 2 to skip the first line, 3
to skip the first two lines, and so on.
-@subcmd{IMPORTCASE} can be used to limit the number of cases read from the
-input file. With the default setting, ALL, all cases in the file are
-read. Specify FIRST @var{max_cases} to read at most @var{max_cases} cases
-from the file. Use @subcmd{PERCENT @var{percent}} to read only @var{percent}
-percent, approximately, of the cases contained in the file. (The
-percentage is approximate, because there is no way to accurately count
-the number of cases in the file without reading the entire file. The
-number of cases in some kinds of unusual files cannot be estimated;
-@pspp{} will read all cases in such files.)
+@subcmd{IMPORTCASES} is ignored, for compatibility. Use @cmd{N OF
+CASES} to limit the number of cases read from a file (@pxref{N OF
+CASES}), or @cmd{SAMPLE} to obtain a random sample of cases
+(@pxref{SAMPLE}).
-@subcmd{FIRSTCASE} and @subcmd{IMPORTCASE} may be used with delimited and fixed-format
-data. The remaining subcommands, which apply only to one of the two file
-arrangements, are described below.
+The remaining subcommands apply only to one of the two file
+arrangements, described below.
@menu
* GET DATA /TYPE=TXT /ARRANGEMENT=DELIMITED::
[/IMPORTCASE=@{ALL,FIRST @var{max_cases},PERCENT @var{percent}@}]
/DELIMITERS="@var{delimiters}"
- [/QUALIFIER="@var{quotes}" [/ESCAPE]]
+ [/QUALIFIER="@var{quotes}"
[/DELCASE=@{LINE,VARIABLES @var{n_variables}@}]
/VARIABLES=@var{del_var1} [@var{del_var2}]@dots{}
where each @var{del_var} takes the form:
instead of terminating it. The ability to specify more than one quote
character is a @pspp{} extension.
-By default, a character specified on @subcmd{QUALIFIER} cannot itself be
-embedded within a field that it quotes, because the quote character
-always terminates the quoted field. With ESCAPE, however, a doubled
-quote character within a quoted field inserts a single instance of the
-quote into the field. For example, if @samp{'} is specified on
-@subcmd{QUALIFIER}, then without ESCAPE @code{'a''b'} specifies a pair of
-fields that contain @samp{a} and @samp{b}, but with ESCAPE it
-specifies a single field that contains @samp{a'b}. ESCAPE is a @pspp{}
-extension.
+The character specified on @subcmd{QUALIFIER} can be embedded within a
+field that it quotes by doubling the qualifier. For example, if
+@samp{'} is specified on @subcmd{QUALIFIER}, then @code{'a''b'}
+specifies a field that contains @samp{a'b}.
The @subcmd{DELCASE} subcommand controls how data may be broken across lines in
the data file. With LINE, the default setting, each line must contain
The @subcmd{VARIABLES} subcommand, which is required, specifies the positions
at which each variable can be found. For each variable, specify its
name, followed by its start and end column separated by @samp{-}
-(e.g.@: @samp{0-9}), followed by an input format type (e.g.@:
-@samp{F}) or a full format specification (e.g.@: @samp{DOLLAR12.2}).
+(@i{e.g.}@: @samp{0-9}), followed by an input format type (@i{e.g.}@:
+@samp{F}) or a full format specification (@i{e.g.}@: @samp{DOLLAR12.2}).
For this command, columns are numbered starting from 0 at
the left column. Introduce the variables in the second and later
lines of a case by a slash followed by the number of the line within
-the case, e.g.@: @samp{/2} for the second line.
+the case, @i{e.g.}@: @samp{/2} for the second line.
@subsubheading Examples
SAVE
/OUTFILE=@{'@var{file_name}',@var{file_handle}@}
/UNSELECTED=@{RETAIN,DELETE@}
- /@{COMPRESSED,UNCOMPRESSED@}
+ /@{UNCOMPRESSED,COMPRESSED,ZCOMPRESSED@}
/PERMISSIONS=@{WRITEABLE,READONLY@}
/DROP=@var{var_list}
/KEEP=@var{var_list}
These can be excluded by specifying @subcmd{DELETE} on the @subcmd{UNSELECTED}
subcommand. Specifying @subcmd{RETAIN} makes the default explicit.
-The @subcmd{COMPRESS} and @subcmd{UNCOMPRESS} subcommand determine whether
-the saved system file is compressed. By default, system files are compressed.
-This default can be changed with the SET command (@pxref{SET}).
+The @subcmd{UNCOMPRESSED}, @subcmd{COMPRESSED}, and
+@subcmd{ZCOMPRESSED} subcommand determine the system file's
+compression level:
+
+@table @code
+@item UNCOMPRESSED
+Data is not compressed. Each numeric value uses 8 bytes of disk
+space. Each string value uses one byte per column width, rounded up
+to a multiple of 8 bytes.
+
+@item COMPRESSED
+Data is compressed with a simple algorithm. Each integer numeric
+value between @minus{}99 and 151, inclusive, or system missing value
+uses one byte of disk space. Each 8-byte segment of a string that
+consists only of spaces uses 1 byte. Any other numeric value or
+8-byte string segment uses 9 bytes of disk space.
+
+@item ZCOMPRESSED
+Data is compressed with the ``deflate'' compression algorithm
+specified in RFC@tie{}1951 (the same algorithm used by
+@command{gzip}). Files written with this compression level cannot be
+read by PSPP 0.8.1 or earlier or by SPSS 20 or earlier.
+@end table
+
+@subcmd{COMPRESSED} is the default compression level. The SET command
+(@pxref{SET}) can change this default.
The @subcmd{PERMISSIONS} subcommand specifies permissions for the new system
file. WRITEABLE, the default, creates the file with read and write
equals sign (@samp{=}) and the names that they should be renamed to.
Multiple parenthesized groups of variable names can be included on a
single @subcmd{RENAME} subcommand. Variables' names may be swapped using a
-@subcmd{RENAME} subcommand of the
+@subcmd{RENAME} subcommand of the
form @subcmd{/RENAME=(@var{A} @var{B}=@var{B} @var{A})}.
Alternate syntax for the @subcmd{RENAME} subcommand allows the parentheses to be
The @subcmd{VERSION} subcommand specifies the version of the file format. Valid
versions are 2 and 3. The default version is 3. In version 2 system
-files, variable names longer than 8 bytes will be truncated. The two
+files, variable names longer than 8 bytes are truncated. The two
versions are otherwise identical.
The @subcmd{NAMES} and @subcmd{MAP} subcommands are currently ignored.
@cmd{SAVE} causes the data to be read. It is a procedure.
+@node SAVE DATA COLLECTION
+@section SAVE DATA COLLECTION
+@vindex SAVE DATA COLLECTION
+
+@display
+SAVE DATA COLLECTION
+ /OUTFILE=@{'@var{file_name}',@var{file_handle}@}
+ /METADATA=@{'@var{file_name}',@var{file_handle}@}
+ /@{UNCOMPRESSED,COMPRESSED,ZCOMPRESSED@}
+ /PERMISSIONS=@{WRITEABLE,READONLY@}
+ /DROP=@var{var_list}
+ /KEEP=@var{var_list}
+ /VERSION=@var{version}
+ /RENAME=(@var{src_names}=@var{target_names})@dots{}
+ /NAMES
+ /MAP
+@end display
+
+Like @cmd{SAVE}, @cmd{SAVE DATA COLLECTION} writes the dictionary and
+data in the active dataset to a system file. In addition, it writes
+metadata to an additional XML metadata file.
+
+OUTFILE is required. Specify the system file to be written as a
+string file name or a file handle (@pxref{File Handles}).
+
+METADATA is also required. Specify the metadata file to be written as
+a string file name or a file handle. Metadata files customarily use a
+@file{.mdd} extension.
+
+The current implementation of this command is experimental. It only
+outputs an approximation of the metadata file format. Please report
+bugs.
+
+Other subcommands are optional. They have the same meanings as in the
+@cmd{SAVE} command.
+
+@cmd{SAVE DATA COLLECTION} causes the data to be read. It is a
+procedure.
+
@node SAVE TRANSLATE
@section SAVE TRANSLATE
@vindex SAVE TRANSLATE
Tab-delimited format.
@end table
-By default, @cmd{SAVE TRANSLATE} will not overwrite an existing file. Use
+By default, @cmd{SAVE TRANSLATE} does not overwrite an existing file. Use
@subcmd{REPLACE} to force an existing file to be overwritten.
With MISSING=IGNORE, the default, @subcmd{SAVE TRANSLATE} treats user-missing
numeric user-missing values like system-missing values and string
user-missing values as all spaces.
-By default, all the variables in the active dataset dictionary are saved
-to the system file, but @subcmd{DROP} or @subcmd{KEEP} can select a subset of variable
-to save. The @subcmd{RENAME} subcommand can also be used to change the names
-under which variables are saved. @subcmd{UNSELECTED} determines whether cases
-filtered out by the @cmd{FILTER} command are written to the output file.
-These subcommands have the same syntax and meaning as on the
-@cmd{SAVE} command (@pxref{SAVE}).
+By default, all the variables in the active dataset dictionary are
+saved to the system file, but @subcmd{DROP} or @subcmd{KEEP} can
+select a subset of variable to save. The @subcmd{RENAME} subcommand
+can also be used to change the names under which variables are saved;
+because they are used only in the output, these names do not have to
+conform to the usual PSPP variable naming rules. @subcmd{UNSELECTED}
+determines whether cases filtered out by the @cmd{FILTER} command are
+written to the output file. These subcommands have the same syntax
+and meaning as on the @cmd{SAVE} command (@pxref{SAVE}).
Each supported file type has additional subcommands, explained in
separate sections below.
The TEXTOPTIONS QUALIFIER setting specifies a character that is output
before and after a value that contains the delimiter character or the
-qualifier character. The default is a double quote (@samp{@@}). A
+qualifier character. The default is a double quote (@samp{"}). A
qualifier character that appears within a value is doubled.
@node SYSFILE INFO
@section SYSFILE INFO
@vindex SYSFILE INFO
-@display
-SYSFILE INFO FILE='@var{file_name}'.
+@display
+SYSFILE INFO FILE='@var{file_name}' [ENCODING='@var{encoding}'].
@end display
-@cmd{SYSFILE INFO} reads the dictionary in a system file and
-displays the information in its dictionary.
+@cmd{SYSFILE INFO} reads the dictionary in an SPSS system file,
+SPSS/PC+ system file, or SPSS portable file, and displays the
+information in its dictionary.
+
+Specify a file name or file handle. @cmd{SYSFILE INFO} reads that
+file and displays information on its dictionary.
-Specify a file name or file handle. @cmd{SYSFILE INFO} reads that file as
-a system file and displays information on its dictionary.
+@pspp{} automatically detects the encoding of string data in the file,
+when possible. The character encoding of old SPSS system files cannot
+always be guessed correctly, and SPSS/PC+ system files do not include
+any indication of their encoding. Specify the @subcmd{ENCODING}
+subcommand with an @acronym{IANA} character set name as its string
+argument to override the default, or specify @code{ENCODING='DETECT'}
+to analyze and report possibly valid encodings for the system file.
+The @subcmd{ENCODING} subcommand is a @pspp{} extension.
@cmd{SYSFILE INFO} does not affect the current active dataset.
/MAP
@end display
-The @cmd{EXPORT} transformation writes the active dataset dictionary and
+The @cmd{XEXPORT} transformation writes the active dataset dictionary and
data to a specified portable file.
This transformation is a @pspp{} extension.
@display
XSAVE
/OUTFILE='@var{file_name}'
- /@{COMPRESSED,UNCOMPRESSED@}
+ /@{UNCOMPRESSED,COMPRESSED,ZCOMPRESSED@}
/PERMISSIONS=@{WRITEABLE,READONLY@}
/DROP=@var{var_list}
/KEEP=@var{var_list}
projects / pspp / blobdiff