that describes how they may be interpreted. This chapter describes
the format of a system file.
-System files use three data types: 8-bit characters, 32-bit integers,
-and 64-bit floating points, called here @code{char}, @code{int32}, and
+System files use four data types: 8-bit characters, 32-bit integers,
+64-bit integers,
+and 64-bit floating points, called here @code{char}, @code{int32},
+@code{int64}, and
@code{flt64}, respectively. Data is not necessarily aligned on a word
or double-word boundary: the long variable name record (@pxref{Long
Variable Names Record}) and very long string records (@pxref{Very Long
Document record, if present.
@item
-Any of the following records, if present, in any order:
-
-@itemize @minus
-@item
-Machine integer info record.
-
-@item
-Machine floating-point info record.
-
-@item
-Variable display parameter record.
-
-@item
-Long variable names record.
-
-@item
-Miscellaneous informational records.
-@end itemize
+Any records not explicitly included in this list, in any order.
@item
Dictionary termination record.
* Document Record::
* Machine Integer Info Record::
* Machine Floating-Point Info Record::
+* Multiple Response Sets Records::
* Variable Display Parameter Record::
* Long Variable Names Record::
* Very Long String Record::
+* Character Encoding Record::
+* Long String Value Labels Record::
+* Data File and Variable Attributes Records::
+* Extended Number of Cases Record::
* Miscellaneous Informational Records::
* Dictionary Termination Record::
* Data Record::
that will be output to a system file at the time that the header is
written. The way that this is dealt with is by writing the entire
system file, including the header, then seeking back to the beginning of
-the file and writing just the @code{ncases} field. For `files' in which
+the file and writing just the @code{ncases} field. For files in which
this is not valid, the seek operation fails. In this case,
@code{ncases} remains -1.
@item int32 label_len;
This field is present only if @code{has_var_label} is set to 1. It is
-set to the length, in characters, of the variable label, which must be a
-number between 0 and 120.
+set to the length, in characters, of the variable label. The
+documented maximum length varies from 120 to 255 based on SPSS
+version, but some files have been seen with longer labels. PSPP
+accepts longer labels and truncates them to 255 bytes on input.
@item char label[];
This field is present only if @code{has_var_label} is set to 1. It has
The first @code{label_len} characters are the variable's variable label.
@item flt64 missing_values[];
-This field is present only if @code{n_missing_values} is not 0. It has
-the same number of elements as the absolute value of
-@code{n_missing_values}. For discrete missing values, each element
-represents one missing value. When a range is present, the first
-element denotes the minimum value in the range, and the second element
-denotes the maximum value in the range. When a range plus a value are
-present, the third element denotes the additional discrete missing
-value. HIGHEST and LOWEST are indicated as described in the chapter
-introduction.
+This field is present only if @code{n_missing_values} is nonzero. It
+has the same number of 8-byte elements as the absolute value of
+@code{n_missing_values}. Each element is interpreted as a number for
+numeric variables (with HIGHEST and LOWEST indicated as described in
+the chapter introduction). For string variables of width less than 8
+bytes, elements are right-padded with spaces; for string variables
+wider than 8 bytes, only the first 8 bytes of each missing value are
+specified, with the remainder implicitly all spaces.
+
+For discrete missing values, each element represents one missing
+value. When a range is present, the first element denotes the minimum
+value in the range, and the second element denotes the maximum value
+in the range. When a range plus a value are present, the third
+element denotes the additional discrete missing value.
@end table
The @code{print} and @code{write} members of sysfile_variable are output
@node Value Labels Records
@section Value Labels Records
+The value label records documented in this section are used for
+numeric and short string variables only. Long string variables may
+have value labels, but their value labels are recorded using a
+different record type (@pxref{Long String Value Labels Record}).
+
The value label record has the following format:
@example
following value label variables record (see below) is read.
@item char label_len;
-The label's length, in bytes.
+The label's length, in bytes. The documented maximum length varies
+from 60 to 120 based on SPSS version. PSPP supports value labels up
+to 255 bytes long.
@item char label[];
@code{label_len} bytes of the actual label, followed by up to 7 bytes
labels (@pxref{Dictionary Index}). There are @code{var_count}
elements.
-String variables wider than 8 bytes may not have value labels.
+String variables wider than 8 bytes may not be specified in this list.
@end table
@node Document Record
Machine endianness. 1 indicates big-endian, 2 indicates little-endian.
@item int32 character_code;
+@anchor{character-code}
Character code. 1 indicates EBCDIC, 2 indicates 7-bit ASCII, 3
indicates 8-bit ASCII, 4 indicates DEC Kanji.
Windows code page numbers are also valid.
+
+Experience has shown that in many files, this field is ignored or incorrect.
+For a more reliable indication of the file's character encoding
+see @ref{Character Encoding Record}.
@end table
@node Machine Floating-Point Info Record
The value used for LOWEST in missing values.
@end table
+@node Multiple Response Sets Records
+@section Multiple Response Sets Records
+
+The system file format has two different types of records that
+represent multiple response sets (@pxref{MRSETS,,,pspp, PSPP Users
+Guide}). The first type of record describes multiple response sets
+that can be understood by SPSS before version 14. The second type of
+record, with a closely related format, is used for multiple dichotomy
+sets that use the CATEGORYLABELS=COUNTEDVALUES feature added in
+version 14.
+
+@example
+/* @r{Header.} */
+int32 rec_type;
+int32 subtype;
+int32 size;
+int32 count;
+
+/* @r{Exactly @code{count} bytes of data.} */
+char mrsets[];
+@end example
+
+@table @code
+@item int32 rec_type;
+Record type. Always set to 7.
+
+@item int32 subtype;
+Record subtype. Set to 7 for records that describe multiple response
+sets understood by SPSS before version 14, or to 19 for records that
+describe dichotomy sets that use the CATEGORYLABELS=COUNTEDVALUES
+feature added in version 14.
+
+@item int32 size;
+The size of each element in the @code{mrsets} member. Always set to 1.
+
+@item int32 count;
+The total number of bytes in @code{mrsets}.
+
+@item char mrsets[];
+A series of multiple response sets, each of which consists of the
+following:
+
+@itemize @bullet
+@item
+The set's name (an identifier that begins with @samp{$}).
+
+@item
+An equals sign (@samp{=}).
+
+@item
+@samp{C} for a multiple category set, @samp{D} for a multiple
+dichotomy set with CATEGORYLABELS=VARLABELS, or @samp{E} for a
+multiple dichotomy set with CATEGORYLABELS=COUNTEDVALUES.
+
+@item
+For a multiple dichotomy set with CATEGORYLABELS=COUNTEDVALUES, a
+space, followed by a number expressed as decimal digits, followed by a
+space. If LABELSOURCE=VARLABEL was specified on MRSETS, then the
+number is 11; otherwise it is 1.@footnote{This part of the format may
+not be fully understood, because only a single example of each
+possibility has been examined.}
+
+@item
+For either kind of multiple dichotomy set, the counted value, as a
+positive integer count specified as decimal digits, followed by a
+space, followed by as many string bytes as specified in the count. If
+the set contains numeric variables, the string consists of the counted
+integer value expressed as decimal digits. If the set contains string
+variables, the string contains the counted string value. Either way,
+the string may be padded on the right with spaces (older versions of
+SPSS seem to always pad to a width of 8 bytes; newer versions don't).
+
+@item
+A space.
+
+@item
+The multiple response set's label, using the same format as for the
+counted value for multiple dichotomy sets. A string of length 0 means
+that the set does not have a label. A string of length 0 is also
+written if LABELSOURCE=VARLABEL was specified.
+
+@item
+A space.
+
+@item
+The names of the variables in the set, each separated from the
+previous by a single space.
+
+@item
+A line feed (byte 0x0a).
+@end itemize
+@end table
+
+Example: Given appropriate variable definitions, consider the
+following MRSETS command:
+
+@example
+MRSETS /MCGROUP NAME=$a LABEL='my mcgroup' VARIABLES=a b c
+ /MDGROUP NAME=$b VARIABLES=g e f d VALUE=55
+ /MDGROUP NAME=$c LABEL='mdgroup #2' VARIABLES=h i j VALUE='Yes'
+ /MDGROUP NAME=$d LABEL='third mdgroup' CATEGORYLABELS=COUNTEDVALUES
+ VARIABLES=k l m VALUE=34
+ /MDGROUP NAME=$e CATEGORYLABELS=COUNTEDVALUES LABELSOURCE=VARLABEL
+ VARIABLES=n o p VALUE='choice'.
+@end example
+
+The above would generate the following multiple response set record of
+subtype 7:
+
+@example
+$a=C 10 my mcgroup a b c
+$b=D2 55 0 g e f d
+$c=D3 Yes 10 mdgroup #2 h i j
+@end example
+
+It would also generate the following multiple response set record with
+subtype 19:
+
+@example
+$d=E 1 2 34 13 third mdgroup k l m
+$e=E 11 6 choice 0 n o p
+@end example
+
@node Variable Display Parameter Record
@section Variable Display Parameter Record
Continuous Scale
@end table
-SPSS 14 sometimes writes a @code{measure} of 0. PSPP interprets this
-as nominal scale.
+SPSS 14 sometimes writes a @code{measure} of 0 for string variables.
+PSPP interprets this as nominal scale.
@item int32 width;
The width of the display column for the variable in characters.
The total length is @code{count} bytes.
@end table
+@node Character Encoding Record
+@section Character Encoding Record
+
+This record, if present, indicates the character encoding for string data,
+long variable names, variable labels, value labels and other strings in the
+file.
+
+@example
+/* @r{Header.} */
+int32 rec_type;
+int32 subtype;
+int32 size;
+int32 count;
+
+/* @r{Exactly @code{count} bytes of data.} */
+char encoding[];
+@end example
+
+@table @code
+@item int32 rec_type;
+Record type. Always set to 7.
+
+@item int32 subtype;
+Record subtype. Always set to 20.
+
+@item int32 size;
+The size of each element in the @code{encoding} member. Always set to 1.
+
+@item int32 count;
+The total number of bytes in @code{encoding}.
+
+@item char encoding[];
+The name of the character encoding. Normally this will be an official IANA characterset name or alias.
+See @url{http://www.iana.org/assignments/character-sets}.
+@end table
+
+This record is not present in files generated by older software.
+See also @ref{character-code}.
+
+@node Long String Value Labels Record
+@section Long String Value Labels Record
+
+This record, if present, specifies value labels for long string
+variables.
+
+@example
+/* @r{Header.} */
+int32 rec_type;
+int32 subtype;
+int32 size;
+int32 count;
+
+/* @r{Repeated up to exactly @code{count} bytes.} */
+int32 var_name_len;
+char var_name[];
+int32 var_width;
+int32 n_labels;
+long_string_label labels[];
+@end example
+
+@table @code
+@item int32 rec_type;
+Record type. Always set to 7.
+
+@item int32 subtype;
+Record subtype. Always set to 21.
+
+@item int32 size;
+Always set to 1.
+
+@item int32 count;
+The number of bytes following the header until the next header.
+
+@item int32 var_name_len;
+@itemx char var_name[];
+The number of bytes in the name of the variable that has long string
+value labels, plus the variable name itself, which consists of exactly
+@code{var_name_len} bytes. The variable name is not padded to any
+particular boundary, nor is it null-terminated.
+
+@item int32 var_width;
+The width of the variable, in bytes, which will be between 9 and
+32767.
+
+@item int32 n_labels;
+@itemx long_string_label labels[];
+The long string labels themselves. The @code{labels} array contains
+exactly @code{n_labels} elements, each of which has the following
+substructure:
+
+@example
+int32 value_len;
+char value[];
+int32 label_len;
+char label[];
+@end example
+
+@table @code
+@item int32 value_len;
+@itemx char value[];
+The string value being labeled. @code{value_len} is the number of
+bytes in @code{value}; it is equal to @code{var_width}. The
+@code{value} array is not padded or null-terminated.
+
+@item int32 label_len;
+@itemx char label[];
+The label for the string value. @code{label_len}, which must be
+between 0 and 120, is the number of bytes in @code{label}. The
+@code{label} array is not padded or null-terminated.
+@end table
+@end table
+
+@node Data File and Variable Attributes Records
+@section Data File and Variable Attributes Records
+
+The data file and variable attributes records represent custom
+attributes for the system file or for individual variables in the
+system file, as defined on the DATAFILE ATTRIBUTE (@pxref{DATAFILE
+ATTRIBUTE,,,pspp, PSPP Users Guide}) and VARIABLE ATTRIBUTE commands
+(@pxref{VARIABLE ATTRIBUTE,,,pspp, PSPP Users Guide}), respectively.
+
+@example
+/* @r{Header.} */
+int32 rec_type;
+int32 subtype;
+int32 size;
+int32 count;
+
+/* @r{Exactly @code{count} bytes of data.} */
+char attributes[];
+@end example
+
+@table @code
+@item int32 rec_type;
+Record type. Always set to 7.
+
+@item int32 subtype;
+Record subtype. Always set to 17 for a data file attribute record or
+to 18 for a variable attributes record.
+
+@item int32 size;
+The size of each element in the @code{attributes} member. Always set to 1.
+
+@item int32 count;
+The total number of bytes in @code{attributes}.
+
+@item char attributes[];
+The attributes, in a text-based format.
+
+In record type 17, this field contains a single attribute set. An
+attribute set is a sequence of one or more attributes concatenated
+together. Each attribute consists of a name, which has the same
+syntax as a variable name, followed by, inside parentheses, a sequence
+of one or more values. Each value consists of a string enclosed in
+single quotes (@code{'}) followed by a line feed (byte 0x0a). A value
+may contain single quote characters, which are not themselves escaped
+or quoted or required to be present in pairs. There is no apparent
+way to embed a line feed in a value. There is no distinction between
+an attribute with a single value and an attribute array with one
+element.
+
+In record type 18, this field contains a sequence of one or more
+variable attribute sets. If more than one variable attribute set is
+present, each one after the first is delimited from the previous by
+@code{/}. Each variable attribute set consists of a (potentially
+long) variable name,
+followed by @code{:}, followed by an attribute set with the same
+syntax as on record type 17.
+
+The total length is @code{count} bytes.
+@end table
+
+@subheading Example
+
+A system file produced with the following VARIABLE ATTRIBUTE commands
+in effect:
+
+@example
+VARIABLE ATTRIBUTE VARIABLES=dummy ATTRIBUTE=fred[1]('23') fred[2]('34').
+VARIABLE ATTRIBUTE VARIABLES=dummy ATTRIBUTE=bert('123').
+@end example
+
+@noindent
+will contain a variable attribute record with the following contents:
+
+@example
+00000000 07 00 00 00 12 00 00 00 01 00 00 00 22 00 00 00 |............"...|
+00000010 64 75 6d 6d 79 3a 66 72 65 64 28 27 32 33 27 0a |dummy:fred('23'.|
+00000020 27 33 34 27 0a 29 62 65 72 74 28 27 31 32 33 27 |'34'.)bert('123'|
+00000030 0a 29 |.) |
+@end example
+
+@node Extended Number of Cases Record
+@section Extended Number of Cases Record
+
+The file header record expresses the number of cases in the system
+file as an int32 (@pxref{File Header Record}). This record allows the
+number of cases in the system file to be expressed as a 64-bit number.
+
+@example
+int32 rec_type;
+int32 subtype;
+int32 size;
+int32 count;
+int64 unknown;
+int64 ncases64;
+@end example
+
+@table @code
+@item int32 rec_type;
+Record type. Always set to 7.
+
+@item int32 subtype;
+Record subtype. Always set to 16.
+
+@item int32 size;
+Size of each element. Always set to 8.
+
+@item int32 count;
+Number of pieces of data in the data part. Alway set to 2.
+
+@item int64 unknown;
+Meaning unknown. Always set to 1.
+
+@item int64 ncases64;
+Number of cases in the file as a 64-bit integer. Presumably this
+could be -1 to indicate that the number of cases is unknown, for the
+same reason as @code{ncases} in the file header record, but this has
+not been observed in the wild.
+@end table
+
@node Miscellaneous Informational Records
@section Miscellaneous Informational Records
projects / pspp-builds.git / blobdiff