1 @node System File Format
2 @appendix System File Format
4 A system file encapsulates a set of cases and dictionary information
5 that describes how they may be interpreted. This chapter describes
6 the format of a system file.
8 System files use three data types: 8-bit characters, 32-bit integers,
9 and 64-bit floating points, called here @code{char}, @code{int32}, and
10 @code{flt64}, respectively. Data is not necessarily aligned on a word
11 or double-word boundary: the long variable name record (@pxref{Long
12 Variable Names Record}) and very long string records (@pxref{Very Long
13 String Record}) have arbitrary byte length and can therefore cause all
14 data coming after them in the file to be misaligned.
16 Integer data in system files may be big-endian or little-endian. A
17 reader may detect the endianness of a system file by examining
18 @code{layout_code} in the file header record
19 (@pxref{layout_code,,@code{layout_code}}).
21 Floating-point data in system files may nominally be in IEEE 754, IBM,
22 or VAX formats. A reader may detect the floating-point format in use
23 by examining @code{bias} in the file header record
24 (@pxref{bias,,@code{bias}}).
26 PSPP detects big-endian and little-endian integer formats in system
27 files and translates as necessary. PSPP also detects the
28 floating-point format in use, as well as the endianness of IEEE 754
29 floating-point numbers, and translates as needed. However, only IEEE
30 754 numbers with the same endianness as integer data in the same file
31 has actually been observed in system files, and it is likely that
32 other formats are obsolete or were never used.
34 The PSPP system-missing value is represented by the largest possible
35 negative number in the floating point format (@code{-DBL_MAX}). Two
36 other values are important for use as missing values: @code{HIGHEST},
37 represented by the largest possible positive number (@code{DBL_MAX}),
38 and @code{LOWEST}, represented by the second-largest negative number
39 (in IEEE 754 format, @code{0xffeffffffffffffe}).
41 System files are divided into records, each of which begins with a
42 4-byte record type, usually regarded as an @code{int32}.
44 The records must appear in the following order:
54 All pairs of value labels records and value label variables records,
58 Document record, if present.
61 Any of the following records, if present, in any order:
65 Machine integer info record.
68 Machine floating-point info record.
71 Variable display parameter record.
74 Long variable names record.
77 Miscellaneous informational records.
81 Dictionary termination record.
87 Each type of record is described separately below.
90 * File Header Record::
92 * Value Labels Records::
94 * Machine Integer Info Record::
95 * Machine Floating-Point Info Record::
96 * Variable Display Parameter Record::
97 * Long Variable Names Record::
98 * Very Long String Record::
99 * Character Encoding Record::
100 * Data File and Variable Attributes Records::
101 * Miscellaneous Informational Records::
102 * Dictionary Termination Record::
106 @node File Header Record
107 @section File Header Record
109 The file header is always the first record in the file. It has the
116 int32 nominal_case_size;
121 char creation_date[9];
122 char creation_time[8];
128 @item char rec_type[4];
129 Record type code, set to @samp{$FL2}.
131 @item char prod_name[60];
132 Product identification string. This always begins with the characters
133 @samp{@@(#) SPSS DATA FILE}. PSPP uses the remaining characters to
134 give its version and the operating system name; for example, @samp{GNU
135 pspp 0.1.4 - sparc-sun-solaris2.5.2}. The string is truncated if it
136 would be longer than 60 characters; otherwise it is padded on the right
140 @item int32 layout_code;
141 Normally set to 2, although a few system files have been spotted in
142 the wild with a value of 3 here. PSPP use this value to determine the
143 file's integer endianness (@pxref{System File Format}).
145 @item int32 nominal_case_size;
146 Number of data elements per case. This is the number of variables,
147 except that long string variables add extra data elements (one for every
148 8 characters after the first 8). However, string variables do not
149 contribute to this value beyond the first 255 bytes. Further, system
150 files written by some systems set this value to -1. In general, it is
151 unsafe for systems reading system files to rely upon this value.
153 @item int32 compressed;
154 Set to 1 if the data in the file is compressed, 0 otherwise.
156 @item int32 weight_index;
157 If one of the variables in the data set is used as a weighting
158 variable, set to the dictionary index of that variable, plus 1
159 (@pxref{Dictionary Index}). Otherwise, set to 0.
162 Set to the number of cases in the file if it is known, or -1 otherwise.
164 In the general case it is not possible to determine the number of cases
165 that will be output to a system file at the time that the header is
166 written. The way that this is dealt with is by writing the entire
167 system file, including the header, then seeking back to the beginning of
168 the file and writing just the @code{ncases} field. For `files' in which
169 this is not valid, the seek operation fails. In this case,
170 @code{ncases} remains -1.
174 Compression bias, ordinarily set to 100. Only integers between
175 @code{1 - bias} and @code{251 - bias} can be compressed.
177 By assuming that its value is 100, PSPP uses @code{bias} to determine
178 the file's floating-point format and endianness (@pxref{System File
179 Format}). If the compression bias is not 100, PSPP cannot auto-detect
180 the floating-point format and assumes that it is IEEE 754 format with
181 the same endianness as the system file's integers, which is correct
182 for all known system files.
184 @item char creation_date[9];
185 Date of creation of the system file, in @samp{dd mmm yy}
186 format, with the month as standard English abbreviations, using an
187 initial capital letter and following with lowercase. If the date is not
188 available then this field is arbitrarily set to @samp{01 Jan 70}.
190 @item char creation_time[8];
191 Time of creation of the system file, in @samp{hh:mm:ss}
192 format and using 24-hour time. If the time is not available then this
193 field is arbitrarily set to @samp{00:00:00}.
195 @item char file_label[64];
196 File label declared by the user, if any (@pxref{FILE LABEL,,,pspp,
197 PSPP Users Guide}). Padded on the right with spaces.
199 @item char padding[3];
200 Ignored padding bytes to make the structure a multiple of 32 bits in
201 length. Set to zeros.
204 @node Variable Record
205 @section Variable Record
207 There must be one variable record for each numeric variable and each
208 string variable with width 8 bytes or less. String variables wider
209 than 8 bytes have one variable record for each 8 bytes, rounding up.
210 The first variable record for a long string specifies the variable's
211 correct dictionary information. Subsequent variable records for a
212 long string are filled with dummy information: a type of -1, no
213 variable label or missing values, print and write formats that are
214 ignored, and an empty string as name. A few system files have been
215 encountered that include a variable label on dummy variable records,
216 so readers should take care to parse dummy variable records in the
217 same way as other variable records.
219 @anchor{Dictionary Index}
220 The @dfn{dictionary index} of a variable is its offset in the set of
221 variable records, including dummy variable records for long string
222 variables. The first variable record has a dictionary index of 0, the
223 second has a dictionary index of 1, and so on.
225 The system file format does not directly support string variables
226 wider than 255 bytes. Such very long string variables are represented
227 by a number of narrower string variables. @xref{Very Long String
228 Record}, for details.
234 int32 n_missing_values;
239 /* @r{Present only if @code{has_var_label} is 1.} */
243 /* @r{Present only if @code{n_missing_values} is nonzero}. */
244 flt64 missing_values[];
248 @item int32 rec_type;
249 Record type code. Always set to 2.
252 Variable type code. Set to 0 for a numeric variable. For a short
253 string variable or the first part of a long string variable, this is set
254 to the width of the string. For the second and subsequent parts of a
255 long string variable, set to -1, and the remaining fields in the
256 structure are ignored.
258 @item int32 has_var_label;
259 If this variable has a variable label, set to 1; otherwise, set to 0.
261 @item int32 n_missing_values;
262 If the variable has no missing values, set to 0. If the variable has
263 one, two, or three discrete missing values, set to 1, 2, or 3,
264 respectively. If the variable has a range for missing variables, set to
265 -2; if the variable has a range for missing variables plus a single
266 discrete value, set to -3.
269 Print format for this variable. See below.
272 Write format for this variable. See below.
275 Variable name. The variable name must begin with a capital letter or
276 the at-sign (@samp{@@}). Subsequent characters may also be digits, octothorpes
277 (@samp{#}), dollar signs (@samp{$}), underscores (@samp{_}), or full
278 stops (@samp{.}). The variable name is padded on the right with spaces.
280 @item int32 label_len;
281 This field is present only if @code{has_var_label} is set to 1. It is
282 set to the length, in characters, of the variable label, which must be a
283 number between 0 and 120.
286 This field is present only if @code{has_var_label} is set to 1. It has
287 length @code{label_len}, rounded up to the nearest multiple of 32 bits.
288 The first @code{label_len} characters are the variable's variable label.
290 @item flt64 missing_values[];
291 This field is present only if @code{n_missing_values} is not 0. It has
292 the same number of elements as the absolute value of
293 @code{n_missing_values}. For discrete missing values, each element
294 represents one missing value. When a range is present, the first
295 element denotes the minimum value in the range, and the second element
296 denotes the maximum value in the range. When a range plus a value are
297 present, the third element denotes the additional discrete missing
298 value. HIGHEST and LOWEST are indicated as described in the chapter
302 The @code{print} and @code{write} members of sysfile_variable are output
303 formats coded into @code{int32} types. The least-significant byte
304 of the @code{int32} represents the number of decimal places, and the
305 next two bytes in order of increasing significance represent field width
306 and format type, respectively. The most-significant byte is not
307 used and should be set to zero.
309 Format types are defined as follows:
312 @multitable {Value} {@code{DATETIME}}
398 @node Value Labels Records
399 @section Value Labels Records
401 The value label record has the following format:
407 /* @r{Repeated @code{label_cnt} times}. */
414 @item int32 rec_type;
415 Record type. Always set to 3.
417 @item int32 label_count;
418 Number of value labels present in this record.
421 The remaining fields are repeated @code{count} times. Each
422 repetition specifies one value label.
426 A numeric value or a short string value padded as necessary to 8 bytes
427 in length. Its type and width cannot be determined until the
428 following value label variables record (see below) is read.
430 @item char label_len;
431 The label's length, in bytes.
434 @code{label_len} bytes of the actual label, followed by up to 7 bytes
435 of padding to bring @code{label} and @code{label_len} together to a
436 multiple of 8 bytes in length.
439 The value label record is always immediately followed by a value label
440 variables record with the following format:
449 @item int32 rec_type;
450 Record type. Always set to 4.
452 @item int32 var_count;
453 Number of variables that the associated value labels from the value
454 label record are to be applied.
457 A list of dictionary indexes of variables to which to apply the value
458 labels (@pxref{Dictionary Index}). There are @code{var_count}
461 String variables wider than 8 bytes may not have value labels.
464 @node Document Record
465 @section Document Record
467 The document record, if present, has the following format:
476 @item int32 rec_type;
477 Record type. Always set to 6.
480 Number of lines of documents present.
482 @item char lines[][80];
483 Document lines. The number of elements is defined by @code{n_lines}.
484 Lines shorter than 80 characters are padded on the right with spaces.
487 @node Machine Integer Info Record
488 @section Machine Integer Info Record
490 The integer info record, if present, has the following format:
502 int32 version_revision;
504 int32 floating_point_rep;
505 int32 compression_code;
507 int32 character_code;
511 @item int32 rec_type;
512 Record type. Always set to 7.
515 Record subtype. Always set to 3.
518 Size of each piece of data in the data part, in bytes. Always set to 4.
521 Number of pieces of data in the data part. Always set to 8.
523 @item int32 version_major;
524 PSPP major version number. In version @var{x}.@var{y}.@var{z}, this
527 @item int32 version_minor;
528 PSPP minor version number. In version @var{x}.@var{y}.@var{z}, this
531 @item int32 version_revision;
532 PSPP version revision number. In version @var{x}.@var{y}.@var{z},
535 @item int32 machine_code;
536 Machine code. PSPP always set this field to value to -1, but other
539 @item int32 floating_point_rep;
540 Floating point representation code. For IEEE 754 systems this is 1.
541 IBM 370 sets this to 2, and DEC VAX E to 3.
543 @item int32 compression_code;
544 Compression code. Always set to 1.
546 @item int32 endianness;
547 Machine endianness. 1 indicates big-endian, 2 indicates little-endian.
549 @item int32 character_code;
550 @anchor{character-code}
551 Character code. 1 indicates EBCDIC, 2 indicates 7-bit ASCII, 3
552 indicates 8-bit ASCII, 4 indicates DEC Kanji.
553 Windows code page numbers are also valid.
555 Experience has shown that in many files, this field is ignored or incorrect.
556 For a more reliable indication of the file's character encoding
557 see @ref{Character Encoding Record}.
560 @node Machine Floating-Point Info Record
561 @section Machine Floating-Point Info Record
563 The floating-point info record, if present, has the following format:
579 @item int32 rec_type;
580 Record type. Always set to 7.
583 Record subtype. Always set to 4.
586 Size of each piece of data in the data part, in bytes. Always set to 8.
589 Number of pieces of data in the data part. Always set to 3.
592 The system missing value.
595 The value used for HIGHEST in missing values.
598 The value used for LOWEST in missing values.
601 @node Variable Display Parameter Record
602 @section Variable Display Parameter Record
604 The variable display parameter record, if present, has the following
614 /* @r{Repeated @code{count} times}. */
616 int32 width; /* @r{Not always present.} */
621 @item int32 rec_type;
622 Record type. Always set to 7.
625 Record subtype. Always set to 11.
628 The size of @code{int32}. Always set to 4.
631 The number of sets of variable display parameters (ordinarily the
632 number of variables in the dictionary), times 2 or 3.
635 The remaining members are repeated @code{count} times, in the same
636 order as the variable records. No element corresponds to variable
637 records that continue long string variables. The meanings of these
638 members are as follows:
642 The measurement type of the variable:
652 SPSS 14 sometimes writes a @code{measure} of 0. PSPP interprets this
656 The width of the display column for the variable in characters.
658 This field is present if @var{count} is 3 times the number of
659 variables in the dictionary. It is omitted if @var{count} is 2 times
660 the number of variables.
662 @item int32 alignment;
663 The alignment of the variable for display purposes:
675 @node Long Variable Names Record
676 @section Long Variable Names Record
678 If present, the long variable names record has the following format:
687 /* @r{Exactly @code{count} bytes of data.} */
688 char var_name_pairs[];
692 @item int32 rec_type;
693 Record type. Always set to 7.
696 Record subtype. Always set to 13.
699 The size of each element in the @code{var_name_pairs} member. Always set to 1.
702 The total number of bytes in @code{var_name_pairs}.
704 @item char var_name_pairs[];
705 A list of @var{key}--@var{value} tuples, where @var{key} is the name
706 of a variable, and @var{value} is its long variable name.
707 The @var{key} field is at most 8 bytes long and must match the
708 name of a variable which appears in the variable record (@pxref{Variable
710 The @var{value} field is at most 64 bytes long.
711 The @var{key} and @var{value} fields are separated by a @samp{=} byte.
712 Each tuple is separated by a byte whose value is 09. There is no
713 trailing separator following the last tuple.
714 The total length is @code{count} bytes.
717 @node Very Long String Record
718 @section Very Long String Record
720 Old versions of SPSS limited string variables to a width of 255 bytes.
721 For backward compatibility with these older versions, the system file
722 format represents a string longer than 255 bytes, called a @dfn{very
723 long string}, as a collection of strings no longer than 255 bytes
724 each. The strings concatenated to make a very long string are called
725 its @dfn{segments}; for consistency, variables other than very long
726 strings are considered to have a single segment.
728 A very long string with a width of @var{w} has @var{n} =
729 (@var{w} + 251) / 252 segments, that is, one segment for every
730 252 bytes of width, rounding up. It would be logical, then, for each
731 of the segments except the last to have a width of 252 and the last
732 segment to have the remainder, but this is not the case. In fact,
733 each segment except the last has a width of 255 bytes. The last
734 segment has width @var{w} - (@var{n} - 1) * 252; some versions
735 of SPSS make it slightly wider, but not wide enough to make the last
736 segment require another 8 bytes of data.
738 Data is packed tightly into segments of a very long string, 255 bytes
739 per segment. Because 255 bytes of segment data are allocated for
740 every 252 bytes of the very long string's width (approximately), some
741 unused space is left over at the end of the allocated segments. Data
742 in unused space is ignored.
744 Example: Consider a very long string of width 20,000. Such a very
745 long string has 20,000 / 252 = 80 (rounding up) segments. The first
746 79 segments have width 255; the last segment has width 20,000 - 79 *
747 252 = 92 or slightly wider (up to 96 bytes, the next multiple of 8).
748 The very long string's data is actually stored in the 19,890 bytes in
749 the first 78 segments, plus the first 110 bytes of the 79th segment
750 (19,890 + 110 = 20,000). The remaining 145 bytes of the 79th segment
751 and all 92 bytes of the 80th segment are unused.
753 The very long string record explains how to stitch together segments
754 to obtain very long string data. For each of the very long string
755 variables in the dictionary, it specifies the name of its first
756 segment's variable and the very long string variable's actual width.
757 The remaining segments immediately follow the named variable in the
758 system file's dictionary.
760 The very long string record, which is present only if the system file
761 contains very long string variables, has the following format:
770 /* @r{Exactly @code{count} bytes of data.} */
771 char string_lengths[];
775 @item int32 rec_type;
776 Record type. Always set to 7.
779 Record subtype. Always set to 14.
782 The size of each element in the @code{string_lengths} member. Always set to 1.
785 The total number of bytes in @code{string_lengths}.
787 @item char string_lengths[];
788 A list of @var{key}--@var{value} tuples, where @var{key} is the name
789 of a variable, and @var{value} is its length.
790 The @var{key} field is at most 8 bytes long and must match the
791 name of a variable which appears in the variable record (@pxref{Variable
793 The @var{value} field is exactly 5 bytes long. It is a zero-padded,
794 ASCII-encoded string that is the length of the variable.
795 The @var{key} and @var{value} fields are separated by a @samp{=} byte.
796 Tuples are delimited by a two-byte sequence @{00, 09@}.
797 After the last tuple, there may be a single byte 00, or @{00, 09@}.
798 The total length is @code{count} bytes.
801 @node Character Encoding Record
802 @section Character Encoding Record
804 This record, if present, indicates the character encoding for string data,
805 long variable names, variable labels, value labels and other strings in the
815 /* @r{Exactly @code{count} bytes of data.} */
820 @item int32 rec_type;
821 Record type. Always set to 7.
824 Record subtype. Always set to 20.
827 The size of each element in the @code{encoding} member. Always set to 1.
830 The total number of bytes in @code{encoding}.
832 @item char encoding[];
833 The name of the character encoding. Normally this will be an official IANA characterset name or alias.
834 See @url{http://www.iana.org/assignments/character-sets}.
837 This record is not present in files generated by older software.
838 See also @ref{character-code}.
841 @node Data File and Variable Attributes Records
842 @section Data File and Variable Attributes Records
844 The data file and variable attributes records represent custom
845 attributes for the system file or for individual variables in the
846 system file, as defined on the DATAFILE ATTRIBUTE (@pxref{DATAFILE
847 ATTRIBUTE,,,pspp, PSPP Users Guide}) and VARIABLE ATTRIBUTE commands
848 (@pxref{VARIABLE ATTRIBUTE,,,pspp, PSPP Users Guide}), respectively.
857 /* @r{Exactly @code{count} bytes of data.} */
862 @item int32 rec_type;
863 Record type. Always set to 7.
866 Record subtype. Always set to 17 for a data file attribute record or
867 to 18 for a variable attributes record.
870 The size of each element in the @code{attributes} member. Always set to 1.
873 The total number of bytes in @code{attributes}.
875 @item char attributes[];
876 The attributes, in a text-based format.
878 In record type 17, this field contains a single attribute set. An
879 attribute set is a sequence of one or more attributes concatenated
880 together. Each attribute consists of a name, which has the same
881 syntax as a variable name, followed by, inside parentheses, a sequence
882 of one or more values. Each value consists of a string enclosed in
883 single quotes (@code{'}) followed by a line feed (byte 0x0a). A value
884 may contain single quote characters, which are not themselves escaped
885 or quoted or required to be present in pairs. There is no apparent
886 way to embed a line feed in a value. There is no distinction between
887 an attribute with a single value and an attribute array with one
890 In record type 18, this field contains a sequence of one or more
891 variable attribute sets. If more than one variable attribute set is
892 present, each one after the first is delimited from the previous by
893 @code{/}. Each variable attribute set consists of a variable name,
894 followed by @code{:}, followed by an attribute set with the same
895 syntax as on record type 17.
897 The total length is @code{count} bytes.
902 A system file produced with the following VARIABLE ATTRIBUTE commands
906 VARIABLE ATTRIBUTE VARIABLES=dummy ATTRIBUTE=fred[1]('23') fred[2]('34').
907 VARIABLE ATTRIBUTE VARIABLES=dummy ATTRIBUTE=bert('123').
911 will contain a variable attribute record with the following contents:
914 00000000 07 00 00 00 12 00 00 00 01 00 00 00 22 00 00 00 |............"...|
915 00000010 64 75 6d 6d 79 3a 66 72 65 64 28 27 32 33 27 0a |dummy:fred('23'.|
916 00000020 27 33 34 27 0a 29 62 65 72 74 28 27 31 32 33 27 |'34'.)bert('123'|
920 @node Miscellaneous Informational Records
921 @section Miscellaneous Informational Records
923 Some specific types of miscellaneous informational records are
924 documented here, but others are known to exist. PSPP ignores unknown
925 miscellaneous informational records when reading system files.
934 /* @r{Exactly @code{size * count} bytes of data.} */
939 @item int32 rec_type;
940 Record type. Always set to 7.
943 Record subtype. May take any value. According to Aapi
944 H@"am@"al@"ainen, value 5 indicates a set of grouped variables and 6
945 indicates date info (probably related to USE).
948 Size of each piece of data in the data part. Should have the value 1,
949 4, or 8, for @code{char}, @code{int32}, and @code{flt64} format data,
953 Number of pieces of data in the data part.
956 Arbitrary data. There must be @code{size} times @code{count} bytes of
960 @node Dictionary Termination Record
961 @section Dictionary Termination Record
963 The dictionary termination record separates all other records from the
972 @item int32 rec_type;
973 Record type. Always set to 999.
976 Ignored padding. Should be set to 0.
982 Data records must follow all other records in the system file. There must
983 be at least one data record in every system file.
985 The format of data records varies depending on whether the data is
986 compressed. Regardless, the data is arranged in a series of 8-byte
989 When data is not compressed,
990 each element corresponds to
991 the variable declared in the respective variable record (@pxref{Variable
992 Record}). Numeric values are given in @code{flt64} format; string
993 values are literal characters string, padded on the right when
994 necessary to fill out 8-byte units.
996 Compressed data is arranged in the following manner: the first 8 bytes
997 in the data section is divided into a series of 1-byte command
998 codes. These codes have meanings as described below:
1002 Ignored. If the program writing the system file accumulates compressed
1003 data in blocks of fixed length, 0 bytes can be used to pad out extra
1004 bytes remaining at the end of a fixed-size block.
1008 value @var{code} - @var{bias}, where
1009 @var{code} is the value of the compression code and @var{bias} is the
1010 variable @code{bias} from the file header. For example,
1011 code 105 with bias 100.0 (the normal value) indicates a numeric variable
1015 End of file. This code may or may not appear at the end of the data
1016 stream. PSPP always outputs this code but its use is not required.
1019 A numeric or string value that is not
1020 compressible. The value is stored in the 8 bytes following the
1021 current block of command bytes. If this value appears twice in a block
1022 of command bytes, then it indicates the second group of 8 bytes following the
1023 command bytes, and so on.
1026 An 8-byte string value that is all spaces.
1029 The system-missing value.
1032 When the end of the an 8-byte group of command bytes is reached, any
1033 blocks of non-compressible values indicated by code 253 are skipped,
1034 and the next element of command bytes is read and interpreted, until
1035 the end of the file or a code with value 252 is reached.
1036 @setfilename ignored