1 @node System File Format
2 @appendix System File Format
4 A system file encapsulates a set of cases and dictionary information
5 that describes how they may be interpreted. This chapter describes
6 the format of a system file.
8 System files use four data types: 8-bit characters, 32-bit integers,
10 and 64-bit floating points, called here @code{char}, @code{int32},
12 @code{flt64}, respectively. Data is not necessarily aligned on a word
13 or double-word boundary: the long variable name record (@pxref{Long
14 Variable Names Record}) and very long string records (@pxref{Very Long
15 String Record}) have arbitrary byte length and can therefore cause all
16 data coming after them in the file to be misaligned.
18 Integer data in system files may be big-endian or little-endian. A
19 reader may detect the endianness of a system file by examining
20 @code{layout_code} in the file header record
21 (@pxref{layout_code,,@code{layout_code}}).
23 Floating-point data in system files may nominally be in IEEE 754, IBM,
24 or VAX formats. A reader may detect the floating-point format in use
25 by examining @code{bias} in the file header record
26 (@pxref{bias,,@code{bias}}).
28 PSPP detects big-endian and little-endian integer formats in system
29 files and translates as necessary. PSPP also detects the
30 floating-point format in use, as well as the endianness of IEEE 754
31 floating-point numbers, and translates as needed. However, only IEEE
32 754 numbers with the same endianness as integer data in the same file
33 have actually been observed in system files, and it is likely that
34 other formats are obsolete or were never used.
36 System files use a few floating point values for special purposes:
40 The system-missing value is represented by the largest possible
41 negative number in the floating point format (@code{-DBL_MAX}).
44 HIGHEST is used as the high end of a missing value range with an
45 unbounded maximum. It is represented by the largest possible positive
46 number (@code{DBL_MAX}).
49 LOWEST is used as the low end of a missing value range with an
50 unbounded minimum. It was originally represented by the
51 second-largest negative number (in IEEE 754 format,
52 @code{0xffeffffffffffffe}). System files written by SPSS 21 and later
53 instead use the largest negative number (@code{-DBL_MAX}), the same
54 value as SYSMIS. This does not lead to ambiguity because LOWEST
55 appears in system files only in missing value ranges, which never
59 System files may use most character encodings based on an 8-bit unit.
60 UTF-16 and UTF-32, based on wider units, appear to be unacceptable.
61 @code{rec_type} in the file header record is sufficient to distinguish
62 between ASCII and EBCDIC based encodings. The best way to determine
63 the specific encoding in use is to consult the character encoding
64 record (@pxref{Character Encoding Record}), if present, and failing
65 that the @code{character_code} in the machine integer info record
66 (@pxref{Machine Integer Info Record}). The same encoding should be
67 used for the dictionary and the data in the file, although it is
68 possible to artificially synthesize files that use different encodings
69 (@pxref{Character Encoding Record}).
72 * System File Record Structure::
73 * File Header Record::
75 * Value Labels Records::
77 * Machine Integer Info Record::
78 * Machine Floating-Point Info Record::
79 * Multiple Response Sets Records::
80 * Extra Product Info Record::
81 * Variable Display Parameter Record::
82 * Long Variable Names Record::
83 * Very Long String Record::
84 * Character Encoding Record::
85 * Long String Value Labels Record::
86 * Long String Missing Values Record::
87 * Data File and Variable Attributes Records::
88 * Extended Number of Cases Record::
89 * Other Informational Records::
90 * Dictionary Termination Record::
94 @node System File Record Structure
95 @section System File Record Structure
97 System files are divided into records with the following format:
104 This header does not identify the length of the @code{data} or any
105 information about what it contains, so the system file reader must
106 understand the format of @code{data} based on @code{type}. However,
107 records with type 7, called @dfn{extension records}, have a stricter
115 char data[size * count];
119 @item int32 rec_type;
120 Record type. Always set to 7.
123 Record subtype. This value identifies a particular kind of extension
127 The size of each piece of data that follows the header, in bytes.
128 Known extension records use 1, 4, or 8, for @code{char}, @code{int32},
129 and @code{flt64} format data, respectively.
132 The number of pieces of data that follow the header.
134 @item char data[size * count];
135 Data, whose format and interpretation depend on the subtype.
138 An extension record contains exactly @code{size * count} bytes of
139 data, which allows a reader that does not understand an extension
140 record to skip it. Extension records provide only nonessential
141 information, so this allows for files written by newer software to
142 preserve backward compatibility with older or less capable readers.
144 Records in a system file must appear in the following order:
154 All pairs of value labels records and value label variables records,
158 Document record, if present.
161 Extension (type 7) records, in ascending numerical order of their
164 System files written by SPSS include at most one of each kind of
165 extension record. This is generally true of system files written by
166 other software as well, with known exceptions noted below in the
167 individual sections about each type of record.
170 Dictionary termination record.
176 We advise authors of programs that read system files to tolerate
177 format variations. Various kinds of misformatting and corruption have
178 been observed in system files written by SPSS and other software
179 alike. In particular, because extension records provide nonessential
180 information, it is generally better to ignore an extension record
181 entirely than to refuse to read a system file.
183 The following sections describe the known kinds of records.
185 @node File Header Record
186 @section File Header Record
188 A system file begins with the file header, with the following format:
194 int32 nominal_case_size;
199 char creation_date[9];
200 char creation_time[8];
206 @item char rec_type[4];
207 Record type code, either @samp{$FL2} for system files with
208 uncompressed data or data compressed with simple bytecode compression,
209 or @samp{$FL3} for system files with ZLIB compressed data.
211 This is truly a character field that uses the character encoding as
212 other strings. Thus, in a file with an ASCII-based character encoding
213 this field contains @code{24 46 4c 32} or @code{24 46 4c 33}, and in a
214 file with an EBCDIC-based encoding this field contains @code{5b c6 d3
215 f2}. (No EBCDIC-based ZLIB-compressed files have been observed.)
217 @item char prod_name[60];
218 Product identification string. This always begins with the characters
219 @samp{@@(#) SPSS DATA FILE}. PSPP uses the remaining characters to
220 give its version and the operating system name; for example, @samp{GNU
221 pspp 0.1.4 - sparc-sun-solaris2.5.2}. The string is truncated if it
222 would be longer than 60 characters; otherwise it is padded on the right
225 The product name field allow readers to behave differently based on
226 quirks in the way that particular software writes system files.
227 @xref{Value Labels Records}, for the detail of the quirk that the PSPP
228 system file reader tolerates in files written by ReadStat, which has
229 @code{https://github.com/WizardMac/ReadStat} in @code{prod_name}.
232 @item int32 layout_code;
233 Normally set to 2, although a few system files have been spotted in
234 the wild with a value of 3 here. PSPP use this value to determine the
235 file's integer endianness (@pxref{System File Format}).
237 @item int32 nominal_case_size;
238 Number of data elements per case. This is the number of variables,
239 except that long string variables add extra data elements (one for every
240 8 characters after the first 8). However, string variables do not
241 contribute to this value beyond the first 255 bytes. Further, some
242 software always writes -1 or 0 in this field. In general, it is
243 unsafe for systems reading system files to rely upon this value.
245 @item int32 compression;
246 Set to 0 if the data in the file is not compressed, 1 if the data is
247 compressed with simple bytecode compression, 2 if the data is ZLIB
248 compressed. This field has value 2 if and only if @code{rec_type} is
251 @item int32 weight_index;
252 If one of the variables in the data set is used as a weighting
253 variable, set to the dictionary index of that variable, plus 1
254 (@pxref{Dictionary Index}). Otherwise, set to 0.
257 Set to the number of cases in the file if it is known, or -1 otherwise.
259 In the general case it is not possible to determine the number of cases
260 that will be output to a system file at the time that the header is
261 written. The way that this is dealt with is by writing the entire
262 system file, including the header, then seeking back to the beginning of
263 the file and writing just the @code{ncases} field. For files in which
264 this is not valid, the seek operation fails. In this case,
265 @code{ncases} remains -1.
269 Compression bias, ordinarily set to 100. Only integers between
270 @code{1 - bias} and @code{251 - bias} can be compressed.
272 By assuming that its value is 100, PSPP uses @code{bias} to determine
273 the file's floating-point format and endianness (@pxref{System File
274 Format}). If the compression bias is not 100, PSPP cannot auto-detect
275 the floating-point format and assumes that it is IEEE 754 format with
276 the same endianness as the system file's integers, which is correct
277 for all known system files.
279 @item char creation_date[9];
280 Date of creation of the system file, in @samp{dd mmm yy}
281 format, with the month as standard English abbreviations, using an
282 initial capital letter and following with lowercase. If the date is not
283 available then this field is arbitrarily set to @samp{01 Jan 70}.
285 @item char creation_time[8];
286 Time of creation of the system file, in @samp{hh:mm:ss}
287 format and using 24-hour time. If the time is not available then this
288 field is arbitrarily set to @samp{00:00:00}.
290 @item char file_label[64];
291 File label declared by the user, if any (@pxref{FILE LABEL,,,pspp,
292 PSPP Users Guide}). Padded on the right with spaces.
294 A product that identifies itself as @code{VOXCO INTERVIEWER 4.3} uses
295 CR-only line ends in this field, rather than the more usual LF-only or
298 @item char padding[3];
299 Ignored padding bytes to make the structure a multiple of 32 bits in
300 length. Set to zeros.
303 @node Variable Record
304 @section Variable Record
306 There must be one variable record for each numeric variable and each
307 string variable with width 8 bytes or less. String variables wider
308 than 8 bytes have one variable record for each 8 bytes, rounding up.
309 The first variable record for a long string specifies the variable's
310 correct dictionary information. Subsequent variable records for a
311 long string are filled with dummy information: a type of -1, no
312 variable label or missing values, print and write formats that are
313 ignored, and an empty string as name. A few system files have been
314 encountered that include a variable label on dummy variable records,
315 so readers should take care to parse dummy variable records in the
316 same way as other variable records.
318 @anchor{Dictionary Index}
319 The @dfn{dictionary index} of a variable is a 1-based offset in the set of
320 variable records, including dummy variable records for long string
321 variables. The first variable record has a dictionary index of 1, the
322 second has a dictionary index of 2, and so on.
324 The system file format does not directly support string variables
325 wider than 255 bytes. Such very long string variables are represented
326 by a number of narrower string variables. @xref{Very Long String
327 Record}, for details.
329 A system file should contain at least one variable and thus at least
330 one variable record, but system files have been observed in the wild
331 without any variables (thus, no data either).
337 int32 n_missing_values;
342 /* @r{Present only if @code{has_var_label} is 1.} */
346 /* @r{Present only if @code{n_missing_values} is nonzero}. */
347 flt64 missing_values[];
351 @item int32 rec_type;
352 Record type code. Always set to 2.
355 Variable type code. Set to 0 for a numeric variable. For a short
356 string variable or the first part of a long string variable, this is set
357 to the width of the string. For the second and subsequent parts of a
358 long string variable, set to -1, and the remaining fields in the
359 structure are ignored.
361 @item int32 has_var_label;
362 If this variable has a variable label, set to 1; otherwise, set to 0.
364 @item int32 n_missing_values;
365 If the variable has no missing values, set to 0. If the variable has
366 one, two, or three discrete missing values, set to 1, 2, or 3,
367 respectively. If the variable has a range for missing variables, set to
368 -2; if the variable has a range for missing variables plus a single
369 discrete value, set to -3.
371 A long string variable always has the value 0 here. A separate record
372 indicates missing values for long string variables (@pxref{Long String
373 Missing Values Record}).
376 Print format for this variable. See below.
379 Write format for this variable. See below.
382 Variable name. The variable name must begin with a capital letter or
383 the at-sign (@samp{@@}). Subsequent characters may also be digits, octothorpes
384 (@samp{#}), dollar signs (@samp{$}), underscores (@samp{_}), or full
385 stops (@samp{.}). The variable name is padded on the right with spaces.
387 The @samp{name} fields should be unique within a system file. System
388 files written by SPSS that contain very long string variables with
389 similar names sometimes contain duplicate names that are later
390 eliminated by resolving the very long string names (@pxref{Very Long
391 String Record}). PSPP handles duplicates by assigning them new,
394 @item int32 label_len;
395 This field is present only if @code{has_var_label} is set to 1. It is
396 set to the length, in characters, of the variable label. The
397 documented maximum length varies from 120 to 255 based on SPSS
398 version, but some files have been seen with longer labels. PSPP
399 accepts labels of any length.
402 This field is present only if @code{has_var_label} is set to 1. It has
403 length @code{label_len}, rounded up to the nearest multiple of 32 bits.
404 The first @code{label_len} characters are the variable's variable label.
406 @item flt64 missing_values[];
407 This field is present only if @code{n_missing_values} is nonzero. It
408 has the same number of 8-byte elements as the absolute value of
409 @code{n_missing_values}. Each element is interpreted as a number for
410 numeric variables (with HIGHEST and LOWEST indicated as described in
411 the chapter introduction). For string variables of width less than 8
412 bytes, elements are right-padded with spaces; for string variables
413 wider than 8 bytes, only the first 8 bytes of each missing value are
414 specified, with the remainder implicitly all spaces.
416 For discrete missing values, each element represents one missing
417 value. When a range is present, the first element denotes the minimum
418 value in the range, and the second element denotes the maximum value
419 in the range. When a range plus a value are present, the third
420 element denotes the additional discrete missing value.
423 @anchor{System File Output Formats}
424 The @code{print} and @code{write} members of sysfile_variable are output
425 formats coded into @code{int32} types. The least-significant byte
426 of the @code{int32} represents the number of decimal places, and the
427 next two bytes in order of increasing significance represent field width
428 and format type, respectively. The most-significant byte is not
429 used and should be set to zero.
431 Format types are defined as follows:
434 @multitable {Value} {@code{DATETIME}}
524 A few system files have been observed in the wild with invalid
525 @code{write} fields, in particular with value 0. Readers should
526 probably treat invalid @code{print} or @code{write} fields as some
529 @node Value Labels Records
530 @section Value Labels Records
532 The value label records documented in this section are used for
533 numeric and short string variables only. Long string variables may
534 have value labels, but their value labels are recorded using a
535 different record type (@pxref{Long String Value Labels Record}).
537 ReadStat (@pxref{File Header Record}) writes value labels that label a
538 single value more than once. In more detail, it emits value labels
539 whose values are longer than string variables' widths, that are
540 identical in the actual width of the variable, e.g.@: labels for
541 values @code{ABC123} and @code{ABC456} for a string variable with
542 width 3. For files written by this software, PSPP ignores such
545 The value label record has the following format:
551 /* @r{Repeated @code{label_cnt} times}. */
558 @item int32 rec_type;
559 Record type. Always set to 3.
561 @item int32 label_count;
562 Number of value labels present in this record.
565 The remaining fields are repeated @code{count} times. Each
566 repetition specifies one value label.
570 A numeric value or a short string value padded as necessary to 8 bytes
571 in length. Its type and width cannot be determined until the
572 following value label variables record (see below) is read.
574 @item char label_len;
575 The label's length, in bytes. The documented maximum length varies
576 from 60 to 120 based on SPSS version. PSPP supports value labels up
580 @code{label_len} bytes of the actual label, followed by up to 7 bytes
581 of padding to bring @code{label} and @code{label_len} together to a
582 multiple of 8 bytes in length.
585 The value label record is always immediately followed by a value label
586 variables record with the following format:
595 @item int32 rec_type;
596 Record type. Always set to 4.
598 @item int32 var_count;
599 Number of variables that the associated value labels from the value
600 label record are to be applied.
603 A list of 1-based dictionary indexes of variables to which to apply the value
604 labels (@pxref{Dictionary Index}). There are @code{var_count}
607 String variables wider than 8 bytes may not be specified in this list.
610 @node Document Record
611 @section Document Record
613 The document record, if present, has the following format:
622 @item int32 rec_type;
623 Record type. Always set to 6.
626 Number of lines of documents present. This should be greater than
627 zero, but ReadStats writes system files with zero @code{n_lines}.
629 @item char lines[][80];
630 Document lines. The number of elements is defined by @code{n_lines}.
631 Lines shorter than 80 characters are padded on the right with spaces.
634 @node Machine Integer Info Record
635 @section Machine Integer Info Record
637 The integer info record, if present, has the following format:
649 int32 version_revision;
651 int32 floating_point_rep;
652 int32 compression_code;
654 int32 character_code;
658 @item int32 rec_type;
659 Record type. Always set to 7.
662 Record subtype. Always set to 3.
665 Size of each piece of data in the data part, in bytes. Always set to 4.
668 Number of pieces of data in the data part. Always set to 8.
670 @item int32 version_major;
671 PSPP major version number. In version @var{x}.@var{y}.@var{z}, this
674 @item int32 version_minor;
675 PSPP minor version number. In version @var{x}.@var{y}.@var{z}, this
678 @item int32 version_revision;
679 PSPP version revision number. In version @var{x}.@var{y}.@var{z},
682 @item int32 machine_code;
683 Machine code. PSPP always set this field to value to -1, but other
686 @item int32 floating_point_rep;
687 Floating point representation code. For IEEE 754 systems this is 1.
688 IBM 370 sets this to 2, and DEC VAX E to 3.
690 @item int32 compression_code;
691 Compression code. Always set to 1, regardless of whether or how the
694 @item int32 endianness;
695 Machine endianness. 1 indicates big-endian, 2 indicates little-endian.
697 @item int32 character_code;
698 @anchor{character-code} Character code. The following values have
699 been actually observed in system files:
709 The @code{windows-1250} code page for Central European and Eastern
713 The @code{windows-1252} code page for Western European languages.
722 The following additional values are known to be defined:
732 Other Windows code page numbers are known to be generally valid.
734 Old versions of SPSS for Unix and Windows always wrote value 2 in this
735 field, regardless of the encoding in use. Newer versions also write
736 the character encoding as a string (see @ref{Character Encoding
740 @node Machine Floating-Point Info Record
741 @section Machine Floating-Point Info Record
743 The floating-point info record, if present, has the following format:
759 @item int32 rec_type;
760 Record type. Always set to 7.
763 Record subtype. Always set to 4.
766 Size of each piece of data in the data part, in bytes. Always set to 8.
769 Number of pieces of data in the data part. Always set to 3.
772 @itemx flt64 highest;
774 The system missing value, the value used for HIGHEST in missing
775 values, and the value used for LOWEST in missing values, respectively.
776 @xref{System File Format}, for more information.
778 The SPSSWriter library in PHP, which identifies itself as @code{FOM
779 SPSS 1.0.0} in the file header record @code{prod_name} field, writes
780 unexpected values to these fields, but it uses the same values
781 consistently throughout the rest of the file.
784 @node Multiple Response Sets Records
785 @section Multiple Response Sets Records
787 The system file format has two different types of records that
788 represent multiple response sets (@pxref{MRSETS,,,pspp, PSPP Users
789 Guide}). The first type of record describes multiple response sets
790 that can be understood by SPSS before version 14. The second type of
791 record, with a closely related format, is used for multiple dichotomy
792 sets that use the CATEGORYLABELS=COUNTEDVALUES feature added in
802 /* @r{Exactly @code{count} bytes of data.} */
807 @item int32 rec_type;
808 Record type. Always set to 7.
811 Record subtype. Set to 7 for records that describe multiple response
812 sets understood by SPSS before version 14, or to 19 for records that
813 describe dichotomy sets that use the CATEGORYLABELS=COUNTEDVALUES
814 feature added in version 14.
817 The size of each element in the @code{mrsets} member. Always set to 1.
820 The total number of bytes in @code{mrsets}.
823 Zero or more line feeds (byte 0x0a), followed by a series of multiple
824 response sets, each of which consists of the following:
828 The set's name (an identifier that begins with @samp{$}), in mixed
829 upper and lower case.
832 An equals sign (@samp{=}).
835 @samp{C} for a multiple category set, @samp{D} for a multiple
836 dichotomy set with CATEGORYLABELS=VARLABELS, or @samp{E} for a
837 multiple dichotomy set with CATEGORYLABELS=COUNTEDVALUES.
840 For a multiple dichotomy set with CATEGORYLABELS=COUNTEDVALUES, a
841 space, followed by a number expressed as decimal digits, followed by a
842 space. If LABELSOURCE=VARLABEL was specified on MRSETS, then the
843 number is 11; otherwise it is 1.@footnote{This part of the format may
844 not be fully understood, because only a single example of each
845 possibility has been examined.}
848 For either kind of multiple dichotomy set, the counted value, as a
849 positive integer count specified as decimal digits, followed by a
850 space, followed by as many string bytes as specified in the count. If
851 the set contains numeric variables, the string consists of the counted
852 integer value expressed as decimal digits. If the set contains string
853 variables, the string contains the counted string value. Either way,
854 the string may be padded on the right with spaces (older versions of
855 SPSS seem to always pad to a width of 8 bytes; newer versions don't).
861 The multiple response set's label, using the same format as for the
862 counted value for multiple dichotomy sets. A string of length 0 means
863 that the set does not have a label. A string of length 0 is also
864 written if LABELSOURCE=VARLABEL was specified.
870 The short names of the variables in the set, converted to lowercase,
871 each separated from the previous by a single space.
873 Even though a multiple response set must have at least two variables,
874 some system files contain multiple response sets with no variables or
875 one variable. The source and meaning of these multiple response sets is
876 unknown. (Perhaps they arise from creating a multiple response set
877 then deleting all the variables that it contains?)
880 One line feed (byte 0x0a). Sometimes multiple, even hundreds, of line
885 Example: Given appropriate variable definitions, consider the
886 following MRSETS command:
889 MRSETS /MCGROUP NAME=$a LABEL='my mcgroup' VARIABLES=a b c
890 /MDGROUP NAME=$b VARIABLES=g e f d VALUE=55
891 /MDGROUP NAME=$c LABEL='mdgroup #2' VARIABLES=h i j VALUE='Yes'
892 /MDGROUP NAME=$d LABEL='third mdgroup' CATEGORYLABELS=COUNTEDVALUES
893 VARIABLES=k l m VALUE=34
894 /MDGROUP NAME=$e CATEGORYLABELS=COUNTEDVALUES LABELSOURCE=VARLABEL
895 VARIABLES=n o p VALUE='choice'.
898 The above would generate the following multiple response set record of
902 $a=C 10 my mcgroup a b c
904 $c=D3 Yes 10 mdgroup #2 h i j
907 It would also generate the following multiple response set record with
911 $d=E 1 2 34 13 third mdgroup k l m
912 $e=E 11 6 choice 0 n o p
915 @node Extra Product Info Record
916 @section Extra Product Info Record
918 This optional record appears to contain a text string that describes
919 the program that wrote the file and the source of the data. (This is
920 redundant with the file label and product info found in the file
930 /* @r{Exactly @code{count} bytes of data.} */
935 @item int32 rec_type;
936 Record type. Always set to 7.
939 Record subtype. Always set to 10.
942 The size of each element in the @code{info} member. Always set to 1.
945 The total number of bytes in @code{info}.
948 A text string. A product that identifies itself as @code{VOXCO
949 INTERVIEWER 4.3} uses CR-only line ends in this field, rather than the
950 more usual LF-only or CR LF line ends.
953 @node Variable Display Parameter Record
954 @section Variable Display Parameter Record
956 The variable display parameter record, if present, has the following
966 /* @r{Repeated @code{count} times}. */
968 int32 width; /* @r{Not always present.} */
973 @item int32 rec_type;
974 Record type. Always set to 7.
977 Record subtype. Always set to 11.
980 The size of @code{int32}. Always set to 4.
983 The number of sets of variable display parameters (ordinarily the
984 number of variables in the dictionary), times 2 or 3.
987 The remaining members are repeated @code{count} times, in the same
988 order as the variable records. No element corresponds to variable
989 records that continue long string variables. The meanings of these
990 members are as follows:
994 The measurement type of the variable:
1004 SPSS sometimes writes a @code{measure} of 0. PSPP interprets this as
1008 The width of the display column for the variable in characters.
1010 This field is present if @var{count} is 3 times the number of
1011 variables in the dictionary. It is omitted if @var{count} is 2 times
1012 the number of variables.
1014 @item int32 alignment;
1015 The alignment of the variable for display purposes:
1027 @node Long Variable Names Record
1028 @section Long Variable Names Record
1030 If present, the long variable names record has the following format:
1039 /* @r{Exactly @code{count} bytes of data.} */
1040 char var_name_pairs[];
1044 @item int32 rec_type;
1045 Record type. Always set to 7.
1047 @item int32 subtype;
1048 Record subtype. Always set to 13.
1051 The size of each element in the @code{var_name_pairs} member. Always set to 1.
1054 The total number of bytes in @code{var_name_pairs}.
1056 @item char var_name_pairs[];
1057 A list of @var{key}--@var{value} tuples, where @var{key} is the name
1058 of a variable, and @var{value} is its long variable name.
1059 The @var{key} field is at most 8 bytes long and must match the
1060 name of a variable which appears in the variable record (@pxref{Variable
1062 The @var{value} field is at most 64 bytes long.
1063 The @var{key} and @var{value} fields are separated by a @samp{=} byte.
1064 Each tuple is separated by a byte whose value is 09. There is no
1065 trailing separator following the last tuple.
1066 The total length is @code{count} bytes.
1069 @node Very Long String Record
1070 @section Very Long String Record
1072 Old versions of SPSS limited string variables to a width of 255 bytes.
1073 For backward compatibility with these older versions, the system file
1074 format represents a string longer than 255 bytes, called a @dfn{very
1075 long string}, as a collection of strings no longer than 255 bytes
1076 each. The strings concatenated to make a very long string are called
1077 its @dfn{segments}; for consistency, variables other than very long
1078 strings are considered to have a single segment.
1080 A very long string with a width of @var{w} has @var{n} =
1081 (@var{w} + 251) / 252 segments, that is, one segment for every
1082 252 bytes of width, rounding up. It would be logical, then, for each
1083 of the segments except the last to have a width of 252 and the last
1084 segment to have the remainder, but this is not the case. In fact,
1085 each segment except the last has a width of 255 bytes. The last
1086 segment has width @var{w} - (@var{n} - 1) * 252; some versions
1087 of SPSS make it slightly wider, but not wide enough to make the last
1088 segment require another 8 bytes of data.
1090 Data is packed tightly into segments of a very long string, 255 bytes
1091 per segment. Because 255 bytes of segment data are allocated for
1092 every 252 bytes of the very long string's width (approximately), some
1093 unused space is left over at the end of the allocated segments. Data
1094 in unused space is ignored.
1096 Example: Consider a very long string of width 20,000. Such a very
1097 long string has 20,000 / 252 = 80 (rounding up) segments. The first
1098 79 segments have width 255; the last segment has width 20,000 - 79 *
1099 252 = 92 or slightly wider (up to 96 bytes, the next multiple of 8).
1100 The very long string's data is actually stored in the 19,890 bytes in
1101 the first 78 segments, plus the first 110 bytes of the 79th segment
1102 (19,890 + 110 = 20,000). The remaining 145 bytes of the 79th segment
1103 and all 92 bytes of the 80th segment are unused.
1105 The very long string record explains how to stitch together segments
1106 to obtain very long string data. For each of the very long string
1107 variables in the dictionary, it specifies the name of its first
1108 segment's variable and the very long string variable's actual width.
1109 The remaining segments immediately follow the named variable in the
1110 system file's dictionary.
1112 The very long string record, which is present only if the system file
1113 contains very long string variables, has the following format:
1122 /* @r{Exactly @code{count} bytes of data.} */
1123 char string_lengths[];
1127 @item int32 rec_type;
1128 Record type. Always set to 7.
1130 @item int32 subtype;
1131 Record subtype. Always set to 14.
1134 The size of each element in the @code{string_lengths} member. Always set to 1.
1137 The total number of bytes in @code{string_lengths}.
1139 @item char string_lengths[];
1140 A list of @var{key}--@var{value} tuples, where @var{key} is the name
1141 of a variable, and @var{value} is its length.
1142 The @var{key} field is at most 8 bytes long and must match the
1143 name of a variable which appears in the variable record (@pxref{Variable
1145 The @var{value} field is exactly 5 bytes long. It is a zero-padded,
1146 ASCII-encoded string that is the length of the variable.
1147 The @var{key} and @var{value} fields are separated by a @samp{=} byte.
1148 Tuples are delimited by a two-byte sequence @{00, 09@}.
1149 After the last tuple, there may be a single byte 00, or @{00, 09@}.
1150 The total length is @code{count} bytes.
1153 @node Character Encoding Record
1154 @section Character Encoding Record
1156 This record, if present, indicates the character encoding for string data,
1157 long variable names, variable labels, value labels and other strings in the
1167 /* @r{Exactly @code{count} bytes of data.} */
1172 @item int32 rec_type;
1173 Record type. Always set to 7.
1175 @item int32 subtype;
1176 Record subtype. Always set to 20.
1179 The size of each element in the @code{encoding} member. Always set to 1.
1182 The total number of bytes in @code{encoding}.
1184 @item char encoding[];
1185 The name of the character encoding. Normally this will be an official
1186 IANA character set name or alias.
1187 See @url{http://www.iana.org/assignments/character-sets}.
1188 Character set names are not case-sensitive, but SPSS appears to write
1189 them in all-uppercase.
1192 This record is not present in files generated by older software. See
1193 also the @code{character_code} field in the machine integer info
1194 record (@pxref{character-code}).
1196 When the character encoding record and the machine integer info record
1197 are both present, all system files observed in practice indicate the
1198 same character encoding, e.g.@: 1252 as @code{character_code} and
1199 @code{windows-1252} as @code{encoding}, 65001 and @code{UTF-8}, etc.
1201 If, for testing purposes, a file is crafted with different
1202 @code{character_code} and @code{encoding}, it seems that
1203 @code{character_code} controls the encoding for all strings in the
1204 system file before the dictionary termination record, including
1205 strings in data (e.g.@: string missing values), and @code{encoding}
1206 controls the encoding for strings following the dictionary termination
1209 @node Long String Value Labels Record
1210 @section Long String Value Labels Record
1212 This record, if present, specifies value labels for long string
1222 /* @r{Repeated up to exactly @code{count} bytes.} */
1227 long_string_label labels[];
1231 @item int32 rec_type;
1232 Record type. Always set to 7.
1234 @item int32 subtype;
1235 Record subtype. Always set to 21.
1241 The number of bytes following the header until the next header.
1243 @item int32 var_name_len;
1244 @itemx char var_name[];
1245 The number of bytes in the name of the variable that has long string
1246 value labels, plus the variable name itself, which consists of exactly
1247 @code{var_name_len} bytes. The variable name is not padded to any
1248 particular boundary, nor is it null-terminated.
1250 @item int32 var_width;
1251 The width of the variable, in bytes, which will be between 9 and
1254 @item int32 n_labels;
1255 @itemx long_string_label labels[];
1256 The long string labels themselves. The @code{labels} array contains
1257 exactly @code{n_labels} elements, each of which has the following
1268 @item int32 value_len;
1269 @itemx char value[];
1270 The string value being labeled. @code{value_len} is the number of
1271 bytes in @code{value}; it is equal to @code{var_width}. The
1272 @code{value} array is not padded or null-terminated.
1274 @item int32 label_len;
1275 @itemx char label[];
1276 The label for the string value. @code{label_len}, which must be
1277 between 0 and 120, is the number of bytes in @code{label}. The
1278 @code{label} array is not padded or null-terminated.
1282 @node Long String Missing Values Record
1283 @section Long String Missing Values Record
1285 This record, if present, specifies missing values for long string
1295 /* @r{Repeated up to exactly @code{count} bytes.} */
1298 char n_missing_values;
1299 long_string_missing_value values[];
1303 @item int32 rec_type;
1304 Record type. Always set to 7.
1306 @item int32 subtype;
1307 Record subtype. Always set to 22.
1313 The number of bytes following the header until the next header.
1315 @item int32 var_name_len;
1316 @itemx char var_name[];
1317 The number of bytes in the name of the long string variable that has
1318 missing values, plus the variable name itself, which consists of
1319 exactly @code{var_name_len} bytes. The variable name is not padded to
1320 any particular boundary, nor is it null-terminated.
1322 @item char n_missing_values;
1323 The number of missing values, either 1, 2, or 3. (This is, unusually,
1324 a single byte instead of a 32-bit number.)
1326 @item long_string_missing_value values[];
1327 The missing values themselves. This array contains exactly
1328 @code{n_missing_values} elements, each of which has the following
1337 @item int32 value_len;
1338 The length of the missing value string, in bytes. This value should
1339 be 8, because long string variables are at least 8 bytes wide (by
1340 definition), only the first 8 bytes of a long string variable's
1341 missing values are allowed to be non-spaces, and any spaces within the
1342 first 8 bytes are included in the missing value here.
1345 The missing value string, exactly @code{value_len} bytes, without
1346 any padding or null terminator.
1350 @node Data File and Variable Attributes Records
1351 @section Data File and Variable Attributes Records
1353 The data file and variable attributes records represent custom
1354 attributes for the system file or for individual variables in the
1355 system file, as defined on the DATAFILE ATTRIBUTE (@pxref{DATAFILE
1356 ATTRIBUTE,,,pspp, PSPP Users Guide}) and VARIABLE ATTRIBUTE commands
1357 (@pxref{VARIABLE ATTRIBUTE,,,pspp, PSPP Users Guide}), respectively.
1366 /* @r{Exactly @code{count} bytes of data.} */
1371 @item int32 rec_type;
1372 Record type. Always set to 7.
1374 @item int32 subtype;
1375 Record subtype. Always set to 17 for a data file attribute record or
1376 to 18 for a variable attributes record.
1379 The size of each element in the @code{attributes} member. Always set to 1.
1382 The total number of bytes in @code{attributes}.
1384 @item char attributes[];
1385 The attributes, in a text-based format.
1387 In record subtype 17, this field contains a single attribute set. An
1388 attribute set is a sequence of one or more attributes concatenated
1389 together. Each attribute consists of a name, which has the same
1390 syntax as a variable name, followed by, inside parentheses, a sequence
1391 of one or more values. Each value consists of a string enclosed in
1392 single quotes (@code{'}) followed by a line feed (byte 0x0a). A value
1393 may contain single quote characters, which are not themselves escaped
1394 or quoted or required to be present in pairs. There is no apparent
1395 way to embed a line feed in a value. There is no distinction between
1396 an attribute with a single value and an attribute array with one
1399 In record subtype 18, this field contains a sequence of one or more
1400 variable attribute sets. If more than one variable attribute set is
1401 present, each one after the first is delimited from the previous by
1402 @code{/}. Each variable attribute set consists of a long
1404 followed by @code{:}, followed by an attribute set with the same
1405 syntax as on record subtype 17.
1407 System files written by @code{Stata 14.1/-savespss- 1.77 by
1408 S.Radyakin} may include multiple records with subtype 18, one per
1409 variable that has variable attributes.
1411 The total length is @code{count} bytes.
1416 A system file produced with the following VARIABLE ATTRIBUTE commands
1420 VARIABLE ATTRIBUTE VARIABLES=dummy ATTRIBUTE=fred[1]('23') fred[2]('34').
1421 VARIABLE ATTRIBUTE VARIABLES=dummy ATTRIBUTE=bert('123').
1425 will contain a variable attribute record with the following contents:
1428 0000 07 00 00 00 12 00 00 00 01 00 00 00 22 00 00 00 |............"...|
1429 0010 64 75 6d 6d 79 3a 66 72 65 64 28 27 32 33 27 0a |dummy:fred('23'.|
1430 0020 27 33 34 27 0a 29 62 65 72 74 28 27 31 32 33 27 |'34'.)bert('123'|
1438 @node Variable Roles
1439 @subsection Variable Roles
1441 A variable's role is represented as an attribute named @code{$@@Role}.
1442 This attribute has a single element whose values and their meanings
1447 Input. This, the default, is the most common role.
1460 @node Extended Number of Cases Record
1461 @section Extended Number of Cases Record
1463 The file header record expresses the number of cases in the system
1464 file as an int32 (@pxref{File Header Record}). This record allows the
1465 number of cases in the system file to be expressed as a 64-bit number.
1477 @item int32 rec_type;
1478 Record type. Always set to 7.
1480 @item int32 subtype;
1481 Record subtype. Always set to 16.
1484 Size of each element. Always set to 8.
1487 Number of pieces of data in the data part. Alway set to 2.
1489 @item int64 unknown;
1490 Meaning unknown. Always set to 1.
1492 @item int64 ncases64;
1493 Number of cases in the file as a 64-bit integer. Presumably this
1494 could be -1 to indicate that the number of cases is unknown, for the
1495 same reason as @code{ncases} in the file header record, but this has
1496 not been observed in the wild.
1499 @node Other Informational Records
1500 @section Other Informational Records
1502 This chapter documents many specific types of extension records are
1503 documented here, but others are known to exist. PSPP ignores unknown
1504 extension records when reading system files.
1506 The following extension record subtypes have also been observed, with
1507 the following believed meanings:
1511 A set of grouped variables (according to Aapi H@"am@"al@"ainen).
1514 Date info, probably related to USE (according to Aapi H@"am@"al@"ainen).
1517 A UUID in the format described in RFC 4122. Only two examples
1518 observed, both written by SPSS 13, and in each case the UUID contained
1519 both upper and lower case.
1522 XML that describes how data in the file should be displayed on-screen.
1525 @node Dictionary Termination Record
1526 @section Dictionary Termination Record
1528 The dictionary termination record separates all other records from the
1537 @item int32 rec_type;
1538 Record type. Always set to 999.
1541 Ignored padding. Should be set to 0.
1545 @section Data Record
1547 The data record must follow all other records in the system file.
1548 Every system file must have a data record that specifies data for at
1549 least one case. The format of the data record varies depending on the
1550 value of @code{compression} in the file header record:
1553 @item 0: no compression
1554 Data is arranged as a series of 8-byte elements.
1555 Each element corresponds to
1556 the variable declared in the respective variable record (@pxref{Variable
1557 Record}). Numeric values are given in @code{flt64} format; string
1558 values are literal characters string, padded on the right when
1559 necessary to fill out 8-byte units.
1561 @item 1: bytecode compression
1563 of the data record is divided into a series of 1-byte command
1564 codes. These codes have meanings as described below:
1568 Ignored. If the program writing the system file accumulates compressed
1569 data in blocks of fixed length, 0 bytes can be used to pad out extra
1570 bytes remaining at the end of a fixed-size block.
1574 value @var{code} - @var{bias}, where
1575 @var{code} is the value of the compression code and @var{bias} is the
1576 variable @code{bias} from the file header. For example,
1577 code 105 with bias 100.0 (the normal value) indicates a numeric variable
1580 A code of 0 (after subtracting the bias) in a string field encodes
1581 null bytes. This is unusual, since a string field normally encodes
1582 text data, but it exists in real system files.
1585 End of file. This code may or may not appear at the end of the data
1586 stream. PSPP always outputs this code but its use is not required.
1589 A numeric or string value that is not
1590 compressible. The value is stored in the 8 bytes following the
1591 current block of command bytes. If this value appears twice in a block
1592 of command bytes, then it indicates the second group of 8 bytes following the
1593 command bytes, and so on.
1596 An 8-byte string value that is all spaces.
1599 The system-missing value.
1602 The end of the 8-byte group of bytecodes is followed by any 8-byte
1603 blocks of non-compressible values indicated by code 253. After that
1604 follows another 8-byte group of bytecodes, then those bytecodes'
1605 non-compressible values. The pattern repeats to the end of the file
1606 or a code with value 252.
1608 @item 2: ZLIB compression
1609 The data record consists of the following, in order:
1613 ZLIB data header, 24 bytes long.
1616 One or more variable-length blocks of ZLIB compressed data.
1619 ZLIB data trailer, with a 24-byte fixed header plus an additional 24
1620 bytes for each preceding ZLIB compressed data block.
1623 The ZLIB data header has the following format:
1632 @item int64 zheader_ofs;
1633 The offset, in bytes, of the beginning of this structure within the
1636 @item int64 ztrailer_ofs;
1637 The offset, in bytes, of the first byte of the ZLIB data trailer.
1639 @item int64 ztrailer_len;
1640 The number of bytes in the ZLIB data trailer. This and the previous
1641 field sum to the size of the system file in bytes.
1644 The data header is followed by @code{(ztrailer_ofs - 24) / 24} ZLIB
1645 compressed data blocks. Each ZLIB compressed data block begins with a
1646 ZLIB header as specified in RFC@tie{}1950, e.g.@: hex bytes @code{78
1647 01} (the only header yet observed in practice). Each block
1648 decompresses to a fixed number of bytes (in practice only
1649 @code{0x3ff000}-byte blocks have been observed), except that the last
1650 block of data may be shorter. The last ZLIB compressed data block
1651 gends just before offset @code{ztrailer_ofs}.
1653 The result of ZLIB decompression is bytecode compressed data as
1654 described above for compression format 1.
1656 The ZLIB data trailer begins with the following 24-byte fixed header:
1666 @item int64 int_bias;
1667 The compression bias as a negative integer, e.g.@: if @code{bias} in
1668 the file header record is 100.0, then @code{int_bias} is @minus{}100
1669 (this is the only value yet observed in practice).
1672 Always observed to be zero.
1674 @item int32 block_size;
1675 The number of bytes in each ZLIB compressed data block, except
1676 possibly the last, following decompression. Only @code{0x3ff000} has
1677 been observed so far.
1679 @item int32 n_blocks;
1680 The number of ZLIB compressed data blocks, always exactly
1681 @code{(ztrailer_ofs - 24) / 24}.
1684 The fixed header is followed by @code{n_blocks} 24-byte ZLIB data
1685 block descriptors, each of which describes the compressed data block
1686 corresponding to its offset. Each block descriptor has the following
1690 int64 uncompressed_ofs;
1691 int64 compressed_ofs;
1692 int32 uncompressed_size;
1693 int32 compressed_size;
1697 @item int64 uncompressed_ofs;
1698 The offset, in bytes, that this block of data would have in a similar
1699 system file that uses compression format 1. This is
1700 @code{zheader_ofs} in the first block descriptor, and in each
1701 succeeding block descriptor it is the sum of the previous desciptor's
1702 @code{uncompressed_ofs} and @code{uncompressed_size}.
1704 @item int64 compressed_ofs;
1705 The offset, in bytes, of the actual beginning of this compressed data
1706 block. This is @code{zheader_ofs + 24} in the first block descriptor,
1707 and in each succeeding block descriptor it is the sum of the previous
1708 descriptor's @code{compressed_ofs} and @code{compressed_size}. The
1709 final block descriptor's @code{compressed_ofs} and
1710 @code{compressed_size} sum to @code{ztrailer_ofs}.
1712 @item int32 uncompressed_size;
1713 The number of bytes in this data block, after decompression. This is
1714 @code{block_size} in every data block except the last, which may be
1717 @item int32 compressed_size;
1718 The number of bytes in this data block, as stored compressed in this
1723 @setfilename ignored