1 @c PSPP - a program for statistical analysis.
2 @c Copyright (C) 2019 Free Software Foundation, Inc.
3 @c Permission is granted to copy, distribute and/or modify this document
4 @c under the terms of the GNU Free Documentation License, Version 1.3
5 @c or any later version published by the Free Software Foundation;
6 @c with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts.
7 @c A copy of the license is included in the section entitled "GNU
8 @c Free Documentation License".
11 @node System File Format
12 @chapter System File Format
14 An SPSS system file holds a set of cases and dictionary information
15 that describes how they may be interpreted. The system file format
16 dates back 40+ years and has evolved greatly over that time to support
17 new features, but in a way to facilitate interchange between even the
18 oldest and newest versions of software. This chapter describes the
21 System files use four data types: 8-bit characters, 32-bit integers,
23 and 64-bit floating points, called here @code{char}, @code{int32},
25 @code{flt64}, respectively. Data is not necessarily aligned on a word
26 or double-word boundary: the long variable name record (@pxref{Long
27 Variable Names Record}) and very long string records (@pxref{Very Long
28 String Record}) have arbitrary byte length and can therefore cause all
29 data coming after them in the file to be misaligned.
31 Integer data in system files may be big-endian or little-endian. A
32 reader may detect the endianness of a system file by examining
33 @code{layout_code} in the file header record
34 (@pxref{layout_code,,@code{layout_code}}).
36 Floating-point data in system files may nominally be in IEEE 754, IBM,
37 or VAX formats. A reader may detect the floating-point format in use
38 by examining @code{bias} in the file header record
39 (@pxref{bias,,@code{bias}}).
41 PSPP detects big-endian and little-endian integer formats in system
42 files and translates as necessary. PSPP also detects the
43 floating-point format in use, as well as the endianness of IEEE 754
44 floating-point numbers, and translates as needed. However, only IEEE
45 754 numbers with the same endianness as integer data in the same file
46 have actually been observed in system files, and it is likely that
47 other formats are obsolete or were never used.
49 System files use a few floating point values for special purposes:
53 The system-missing value is represented by the largest possible
54 negative number in the floating point format (@code{-DBL_MAX}).
57 HIGHEST is used as the high end of a missing value range with an
58 unbounded maximum. It is represented by the largest possible positive
59 number (@code{DBL_MAX}).
62 LOWEST is used as the low end of a missing value range with an
63 unbounded minimum. It was originally represented by the
64 second-largest negative number (in IEEE 754 format,
65 @code{0xffeffffffffffffe}). System files written by SPSS 21 and later
66 instead use the largest negative number (@code{-DBL_MAX}), the same
67 value as SYSMIS. This does not lead to ambiguity because LOWEST
68 appears in system files only in missing value ranges, which never
72 System files may use most character encodings based on an 8-bit unit.
73 UTF-16 and UTF-32, based on wider units, appear to be unacceptable.
74 @code{rec_type} in the file header record is sufficient to distinguish
75 between ASCII and EBCDIC based encodings. The best way to determine
76 the specific encoding in use is to consult the character encoding
77 record (@pxref{Character Encoding Record}), if present, and failing
78 that the @code{character_code} in the machine integer info record
79 (@pxref{Machine Integer Info Record}). The same encoding should be
80 used for the dictionary and the data in the file, although it is
81 possible to artificially synthesize files that use different encodings
82 (@pxref{Character Encoding Record}).
85 * System File Record Structure::
86 * File Header Record::
88 * Value Labels Records::
90 * Machine Integer Info Record::
91 * Machine Floating-Point Info Record::
92 * Multiple Response Sets Records::
93 * Extra Product Info Record::
94 * Variable Display Parameter Record::
95 * Variable Sets Record::
96 * Long Variable Names Record::
97 * Very Long String Record::
98 * Character Encoding Record::
99 * Long String Value Labels Record::
100 * Long String Missing Values Record::
101 * Data File and Variable Attributes Records::
102 * Extended Number of Cases Record::
103 * Other Informational Records::
104 * Dictionary Termination Record::
108 @node System File Record Structure
109 @section System File Record Structure
111 System files are divided into records with the following format:
118 This header does not identify the length of the @code{data} or any
119 information about what it contains, so the system file reader must
120 understand the format of @code{data} based on @code{type}. However,
121 records with type 7, called @dfn{extension records}, have a stricter
129 char data[size * count];
133 @item int32 rec_type;
134 Record type. Always set to 7.
137 Record subtype. This value identifies a particular kind of extension
141 The size of each piece of data that follows the header, in bytes.
142 Known extension records use 1, 4, or 8, for @code{char}, @code{int32},
143 and @code{flt64} format data, respectively.
146 The number of pieces of data that follow the header.
148 @item char data[size * count];
149 Data, whose format and interpretation depend on the subtype.
152 An extension record contains exactly @code{size * count} bytes of
153 data, which allows a reader that does not understand an extension
154 record to skip it. Extension records provide only nonessential
155 information, so this allows for files written by newer software to
156 preserve backward compatibility with older or less capable readers.
158 Records in a system file must appear in the following order:
168 All pairs of value labels records and value label variables records,
172 Document record, if present.
175 Extension (type 7) records, in ascending numerical order of their
178 System files written by SPSS include at most one of each kind of
179 extension record. This is generally true of system files written by
180 other software as well, with known exceptions noted below in the
181 individual sections about each type of record.
184 Dictionary termination record.
190 We advise authors of programs that read system files to tolerate
191 format variations. Various kinds of misformatting and corruption have
192 been observed in system files written by SPSS and other software
193 alike. In particular, because extension records provide nonessential
194 information, it is generally better to ignore an extension record
195 entirely than to refuse to read a system file.
197 The following sections describe the known kinds of records.
199 @node File Header Record
200 @section File Header Record
202 A system file begins with the file header, with the following format:
208 int32 nominal_case_size;
213 char creation_date[9];
214 char creation_time[8];
220 @item char rec_type[4];
221 Record type code, either @samp{$FL2} for system files with
222 uncompressed data or data compressed with simple bytecode compression,
223 or @samp{$FL3} for system files with ZLIB compressed data.
225 This is truly a character field that uses the character encoding as
226 other strings. Thus, in a file with an ASCII-based character encoding
227 this field contains @code{24 46 4c 32} or @code{24 46 4c 33}, and in a
228 file with an EBCDIC-based encoding this field contains @code{5b c6 d3
229 f2}. (No EBCDIC-based ZLIB-compressed files have been observed.)
231 @item char prod_name[60];
232 Product identification string. This always begins with the characters
233 @samp{@@(#) SPSS DATA FILE}. PSPP uses the remaining characters to
234 give its version and the operating system name; for example, @samp{GNU
235 pspp 0.1.4 - sparc-sun-solaris2.5.2}. The string is truncated if it
236 would be longer than 60 characters; otherwise it is padded on the right
239 The product name field allow readers to behave differently based on
240 quirks in the way that particular software writes system files.
241 @xref{Value Labels Records}, for the detail of the quirk that the PSPP
242 system file reader tolerates in files written by ReadStat, which has
243 @code{https://github.com/WizardMac/ReadStat} in @code{prod_name}.
246 @item int32 layout_code;
247 Normally set to 2, although a few system files have been spotted in
248 the wild with a value of 3 here. PSPP use this value to determine the
249 file's integer endianness (@pxref{System File Format}).
251 @item int32 nominal_case_size;
252 Number of data elements per case. This is the number of variables,
253 except that long string variables add extra data elements (one for every
254 8 characters after the first 8). However, string variables do not
255 contribute to this value beyond the first 255 bytes. Further, some
256 software always writes -1 or 0 in this field. In general, it is
257 unsafe for systems reading system files to rely upon this value.
259 @item int32 compression;
260 Set to 0 if the data in the file is not compressed, 1 if the data is
261 compressed with simple bytecode compression, 2 if the data is ZLIB
262 compressed. This field has value 2 if and only if @code{rec_type} is
265 @item int32 weight_index;
266 If one of the variables in the data set is used as a weighting
267 variable, set to the dictionary index of that variable, plus 1
268 (@pxref{Dictionary Index}). Otherwise, set to 0.
271 Set to the number of cases in the file if it is known, or -1 otherwise.
273 In the general case it is not possible to determine the number of cases
274 that will be output to a system file at the time that the header is
275 written. The way that this is dealt with is by writing the entire
276 system file, including the header, then seeking back to the beginning of
277 the file and writing just the @code{ncases} field. For files in which
278 this is not valid, the seek operation fails. In this case,
279 @code{ncases} remains -1.
283 Compression bias, ordinarily set to 100. Only integers between
284 @code{1 - bias} and @code{251 - bias} can be compressed.
286 By assuming that its value is 100, PSPP uses @code{bias} to determine
287 the file's floating-point format and endianness (@pxref{System File
288 Format}). If the compression bias is not 100, PSPP cannot auto-detect
289 the floating-point format and assumes that it is IEEE 754 format with
290 the same endianness as the system file's integers, which is correct
291 for all known system files.
293 @item char creation_date[9];
294 Date of creation of the system file, in @samp{dd mmm yy}
295 format, with the month as standard English abbreviations, using an
296 initial capital letter and following with lowercase. If the date is not
297 available then this field is arbitrarily set to @samp{01 Jan 70}.
299 @item char creation_time[8];
300 Time of creation of the system file, in @samp{hh:mm:ss}
301 format and using 24-hour time. If the time is not available then this
302 field is arbitrarily set to @samp{00:00:00}.
304 @item char file_label[64];
305 File label declared by the user, if any (@pxref{FILE LABEL,,,pspp,
306 PSPP Users Guide}). Padded on the right with spaces.
308 A product that identifies itself as @code{VOXCO INTERVIEWER 4.3} uses
309 CR-only line ends in this field, rather than the more usual LF-only or
312 @item char padding[3];
313 Ignored padding bytes to make the structure a multiple of 32 bits in
314 length. Set to zeros.
317 @node Variable Record
318 @section Variable Record
320 There must be one variable record for each numeric variable and each
321 string variable with width 8 bytes or less. String variables wider
322 than 8 bytes have one variable record for each 8 bytes, rounding up.
323 The first variable record for a long string specifies the variable's
324 correct dictionary information. Subsequent variable records for a
325 long string are filled with dummy information: a type of -1, no
326 variable label or missing values, print and write formats that are
327 ignored, and an empty string as name. A few system files have been
328 encountered that include a variable label on dummy variable records,
329 so readers should take care to parse dummy variable records in the
330 same way as other variable records.
332 @anchor{Dictionary Index}
333 The @dfn{dictionary index} of a variable is a 1-based offset in the set of
334 variable records, including dummy variable records for long string
335 variables. The first variable record has a dictionary index of 1, the
336 second has a dictionary index of 2, and so on.
338 The system file format does not directly support string variables
339 wider than 255 bytes. Such very long string variables are represented
340 by a number of narrower string variables. @xref{Very Long String
341 Record}, for details.
343 A system file should contain at least one variable and thus at least
344 one variable record, but system files have been observed in the wild
345 without any variables (thus, no data either).
351 int32 n_missing_values;
356 /* @r{Present only if @code{has_var_label} is 1.} */
360 /* @r{Present only if @code{n_missing_values} is nonzero}. */
361 flt64 missing_values[];
365 @item int32 rec_type;
366 Record type code. Always set to 2.
369 Variable type code. Set to 0 for a numeric variable. For a short
370 string variable or the first part of a long string variable, this is set
371 to the width of the string. For the second and subsequent parts of a
372 long string variable, set to -1, and the remaining fields in the
373 structure are ignored.
375 @item int32 has_var_label;
376 If this variable has a variable label, set to 1; otherwise, set to 0.
378 @item int32 n_missing_values;
379 If the variable has no missing values, set to 0. If the variable has
380 one, two, or three discrete missing values, set to 1, 2, or 3,
381 respectively. If the variable has a range for missing variables, set to
382 -2; if the variable has a range for missing variables plus a single
383 discrete value, set to -3.
385 A long string variable always has the value 0 here. A separate record
386 indicates missing values for long string variables (@pxref{Long String
387 Missing Values Record}).
390 Print format for this variable. See below.
393 Write format for this variable. See below.
396 Variable name. The variable name must begin with a capital letter or
397 the at-sign (@samp{@@}). Subsequent characters may also be digits, octothorpes
398 (@samp{#}), dollar signs (@samp{$}), underscores (@samp{_}), or full
399 stops (@samp{.}). The variable name is padded on the right with spaces.
401 The @samp{name} fields should be unique within a system file. System
402 files written by SPSS that contain very long string variables with
403 similar names sometimes contain duplicate names that are later
404 eliminated by resolving the very long string names (@pxref{Very Long
405 String Record}). PSPP handles duplicates by assigning them new,
408 @item int32 label_len;
409 This field is present only if @code{has_var_label} is set to 1. It is
410 set to the length, in characters, of the variable label. The
411 documented maximum length varies from 120 to 255 based on SPSS
412 version, but some files have been seen with longer labels. PSPP
413 accepts labels of any length.
416 This field is present only if @code{has_var_label} is set to 1. It has
417 length @code{label_len}, rounded up to the nearest multiple of 32 bits.
418 The first @code{label_len} characters are the variable's variable label.
420 @item flt64 missing_values[];
421 This field is present only if @code{n_missing_values} is nonzero. It
422 has the same number of 8-byte elements as the absolute value of
423 @code{n_missing_values}. Each element is interpreted as a number for
424 numeric variables (with HIGHEST and LOWEST indicated as described in
425 the chapter introduction). For string variables of width less than 8
426 bytes, elements are right-padded with spaces; for string variables
427 wider than 8 bytes, only the first 8 bytes of each missing value are
428 specified, with the remainder implicitly all spaces.
430 For discrete missing values, each element represents one missing
431 value. When a range is present, the first element denotes the minimum
432 value in the range, and the second element denotes the maximum value
433 in the range. When a range plus a value are present, the third
434 element denotes the additional discrete missing value.
437 @anchor{System File Output Formats}
438 The @code{print} and @code{write} members of sysfile_variable are output
439 formats coded into @code{int32} types. The least-significant byte
440 of the @code{int32} represents the number of decimal places, and the
441 next two bytes in order of increasing significance represent field width
442 and format type, respectively. The most-significant byte is not
443 used and should be set to zero.
445 Format types are defined as follows:
448 @multitable {Value} {@code{DATETIME}}
538 A few system files have been observed in the wild with invalid
539 @code{write} fields, in particular with value 0. Readers should
540 probably treat invalid @code{print} or @code{write} fields as some
543 @node Value Labels Records
544 @section Value Labels Records
546 The value label records documented in this section are used for
547 numeric and short string variables only. Long string variables may
548 have value labels, but their value labels are recorded using a
549 different record type (@pxref{Long String Value Labels Record}).
551 ReadStat (@pxref{File Header Record}) writes value labels that label a
552 single value more than once. In more detail, it emits value labels
553 whose values are longer than string variables' widths, that are
554 identical in the actual width of the variable, e.g.@: labels for
555 values @code{ABC123} and @code{ABC456} for a string variable with
556 width 3. For files written by this software, PSPP ignores such
559 The value label record has the following format:
565 /* @r{Repeated @code{n_label} times}. */
572 @item int32 rec_type;
573 Record type. Always set to 3.
575 @item int32 label_count;
576 Number of value labels present in this record.
579 The remaining fields are repeated @code{count} times. Each
580 repetition specifies one value label.
584 A numeric value or a short string value padded as necessary to 8 bytes
585 in length. Its type and width cannot be determined until the
586 following value label variables record (see below) is read.
588 @item char label_len;
589 The label's length, in bytes. The documented maximum length varies
590 from 60 to 120 based on SPSS version. PSPP supports value labels up
594 @code{label_len} bytes of the actual label, followed by up to 7 bytes
595 of padding to bring @code{label} and @code{label_len} together to a
596 multiple of 8 bytes in length.
599 The value label record is always immediately followed by a value label
600 variables record with the following format:
609 @item int32 rec_type;
610 Record type. Always set to 4.
612 @item int32 var_count;
613 Number of variables that the associated value labels from the value
614 label record are to be applied.
617 A list of 1-based dictionary indexes of variables to which to apply the value
618 labels (@pxref{Dictionary Index}). There are @code{var_count}
621 String variables wider than 8 bytes may not be specified in this list.
624 @node Document Record
625 @section Document Record
627 The document record, if present, has the following format:
636 @item int32 rec_type;
637 Record type. Always set to 6.
640 Number of lines of documents present. This should be greater than
641 zero, but ReadStats writes system files with zero @code{n_lines}.
643 @item char lines[][80];
644 Document lines. The number of elements is defined by @code{n_lines}.
645 Lines shorter than 80 characters are padded on the right with spaces.
648 @node Machine Integer Info Record
649 @section Machine Integer Info Record
651 The integer info record, if present, has the following format:
663 int32 version_revision;
665 int32 floating_point_rep;
666 int32 compression_code;
668 int32 character_code;
672 @item int32 rec_type;
673 Record type. Always set to 7.
676 Record subtype. Always set to 3.
679 Size of each piece of data in the data part, in bytes. Always set to 4.
682 Number of pieces of data in the data part. Always set to 8.
684 @item int32 version_major;
685 PSPP major version number. In version @var{x}.@var{y}.@var{z}, this
688 @item int32 version_minor;
689 PSPP minor version number. In version @var{x}.@var{y}.@var{z}, this
692 @item int32 version_revision;
693 PSPP version revision number. In version @var{x}.@var{y}.@var{z},
696 @item int32 machine_code;
697 Machine code. PSPP always set this field to value to -1, but other
700 @item int32 floating_point_rep;
701 Floating point representation code. For IEEE 754 systems this is 1.
702 IBM 370 sets this to 2, and DEC VAX E to 3.
704 @item int32 compression_code;
705 Compression code. Always set to 1, regardless of whether or how the
708 @item int32 endianness;
709 Machine endianness. 1 indicates big-endian, 2 indicates little-endian.
711 @item int32 character_code;
712 @anchor{character-code} Character code. The following values have
713 been actually observed in system files:
717 EBCDIC. Only one example has been observed.
720 7-bit ASCII. Old versions of SPSS for Unix and Windows always wrote
721 value 2 in this field, regardless of the encoding in use, so it is not
722 reliable and should be ignored.
728 ISO 8859-1 (IBM AIX code page number).
732 The @code{windows-874} code page for Thai.
735 The @code{windows-932} code page for Japanese (aka @code{Shift_JIS}).
738 The @code{windows-936} code page for simplified Chinese (aka @code{GBK}).
741 Probably @code{ks_c_5601-1987}, Unified Hangul Code.
744 The @code{big5} code page for traditional Chinese.
747 The @code{windows-1250} code page for Central European and Eastern
751 The @code{windows-1251} code page for Cyrillic languages.
754 The @code{windows-1252} code page for Western European languages.
757 The @code{windows-1253} code page for modern Greek.
760 The @code{windows-1254} code page for Turkish.
763 The @code{windows-1255} code page for Hebrew.
766 The @code{windows-1256} code page for Arabic script.
769 The @code{windows-1257} code page for Estonian, Latvian, and
773 The @code{windows-1258} code page for Vietnamese.
779 ISO 8859-1 (Latin-1).
782 ISO 8859-2 (Central European).
785 ISO 8895-9 (Latin-9).
788 The @code{euc-kr} code page for Korean.
794 The following additional values are known to be defined:
804 The most common values observed, from most to least common, are 1252,
807 Other Windows code page numbers are known to be generally valid.
809 Newer versions also write the character encoding as a string (see
810 @ref{Character Encoding Record}).
813 @node Machine Floating-Point Info Record
814 @section Machine Floating-Point Info Record
816 The floating-point info record, if present, has the following format:
832 @item int32 rec_type;
833 Record type. Always set to 7.
836 Record subtype. Always set to 4.
839 Size of each piece of data in the data part, in bytes. Always set to 8.
842 Number of pieces of data in the data part. Always set to 3.
845 @itemx flt64 highest;
847 The system missing value, the value used for HIGHEST in missing
848 values, and the value used for LOWEST in missing values, respectively.
849 @xref{System File Format}, for more information.
851 The SPSSWriter library in PHP, which identifies itself as @code{FOM
852 SPSS 1.0.0} in the file header record @code{prod_name} field, writes
853 unexpected values to these fields, but it uses the same values
854 consistently throughout the rest of the file.
857 @node Multiple Response Sets Records
858 @section Multiple Response Sets Records
860 The system file format has two different types of records that
861 represent multiple response sets (@pxref{MRSETS,,,pspp, PSPP Users
862 Guide}). The first type of record describes multiple response sets
863 that can be understood by SPSS before version 14. The second type of
864 record, with a closely related format, is used for multiple dichotomy
865 sets that use the CATEGORYLABELS=COUNTEDVALUES feature added in
875 /* @r{Exactly @code{count} bytes of data.} */
880 @item int32 rec_type;
881 Record type. Always set to 7.
884 Record subtype. Set to 7 for records that describe multiple response
885 sets understood by SPSS before version 14, or to 19 for records that
886 describe dichotomy sets that use the CATEGORYLABELS=COUNTEDVALUES
887 feature added in version 14.
890 The size of each element in the @code{mrsets} member. Always set to 1.
893 The total number of bytes in @code{mrsets}.
896 Zero or more line feeds (byte 0x0a), followed by a series of multiple
897 response sets, each of which consists of the following:
901 The set's name (an identifier that begins with @samp{$}), in mixed
902 upper and lower case.
905 An equals sign (@samp{=}).
908 @samp{C} for a multiple category set, @samp{D} for a multiple
909 dichotomy set with CATEGORYLABELS=VARLABELS, or @samp{E} for a
910 multiple dichotomy set with CATEGORYLABELS=COUNTEDVALUES.
913 For a multiple dichotomy set with CATEGORYLABELS=COUNTEDVALUES, a
914 space, followed by a number expressed as decimal digits, followed by a
915 space. If LABELSOURCE=VARLABEL was specified on MRSETS, then the
916 number is 11; otherwise it is 1.@footnote{This part of the format may
917 not be fully understood, because only a single example of each
918 possibility has been examined.}
921 For either kind of multiple dichotomy set, the counted value, as a
922 positive integer count specified as decimal digits, followed by a
923 space, followed by as many string bytes as specified in the count. If
924 the set contains numeric variables, the string consists of the counted
925 integer value expressed as decimal digits. If the set contains string
926 variables, the string contains the counted string value. Either way,
927 the string may be padded on the right with spaces (older versions of
928 SPSS seem to always pad to a width of 8 bytes; newer versions don't).
934 The multiple response set's label, using the same format as for the
935 counted value for multiple dichotomy sets. A string of length 0 means
936 that the set does not have a label. A string of length 0 is also
937 written if LABELSOURCE=VARLABEL was specified.
940 The short names of the variables in the set, converted to lowercase,
941 each preceded by a single space.
943 Even though a multiple response set must have at least two variables,
944 some system files contain multiple response sets with no variables or
945 one variable. The source and meaning of these multiple response sets is
946 unknown. (Perhaps they arise from creating a multiple response set
947 then deleting all the variables that it contains?)
950 One line feed (byte 0x0a). Sometimes multiple, even hundreds, of line
955 Example: Given appropriate variable definitions, consider the
956 following MRSETS command:
959 MRSETS /MCGROUP NAME=$a LABEL='my mcgroup' VARIABLES=a b c
960 /MDGROUP NAME=$b VARIABLES=g e f d VALUE=55
961 /MDGROUP NAME=$c LABEL='mdgroup #2' VARIABLES=h i j VALUE='Yes'
962 /MDGROUP NAME=$d LABEL='third mdgroup' CATEGORYLABELS=COUNTEDVALUES
963 VARIABLES=k l m VALUE=34
964 /MDGROUP NAME=$e CATEGORYLABELS=COUNTEDVALUES LABELSOURCE=VARLABEL
965 VARIABLES=n o p VALUE='choice'.
968 The above would generate the following multiple response set record of
972 $a=C 10 my mcgroup a b c
974 $c=D3 Yes 10 mdgroup #2 h i j
977 It would also generate the following multiple response set record with
981 $d=E 1 2 34 13 third mdgroup k l m
982 $e=E 11 6 choice 0 n o p
985 @node Extra Product Info Record
986 @section Extra Product Info Record
988 This optional record appears to contain a text string that describes
989 the program that wrote the file and the source of the data. (This is
990 redundant with the file label and product info found in the file
1000 /* @r{Exactly @code{count} bytes of data.} */
1005 @item int32 rec_type;
1006 Record type. Always set to 7.
1008 @item int32 subtype;
1009 Record subtype. Always set to 10.
1012 The size of each element in the @code{info} member. Always set to 1.
1015 The total number of bytes in @code{info}.
1018 A text string. A product that identifies itself as @code{VOXCO
1019 INTERVIEWER 4.3} uses CR-only line ends in this field, rather than the
1020 more usual LF-only or CR LF line ends.
1023 @node Variable Display Parameter Record
1024 @section Variable Display Parameter Record
1026 The variable display parameter record, if present, has the following
1036 /* @r{Repeated @code{count} times}. */
1038 int32 width; /* @r{Not always present.} */
1043 @item int32 rec_type;
1044 Record type. Always set to 7.
1046 @item int32 subtype;
1047 Record subtype. Always set to 11.
1050 The size of @code{int32}. Always set to 4.
1053 The number of sets of variable display parameters (ordinarily the
1054 number of variables in the dictionary), times 2 or 3.
1057 The remaining members are repeated @code{count} times, in the same
1058 order as the variable records. No element corresponds to variable
1059 records that continue long string variables. The meanings of these
1060 members are as follows:
1063 @item int32 measure;
1064 The measurement level of the variable:
1076 An ``unknown'' @code{measure} of 0 means that the variable was created
1077 in some way that doesn't make the measurement level clear, e.g.@: with
1078 a @code{COMPUTE} transformation. PSPP sets the measurement level the
1079 first time it reads the data using the rules documented in
1080 @ref{Measurement Level,,,pspp, PSPP Users Guide}, so this should
1084 The width of the display column for the variable in characters.
1086 This field is present if @var{count} is 3 times the number of
1087 variables in the dictionary. It is omitted if @var{count} is 2 times
1088 the number of variables.
1090 @item int32 alignment;
1091 The alignment of the variable for display purposes:
1103 @node Variable Sets Record
1104 @section Variable Sets Record
1106 The SPSS GUI offers users the ability to arrange variables in sets.
1107 Users may enable and disable sets individually, and the data editor
1108 and analysis dialog boxes only show enabled sets. Syntax does not use
1111 The variable sets record, if present, has the following format:
1120 /* @r{Exactly @code{count} bytes of text.} */
1125 @item int32 rec_type;
1126 Record type. Always set to 7.
1128 @item int32 subtype;
1129 Record subtype. Always set to 5.
1135 The total number of bytes in @code{text}.
1138 The variable sets, in a text-based format.
1140 Each variable set occupies one line of text, each of which ends with a
1141 line feed (byte 0x0a), optionally preceded by a carriage return (byte
1144 Each line begins with the name of the variable set, followed by an
1145 equals sign (@samp{=}) and a space (byte 0x20), followed by the long
1146 variable names of the members of the set, separated by spaces. A
1147 variable set may be empty, in which case the equals sign and the space
1148 following it are still present.
1151 @node Long Variable Names Record
1152 @section Long Variable Names Record
1154 If present, the long variable names record has the following format:
1163 /* @r{Exactly @code{count} bytes of data.} */
1164 char var_name_pairs[];
1168 @item int32 rec_type;
1169 Record type. Always set to 7.
1171 @item int32 subtype;
1172 Record subtype. Always set to 13.
1175 The size of each element in the @code{var_name_pairs} member. Always set to 1.
1178 The total number of bytes in @code{var_name_pairs}.
1180 @item char var_name_pairs[];
1181 A list of @var{key}--@var{value} tuples, where @var{key} is the name
1182 of a variable, and @var{value} is its long variable name.
1183 The @var{key} field is at most 8 bytes long and must match the
1184 name of a variable which appears in the variable record (@pxref{Variable
1186 The @var{value} field is at most 64 bytes long.
1187 The @var{key} and @var{value} fields are separated by a @samp{=} byte.
1188 Each tuple is separated by a byte whose value is 09. There is no
1189 trailing separator following the last tuple.
1190 The total length is @code{count} bytes.
1193 @node Very Long String Record
1194 @section Very Long String Record
1196 Old versions of SPSS limited string variables to a width of 255 bytes.
1197 For backward compatibility with these older versions, the system file
1198 format represents a string longer than 255 bytes, called a @dfn{very
1199 long string}, as a collection of strings no longer than 255 bytes
1200 each. The strings concatenated to make a very long string are called
1201 its @dfn{segments}; for consistency, variables other than very long
1202 strings are considered to have a single segment.
1204 A very long string with a width of @var{w} has @var{n} =
1205 (@var{w} + 251) / 252 segments, that is, one segment for every
1206 252 bytes of width, rounding up. It would be logical, then, for each
1207 of the segments except the last to have a width of 252 and the last
1208 segment to have the remainder, but this is not the case. In fact,
1209 each segment except the last has a width of 255 bytes. The last
1210 segment has width @var{w} - (@var{n} - 1) * 252; some versions
1211 of SPSS make it slightly wider, but not wide enough to make the last
1212 segment require another 8 bytes of data.
1214 Data is packed tightly into segments of a very long string, 255 bytes
1215 per segment. Because 255 bytes of segment data are allocated for
1216 every 252 bytes of the very long string's width (approximately), some
1217 unused space is left over at the end of the allocated segments. Data
1218 in unused space is ignored.
1220 Example: Consider a very long string of width 20,000. Such a very
1221 long string has 20,000 / 252 = 80 (rounding up) segments. The first
1222 79 segments have width 255; the last segment has width 20,000 - 79 *
1223 252 = 92 or slightly wider (up to 96 bytes, the next multiple of 8).
1224 The very long string's data is actually stored in the 19,890 bytes in
1225 the first 78 segments, plus the first 110 bytes of the 79th segment
1226 (19,890 + 110 = 20,000). The remaining 145 bytes of the 79th segment
1227 and all 92 bytes of the 80th segment are unused.
1229 The very long string record explains how to stitch together segments
1230 to obtain very long string data. For each of the very long string
1231 variables in the dictionary, it specifies the name of its first
1232 segment's variable and the very long string variable's actual width.
1233 The remaining segments immediately follow the named variable in the
1234 system file's dictionary.
1236 The very long string record, which is present only if the system file
1237 contains very long string variables, has the following format:
1246 /* @r{Exactly @code{count} bytes of data.} */
1247 char string_lengths[];
1251 @item int32 rec_type;
1252 Record type. Always set to 7.
1254 @item int32 subtype;
1255 Record subtype. Always set to 14.
1258 The size of each element in the @code{string_lengths} member. Always set to 1.
1261 The total number of bytes in @code{string_lengths}.
1263 @item char string_lengths[];
1264 A list of @var{key}--@var{value} tuples, where @var{key} is the name
1265 of a variable, and @var{value} is its length.
1266 The @var{key} field is at most 8 bytes long and must match the
1267 name of a variable which appears in the variable record (@pxref{Variable
1269 The @var{value} field is exactly 5 bytes long. It is a zero-padded,
1270 ASCII-encoded string that is the length of the variable.
1271 The @var{key} and @var{value} fields are separated by a @samp{=} byte.
1272 Tuples are delimited by a two-byte sequence @{00, 09@}.
1273 After the last tuple, there may be a single byte 00, or @{00, 09@}.
1274 The total length is @code{count} bytes.
1277 @node Character Encoding Record
1278 @section Character Encoding Record
1280 This record, if present, indicates the character encoding for string data,
1281 long variable names, variable labels, value labels and other strings in the
1291 /* @r{Exactly @code{count} bytes of data.} */
1296 @item int32 rec_type;
1297 Record type. Always set to 7.
1299 @item int32 subtype;
1300 Record subtype. Always set to 20.
1303 The size of each element in the @code{encoding} member. Always set to 1.
1306 The total number of bytes in @code{encoding}.
1308 @item char encoding[];
1309 The name of the character encoding. Normally this will be an official
1310 IANA character set name or alias. See
1311 @url{http://www.iana.org/assignments/character-sets}. Character set
1312 names are not case-sensitive, and SPSS is not consistent, e.g.@: both
1313 @code{windows-1251} and @code{WINDOWS-1252} have both been observed,
1314 as have @code{Big5} and @code{BIG5}.
1317 This record is not present in files generated by older software. See
1318 also the @code{character_code} field in the machine integer info
1319 record (@pxref{character-code}).
1321 The following character encoding names have been observed. The names
1322 are shown in lowercase, even though they were not always in lowercase
1323 in the file. Alternative names for the same encoding are, when known,
1324 listed together. For each encoding, the @code{character_code} values
1325 that they were observed paired with are also listed. First, the
1326 following are strictly single-byte, ASCII-compatible encodings:
1329 @item (encoding record missing)
1330 0, 2, 3, 874, 1250, 1251, 1252, 1253, 1254, 1255, 1256, 20127, 28591,
1333 @item @code{ansi_x3.4-1968}
1337 @item @code{cp28605}
1343 @item @code{iso-8859-1}
1346 @item @code{windows-874}
1349 @item @code{windows-1250}
1352 @item @code{windows-1251}
1356 @itemx @code{windows-1252}
1360 @itemx @code{windows-1253}
1363 @item @code{windows-1254}
1366 @item @code{windows-1255}
1369 @item @code{windows-1256}
1372 @item @code{windows-1257}
1375 @item @code{windows-1258}
1379 The others are multibyte encodings, in which some code points occupy a
1380 single byte and others multiple bytes. The following multibyte
1381 encodings are ``ASCII compatible,'' that is, they use ASCII values
1382 only to indicate ASCII:
1385 @item (encoding record missing)
1392 0, 2, 1250, 1251, 1252, 1256, 65001
1395 The following multibyte encodings are not ASCII compatible, that is,
1396 while they encode ASCII characters as their native values, they also
1397 use ASCII values as second or later bytes in multibyte sequences:
1400 @item (encoding record missing)
1411 @itemx @code{windows-31j}
1415 As the tables above show, when the character encoding record and the
1416 machine integer info record are both present, they can contradict each
1417 other. Observations show that, in this case, the character encoding
1418 record should be honored.
1420 If, for testing purposes, a file is crafted with different
1421 @code{character_code} and @code{encoding}, it seems that
1422 @code{character_code} controls the encoding for all strings in the
1423 system file before the dictionary termination record, including
1424 strings in data (e.g.@: string missing values), and @code{encoding}
1425 controls the encoding for strings following the dictionary termination
1428 @node Long String Value Labels Record
1429 @section Long String Value Labels Record
1431 This record, if present, specifies value labels for long string
1441 /* @r{Repeated up to exactly @code{count} bytes.} */
1446 long_string_label labels[];
1450 @item int32 rec_type;
1451 Record type. Always set to 7.
1453 @item int32 subtype;
1454 Record subtype. Always set to 21.
1460 The number of bytes following the header until the next header.
1462 @item int32 var_name_len;
1463 @itemx char var_name[];
1464 The number of bytes in the name of the variable that has long string
1465 value labels, plus the variable name itself, which consists of exactly
1466 @code{var_name_len} bytes. The variable name is not padded to any
1467 particular boundary, nor is it null-terminated.
1469 @item int32 var_width;
1470 The width of the variable, in bytes, which will be between 9 and
1473 @item int32 n_labels;
1474 @itemx long_string_label labels[];
1475 The long string labels themselves. The @code{labels} array contains
1476 exactly @code{n_labels} elements, each of which has the following
1487 @item int32 value_len;
1488 @itemx char value[];
1489 The string value being labeled. @code{value_len} is the number of
1490 bytes in @code{value}; it is equal to @code{var_width}. The
1491 @code{value} array is not padded or null-terminated.
1493 @item int32 label_len;
1494 @itemx char label[];
1495 The label for the string value. @code{label_len}, which must be
1496 between 0 and 120, is the number of bytes in @code{label}. The
1497 @code{label} array is not padded or null-terminated.
1501 @node Long String Missing Values Record
1502 @section Long String Missing Values Record
1504 This record, if present, specifies missing values for long string
1514 /* @r{Repeated up to exactly @code{count} bytes.} */
1517 char n_missing_values;
1519 char values[value_len * n_missing_values];
1523 @item int32 rec_type;
1524 Record type. Always set to 7.
1526 @item int32 subtype;
1527 Record subtype. Always set to 22.
1533 The number of bytes following the header until the next header.
1535 @item int32 var_name_len;
1536 @itemx char var_name[];
1537 The number of bytes in the name of the long string variable that has
1538 missing values, plus the variable name itself, which consists of
1539 exactly @code{var_name_len} bytes. The variable name is not padded to
1540 any particular boundary, nor is it null-terminated.
1542 @item char n_missing_values;
1543 The number of missing values, either 1, 2, or 3. (This is, unusually,
1544 a single byte instead of a 32-bit number.)
1546 @item int32 value_len;
1547 The length of each missing value string, in bytes. This value should
1548 be 8, because long string variables are at least 8 bytes wide (by
1549 definition), only the first 8 bytes of a long string variable's
1550 missing values are allowed to be non-spaces, and any spaces within the
1551 first 8 bytes are included in the missing value here.
1553 @item char values[value_len * n_missing_values]
1554 The missing values themselves, without any padding or null
1558 An earlier version of this document stated that @code{value_len} was
1559 repeated before each of the missing values, so that there was an extra
1560 @code{int32} value of 8 before each missing value after the first.
1561 Old versions of PSPP wrote data files in this format. Readers can
1562 tolerate this mistake, if they wish, by noticing and skipping the
1563 extra @code{int32} values, which wouldn't ordinarily occur in strings.
1565 @node Data File and Variable Attributes Records
1566 @section Data File and Variable Attributes Records
1568 The data file and variable attributes records represent custom
1569 attributes for the system file or for individual variables in the
1570 system file, as defined on the DATAFILE ATTRIBUTE (@pxref{DATAFILE
1571 ATTRIBUTE,,,pspp, PSPP Users Guide}) and VARIABLE ATTRIBUTE commands
1572 (@pxref{VARIABLE ATTRIBUTE,,,pspp, PSPP Users Guide}), respectively.
1581 /* @r{Exactly @code{count} bytes of data.} */
1586 @item int32 rec_type;
1587 Record type. Always set to 7.
1589 @item int32 subtype;
1590 Record subtype. Always set to 17 for a data file attribute record or
1591 to 18 for a variable attributes record.
1594 The size of each element in the @code{attributes} member. Always set to 1.
1597 The total number of bytes in @code{attributes}.
1599 @item char attributes[];
1600 The attributes, in a text-based format.
1602 In record subtype 17, this field contains a single attribute set. An
1603 attribute set is a sequence of one or more attributes concatenated
1604 together. Each attribute consists of a name, which has the same
1605 syntax as a variable name, followed by, inside parentheses, a sequence
1606 of one or more values. Each value consists of a string enclosed in
1607 single quotes (@code{'}) followed by a line feed (byte 0x0a). A value
1608 may contain single quote characters, which are not themselves escaped
1609 or quoted or required to be present in pairs. There is no apparent
1610 way to embed a line feed in a value. There is no distinction between
1611 an attribute with a single value and an attribute array with one
1614 In record subtype 18, this field contains a sequence of one or more
1615 variable attribute sets. If more than one variable attribute set is
1616 present, each one after the first is delimited from the previous by
1617 @code{/}. Each variable attribute set consists of a long
1619 followed by @code{:}, followed by an attribute set with the same
1620 syntax as on record subtype 17.
1622 System files written by @code{Stata 14.1/-savespss- 1.77 by
1623 S.Radyakin} may include multiple records with subtype 18, one per
1624 variable that has variable attributes.
1626 The total length is @code{count} bytes.
1631 A system file produced with the following VARIABLE ATTRIBUTE commands
1635 VARIABLE ATTRIBUTE VARIABLES=dummy ATTRIBUTE=fred[1]('23') fred[2]('34').
1636 VARIABLE ATTRIBUTE VARIABLES=dummy ATTRIBUTE=bert('123').
1640 will contain a variable attribute record with the following contents:
1643 0000 07 00 00 00 12 00 00 00 01 00 00 00 22 00 00 00 |............"...|
1644 0010 64 75 6d 6d 79 3a 66 72 65 64 28 27 32 33 27 0a |dummy:fred('23'.|
1645 0020 27 33 34 27 0a 29 62 65 72 74 28 27 31 32 33 27 |'34'.)bert('123'|
1653 @node Variable Roles
1654 @subsection Variable Roles
1656 A variable's role is represented as an attribute named @code{$@@Role}.
1657 This attribute has a single element whose values and their meanings
1662 Input. This, the default, is the most common role.
1675 @node Extended Number of Cases Record
1676 @section Extended Number of Cases Record
1678 The file header record expresses the number of cases in the system
1679 file as an int32 (@pxref{File Header Record}). This record allows the
1680 number of cases in the system file to be expressed as a 64-bit number.
1692 @item int32 rec_type;
1693 Record type. Always set to 7.
1695 @item int32 subtype;
1696 Record subtype. Always set to 16.
1699 Size of each element. Always set to 8.
1702 Number of pieces of data in the data part. Alway set to 2.
1704 @item int64 unknown;
1705 Meaning unknown. Always set to 1.
1707 @item int64 ncases64;
1708 Number of cases in the file as a 64-bit integer. Presumably this
1709 could be -1 to indicate that the number of cases is unknown, for the
1710 same reason as @code{ncases} in the file header record, but this has
1711 not been observed in the wild.
1714 @node Other Informational Records
1715 @section Other Informational Records
1717 This chapter documents many specific types of extension records are
1718 documented here, but others are known to exist. PSPP ignores unknown
1719 extension records when reading system files.
1721 The following extension record subtypes have also been observed, with
1722 the following believed meanings:
1726 Date info, probably related to USE (according to Aapi H@"am@"al@"ainen).
1729 A UUID in the format described in RFC 4122. Only two examples
1730 observed, both written by SPSS 13, and in each case the UUID contained
1731 both upper and lower case.
1734 XML that describes how data in the file should be displayed on-screen.
1737 @node Dictionary Termination Record
1738 @section Dictionary Termination Record
1740 The dictionary termination record separates all other records from the
1749 @item int32 rec_type;
1750 Record type. Always set to 999.
1753 Ignored padding. Should be set to 0.
1757 @section Data Record
1759 The data record must follow all other records in the system file.
1760 Every system file must have a data record that specifies data for at
1761 least one case. The format of the data record varies depending on the
1762 value of @code{compression} in the file header record:
1765 @item 0: no compression
1766 Data is arranged as a series of 8-byte elements.
1767 Each element corresponds to
1768 the variable declared in the respective variable record (@pxref{Variable
1769 Record}). Numeric values are given in @code{flt64} format; string
1770 values are literal characters string, padded on the right when
1771 necessary to fill out 8-byte units.
1773 @item 1: bytecode compression
1775 of the data record is divided into a series of 1-byte command
1776 codes. These codes have meanings as described below:
1780 Ignored. If the program writing the system file accumulates compressed
1781 data in blocks of fixed length, 0 bytes can be used to pad out extra
1782 bytes remaining at the end of a fixed-size block.
1786 value @var{code} - @var{bias}, where
1787 @var{code} is the value of the compression code and @var{bias} is the
1788 variable @code{bias} from the file header. For example,
1789 code 105 with bias 100.0 (the normal value) indicates a numeric variable
1792 A code of 0 (after subtracting the bias) in a string field encodes
1793 null bytes. This is unusual, since a string field normally encodes
1794 text data, but it exists in real system files.
1797 End of file. This code may or may not appear at the end of the data
1798 stream. PSPP always outputs this code but its use is not required.
1801 A numeric or string value that is not
1802 compressible. The value is stored in the 8 bytes following the
1803 current block of command bytes. If this value appears twice in a block
1804 of command bytes, then it indicates the second group of 8 bytes following the
1805 command bytes, and so on.
1808 An 8-byte string value that is all spaces.
1811 The system-missing value.
1814 The end of the 8-byte group of bytecodes is followed by any 8-byte
1815 blocks of non-compressible values indicated by code 253. After that
1816 follows another 8-byte group of bytecodes, then those bytecodes'
1817 non-compressible values. The pattern repeats to the end of the file
1818 or a code with value 252.
1820 @item 2: ZLIB compression
1821 The data record consists of the following, in order:
1825 ZLIB data header, 24 bytes long.
1828 One or more variable-length blocks of ZLIB compressed data.
1831 ZLIB data trailer, with a 24-byte fixed header plus an additional 24
1832 bytes for each preceding ZLIB compressed data block.
1835 The ZLIB data header has the following format:
1844 @item int64 zheader_ofs;
1845 The offset, in bytes, of the beginning of this structure within the
1848 @item int64 ztrailer_ofs;
1849 The offset, in bytes, of the first byte of the ZLIB data trailer.
1851 @item int64 ztrailer_len;
1852 The number of bytes in the ZLIB data trailer. This and the previous
1853 field sum to the size of the system file in bytes.
1856 The data header is followed by @code{(ztrailer_len - 24) / 24} ZLIB
1857 compressed data blocks. Each ZLIB compressed data block begins with a
1858 ZLIB header as specified in RFC@tie{}1950, e.g.@: hex bytes @code{78
1859 01} (the only header yet observed in practice). Each block
1860 decompresses to a fixed number of bytes (in practice only
1861 @code{0x3ff000}-byte blocks have been observed), except that the last
1862 block of data may be shorter. The last ZLIB compressed data block
1863 gends just before offset @code{ztrailer_ofs}.
1865 The result of ZLIB decompression is bytecode compressed data as
1866 described above for compression format 1.
1868 The ZLIB data trailer begins with the following 24-byte fixed header:
1878 @item int64 int_bias;
1879 The compression bias as a negative integer, e.g.@: if @code{bias} in
1880 the file header record is 100.0, then @code{int_bias} is @minus{}100
1881 (this is the only value yet observed in practice).
1884 Always observed to be zero.
1886 @item int32 block_size;
1887 The number of bytes in each ZLIB compressed data block, except
1888 possibly the last, following decompression. Only @code{0x3ff000} has
1889 been observed so far.
1891 @item int32 n_blocks;
1892 The number of ZLIB compressed data blocks, always exactly
1893 @code{(ztrailer_len - 24) / 24}.
1896 The fixed header is followed by @code{n_blocks} 24-byte ZLIB data
1897 block descriptors, each of which describes the compressed data block
1898 corresponding to its offset. Each block descriptor has the following
1902 int64 uncompressed_ofs;
1903 int64 compressed_ofs;
1904 int32 uncompressed_size;
1905 int32 compressed_size;
1909 @item int64 uncompressed_ofs;
1910 The offset, in bytes, that this block of data would have in a similar
1911 system file that uses compression format 1. This is
1912 @code{zheader_ofs} in the first block descriptor, and in each
1913 succeeding block descriptor it is the sum of the previous desciptor's
1914 @code{uncompressed_ofs} and @code{uncompressed_size}.
1916 @item int64 compressed_ofs;
1917 The offset, in bytes, of the actual beginning of this compressed data
1918 block. This is @code{zheader_ofs + 24} in the first block descriptor,
1919 and in each succeeding block descriptor it is the sum of the previous
1920 descriptor's @code{compressed_ofs} and @code{compressed_size}. The
1921 final block descriptor's @code{compressed_ofs} and
1922 @code{compressed_size} sum to @code{ztrailer_ofs}.
1924 @item int32 uncompressed_size;
1925 The number of bytes in this data block, after decompression. This is
1926 @code{block_size} in every data block except the last, which may be
1929 @item int32 compressed_size;
1930 The number of bytes in this data block, as stored compressed in this
1935 @setfilename ignored