1 @node System File Format
2 @appendix System File Format
4 A system file encapsulates a set of cases and dictionary information
5 that describes how they may be interpreted. This chapter describes
6 the format of a system file.
8 System files use four data types: 8-bit characters, 32-bit integers,
10 and 64-bit floating points, called here @code{char}, @code{int32},
12 @code{flt64}, respectively. Data is not necessarily aligned on a word
13 or double-word boundary: the long variable name record (@pxref{Long
14 Variable Names Record}) and very long string records (@pxref{Very Long
15 String Record}) have arbitrary byte length and can therefore cause all
16 data coming after them in the file to be misaligned.
18 Integer data in system files may be big-endian or little-endian. A
19 reader may detect the endianness of a system file by examining
20 @code{layout_code} in the file header record
21 (@pxref{layout_code,,@code{layout_code}}).
23 Floating-point data in system files may nominally be in IEEE 754, IBM,
24 or VAX formats. A reader may detect the floating-point format in use
25 by examining @code{bias} in the file header record
26 (@pxref{bias,,@code{bias}}).
28 PSPP detects big-endian and little-endian integer formats in system
29 files and translates as necessary. PSPP also detects the
30 floating-point format in use, as well as the endianness of IEEE 754
31 floating-point numbers, and translates as needed. However, only IEEE
32 754 numbers with the same endianness as integer data in the same file
33 have actually been observed in system files, and it is likely that
34 other formats are obsolete or were never used.
36 System files use a few floating point values for special purposes:
40 The system-missing value is represented by the largest possible
41 negative number in the floating point format (@code{-DBL_MAX}).
44 HIGHEST is used as the high end of a missing value range with an
45 unbounded maximum. It is represented by the largest possible positive
46 number (@code{DBL_MAX}).
49 LOWEST is used as the low end of a missing value range with an
50 unbounded minimum. It was originally represented by the
51 second-largest negative number (in IEEE 754 format,
52 @code{0xffeffffffffffffe}). System files written by SPSS 21 and later
53 instead use the largest negative number (@code{-DBL_MAX}), the same
54 value as SYSMIS. This does not lead to ambiguity because LOWEST
55 appears in system files only in missing value ranges, which never
59 System files may use most character encodings based on an 8-bit unit.
60 UTF-16 and UTF-32, based on wider units, appear to be unacceptable.
61 @code{rec_type} in the file header record is sufficient to distinguish
62 between ASCII and EBCDIC based encodings. The best way to determine
63 the specific encoding in use is to consult the character encoding
64 record (@pxref{Character Encoding Record}), if present, and failing
65 that the @code{character_code} in the machine integer info record
66 (@pxref{Machine Integer Info Record}). The same encoding should be
67 used for the dictionary and the data in the file, although it is
68 possible to artificially synthesize files that use different encodings
69 (@pxref{Character Encoding Record}).
72 * System File Record Structure::
73 * File Header Record::
75 * Value Labels Records::
77 * Machine Integer Info Record::
78 * Machine Floating-Point Info Record::
79 * Multiple Response Sets Records::
80 * Extra Product Info Record::
81 * Variable Display Parameter Record::
82 * Long Variable Names Record::
83 * Very Long String Record::
84 * Character Encoding Record::
85 * Long String Value Labels Record::
86 * Long String Missing Values Record::
87 * Data File and Variable Attributes Records::
88 * Extended Number of Cases Record::
89 * Other Informational Records::
90 * Dictionary Termination Record::
92 * Encrypted System Files::
95 @node System File Record Structure
96 @section System File Record Structure
98 System files are divided into records with the following format:
105 This header does not identify the length of the @code{data} or any
106 information about what it contains, so the system file reader must
107 understand the format of @code{data} based on @code{type}. However,
108 records with type 7, called @dfn{extension records}, have a stricter
116 char data[size * count];
120 @item int32 rec_type;
121 Record type. Always set to 7.
124 Record subtype. This value identifies a particular kind of extension
128 The size of each piece of data that follows the header, in bytes.
129 Known extension records use 1, 4, or 8, for @code{char}, @code{int32},
130 and @code{flt64} format data, respectively.
133 The number of pieces of data that follow the header.
135 @item char data[size * count];
136 Data, whose format and interpretation depend on the subtype.
139 An extension record contains exactly @code{size * count} bytes of
140 data, which allows a reader that does not understand an extension
141 record to skip it. Extension records provide only nonessential
142 information, so this allows for files written by newer software to
143 preserve backward compatibility with older or less capable readers.
145 Records in a system file must appear in the following order:
155 All pairs of value labels records and value label variables records,
159 Document record, if present.
162 Extension (type 7) records, in ascending numerical order of their
166 Dictionary termination record.
172 We advise authors of programs that read system files to tolerate
173 format variations. Various kinds of misformatting and corruption have
174 been observed in system files written by SPSS and other software
175 alike. In particular, because extension records provide nonessential
176 information, it is generally better to ignore an extension record
177 entirely than to refuse to read a system file.
179 The following sections describe the known kinds of records.
181 @node File Header Record
182 @section File Header Record
184 A system file begins with the file header, with the following format:
190 int32 nominal_case_size;
195 char creation_date[9];
196 char creation_time[8];
202 @item char rec_type[4];
203 Record type code, either @samp{$FL2} for system files with
204 uncompressed data or data compressed with simple bytecode compression,
205 or @samp{$FL3} for system files with ZLIB compressed data.
207 This is truly a character field that uses the character encoding as
208 other strings. Thus, in a file with an ASCII-based character encoding
209 this field contains @code{24 46 4c 32} or @code{24 46 4c 33}, and in a
210 file with an EBCDIC-based encoding this field contains @code{5b c6 d3
211 f2}. (No EBCDIC-based ZLIB-compressed files have been observed.)
213 @item char prod_name[60];
214 Product identification string. This always begins with the characters
215 @samp{@@(#) SPSS DATA FILE}. PSPP uses the remaining characters to
216 give its version and the operating system name; for example, @samp{GNU
217 pspp 0.1.4 - sparc-sun-solaris2.5.2}. The string is truncated if it
218 would be longer than 60 characters; otherwise it is padded on the right
222 @item int32 layout_code;
223 Normally set to 2, although a few system files have been spotted in
224 the wild with a value of 3 here. PSPP use this value to determine the
225 file's integer endianness (@pxref{System File Format}).
227 @item int32 nominal_case_size;
228 Number of data elements per case. This is the number of variables,
229 except that long string variables add extra data elements (one for every
230 8 characters after the first 8). However, string variables do not
231 contribute to this value beyond the first 255 bytes. Further, system
232 files written by some systems set this value to -1. In general, it is
233 unsafe for systems reading system files to rely upon this value.
235 @item int32 compression;
236 Set to 0 if the data in the file is not compressed, 1 if the data is
237 compressed with simple bytecode compression, 2 if the data is ZLIB
238 compressed. This field has value 2 if and only if @code{rec_type} is
241 @item int32 weight_index;
242 If one of the variables in the data set is used as a weighting
243 variable, set to the dictionary index of that variable, plus 1
244 (@pxref{Dictionary Index}). Otherwise, set to 0.
247 Set to the number of cases in the file if it is known, or -1 otherwise.
249 In the general case it is not possible to determine the number of cases
250 that will be output to a system file at the time that the header is
251 written. The way that this is dealt with is by writing the entire
252 system file, including the header, then seeking back to the beginning of
253 the file and writing just the @code{ncases} field. For files in which
254 this is not valid, the seek operation fails. In this case,
255 @code{ncases} remains -1.
259 Compression bias, ordinarily set to 100. Only integers between
260 @code{1 - bias} and @code{251 - bias} can be compressed.
262 By assuming that its value is 100, PSPP uses @code{bias} to determine
263 the file's floating-point format and endianness (@pxref{System File
264 Format}). If the compression bias is not 100, PSPP cannot auto-detect
265 the floating-point format and assumes that it is IEEE 754 format with
266 the same endianness as the system file's integers, which is correct
267 for all known system files.
269 @item char creation_date[9];
270 Date of creation of the system file, in @samp{dd mmm yy}
271 format, with the month as standard English abbreviations, using an
272 initial capital letter and following with lowercase. If the date is not
273 available then this field is arbitrarily set to @samp{01 Jan 70}.
275 @item char creation_time[8];
276 Time of creation of the system file, in @samp{hh:mm:ss}
277 format and using 24-hour time. If the time is not available then this
278 field is arbitrarily set to @samp{00:00:00}.
280 @item char file_label[64];
281 File label declared by the user, if any (@pxref{FILE LABEL,,,pspp,
282 PSPP Users Guide}). Padded on the right with spaces.
284 A product that identifies itself as @code{VOXCO INTERVIEWER 4.3} uses
285 CR-only line ends in this field, rather than the more usual LF-only or
288 @item char padding[3];
289 Ignored padding bytes to make the structure a multiple of 32 bits in
290 length. Set to zeros.
293 @node Variable Record
294 @section Variable Record
296 There must be one variable record for each numeric variable and each
297 string variable with width 8 bytes or less. String variables wider
298 than 8 bytes have one variable record for each 8 bytes, rounding up.
299 The first variable record for a long string specifies the variable's
300 correct dictionary information. Subsequent variable records for a
301 long string are filled with dummy information: a type of -1, no
302 variable label or missing values, print and write formats that are
303 ignored, and an empty string as name. A few system files have been
304 encountered that include a variable label on dummy variable records,
305 so readers should take care to parse dummy variable records in the
306 same way as other variable records.
308 @anchor{Dictionary Index}
309 The @dfn{dictionary index} of a variable is its offset in the set of
310 variable records, including dummy variable records for long string
311 variables. The first variable record has a dictionary index of 0, the
312 second has a dictionary index of 1, and so on.
314 The system file format does not directly support string variables
315 wider than 255 bytes. Such very long string variables are represented
316 by a number of narrower string variables. @xref{Very Long String
317 Record}, for details.
319 A system file should contain at least one variable and thus at least
320 one variable record, but system files have been observed in the wild
321 without any variables (thus, no data either).
327 int32 n_missing_values;
332 /* @r{Present only if @code{has_var_label} is 1.} */
336 /* @r{Present only if @code{n_missing_values} is nonzero}. */
337 flt64 missing_values[];
341 @item int32 rec_type;
342 Record type code. Always set to 2.
345 Variable type code. Set to 0 for a numeric variable. For a short
346 string variable or the first part of a long string variable, this is set
347 to the width of the string. For the second and subsequent parts of a
348 long string variable, set to -1, and the remaining fields in the
349 structure are ignored.
351 @item int32 has_var_label;
352 If this variable has a variable label, set to 1; otherwise, set to 0.
354 @item int32 n_missing_values;
355 If the variable has no missing values, set to 0. If the variable has
356 one, two, or three discrete missing values, set to 1, 2, or 3,
357 respectively. If the variable has a range for missing variables, set to
358 -2; if the variable has a range for missing variables plus a single
359 discrete value, set to -3.
361 A long string variable always has the value 0 here. A separate record
362 indicates missing values for long string variables (@pxref{Long String
363 Missing Values Record}).
366 Print format for this variable. See below.
369 Write format for this variable. See below.
372 Variable name. The variable name must begin with a capital letter or
373 the at-sign (@samp{@@}). Subsequent characters may also be digits, octothorpes
374 (@samp{#}), dollar signs (@samp{$}), underscores (@samp{_}), or full
375 stops (@samp{.}). The variable name is padded on the right with spaces.
377 The @samp{name} fields should be unique within a system file. System
378 files written by SPSS that contain very long string variables with
379 similar names sometimes contain duplicate names that are later
380 eliminated by resolving the very long string names (@pxref{Very Long
381 String Record}). PSPP handles duplicates by assigning them new,
384 @item int32 label_len;
385 This field is present only if @code{has_var_label} is set to 1. It is
386 set to the length, in characters, of the variable label. The
387 documented maximum length varies from 120 to 255 based on SPSS
388 version, but some files have been seen with longer labels. PSPP
389 accepts labels of any length.
392 This field is present only if @code{has_var_label} is set to 1. It has
393 length @code{label_len}, rounded up to the nearest multiple of 32 bits.
394 The first @code{label_len} characters are the variable's variable label.
396 @item flt64 missing_values[];
397 This field is present only if @code{n_missing_values} is nonzero. It
398 has the same number of 8-byte elements as the absolute value of
399 @code{n_missing_values}. Each element is interpreted as a number for
400 numeric variables (with HIGHEST and LOWEST indicated as described in
401 the chapter introduction). For string variables of width less than 8
402 bytes, elements are right-padded with spaces; for string variables
403 wider than 8 bytes, only the first 8 bytes of each missing value are
404 specified, with the remainder implicitly all spaces.
406 For discrete missing values, each element represents one missing
407 value. When a range is present, the first element denotes the minimum
408 value in the range, and the second element denotes the maximum value
409 in the range. When a range plus a value are present, the third
410 element denotes the additional discrete missing value.
413 @anchor{System File Output Formats}
414 The @code{print} and @code{write} members of sysfile_variable are output
415 formats coded into @code{int32} types. The least-significant byte
416 of the @code{int32} represents the number of decimal places, and the
417 next two bytes in order of increasing significance represent field width
418 and format type, respectively. The most-significant byte is not
419 used and should be set to zero.
421 Format types are defined as follows:
424 @multitable {Value} {@code{DATETIME}}
510 A few system files have been observed in the wild with invalid
511 @code{write} fields, in particular with value 0. Readers should
512 probably treat invalid @code{print} or @code{write} fields as some
515 @node Value Labels Records
516 @section Value Labels Records
518 The value label records documented in this section are used for
519 numeric and short string variables only. Long string variables may
520 have value labels, but their value labels are recorded using a
521 different record type (@pxref{Long String Value Labels Record}).
523 The value label record has the following format:
529 /* @r{Repeated @code{label_cnt} times}. */
536 @item int32 rec_type;
537 Record type. Always set to 3.
539 @item int32 label_count;
540 Number of value labels present in this record.
543 The remaining fields are repeated @code{count} times. Each
544 repetition specifies one value label.
548 A numeric value or a short string value padded as necessary to 8 bytes
549 in length. Its type and width cannot be determined until the
550 following value label variables record (see below) is read.
552 @item char label_len;
553 The label's length, in bytes. The documented maximum length varies
554 from 60 to 120 based on SPSS version. PSPP supports value labels up
558 @code{label_len} bytes of the actual label, followed by up to 7 bytes
559 of padding to bring @code{label} and @code{label_len} together to a
560 multiple of 8 bytes in length.
563 The value label record is always immediately followed by a value label
564 variables record with the following format:
573 @item int32 rec_type;
574 Record type. Always set to 4.
576 @item int32 var_count;
577 Number of variables that the associated value labels from the value
578 label record are to be applied.
581 A list of dictionary indexes of variables to which to apply the value
582 labels (@pxref{Dictionary Index}). There are @code{var_count}
585 String variables wider than 8 bytes may not be specified in this list.
588 @node Document Record
589 @section Document Record
591 The document record, if present, has the following format:
600 @item int32 rec_type;
601 Record type. Always set to 6.
604 Number of lines of documents present.
606 @item char lines[][80];
607 Document lines. The number of elements is defined by @code{n_lines}.
608 Lines shorter than 80 characters are padded on the right with spaces.
611 @node Machine Integer Info Record
612 @section Machine Integer Info Record
614 The integer info record, if present, has the following format:
626 int32 version_revision;
628 int32 floating_point_rep;
629 int32 compression_code;
631 int32 character_code;
635 @item int32 rec_type;
636 Record type. Always set to 7.
639 Record subtype. Always set to 3.
642 Size of each piece of data in the data part, in bytes. Always set to 4.
645 Number of pieces of data in the data part. Always set to 8.
647 @item int32 version_major;
648 PSPP major version number. In version @var{x}.@var{y}.@var{z}, this
651 @item int32 version_minor;
652 PSPP minor version number. In version @var{x}.@var{y}.@var{z}, this
655 @item int32 version_revision;
656 PSPP version revision number. In version @var{x}.@var{y}.@var{z},
659 @item int32 machine_code;
660 Machine code. PSPP always set this field to value to -1, but other
663 @item int32 floating_point_rep;
664 Floating point representation code. For IEEE 754 systems this is 1.
665 IBM 370 sets this to 2, and DEC VAX E to 3.
667 @item int32 compression_code;
668 Compression code. Always set to 1, regardless of whether or how the
671 @item int32 endianness;
672 Machine endianness. 1 indicates big-endian, 2 indicates little-endian.
674 @item int32 character_code;
675 @anchor{character-code} Character code. The following values have
676 been actually observed in system files:
686 The @code{windows-1250} code page for Central European and Eastern
690 The @code{windows-1252} code page for Western European languages.
699 The following additional values are known to be defined:
709 Other Windows code page numbers are known to be generally valid.
711 Old versions of SPSS for Unix and Windows always wrote value 2 in this
712 field, regardless of the encoding in use. Newer versions also write
713 the character encoding as a string (see @ref{Character Encoding
717 @node Machine Floating-Point Info Record
718 @section Machine Floating-Point Info Record
720 The floating-point info record, if present, has the following format:
736 @item int32 rec_type;
737 Record type. Always set to 7.
740 Record subtype. Always set to 4.
743 Size of each piece of data in the data part, in bytes. Always set to 8.
746 Number of pieces of data in the data part. Always set to 3.
749 The system missing value.
752 The value used for HIGHEST in missing values.
755 The value used for LOWEST in missing values.
758 @node Multiple Response Sets Records
759 @section Multiple Response Sets Records
761 The system file format has two different types of records that
762 represent multiple response sets (@pxref{MRSETS,,,pspp, PSPP Users
763 Guide}). The first type of record describes multiple response sets
764 that can be understood by SPSS before version 14. The second type of
765 record, with a closely related format, is used for multiple dichotomy
766 sets that use the CATEGORYLABELS=COUNTEDVALUES feature added in
776 /* @r{Exactly @code{count} bytes of data.} */
781 @item int32 rec_type;
782 Record type. Always set to 7.
785 Record subtype. Set to 7 for records that describe multiple response
786 sets understood by SPSS before version 14, or to 19 for records that
787 describe dichotomy sets that use the CATEGORYLABELS=COUNTEDVALUES
788 feature added in version 14.
791 The size of each element in the @code{mrsets} member. Always set to 1.
794 The total number of bytes in @code{mrsets}.
797 Zero or more line feeds (byte 0x0a), followed by a series of multiple
798 response sets, each of which consists of the following:
802 The set's name (an identifier that begins with @samp{$}), in mixed
803 upper and lower case.
806 An equals sign (@samp{=}).
809 @samp{C} for a multiple category set, @samp{D} for a multiple
810 dichotomy set with CATEGORYLABELS=VARLABELS, or @samp{E} for a
811 multiple dichotomy set with CATEGORYLABELS=COUNTEDVALUES.
814 For a multiple dichotomy set with CATEGORYLABELS=COUNTEDVALUES, a
815 space, followed by a number expressed as decimal digits, followed by a
816 space. If LABELSOURCE=VARLABEL was specified on MRSETS, then the
817 number is 11; otherwise it is 1.@footnote{This part of the format may
818 not be fully understood, because only a single example of each
819 possibility has been examined.}
822 For either kind of multiple dichotomy set, the counted value, as a
823 positive integer count specified as decimal digits, followed by a
824 space, followed by as many string bytes as specified in the count. If
825 the set contains numeric variables, the string consists of the counted
826 integer value expressed as decimal digits. If the set contains string
827 variables, the string contains the counted string value. Either way,
828 the string may be padded on the right with spaces (older versions of
829 SPSS seem to always pad to a width of 8 bytes; newer versions don't).
835 The multiple response set's label, using the same format as for the
836 counted value for multiple dichotomy sets. A string of length 0 means
837 that the set does not have a label. A string of length 0 is also
838 written if LABELSOURCE=VARLABEL was specified.
844 The short names of the variables in the set, converted to lowercase,
845 each separated from the previous by a single space.
847 Even though a multiple response set must have at least two variables,
848 some system files contain multiple response sets with no variables at
849 all. The source and meaning of these multiple response sets is
850 unknown. (Perhaps they arise from creating a multiple response set
851 then deleting all the variables that it contains?)
854 One line feed (byte 0x0a). Sometimes multiple, even hundreds, of line
859 Example: Given appropriate variable definitions, consider the
860 following MRSETS command:
863 MRSETS /MCGROUP NAME=$a LABEL='my mcgroup' VARIABLES=a b c
864 /MDGROUP NAME=$b VARIABLES=g e f d VALUE=55
865 /MDGROUP NAME=$c LABEL='mdgroup #2' VARIABLES=h i j VALUE='Yes'
866 /MDGROUP NAME=$d LABEL='third mdgroup' CATEGORYLABELS=COUNTEDVALUES
867 VARIABLES=k l m VALUE=34
868 /MDGROUP NAME=$e CATEGORYLABELS=COUNTEDVALUES LABELSOURCE=VARLABEL
869 VARIABLES=n o p VALUE='choice'.
872 The above would generate the following multiple response set record of
876 $a=C 10 my mcgroup a b c
878 $c=D3 Yes 10 mdgroup #2 h i j
881 It would also generate the following multiple response set record with
885 $d=E 1 2 34 13 third mdgroup k l m
886 $e=E 11 6 choice 0 n o p
889 @node Extra Product Info Record
890 @section Extra Product Info Record
892 This optional record appears to contain a text string that describes
893 the program that wrote the file and the source of the data. (This is
894 redundant with the file label and product info found in the file
904 /* @r{Exactly @code{count} bytes of data.} */
909 @item int32 rec_type;
910 Record type. Always set to 7.
913 Record subtype. Always set to 10.
916 The size of each element in the @code{info} member. Always set to 1.
919 The total number of bytes in @code{info}.
922 A text string. A product that identifies itself as @code{VOXCO
923 INTERVIEWER 4.3} uses CR-only line ends in this field, rather than the
924 more usual LF-only or CR LF line ends.
927 @node Variable Display Parameter Record
928 @section Variable Display Parameter Record
930 The variable display parameter record, if present, has the following
940 /* @r{Repeated @code{count} times}. */
942 int32 width; /* @r{Not always present.} */
947 @item int32 rec_type;
948 Record type. Always set to 7.
951 Record subtype. Always set to 11.
954 The size of @code{int32}. Always set to 4.
957 The number of sets of variable display parameters (ordinarily the
958 number of variables in the dictionary), times 2 or 3.
961 The remaining members are repeated @code{count} times, in the same
962 order as the variable records. No element corresponds to variable
963 records that continue long string variables. The meanings of these
964 members are as follows:
968 The measurement type of the variable:
978 SPSS sometimes writes a @code{measure} of 0. PSPP interprets this as
982 The width of the display column for the variable in characters.
984 This field is present if @var{count} is 3 times the number of
985 variables in the dictionary. It is omitted if @var{count} is 2 times
986 the number of variables.
988 @item int32 alignment;
989 The alignment of the variable for display purposes:
1001 @node Long Variable Names Record
1002 @section Long Variable Names Record
1004 If present, the long variable names record has the following format:
1013 /* @r{Exactly @code{count} bytes of data.} */
1014 char var_name_pairs[];
1018 @item int32 rec_type;
1019 Record type. Always set to 7.
1021 @item int32 subtype;
1022 Record subtype. Always set to 13.
1025 The size of each element in the @code{var_name_pairs} member. Always set to 1.
1028 The total number of bytes in @code{var_name_pairs}.
1030 @item char var_name_pairs[];
1031 A list of @var{key}--@var{value} tuples, where @var{key} is the name
1032 of a variable, and @var{value} is its long variable name.
1033 The @var{key} field is at most 8 bytes long and must match the
1034 name of a variable which appears in the variable record (@pxref{Variable
1036 The @var{value} field is at most 64 bytes long.
1037 The @var{key} and @var{value} fields are separated by a @samp{=} byte.
1038 Each tuple is separated by a byte whose value is 09. There is no
1039 trailing separator following the last tuple.
1040 The total length is @code{count} bytes.
1043 @node Very Long String Record
1044 @section Very Long String Record
1046 Old versions of SPSS limited string variables to a width of 255 bytes.
1047 For backward compatibility with these older versions, the system file
1048 format represents a string longer than 255 bytes, called a @dfn{very
1049 long string}, as a collection of strings no longer than 255 bytes
1050 each. The strings concatenated to make a very long string are called
1051 its @dfn{segments}; for consistency, variables other than very long
1052 strings are considered to have a single segment.
1054 A very long string with a width of @var{w} has @var{n} =
1055 (@var{w} + 251) / 252 segments, that is, one segment for every
1056 252 bytes of width, rounding up. It would be logical, then, for each
1057 of the segments except the last to have a width of 252 and the last
1058 segment to have the remainder, but this is not the case. In fact,
1059 each segment except the last has a width of 255 bytes. The last
1060 segment has width @var{w} - (@var{n} - 1) * 252; some versions
1061 of SPSS make it slightly wider, but not wide enough to make the last
1062 segment require another 8 bytes of data.
1064 Data is packed tightly into segments of a very long string, 255 bytes
1065 per segment. Because 255 bytes of segment data are allocated for
1066 every 252 bytes of the very long string's width (approximately), some
1067 unused space is left over at the end of the allocated segments. Data
1068 in unused space is ignored.
1070 Example: Consider a very long string of width 20,000. Such a very
1071 long string has 20,000 / 252 = 80 (rounding up) segments. The first
1072 79 segments have width 255; the last segment has width 20,000 - 79 *
1073 252 = 92 or slightly wider (up to 96 bytes, the next multiple of 8).
1074 The very long string's data is actually stored in the 19,890 bytes in
1075 the first 78 segments, plus the first 110 bytes of the 79th segment
1076 (19,890 + 110 = 20,000). The remaining 145 bytes of the 79th segment
1077 and all 92 bytes of the 80th segment are unused.
1079 The very long string record explains how to stitch together segments
1080 to obtain very long string data. For each of the very long string
1081 variables in the dictionary, it specifies the name of its first
1082 segment's variable and the very long string variable's actual width.
1083 The remaining segments immediately follow the named variable in the
1084 system file's dictionary.
1086 The very long string record, which is present only if the system file
1087 contains very long string variables, has the following format:
1096 /* @r{Exactly @code{count} bytes of data.} */
1097 char string_lengths[];
1101 @item int32 rec_type;
1102 Record type. Always set to 7.
1104 @item int32 subtype;
1105 Record subtype. Always set to 14.
1108 The size of each element in the @code{string_lengths} member. Always set to 1.
1111 The total number of bytes in @code{string_lengths}.
1113 @item char string_lengths[];
1114 A list of @var{key}--@var{value} tuples, where @var{key} is the name
1115 of a variable, and @var{value} is its length.
1116 The @var{key} field is at most 8 bytes long and must match the
1117 name of a variable which appears in the variable record (@pxref{Variable
1119 The @var{value} field is exactly 5 bytes long. It is a zero-padded,
1120 ASCII-encoded string that is the length of the variable.
1121 The @var{key} and @var{value} fields are separated by a @samp{=} byte.
1122 Tuples are delimited by a two-byte sequence @{00, 09@}.
1123 After the last tuple, there may be a single byte 00, or @{00, 09@}.
1124 The total length is @code{count} bytes.
1127 @node Character Encoding Record
1128 @section Character Encoding Record
1130 This record, if present, indicates the character encoding for string data,
1131 long variable names, variable labels, value labels and other strings in the
1141 /* @r{Exactly @code{count} bytes of data.} */
1146 @item int32 rec_type;
1147 Record type. Always set to 7.
1149 @item int32 subtype;
1150 Record subtype. Always set to 20.
1153 The size of each element in the @code{encoding} member. Always set to 1.
1156 The total number of bytes in @code{encoding}.
1158 @item char encoding[];
1159 The name of the character encoding. Normally this will be an official
1160 IANA character set name or alias.
1161 See @url{http://www.iana.org/assignments/character-sets}.
1162 Character set names are not case-sensitive, but SPSS appears to write
1163 them in all-uppercase.
1166 This record is not present in files generated by older software. See
1167 also the @code{character_code} field in the machine integer info
1168 record (@pxref{character-code}).
1170 When the character encoding record and the machine integer info record
1171 are both present, all system files observed in practice indicate the
1172 same character encoding, e.g.@: 1252 as @code{character_code} and
1173 @code{windows-1252} as @code{encoding}, 65001 and @code{UTF-8}, etc.
1175 If, for testing purposes, a file is crafted with different
1176 @code{character_code} and @code{encoding}, it seems that
1177 @code{character_code} controls the encoding for all strings in the
1178 system file before the dictionary termination record, including
1179 strings in data (e.g.@: string missing values), and @code{encoding}
1180 controls the encoding for strings following the dictionary termination
1183 @node Long String Value Labels Record
1184 @section Long String Value Labels Record
1186 This record, if present, specifies value labels for long string
1196 /* @r{Repeated up to exactly @code{count} bytes.} */
1201 long_string_label labels[];
1205 @item int32 rec_type;
1206 Record type. Always set to 7.
1208 @item int32 subtype;
1209 Record subtype. Always set to 21.
1215 The number of bytes following the header until the next header.
1217 @item int32 var_name_len;
1218 @itemx char var_name[];
1219 The number of bytes in the name of the variable that has long string
1220 value labels, plus the variable name itself, which consists of exactly
1221 @code{var_name_len} bytes. The variable name is not padded to any
1222 particular boundary, nor is it null-terminated.
1224 @item int32 var_width;
1225 The width of the variable, in bytes, which will be between 9 and
1228 @item int32 n_labels;
1229 @itemx long_string_label labels[];
1230 The long string labels themselves. The @code{labels} array contains
1231 exactly @code{n_labels} elements, each of which has the following
1242 @item int32 value_len;
1243 @itemx char value[];
1244 The string value being labeled. @code{value_len} is the number of
1245 bytes in @code{value}; it is equal to @code{var_width}. The
1246 @code{value} array is not padded or null-terminated.
1248 @item int32 label_len;
1249 @itemx char label[];
1250 The label for the string value. @code{label_len}, which must be
1251 between 0 and 120, is the number of bytes in @code{label}. The
1252 @code{label} array is not padded or null-terminated.
1256 @node Long String Missing Values Record
1257 @section Long String Missing Values Record
1259 This record, if present, specifies missing values for long string
1269 /* @r{Repeated up to exactly @code{count} bytes.} */
1272 char n_missing_values;
1273 long_string_missing_value values[];
1277 @item int32 rec_type;
1278 Record type. Always set to 7.
1280 @item int32 subtype;
1281 Record subtype. Always set to 22.
1287 The number of bytes following the header until the next header.
1289 @item int32 var_name_len;
1290 @itemx char var_name[];
1291 The number of bytes in the name of the long string variable that has
1292 missing values, plus the variable name itself, which consists of
1293 exactly @code{var_name_len} bytes. The variable name is not padded to
1294 any particular boundary, nor is it null-terminated.
1296 @item char n_missing_values;
1297 The number of missing values, either 1, 2, or 3. (This is, unusually,
1298 a single byte instead of a 32-bit number.)
1300 @item long_string_missing_value values[];
1301 The missing values themselves. This array contains exactly
1302 @code{n_missing_values} elements, each of which has the following
1311 @item int32 value_len;
1312 The length of the missing value string, in bytes. This value should
1313 be 8, because long string variables are at least 8 bytes wide (by
1314 definition), only the first 8 bytes of a long string variable's
1315 missing values are allowed to be non-spaces, and any spaces within the
1316 first 8 bytes are included in the missing value here.
1319 The missing value string, exactly @code{value_len} bytes, without
1320 any padding or null terminator.
1324 @node Data File and Variable Attributes Records
1325 @section Data File and Variable Attributes Records
1327 The data file and variable attributes records represent custom
1328 attributes for the system file or for individual variables in the
1329 system file, as defined on the DATAFILE ATTRIBUTE (@pxref{DATAFILE
1330 ATTRIBUTE,,,pspp, PSPP Users Guide}) and VARIABLE ATTRIBUTE commands
1331 (@pxref{VARIABLE ATTRIBUTE,,,pspp, PSPP Users Guide}), respectively.
1340 /* @r{Exactly @code{count} bytes of data.} */
1345 @item int32 rec_type;
1346 Record type. Always set to 7.
1348 @item int32 subtype;
1349 Record subtype. Always set to 17 for a data file attribute record or
1350 to 18 for a variable attributes record.
1353 The size of each element in the @code{attributes} member. Always set to 1.
1356 The total number of bytes in @code{attributes}.
1358 @item char attributes[];
1359 The attributes, in a text-based format.
1361 In record type 17, this field contains a single attribute set. An
1362 attribute set is a sequence of one or more attributes concatenated
1363 together. Each attribute consists of a name, which has the same
1364 syntax as a variable name, followed by, inside parentheses, a sequence
1365 of one or more values. Each value consists of a string enclosed in
1366 single quotes (@code{'}) followed by a line feed (byte 0x0a). A value
1367 may contain single quote characters, which are not themselves escaped
1368 or quoted or required to be present in pairs. There is no apparent
1369 way to embed a line feed in a value. There is no distinction between
1370 an attribute with a single value and an attribute array with one
1373 In record type 18, this field contains a sequence of one or more
1374 variable attribute sets. If more than one variable attribute set is
1375 present, each one after the first is delimited from the previous by
1376 @code{/}. Each variable attribute set consists of a long
1378 followed by @code{:}, followed by an attribute set with the same
1379 syntax as on record type 17.
1381 The total length is @code{count} bytes.
1386 A system file produced with the following VARIABLE ATTRIBUTE commands
1390 VARIABLE ATTRIBUTE VARIABLES=dummy ATTRIBUTE=fred[1]('23') fred[2]('34').
1391 VARIABLE ATTRIBUTE VARIABLES=dummy ATTRIBUTE=bert('123').
1395 will contain a variable attribute record with the following contents:
1398 0000 07 00 00 00 12 00 00 00 01 00 00 00 22 00 00 00 |............"...|
1399 0010 64 75 6d 6d 79 3a 66 72 65 64 28 27 32 33 27 0a |dummy:fred('23'.|
1400 0020 27 33 34 27 0a 29 62 65 72 74 28 27 31 32 33 27 |'34'.)bert('123'|
1408 @node Variable Roles
1409 @subsection Variable Roles
1411 A variable's role is represented as an attribute named @code{$@@Role}.
1412 This attribute has a single element whose values and their meanings
1417 Input. This, the default, is the most common role.
1430 @node Extended Number of Cases Record
1431 @section Extended Number of Cases Record
1433 The file header record expresses the number of cases in the system
1434 file as an int32 (@pxref{File Header Record}). This record allows the
1435 number of cases in the system file to be expressed as a 64-bit number.
1447 @item int32 rec_type;
1448 Record type. Always set to 7.
1450 @item int32 subtype;
1451 Record subtype. Always set to 16.
1454 Size of each element. Always set to 8.
1457 Number of pieces of data in the data part. Alway set to 2.
1459 @item int64 unknown;
1460 Meaning unknown. Always set to 1.
1462 @item int64 ncases64;
1463 Number of cases in the file as a 64-bit integer. Presumably this
1464 could be -1 to indicate that the number of cases is unknown, for the
1465 same reason as @code{ncases} in the file header record, but this has
1466 not been observed in the wild.
1469 @node Other Informational Records
1470 @section Other Informational Records
1472 This chapter documents many specific types of extension records are
1473 documented here, but others are known to exist. PSPP ignores unknown
1474 extension records when reading system files.
1476 The following extension record subtypes have also been observed, with
1477 the following believed meanings:
1481 A set of grouped variables (according to Aapi H@"am@"al@"ainen).
1484 Date info, probably related to USE (according to Aapi H@"am@"al@"ainen).
1487 XML that describes how data in the file should be displayed on-screen.
1490 @node Dictionary Termination Record
1491 @section Dictionary Termination Record
1493 The dictionary termination record separates all other records from the
1502 @item int32 rec_type;
1503 Record type. Always set to 999.
1506 Ignored padding. Should be set to 0.
1510 @section Data Record
1512 The data record must follow all other records in the system file.
1513 Every system file must have a data record that specifies data for at
1514 least one case. The format of the data record varies depending on the
1515 value of @code{compression} in the file header record:
1518 @item 0: no compression
1519 Data is arranged as a series of 8-byte elements.
1520 Each element corresponds to
1521 the variable declared in the respective variable record (@pxref{Variable
1522 Record}). Numeric values are given in @code{flt64} format; string
1523 values are literal characters string, padded on the right when
1524 necessary to fill out 8-byte units.
1526 @item 1: bytecode compression
1528 of the data record is divided into a series of 1-byte command
1529 codes. These codes have meanings as described below:
1533 Ignored. If the program writing the system file accumulates compressed
1534 data in blocks of fixed length, 0 bytes can be used to pad out extra
1535 bytes remaining at the end of a fixed-size block.
1539 value @var{code} - @var{bias}, where
1540 @var{code} is the value of the compression code and @var{bias} is the
1541 variable @code{bias} from the file header. For example,
1542 code 105 with bias 100.0 (the normal value) indicates a numeric variable
1544 One file has been seen written by SPSS 14 that contained such a code
1545 in a @emph{string} field with the value 0 (after the bias is
1546 subtracted) as a way of encoding null bytes.
1549 End of file. This code may or may not appear at the end of the data
1550 stream. PSPP always outputs this code but its use is not required.
1553 A numeric or string value that is not
1554 compressible. The value is stored in the 8 bytes following the
1555 current block of command bytes. If this value appears twice in a block
1556 of command bytes, then it indicates the second group of 8 bytes following the
1557 command bytes, and so on.
1560 An 8-byte string value that is all spaces.
1563 The system-missing value.
1566 The end of the 8-byte group of bytecodes is followed by any 8-byte
1567 blocks of non-compressible values indicated by code 253. After that
1568 follows another 8-byte group of bytecodes, then those bytecodes'
1569 non-compressible values. The pattern repeats to the end of the file
1570 or a code with value 252.
1572 @item 2: ZLIB compression
1573 The data record consists of the following, in order:
1577 ZLIB data header, 24 bytes long.
1580 One or more variable-length blocks of ZLIB compressed data.
1583 ZLIB data trailer, with a 24-byte fixed header plus an additional 24
1584 bytes for each preceding ZLIB compressed data block.
1587 The ZLIB data header has the following format:
1596 @item int64 zheader_ofs;
1597 The offset, in bytes, of the beginning of this structure within the
1600 @item int64 ztrailer_ofs;
1601 The offset, in bytes, of the first byte of the ZLIB data trailer.
1603 @item int64 ztrailer_len;
1604 The number of bytes in the ZLIB data trailer. This and the previous
1605 field sum to the size of the system file in bytes.
1608 The data header is followed by @code{(ztrailer_ofs - 24) / 24} ZLIB
1609 compressed data blocks. Each ZLIB compressed data block begins with a
1610 ZLIB header as specified in RFC@tie{}1950, e.g.@: hex bytes @code{78
1611 01} (the only header yet observed in practice). Each block
1612 decompresses to a fixed number of bytes (in practice only
1613 @code{0x3ff000}-byte blocks have been observed), except that the last
1614 block of data may be shorter. The last ZLIB compressed data block
1615 gends just before offset @code{ztrailer_ofs}.
1617 The result of ZLIB decompression is bytecode compressed data as
1618 described above for compression format 1.
1620 The ZLIB data trailer begins with the following 24-byte fixed header:
1630 @item int64 int_bias;
1631 The compression bias as a negative integer, e.g.@: if @code{bias} in
1632 the file header record is 100.0, then @code{int_bias} is @minus{}100
1633 (this is the only value yet observed in practice).
1636 Always observed to be zero.
1638 @item int32 block_size;
1639 The number of bytes in each ZLIB compressed data block, except
1640 possibly the last, following decompression. Only @code{0x3ff000} has
1641 been observed so far.
1643 @item int32 n_blocks;
1644 The number of ZLIB compressed data blocks, always exactly
1645 @code{(ztrailer_ofs - 24) / 24}.
1648 The fixed header is followed by @code{n_blocks} 24-byte ZLIB data
1649 block descriptors, each of which describes the compressed data block
1650 corresponding to its offset. Each block descriptor has the following
1654 int64 uncompressed_ofs;
1655 int64 compressed_ofs;
1656 int32 uncompressed_size;
1657 int32 compressed_size;
1661 @item int64 uncompressed_ofs;
1662 The offset, in bytes, that this block of data would have in a similar
1663 system file that uses compression format 1. This is
1664 @code{zheader_ofs} in the first block descriptor, and in each
1665 succeeding block descriptor it is the sum of the previous desciptor's
1666 @code{uncompressed_ofs} and @code{uncompressed_size}.
1668 @item int64 compressed_ofs;
1669 The offset, in bytes, of the actual beginning of this compressed data
1670 block. This is @code{zheader_ofs + 24} in the first block descriptor,
1671 and in each succeeding block descriptor it is the sum of the previous
1672 descriptor's @code{compressed_ofs} and @code{compressed_size}. The
1673 final block descriptor's @code{compressed_ofs} and
1674 @code{compressed_size} sum to @code{ztrailer_ofs}.
1676 @item int32 uncompressed_size;
1677 The number of bytes in this data block, after decompression. This is
1678 @code{block_size} in every data block except the last, which may be
1681 @item int32 compressed_size;
1682 The number of bytes in this data block, as stored compressed in this
1687 @setfilename ignored
1689 @node Encrypted System Files
1690 @section Encrypted System Files
1692 SPSS 21 and later support an encrypted system file format.
1695 The SPSS encrypted file format is poorly designed. It is much cheaper
1696 and faster to decrypt a file encrypted this way than if a well
1697 designed alternative were used. If you must use this format, use a
1698 10-byte randomly generated password.
1701 @subheading Encrypted File Format
1703 Encrypted system files begin with the following 36-byte fixed header:
1706 0000 1c 00 00 00 00 00 00 00 45 4e 43 52 59 50 54 45 |........ENCRYPTE|
1707 0010 44 53 41 56 15 00 00 00 00 00 00 00 00 00 00 00 |DSAV............|
1708 0020 00 00 00 00 |....|
1711 Following the fixed header is a complete system file in the usual
1712 format, except that each 16-byte block is encrypted with AES-256 in
1713 ECB mode. The AES-256 key is derived from a password in the following
1718 Start from the literal password typed by the user. Truncate it to at
1719 most 10 bytes, then append (between 1 and 22) null bytes until there
1720 are exactly 32 bytes. Call this @var{password}.
1723 Let @var{constant} be the following 73-byte constant:
1726 0000 00 00 00 01 35 27 13 cc 53 a7 78 89 87 53 22 11
1727 0010 d6 5b 31 58 dc fe 2e 7e 94 da 2f 00 cc 15 71 80
1728 0020 0a 6c 63 53 00 38 c3 38 ac 22 f3 63 62 0e ce 85
1729 0030 3f b8 07 4c 4e 2b 77 c7 21 f5 1a 80 1d 67 fb e1
1730 0040 e1 83 07 d8 0d 00 00 01 00
1734 Compute CMAC-AES-256(@var{password}, @var{constant}). Call the
1735 16-byte result @var{cmac}.
1738 The 32-byte AES-256 key is @var{cmac} || @var{cmac}, that is,
1739 @var{cmac} repeated twice.
1742 @subsubheading Example
1744 Consider the password @samp{pspp}. @var{password} is:
1747 0000 70 73 70 70 00 00 00 00 00 00 00 00 00 00 00 00 |pspp............|
1748 0010 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
1755 0000 3e da 09 8e 66 04 d4 fd f9 63 0c 2c a8 6f b0 45
1762 0000 3e da 09 8e 66 04 d4 fd f9 63 0c 2c a8 6f b0 45
1763 0010 3e da 09 8e 66 04 d4 fd f9 63 0c 2c a8 6f b0 45
1766 @subheading Password Encoding
1768 SPSS also supports what it calls ``encrypted passwords.'' These are
1769 not encrypted. They are encoded with a simple, fixed scheme. An
1770 encoded password is always a multiple of 2 characters long, and never
1771 longer than 20 characters. The characters in an encoded password are
1772 always in the graphic ASCII range 33 through 126. Each successive
1773 pair of characters in the password encodes a single byte in the
1776 Use the following algorithm to decode a pair of characters:
1780 Let @var{a} be the ASCII code of the first character, and @var{b} be
1781 the ASCII code of the second character.
1784 Let @var{ah} be the most significant 4 bits of @var{a}. Find the line
1785 in the table below that has @var{ah} on the left side. The right side
1786 of the line is a set of possible values for the most significant 4
1787 bits of the decoded byte.
1790 @t{2 } @result{} @t{2367}
1791 @t{3 } @result{} @t{0145}
1792 @t{47} @result{} @t{89cd}
1793 @t{56} @result{} @t{abef}
1797 Let @var{bh} be the most significant 4 bits of @var{b}. Find the line
1798 in the second table below that has @var{bh} on the left side. The
1799 right side of the line is a set of possible values for the most
1800 significant 4 bits of the decoded byte. Together with the results of
1801 the previous step, only a single possibility is left.
1804 @t{2 } @result{} @t{139b}
1805 @t{3 } @result{} @t{028a}
1806 @t{47} @result{} @t{46ce}
1807 @t{56} @result{} @t{57df}
1811 Let @var{al} be the least significant 4 bits of @var{a}. Find the
1812 line in the table below that has @var{al} on the left side. The right
1813 side of the line is a set of possible values for the least significant
1814 4 bits of the decoded byte.
1817 @t{03cf} @result{} @t{0145}
1818 @t{12de} @result{} @t{2367}
1819 @t{478b} @result{} @t{89cd}
1820 @t{569a} @result{} @t{abef}
1824 Let @var{bl} be the least significant 4 bits of @var{b}. Find the
1825 line in the table below that has @var{bl} on the left side. The right
1826 side of the line is a set of possible values for the least significant
1827 4 bits of the decoded byte. Together with the results of the previous
1828 step, only a single possibility is left.
1831 @t{03cf} @result{} @t{028a}
1832 @t{12de} @result{} @t{139b}
1833 @t{478b} @result{} @t{46ce}
1834 @t{569a} @result{} @t{57df}
1838 @subsubheading Example
1840 Consider the encoded character pair @samp{-|}. @var{a} is
1841 0x2d and @var{b} is 0x7c, so @var{ah} is 2, @var{bh} is 7, @var{al} is
1842 0xd, and @var{bl} is 0xc. @var{ah} means that the most significant
1843 four bits of the decoded character is 2, 3, 6, or 7, and @var{bh}
1844 means that they are 4, 6, 0xc, or 0xe. The single possibility in
1845 common is 6, so the most significant four bits are 6. Similarly,
1846 @var{al} means that the least significant four bits are 2, 3, 6, or 7,
1847 and @var{bl} means they are 0, 2, 8, or 0xa, so the least significant
1848 four bits are 2. The decoded character is therefore 0x62, the letter