1 @node System File Format
2 @appendix System File Format
4 A system file encapsulates a set of cases and dictionary information
5 that describes how they may be interpreted. This chapter describes
6 the format of a system file.
8 System files use four data types: 8-bit characters, 32-bit integers,
10 and 64-bit floating points, called here @code{char}, @code{int32},
12 @code{flt64}, respectively. Data is not necessarily aligned on a word
13 or double-word boundary: the long variable name record (@pxref{Long
14 Variable Names Record}) and very long string records (@pxref{Very Long
15 String Record}) have arbitrary byte length and can therefore cause all
16 data coming after them in the file to be misaligned.
18 Integer data in system files may be big-endian or little-endian. A
19 reader may detect the endianness of a system file by examining
20 @code{layout_code} in the file header record
21 (@pxref{layout_code,,@code{layout_code}}).
23 Floating-point data in system files may nominally be in IEEE 754, IBM,
24 or VAX formats. A reader may detect the floating-point format in use
25 by examining @code{bias} in the file header record
26 (@pxref{bias,,@code{bias}}).
28 PSPP detects big-endian and little-endian integer formats in system
29 files and translates as necessary. PSPP also detects the
30 floating-point format in use, as well as the endianness of IEEE 754
31 floating-point numbers, and translates as needed. However, only IEEE
32 754 numbers with the same endianness as integer data in the same file
33 has actually been observed in system files, and it is likely that
34 other formats are obsolete or were never used.
36 System files use a few floating point values for special purposes:
40 The system-missing value is represented by the largest possible
41 negative number in the floating point format (@code{-DBL_MAX}).
44 HIGHEST is used as the high end of a missing value range with an
45 unbounded maximum. It is represented by the largest possible positive
46 number (@code{DBL_MAX}).
49 LOWEST is used as the low end of a missing value range with an
50 unbounded minimum. It was originally represented by the
51 second-largest negative number (in IEEE 754 format,
52 @code{0xffeffffffffffffe}). System files written by SPSS 21 and later
53 instead use the largest negative number (@code{-DBL_MAX}), the same
54 value as SYSMIS. This does not lead to ambiguity because LOWEST
55 appears in system files only in missing value ranges, which never
59 System files may use most character encodings based on an 8-bit unit.
60 UTF-16 and UTF-32, based on wider units, appear to be unacceptable.
61 @code{rec_type} in the file header record is sufficient to distinguish
62 between ASCII and EBCDIC based encodings. The best way to determine
63 the specific encoding in use is to consult the character encoding
64 record (@pxref{Character Encoding Record}), if present, and failing
65 that the @code{character_code} in the machine integer info record
66 (@pxref{Machine Integer Info Record}). The same encoding should be
67 used for the dictionary and the data in the file, although it is
68 possible to artificially synthesize files that use different encodings
69 (@pxref{Character Encoding Record}).
71 System files are divided into records, each of which begins with a
72 4-byte record type, usually regarded as an @code{int32}.
74 The records must appear in the following order:
84 All pairs of value labels records and value label variables records,
88 Document record, if present.
91 Extension (type 7) records, in ascending numerical order of their
95 Dictionary termination record.
101 Each type of record is described separately below.
104 * File Header Record::
106 * Value Labels Records::
108 * Machine Integer Info Record::
109 * Machine Floating-Point Info Record::
110 * Multiple Response Sets Records::
111 * Extra Product Info Record::
112 * Variable Display Parameter Record::
113 * Long Variable Names Record::
114 * Very Long String Record::
115 * Character Encoding Record::
116 * Long String Value Labels Record::
117 * Long String Missing Values Record::
118 * Data File and Variable Attributes Records::
119 * Extended Number of Cases Record::
120 * Miscellaneous Informational Records::
121 * Dictionary Termination Record::
125 @node File Header Record
126 @section File Header Record
128 The file header is always the first record in the file. It has the
135 int32 nominal_case_size;
140 char creation_date[9];
141 char creation_time[8];
147 @item char rec_type[4];
148 Record type code, either @samp{$FL2} for system files with
149 uncompressed data or data compressed with simple bytecode compression,
150 or @samp{$FL3} for system files with ZLIB compressed data.
152 This is truly a character field that uses the character encoding as
153 other strings. Thus, in a file with an ASCII-based character encoding
154 this field contains @code{24 46 4c 32} or @code{24 46 4c 33}, and in a
155 file with an EBCDIC-based encoding this field contains @code{5b c6 d3
156 f2}. (No EBCDIC-based ZLIB-compressed files have been observed.)
158 @item char prod_name[60];
159 Product identification string. This always begins with the characters
160 @samp{@@(#) SPSS DATA FILE}. PSPP uses the remaining characters to
161 give its version and the operating system name; for example, @samp{GNU
162 pspp 0.1.4 - sparc-sun-solaris2.5.2}. The string is truncated if it
163 would be longer than 60 characters; otherwise it is padded on the right
167 @item int32 layout_code;
168 Normally set to 2, although a few system files have been spotted in
169 the wild with a value of 3 here. PSPP use this value to determine the
170 file's integer endianness (@pxref{System File Format}).
172 @item int32 nominal_case_size;
173 Number of data elements per case. This is the number of variables,
174 except that long string variables add extra data elements (one for every
175 8 characters after the first 8). However, string variables do not
176 contribute to this value beyond the first 255 bytes. Further, system
177 files written by some systems set this value to -1. In general, it is
178 unsafe for systems reading system files to rely upon this value.
180 @item int32 compressed;
181 Set to 0 if the data in the file is not compressed, 1 if the data is
182 compressed with simple bytecode compression, 2 if the data is ZLIB
183 compressed. This field has value 2 if and only if @code{rec_type} is
186 @item int32 weight_index;
187 If one of the variables in the data set is used as a weighting
188 variable, set to the dictionary index of that variable, plus 1
189 (@pxref{Dictionary Index}). Otherwise, set to 0.
192 Set to the number of cases in the file if it is known, or -1 otherwise.
194 In the general case it is not possible to determine the number of cases
195 that will be output to a system file at the time that the header is
196 written. The way that this is dealt with is by writing the entire
197 system file, including the header, then seeking back to the beginning of
198 the file and writing just the @code{ncases} field. For files in which
199 this is not valid, the seek operation fails. In this case,
200 @code{ncases} remains -1.
204 Compression bias, ordinarily set to 100. Only integers between
205 @code{1 - bias} and @code{251 - bias} can be compressed.
207 By assuming that its value is 100, PSPP uses @code{bias} to determine
208 the file's floating-point format and endianness (@pxref{System File
209 Format}). If the compression bias is not 100, PSPP cannot auto-detect
210 the floating-point format and assumes that it is IEEE 754 format with
211 the same endianness as the system file's integers, which is correct
212 for all known system files.
214 @item char creation_date[9];
215 Date of creation of the system file, in @samp{dd mmm yy}
216 format, with the month as standard English abbreviations, using an
217 initial capital letter and following with lowercase. If the date is not
218 available then this field is arbitrarily set to @samp{01 Jan 70}.
220 @item char creation_time[8];
221 Time of creation of the system file, in @samp{hh:mm:ss}
222 format and using 24-hour time. If the time is not available then this
223 field is arbitrarily set to @samp{00:00:00}.
225 @item char file_label[64];
226 File label declared by the user, if any (@pxref{FILE LABEL,,,pspp,
227 PSPP Users Guide}). Padded on the right with spaces.
229 A product that identifies itself as @code{VOXCO INTERVIEWER 4.3} uses
230 CR-only line ends in this field, rather than the more usual LF-only or
233 @item char padding[3];
234 Ignored padding bytes to make the structure a multiple of 32 bits in
235 length. Set to zeros.
238 @node Variable Record
239 @section Variable Record
241 There must be one variable record for each numeric variable and each
242 string variable with width 8 bytes or less. String variables wider
243 than 8 bytes have one variable record for each 8 bytes, rounding up.
244 The first variable record for a long string specifies the variable's
245 correct dictionary information. Subsequent variable records for a
246 long string are filled with dummy information: a type of -1, no
247 variable label or missing values, print and write formats that are
248 ignored, and an empty string as name. A few system files have been
249 encountered that include a variable label on dummy variable records,
250 so readers should take care to parse dummy variable records in the
251 same way as other variable records.
253 @anchor{Dictionary Index}
254 The @dfn{dictionary index} of a variable is its offset in the set of
255 variable records, including dummy variable records for long string
256 variables. The first variable record has a dictionary index of 0, the
257 second has a dictionary index of 1, and so on.
259 The system file format does not directly support string variables
260 wider than 255 bytes. Such very long string variables are represented
261 by a number of narrower string variables. @xref{Very Long String
262 Record}, for details.
268 int32 n_missing_values;
273 /* @r{Present only if @code{has_var_label} is 1.} */
277 /* @r{Present only if @code{n_missing_values} is nonzero}. */
278 flt64 missing_values[];
282 @item int32 rec_type;
283 Record type code. Always set to 2.
286 Variable type code. Set to 0 for a numeric variable. For a short
287 string variable or the first part of a long string variable, this is set
288 to the width of the string. For the second and subsequent parts of a
289 long string variable, set to -1, and the remaining fields in the
290 structure are ignored.
292 @item int32 has_var_label;
293 If this variable has a variable label, set to 1; otherwise, set to 0.
295 @item int32 n_missing_values;
296 If the variable has no missing values, set to 0. If the variable has
297 one, two, or three discrete missing values, set to 1, 2, or 3,
298 respectively. If the variable has a range for missing variables, set to
299 -2; if the variable has a range for missing variables plus a single
300 discrete value, set to -3.
302 A long string variable always has the value 0 here. A separate record
303 indicates missing values for long string variables (@pxref{Long String
304 Missing Values Record}).
307 Print format for this variable. See below.
310 Write format for this variable. See below.
313 Variable name. The variable name must begin with a capital letter or
314 the at-sign (@samp{@@}). Subsequent characters may also be digits, octothorpes
315 (@samp{#}), dollar signs (@samp{$}), underscores (@samp{_}), or full
316 stops (@samp{.}). The variable name is padded on the right with spaces.
318 @item int32 label_len;
319 This field is present only if @code{has_var_label} is set to 1. It is
320 set to the length, in characters, of the variable label. The
321 documented maximum length varies from 120 to 255 based on SPSS
322 version, but some files have been seen with longer labels. PSPP
323 accepts longer labels and truncates them to 255 bytes on input.
326 This field is present only if @code{has_var_label} is set to 1. It has
327 length @code{label_len}, rounded up to the nearest multiple of 32 bits.
328 The first @code{label_len} characters are the variable's variable label.
330 @item flt64 missing_values[];
331 This field is present only if @code{n_missing_values} is nonzero. It
332 has the same number of 8-byte elements as the absolute value of
333 @code{n_missing_values}. Each element is interpreted as a number for
334 numeric variables (with HIGHEST and LOWEST indicated as described in
335 the chapter introduction). For string variables of width less than 8
336 bytes, elements are right-padded with spaces; for string variables
337 wider than 8 bytes, only the first 8 bytes of each missing value are
338 specified, with the remainder implicitly all spaces.
340 For discrete missing values, each element represents one missing
341 value. When a range is present, the first element denotes the minimum
342 value in the range, and the second element denotes the maximum value
343 in the range. When a range plus a value are present, the third
344 element denotes the additional discrete missing value.
347 The @code{print} and @code{write} members of sysfile_variable are output
348 formats coded into @code{int32} types. The least-significant byte
349 of the @code{int32} represents the number of decimal places, and the
350 next two bytes in order of increasing significance represent field width
351 and format type, respectively. The most-significant byte is not
352 used and should be set to zero.
354 Format types are defined as follows:
357 @multitable {Value} {@code{DATETIME}}
443 A few system files have been observed in the wild with invalid
444 @code{write} fields, in particular with value 0. Readers should
445 probably treat invalid @code{print} or @code{write} fields as some
448 @node Value Labels Records
449 @section Value Labels Records
451 The value label records documented in this section are used for
452 numeric and short string variables only. Long string variables may
453 have value labels, but their value labels are recorded using a
454 different record type (@pxref{Long String Value Labels Record}).
456 The value label record has the following format:
462 /* @r{Repeated @code{label_cnt} times}. */
469 @item int32 rec_type;
470 Record type. Always set to 3.
472 @item int32 label_count;
473 Number of value labels present in this record.
476 The remaining fields are repeated @code{count} times. Each
477 repetition specifies one value label.
481 A numeric value or a short string value padded as necessary to 8 bytes
482 in length. Its type and width cannot be determined until the
483 following value label variables record (see below) is read.
485 @item char label_len;
486 The label's length, in bytes. The documented maximum length varies
487 from 60 to 120 based on SPSS version. PSPP supports value labels up
491 @code{label_len} bytes of the actual label, followed by up to 7 bytes
492 of padding to bring @code{label} and @code{label_len} together to a
493 multiple of 8 bytes in length.
496 The value label record is always immediately followed by a value label
497 variables record with the following format:
506 @item int32 rec_type;
507 Record type. Always set to 4.
509 @item int32 var_count;
510 Number of variables that the associated value labels from the value
511 label record are to be applied.
514 A list of dictionary indexes of variables to which to apply the value
515 labels (@pxref{Dictionary Index}). There are @code{var_count}
518 String variables wider than 8 bytes may not be specified in this list.
521 @node Document Record
522 @section Document Record
524 The document record, if present, has the following format:
533 @item int32 rec_type;
534 Record type. Always set to 6.
537 Number of lines of documents present.
539 @item char lines[][80];
540 Document lines. The number of elements is defined by @code{n_lines}.
541 Lines shorter than 80 characters are padded on the right with spaces.
544 @node Machine Integer Info Record
545 @section Machine Integer Info Record
547 The integer info record, if present, has the following format:
559 int32 version_revision;
561 int32 floating_point_rep;
562 int32 compression_code;
564 int32 character_code;
568 @item int32 rec_type;
569 Record type. Always set to 7.
572 Record subtype. Always set to 3.
575 Size of each piece of data in the data part, in bytes. Always set to 4.
578 Number of pieces of data in the data part. Always set to 8.
580 @item int32 version_major;
581 PSPP major version number. In version @var{x}.@var{y}.@var{z}, this
584 @item int32 version_minor;
585 PSPP minor version number. In version @var{x}.@var{y}.@var{z}, this
588 @item int32 version_revision;
589 PSPP version revision number. In version @var{x}.@var{y}.@var{z},
592 @item int32 machine_code;
593 Machine code. PSPP always set this field to value to -1, but other
596 @item int32 floating_point_rep;
597 Floating point representation code. For IEEE 754 systems this is 1.
598 IBM 370 sets this to 2, and DEC VAX E to 3.
600 @item int32 compression_code;
601 Compression code. Always set to 1, regardless of whether or how the
604 @item int32 endianness;
605 Machine endianness. 1 indicates big-endian, 2 indicates little-endian.
607 @item int32 character_code;
608 @anchor{character-code} Character code. The following values have
609 been actually observed in system files:
619 The @code{windows-1250} code page for Central European and Eastern
623 The @code{windows-1252} code page for Western European languages.
632 The following additional values are known to be defined:
642 Other Windows code page numbers are known to be generally valid.
644 Old versions of SPSS for Unix and Windows always wrote value 2 in this
645 field, regardless of the encoding in use. Newer versions also write
646 the character encoding as a string (see @ref{Character Encoding
650 @node Machine Floating-Point Info Record
651 @section Machine Floating-Point Info Record
653 The floating-point info record, if present, has the following format:
669 @item int32 rec_type;
670 Record type. Always set to 7.
673 Record subtype. Always set to 4.
676 Size of each piece of data in the data part, in bytes. Always set to 8.
679 Number of pieces of data in the data part. Always set to 3.
682 The system missing value.
685 The value used for HIGHEST in missing values.
688 The value used for LOWEST in missing values.
691 @node Multiple Response Sets Records
692 @section Multiple Response Sets Records
694 The system file format has two different types of records that
695 represent multiple response sets (@pxref{MRSETS,,,pspp, PSPP Users
696 Guide}). The first type of record describes multiple response sets
697 that can be understood by SPSS before version 14. The second type of
698 record, with a closely related format, is used for multiple dichotomy
699 sets that use the CATEGORYLABELS=COUNTEDVALUES feature added in
709 /* @r{Exactly @code{count} bytes of data.} */
714 @item int32 rec_type;
715 Record type. Always set to 7.
718 Record subtype. Set to 7 for records that describe multiple response
719 sets understood by SPSS before version 14, or to 19 for records that
720 describe dichotomy sets that use the CATEGORYLABELS=COUNTEDVALUES
721 feature added in version 14.
724 The size of each element in the @code{mrsets} member. Always set to 1.
727 The total number of bytes in @code{mrsets}.
730 A series of multiple response sets, each of which consists of the
735 The set's name (an identifier that begins with @samp{$}), in mixed
736 upper and lower case.
739 An equals sign (@samp{=}).
742 @samp{C} for a multiple category set, @samp{D} for a multiple
743 dichotomy set with CATEGORYLABELS=VARLABELS, or @samp{E} for a
744 multiple dichotomy set with CATEGORYLABELS=COUNTEDVALUES.
747 For a multiple dichotomy set with CATEGORYLABELS=COUNTEDVALUES, a
748 space, followed by a number expressed as decimal digits, followed by a
749 space. If LABELSOURCE=VARLABEL was specified on MRSETS, then the
750 number is 11; otherwise it is 1.@footnote{This part of the format may
751 not be fully understood, because only a single example of each
752 possibility has been examined.}
755 For either kind of multiple dichotomy set, the counted value, as a
756 positive integer count specified as decimal digits, followed by a
757 space, followed by as many string bytes as specified in the count. If
758 the set contains numeric variables, the string consists of the counted
759 integer value expressed as decimal digits. If the set contains string
760 variables, the string contains the counted string value. Either way,
761 the string may be padded on the right with spaces (older versions of
762 SPSS seem to always pad to a width of 8 bytes; newer versions don't).
768 The multiple response set's label, using the same format as for the
769 counted value for multiple dichotomy sets. A string of length 0 means
770 that the set does not have a label. A string of length 0 is also
771 written if LABELSOURCE=VARLABEL was specified.
777 The short names of the variables in the set, converted to lowercase,
778 each separated from the previous by a single space.
781 A line feed (byte 0x0a).
785 Example: Given appropriate variable definitions, consider the
786 following MRSETS command:
789 MRSETS /MCGROUP NAME=$a LABEL='my mcgroup' VARIABLES=a b c
790 /MDGROUP NAME=$b VARIABLES=g e f d VALUE=55
791 /MDGROUP NAME=$c LABEL='mdgroup #2' VARIABLES=h i j VALUE='Yes'
792 /MDGROUP NAME=$d LABEL='third mdgroup' CATEGORYLABELS=COUNTEDVALUES
793 VARIABLES=k l m VALUE=34
794 /MDGROUP NAME=$e CATEGORYLABELS=COUNTEDVALUES LABELSOURCE=VARLABEL
795 VARIABLES=n o p VALUE='choice'.
798 The above would generate the following multiple response set record of
802 $a=C 10 my mcgroup a b c
804 $c=D3 Yes 10 mdgroup #2 h i j
807 It would also generate the following multiple response set record with
811 $d=E 1 2 34 13 third mdgroup k l m
812 $e=E 11 6 choice 0 n o p
815 @node Extra Product Info Record
816 @section Extra Product Info Record
818 This optional record appears to contain a text string that describes
819 the program that wrote the file and the source of the data. (This is
820 redundant with the file label and product info found in the file
830 /* @r{Exactly @code{count} bytes of data.} */
835 @item int32 rec_type;
836 Record type. Always set to 7.
839 Record subtype. Always set to 10.
842 The size of each element in the @code{info} member. Always set to 1.
845 The total number of bytes in @code{info}.
848 A text string. A product that identifies itself as @code{VOXCO
849 INTERVIEWER 4.3} uses CR-only line ends in this field, rather than the
850 more usual LF-only or CR LF line ends.
853 @node Variable Display Parameter Record
854 @section Variable Display Parameter Record
856 The variable display parameter record, if present, has the following
866 /* @r{Repeated @code{count} times}. */
868 int32 width; /* @r{Not always present.} */
873 @item int32 rec_type;
874 Record type. Always set to 7.
877 Record subtype. Always set to 11.
880 The size of @code{int32}. Always set to 4.
883 The number of sets of variable display parameters (ordinarily the
884 number of variables in the dictionary), times 2 or 3.
887 The remaining members are repeated @code{count} times, in the same
888 order as the variable records. No element corresponds to variable
889 records that continue long string variables. The meanings of these
890 members are as follows:
894 The measurement type of the variable:
904 SPSS sometimes writes a @code{measure} of 0. PSPP interprets this as
908 The width of the display column for the variable in characters.
910 This field is present if @var{count} is 3 times the number of
911 variables in the dictionary. It is omitted if @var{count} is 2 times
912 the number of variables.
914 @item int32 alignment;
915 The alignment of the variable for display purposes:
927 @node Long Variable Names Record
928 @section Long Variable Names Record
930 If present, the long variable names record has the following format:
939 /* @r{Exactly @code{count} bytes of data.} */
940 char var_name_pairs[];
944 @item int32 rec_type;
945 Record type. Always set to 7.
948 Record subtype. Always set to 13.
951 The size of each element in the @code{var_name_pairs} member. Always set to 1.
954 The total number of bytes in @code{var_name_pairs}.
956 @item char var_name_pairs[];
957 A list of @var{key}--@var{value} tuples, where @var{key} is the name
958 of a variable, and @var{value} is its long variable name.
959 The @var{key} field is at most 8 bytes long and must match the
960 name of a variable which appears in the variable record (@pxref{Variable
962 The @var{value} field is at most 64 bytes long.
963 The @var{key} and @var{value} fields are separated by a @samp{=} byte.
964 Each tuple is separated by a byte whose value is 09. There is no
965 trailing separator following the last tuple.
966 The total length is @code{count} bytes.
969 @node Very Long String Record
970 @section Very Long String Record
972 Old versions of SPSS limited string variables to a width of 255 bytes.
973 For backward compatibility with these older versions, the system file
974 format represents a string longer than 255 bytes, called a @dfn{very
975 long string}, as a collection of strings no longer than 255 bytes
976 each. The strings concatenated to make a very long string are called
977 its @dfn{segments}; for consistency, variables other than very long
978 strings are considered to have a single segment.
980 A very long string with a width of @var{w} has @var{n} =
981 (@var{w} + 251) / 252 segments, that is, one segment for every
982 252 bytes of width, rounding up. It would be logical, then, for each
983 of the segments except the last to have a width of 252 and the last
984 segment to have the remainder, but this is not the case. In fact,
985 each segment except the last has a width of 255 bytes. The last
986 segment has width @var{w} - (@var{n} - 1) * 252; some versions
987 of SPSS make it slightly wider, but not wide enough to make the last
988 segment require another 8 bytes of data.
990 Data is packed tightly into segments of a very long string, 255 bytes
991 per segment. Because 255 bytes of segment data are allocated for
992 every 252 bytes of the very long string's width (approximately), some
993 unused space is left over at the end of the allocated segments. Data
994 in unused space is ignored.
996 Example: Consider a very long string of width 20,000. Such a very
997 long string has 20,000 / 252 = 80 (rounding up) segments. The first
998 79 segments have width 255; the last segment has width 20,000 - 79 *
999 252 = 92 or slightly wider (up to 96 bytes, the next multiple of 8).
1000 The very long string's data is actually stored in the 19,890 bytes in
1001 the first 78 segments, plus the first 110 bytes of the 79th segment
1002 (19,890 + 110 = 20,000). The remaining 145 bytes of the 79th segment
1003 and all 92 bytes of the 80th segment are unused.
1005 The very long string record explains how to stitch together segments
1006 to obtain very long string data. For each of the very long string
1007 variables in the dictionary, it specifies the name of its first
1008 segment's variable and the very long string variable's actual width.
1009 The remaining segments immediately follow the named variable in the
1010 system file's dictionary.
1012 The very long string record, which is present only if the system file
1013 contains very long string variables, has the following format:
1022 /* @r{Exactly @code{count} bytes of data.} */
1023 char string_lengths[];
1027 @item int32 rec_type;
1028 Record type. Always set to 7.
1030 @item int32 subtype;
1031 Record subtype. Always set to 14.
1034 The size of each element in the @code{string_lengths} member. Always set to 1.
1037 The total number of bytes in @code{string_lengths}.
1039 @item char string_lengths[];
1040 A list of @var{key}--@var{value} tuples, where @var{key} is the name
1041 of a variable, and @var{value} is its length.
1042 The @var{key} field is at most 8 bytes long and must match the
1043 name of a variable which appears in the variable record (@pxref{Variable
1045 The @var{value} field is exactly 5 bytes long. It is a zero-padded,
1046 ASCII-encoded string that is the length of the variable.
1047 The @var{key} and @var{value} fields are separated by a @samp{=} byte.
1048 Tuples are delimited by a two-byte sequence @{00, 09@}.
1049 After the last tuple, there may be a single byte 00, or @{00, 09@}.
1050 The total length is @code{count} bytes.
1053 @node Character Encoding Record
1054 @section Character Encoding Record
1056 This record, if present, indicates the character encoding for string data,
1057 long variable names, variable labels, value labels and other strings in the
1067 /* @r{Exactly @code{count} bytes of data.} */
1072 @item int32 rec_type;
1073 Record type. Always set to 7.
1075 @item int32 subtype;
1076 Record subtype. Always set to 20.
1079 The size of each element in the @code{encoding} member. Always set to 1.
1082 The total number of bytes in @code{encoding}.
1084 @item char encoding[];
1085 The name of the character encoding. Normally this will be an official
1086 IANA character set name or alias.
1087 See @url{http://www.iana.org/assignments/character-sets}.
1088 Character set names are not case-sensitive, but SPSS appears to write
1089 them in all-uppercase.
1092 This record is not present in files generated by older software. See
1093 also the @code{character_code} field in the machine integer info
1094 record (@pxref{character-code}).
1096 When the character encoding record and the machine integer info record
1097 are both present, all system files observed in practice indicate the
1098 same character encoding, e.g.@: 1252 as @code{character_code} and
1099 @code{windows-1252} as @code{encoding}, 65001 and @code{UTF-8}, etc.
1101 If, for testing purposes, a file is crafted with different
1102 @code{character_code} and @code{encoding}, it seems that
1103 @code{character_code} controls the encoding for all strings in the
1104 system file before the dictionary termination record, including
1105 strings in data (e.g.@: string missing values), and @code{encoding}
1106 controls the encoding for strings following the dictionary termination
1109 @node Long String Value Labels Record
1110 @section Long String Value Labels Record
1112 This record, if present, specifies value labels for long string
1122 /* @r{Repeated up to exactly @code{count} bytes.} */
1127 long_string_label labels[];
1131 @item int32 rec_type;
1132 Record type. Always set to 7.
1134 @item int32 subtype;
1135 Record subtype. Always set to 21.
1141 The number of bytes following the header until the next header.
1143 @item int32 var_name_len;
1144 @itemx char var_name[];
1145 The number of bytes in the name of the variable that has long string
1146 value labels, plus the variable name itself, which consists of exactly
1147 @code{var_name_len} bytes. The variable name is not padded to any
1148 particular boundary, nor is it null-terminated.
1150 @item int32 var_width;
1151 The width of the variable, in bytes, which will be between 9 and
1154 @item int32 n_labels;
1155 @itemx long_string_label labels[];
1156 The long string labels themselves. The @code{labels} array contains
1157 exactly @code{n_labels} elements, each of which has the following
1168 @item int32 value_len;
1169 @itemx char value[];
1170 The string value being labeled. @code{value_len} is the number of
1171 bytes in @code{value}; it is equal to @code{var_width}. The
1172 @code{value} array is not padded or null-terminated.
1174 @item int32 label_len;
1175 @itemx char label[];
1176 The label for the string value. @code{label_len}, which must be
1177 between 0 and 120, is the number of bytes in @code{label}. The
1178 @code{label} array is not padded or null-terminated.
1182 @node Long String Missing Values Record
1183 @section Long String Missing Values Record
1185 This record, if present, specifies missing values for long string
1195 /* @r{Repeated up to exactly @code{count} bytes.} */
1198 char n_missing_values;
1199 long_string_missing_value values[];
1203 @item int32 rec_type;
1204 Record type. Always set to 7.
1206 @item int32 subtype;
1207 Record subtype. Always set to 22.
1213 The number of bytes following the header until the next header.
1215 @item int32 var_name_len;
1216 @itemx char var_name[];
1217 The number of bytes in the name of the long string variable that has
1218 missing values, plus the variable name itself, which consists of
1219 exactly @code{var_name_len} bytes. The variable name is not padded to
1220 any particular boundary, nor is it null-terminated.
1222 @item char n_missing_values;
1223 The number of missing values, either 1, 2, or 3. (This is, unusually,
1224 a single byte instead of a 32-bit number.)
1226 @item long_string_missing_value values[];
1227 The missing values themselves. This array contains exactly
1228 @code{n_missing_values} elements, each of which has the following
1237 @item int32 value_len;
1238 The length of the missing value string, in bytes. This value should
1239 be 8, because long string variables are at least 8 bytes wide (by
1240 definition), only the first 8 bytes of a long string variable's
1241 missing values are allowed to be non-spaces, and any spaces within the
1242 first 8 bytes are included in the missing value here.
1245 The missing value string, exactly @code{value_len} bytes, without
1246 any padding or null terminator.
1250 @node Data File and Variable Attributes Records
1251 @section Data File and Variable Attributes Records
1253 The data file and variable attributes records represent custom
1254 attributes for the system file or for individual variables in the
1255 system file, as defined on the DATAFILE ATTRIBUTE (@pxref{DATAFILE
1256 ATTRIBUTE,,,pspp, PSPP Users Guide}) and VARIABLE ATTRIBUTE commands
1257 (@pxref{VARIABLE ATTRIBUTE,,,pspp, PSPP Users Guide}), respectively.
1266 /* @r{Exactly @code{count} bytes of data.} */
1271 @item int32 rec_type;
1272 Record type. Always set to 7.
1274 @item int32 subtype;
1275 Record subtype. Always set to 17 for a data file attribute record or
1276 to 18 for a variable attributes record.
1279 The size of each element in the @code{attributes} member. Always set to 1.
1282 The total number of bytes in @code{attributes}.
1284 @item char attributes[];
1285 The attributes, in a text-based format.
1287 In record type 17, this field contains a single attribute set. An
1288 attribute set is a sequence of one or more attributes concatenated
1289 together. Each attribute consists of a name, which has the same
1290 syntax as a variable name, followed by, inside parentheses, a sequence
1291 of one or more values. Each value consists of a string enclosed in
1292 single quotes (@code{'}) followed by a line feed (byte 0x0a). A value
1293 may contain single quote characters, which are not themselves escaped
1294 or quoted or required to be present in pairs. There is no apparent
1295 way to embed a line feed in a value. There is no distinction between
1296 an attribute with a single value and an attribute array with one
1299 In record type 18, this field contains a sequence of one or more
1300 variable attribute sets. If more than one variable attribute set is
1301 present, each one after the first is delimited from the previous by
1302 @code{/}. Each variable attribute set consists of a long
1304 followed by @code{:}, followed by an attribute set with the same
1305 syntax as on record type 17.
1307 The total length is @code{count} bytes.
1312 A system file produced with the following VARIABLE ATTRIBUTE commands
1316 VARIABLE ATTRIBUTE VARIABLES=dummy ATTRIBUTE=fred[1]('23') fred[2]('34').
1317 VARIABLE ATTRIBUTE VARIABLES=dummy ATTRIBUTE=bert('123').
1321 will contain a variable attribute record with the following contents:
1324 00000000 07 00 00 00 12 00 00 00 01 00 00 00 22 00 00 00 |............"...|
1325 00000010 64 75 6d 6d 79 3a 66 72 65 64 28 27 32 33 27 0a |dummy:fred('23'.|
1326 00000020 27 33 34 27 0a 29 62 65 72 74 28 27 31 32 33 27 |'34'.)bert('123'|
1327 00000030 0a 29 |.) |
1334 @node Variable Roles
1335 @subsection Variable Roles
1337 A variable's role is represented as an attribute named @code{$@@Role}.
1338 This attribute has a single element whose values and their meanings
1343 Input. This, the default, is the most common role.
1356 @node Extended Number of Cases Record
1357 @section Extended Number of Cases Record
1359 The file header record expresses the number of cases in the system
1360 file as an int32 (@pxref{File Header Record}). This record allows the
1361 number of cases in the system file to be expressed as a 64-bit number.
1373 @item int32 rec_type;
1374 Record type. Always set to 7.
1376 @item int32 subtype;
1377 Record subtype. Always set to 16.
1380 Size of each element. Always set to 8.
1383 Number of pieces of data in the data part. Alway set to 2.
1385 @item int64 unknown;
1386 Meaning unknown. Always set to 1.
1388 @item int64 ncases64;
1389 Number of cases in the file as a 64-bit integer. Presumably this
1390 could be -1 to indicate that the number of cases is unknown, for the
1391 same reason as @code{ncases} in the file header record, but this has
1392 not been observed in the wild.
1395 @node Miscellaneous Informational Records
1396 @section Miscellaneous Informational Records
1398 Some specific types of miscellaneous informational records are
1399 documented here, but others are known to exist. PSPP ignores unknown
1400 miscellaneous informational records when reading system files.
1409 /* @r{Exactly @code{size * count} bytes of data.} */
1414 @item int32 rec_type;
1415 Record type. Always set to 7.
1417 @item int32 subtype;
1418 Record subtype. May take any value. According to Aapi
1419 H@"am@"al@"ainen, value 5 indicates a set of grouped variables and 6
1420 indicates date info (probably related to USE). Subtype 24 appears to
1421 contain XML that describes how data in the file should be displayed
1425 Size of each piece of data in the data part. Should have the value 1,
1426 4, or 8, for @code{char}, @code{int32}, and @code{flt64} format data,
1430 Number of pieces of data in the data part.
1433 Arbitrary data. There must be @code{size} times @code{count} bytes of
1437 @node Dictionary Termination Record
1438 @section Dictionary Termination Record
1440 The dictionary termination record separates all other records from the
1449 @item int32 rec_type;
1450 Record type. Always set to 999.
1453 Ignored padding. Should be set to 0.
1457 @section Data Record
1459 The data record must follow all other records in the system file.
1460 Every system file must have a data record that specifies data for at
1461 least one case. The format of the data record varies depending on the
1462 value of @code{compression} in the file header record:
1465 @item 0: no compression
1466 Data is arranged as a series of 8-byte elements.
1467 Each element corresponds to
1468 the variable declared in the respective variable record (@pxref{Variable
1469 Record}). Numeric values are given in @code{flt64} format; string
1470 values are literal characters string, padded on the right when
1471 necessary to fill out 8-byte units.
1473 @item 1: bytecode compression
1475 of the data record is divided into a series of 1-byte command
1476 codes. These codes have meanings as described below:
1480 Ignored. If the program writing the system file accumulates compressed
1481 data in blocks of fixed length, 0 bytes can be used to pad out extra
1482 bytes remaining at the end of a fixed-size block.
1486 value @var{code} - @var{bias}, where
1487 @var{code} is the value of the compression code and @var{bias} is the
1488 variable @code{bias} from the file header. For example,
1489 code 105 with bias 100.0 (the normal value) indicates a numeric variable
1491 One file has been seen written by SPSS 14 that contained such a code
1492 in a @emph{string} field with the value 0 (after the bias is
1493 subtracted) as a way of encoding null bytes.
1496 End of file. This code may or may not appear at the end of the data
1497 stream. PSPP always outputs this code but its use is not required.
1500 A numeric or string value that is not
1501 compressible. The value is stored in the 8 bytes following the
1502 current block of command bytes. If this value appears twice in a block
1503 of command bytes, then it indicates the second group of 8 bytes following the
1504 command bytes, and so on.
1507 An 8-byte string value that is all spaces.
1510 The system-missing value.
1513 The end of the 8-byte group of bytecodes is followed by any 8-byte
1514 blocks of non-compressible values indicated by code 253. After that
1515 follows another 8-byte group of bytecodes, then those bytecodes'
1516 non-compressible values. The pattern repeats to the end of the file
1517 or a code with value 252.
1519 @item 2: ZLIB compression
1520 The data record consists of the following, in order:
1524 ZLIB data header, 24 bytes long.
1527 One or more variable-length blocks of ZLIB compressed data.
1530 ZLIB data trailer, with a 24-byte fixed header plus an additional 24
1531 bytes for each preceding ZLIB compressed data block.
1534 The ZLIB data header has the following format:
1543 @item int64 zheader_ofs;
1544 The offset, in bytes, of the beginning of this structure within the
1547 @item int64 ztrailer_ofs;
1548 The offset, in bytes, of the first byte of the ZLIB data trailer.
1550 @item int64 ztrailer_len;
1551 The number of bytes in the ZLIB data trailer. This and the previous
1552 field sum to the size of the system file in bytes.
1555 The data header is followed by @code{(ztrailer_ofs - 24) / 24} ZLIB
1556 compressed data blocks. Each ZLIB compressed data block begins with a
1557 ZLIB header as specified in RFC@tie{}1950, e.g.@: hex bytes @code{78
1558 01} (the only header yet observed in practice). Each block
1559 decompresses to a fixed number of bytes (in practice only
1560 @code{0x3ff000}-byte blocks have been observed), except that the last
1561 block of data may be shorter. The last ZLIB compressed data block
1562 gends just before offset @code{ztrailer_ofs}.
1564 The result of ZLIB decompression is bytecode compressed data as
1565 described above for compression format 1.
1567 The ZLIB data trailer begins with the following 24-byte fixed header:
1577 @item int64 int_bias;
1578 The compression bias as a negative integer, e.g.@: if @code{bias} in
1579 the file header record is 100.0, then @code{int_bias} is @minus{}100
1580 (this is the only value yet observed in practice).
1583 Always observed to be zero.
1585 @item int32 block_size;
1586 The number of bytes in each ZLIB compressed data block, except
1587 possibly the last, following decompression. Only @code{0x3ff000} has
1588 been observed so far.
1590 @item int32 n_blocks;
1591 The number of ZLIB compressed data blocks, always exactly
1592 @code{(ztrailer_ofs - 24) / 24}.
1595 The fixed header is followed by @code{n_blocks} 24-byte ZLIB data
1596 block descriptors, each of which describes the compressed data block
1597 corresponding to its offset. Each block descriptor has the following
1601 int64 uncompressed_ofs;
1602 int64 compressed_ofs;
1603 int32 uncompressed_size;
1604 int32 compressed_size;
1608 @item int64 uncompressed_ofs;
1609 The offset, in bytes, that this block of data would have in a similar
1610 system file that uses compression format 1. This is
1611 @code{zheader_ofs} in the first block descriptor, and in each
1612 succeeding block descriptor it is the sum of the previous desciptor's
1613 @code{uncompressed_ofs} and @code{uncompressed_size}.
1615 @item int64 compressed_ofs;
1616 The offset, in bytes, of the actual beginning of this compressed data
1617 block. This is @code{zheader_ofs + 24} in the first block descriptor,
1618 and in each succeeding block descriptor it is the sum of the previous
1619 descriptor's @code{compressed_ofs} and @code{compressed_size}. The
1620 final block descriptor's @code{compressed_ofs} and
1621 @code{compressed_size} sum to @code{ztrailer_ofs}.
1623 @item int32 uncompressed_size;
1624 The number of bytes in this data block, after decompression. This is
1625 @code{block_size} in every data block except the last, which may be
1628 @item int32 compressed_size;
1629 The number of bytes in this data block, as stored compressed in this
1634 @setfilename ignored