Data File Format
****************

PSPP necessarily uses the same format for system files as do the products with which it is compatible.  This chapter is a description of that format.

There are three data types used in system files: 32-bit integers, 64-bit floating points, and 1-byte characters.  In this document these will simply be referred to as int32, flt64, and char, the names that are used in the PSPP source code.  Every field of type int32 or flt64 is aligned on a 32-bit boundary relative to the start of the record.

The endianness of data in PSPP system files is not specified.  System files output on a computer of a particular endianness will have the endianness of that computer.  However, PSPP can read files of either endianness, regardless of its host computer's endianness.  PSPP translates endianness for both integer and floating point numbers.

Floating point formats are also not specified.  PSPP does not translate between floating point formats.  This is unlikely to be a problem as all modern computer architectures use IEEE 754 format for floating point representation.

The PSPP system-missing value is represented by the largest possible negative number in the floating point format; in C, this is most likely -DBL_MAX.  There are two other important values used in missing values: HIGHEST and LOWEST.  These are represented by the largest possible positive number (probably DBL_MAX) and the second-largest negative number.  The latter must be determined in a system-dependent manner; in IEEE 754 format it is represented by value 0xffeffffffffffffe.

System files are divided into records.  Each record begins with an int32 giving a numeric record type.  Individual record types are described below:

File Header Record
==================

The file header is always the first record in the file.

     struct sysfile_header
       {
         char                rec_type[4];
         char                prod_name[60];
         int32               layout_code;
         int32               nominal_case_size;
         int32               compressed;
         int32               weight_index;
         int32               ncases;
         flt64               bias;
         char                creation_date[9];
         char                creation_time[8];
         char                file_label[64];
         char                padding[3];
       };

char rec_type[4];
     Record type code.  Always set to $FL2.  This is the only record for which the record type is not of type int32.

char prod_name[60];
     Product identification string.  This always begins with the characters @(#) SPSS DATA FILE.  PSPP uses the remaining characters to give its version and the operating system name; for example, GNU pspp 0.1.4 - sparc-sun-solaris2.5.2.  The string is truncated if it would be longer than 60 characters; otherwise it is padded on the right with spaces.

int32 layout_code;
     Always set to 2.  PSPP reads this value to determine the file's endianness.

int32 nominal_case_size;
     Number of data elements per case.  This is the number of variables, except that long string variables add extra data elements (one for every 8 characters after the first 8).  However, string variables do not contribute to this value beyond the first 255 bytes.   Further, system files written by some systems set this value to -1.  In general, it is unsafe for systems reading system files to rely upon this value.

int32 compressed;
     Set to 1 if the data in the file is compressed, 0 otherwise.

int32 weight_index;
     If one of the variables in the data set is used as a weighting variable, set to the index of that variable.  Otherwise, set to 0.

int32 ncases;
     Set to the number of cases in the file if it is known, or -1 otherwise.

     In the general case it is not possible to determine the number of cases that will be output to a system file at the time that the header is written.  The way that this is dealt with is by writing the entire system file, including the header, then seeking back to the beginning of the file and writing just the ncases field.  For `files' in which this is not valid, the seek operation fails.  In this case, ncases remains -1.

flt64 bias;
     Compression bias.  Always set to 100.  The significance of this value is that only numbers between (1 - bias) and (251 - bias) can be compressed.

char creation_date[9];
     Set to the date of creation of the system file, in dd mmm yy format, with the month as standard English abbreviations, using an initial capital letter and following with lowercase.  If the date is not available then this field is arbitrarily set to 01 Jan 70.

char creation_time[8];
     Set to the time of creation of the system file, in hh:mm:ss format and using 24-hour time.  If the time is not available then this field is arbitrarily set to 00:00:00.

char file_label[64];
     Set the the file label declared by the user, if any (see FILE LABEL). Padded on the right with spaces.

char padding[3];
     Ignored padding bytes to make the structure a multiple of 32 bits in length.  Set to zeros.

Variable Record
===============

Immediately following the header must come the variable records.  There must be one variable record for every variable and every 8 characters in a long string beyond the first 8.

     struct sysfile_variable
       {
         int32               rec_type;
         int32               type;
         int32               has_var_label;
         int32               n_missing_values;
         int32               print;
         int32               write;
         char                name[8];

         /* The following two fields are present
            only if has_var_label is 1. */
         int32               label_len;
         char                label[/* variable length */];

         /* The following field is present only
            if n_missing_values is not 0. */
         flt64               missing_values[/* variable length */];
       };

int32 rec_type;
     Record type code.  Always set to 2.

int32 type;
     Variable type code.  Set to 0 for a numeric variable.  For a short string variable or the first part of a long string variable, this is set to the width of the string.  For the second and subsequent parts of a long string variable, set to -1, and the remaining fields in the structure are ignored.

int32 has_var_label;
     If this variable has a variable label, set to 1; otherwise, set to 0.

int32 n_missing_values;
     If the variable has no missing values, set to 0.  If the variable has one, two, or three discrete missing values, set to 1, 2, or 3, respectively.  If the variable has a range for missing variables, set to -2; if the variable has a range for missing variables plus a single discrete value, set to -3.

int32 print;
     Print format for this variable.  See below.

int32 write;
     Write format for this variable.  See below.

char name[8];
     Variable name.  The variable name must begin with a capital letter or the at-sign (@).  Subsequent characters may also be octothorpes (#), dollar signs ($), underscores (_), or full stops (.).  The variable name is padded on the right with spaces.

int32 label_len;
     This field is present only if has_var_label is set to 1.  It is set to the length, in characters, of the variable label, which must be a number between 0 and 120.

char label[/* variable length */];
     This field is present only if has_var_label is set to 1.  It has length label_len, rounded up to the nearest multiple of 32 bits. The first label_len characters are the variable's variable label.

flt64 missing_values[/* variable length */];
     This field is present only if n_missing_values is not 0.  It has the same number of elements as the absolute value of n_missing_values.  For discrete missing values, each element represents one missing value.  When a range is present, the first element denotes the minimum value in the range, and the second element denotes the maximum value in the range.  When a range plus a value are present, the third element denotes the additional discrete missing value.  HIGHEST and LOWEST are indicated as described in the chapter introduction.

The print and write members of sysfile_variable are output formats coded into int32 types.  The LSB (least-significant byte) of the int32 represents the number of decimal places, and the next two bytes in order of increasing significance represent field width and format type, respectively.  The MSB (most-significant byte) is not used and should be set to zero.

Format types are defined as follows:

0
     Not used.

1
     A

2
     AHEX

3
     COMMA

4
     DOLLAR

5
     F

6
     IB

7
     PIBHEX

8
     P

9
     PIB

10
     PK

11
     RB

12
     RBHEX

13
     Not used.

14
     Not used.

15
     Z

16
     N

17
     E

18
     Not used.

19
     Not used.

20
     DATE

21
     TIME

22
     DATETIME

23
     ADATE

24
     JDATE

25
     DTIME

26
     WKDAY

27
     MONTH

28
     MOYR

29
     QYR

30
     WKYR

31
     PCT

32
     DOT

33
     CCA

34
     CCB

35
     CCC

36
     CCD

37
     CCE

38
     EDATE

39
     SDATE

Value Label Record
==================

Value label records must follow the variable records and must precede the header termination record.  Other than this, they may appear anywhere in the system file.  Every value label record must be immediately followed by a label variable record, described below.

Value label records begin with rec_type, an int32 value set to the record type of 3.  This is followed by count, an int32 value set to the number of value labels present in this record.

These two fields are followed by a series of count tuples.  Each tuple is divided into two fields, the value and the label.  The first of these, the value, is composed of a 64-bit value, which is either a flt64 value or up to 8 characters (padded on the right to 8 bytes) denoting a short string value.  Whether the value is a flt64 or a character string is not defined inside the value label record.

The second field in the tuple, the label, has variable length.  The first char is a count of the number of characters in the value label.  The remainder of the field is the label itself.  The field is padded on the right to a multiple of 64 bits in length.

Value Label Variable Record
===========================

Every value label variable record must be immediately preceded by a value label record, described above.

     struct sysfile_value_label_variable
       {
          int32              rec_type;
          int32              count;
          int32              vars[/* variable length */];
       };

int32 rec_type;
     Record type.  Always set to 4.

int32 count;
     Number of variables that the associated value labels from the value label record are to be applied.

int32 vars[/* variable length */];
     A list of variables to which to apply the value labels.  There are count elements.  Each element identifies a variable record, where the first element is numbered 1 and long string variables are considered to occupy multiple indexes.

Document Record
===============

There must be no more than one document record per system file. Document records must follow the variable records and precede the dictionary termination record.

     struct sysfile_document
       {
         int32               rec_type;
         int32               n_lines;
         char                lines[/* variable length */][80];
       };

int32 rec_type;
     Record type.  Always set to 6.

int32 n_lines;
     Number of lines of documents present.

char lines[/* variable length */][80];
     Document lines.  The number of elements is defined by n_lines. Lines shorter than 80 characters are padded on the right with spaces.

Machine int32 Info Record
=========================

There must be no more than one machine int32 info record per system file.  Machine int32 info records must follow the variable records and precede the dictionary termination record.

     struct sysfile_machine_int32_info
       {
         /* Header. */
         int32               rec_type;
         int32               subtype;
         int32               size;
         int32               count;

         /* Data. */
         int32               version_major;
         int32               version_minor;
         int32               version_revision;
         int32               machine_code;
         int32               floating_point_rep;
         int32               compression_code;
         int32               endianness;
         int32               character_code;
       };

int32 rec_type;
     Record type.  Always set to 7.

int32 subtype;
     Record subtype.  Always set to 3.

int32 size;
     Size of each piece of data in the data part, in bytes.  Always set to 4.

int32 count;
     Number of pieces of data in the data part.  Always set to 8.

int32 version_major;
     PSPP major version number.  In version X.Y.Z, this is X.

int32 version_minor;
     PSPP minor version number.  In version X.Y.Z, this is Y.

int32 version_revision;
     PSPP version revision number.  In version X.Y.Z, this is Z.

int32 machine_code;
     Machine code.  PSPP always set this field to value to -1, but other values may appear.

int32 floating_point_rep;
     Floating point representation code.  For IEEE 754 systems this is 1. IBM 370 sets this to 2, and DEC VAX E to 3.

int32 compression_code;
     Compression code.  Always set to 1.

int32 endianness;
     Machine endianness.  1 indicates big-endian, 2 indicates little-endian.

int32 character_code;
     Character code.  1 indicates EBCDIC, 2 indicates 7-bit ASCII, 3 indicates 8-bit ASCII, 4 indicates DEC Kanji.

Machine flt64 Info Record
=========================

There must be no more than one machine flt64 info record per system file.  Machine flt64 info records must follow the variable records and precede the dictionary termination record.

     struct sysfile_machine_flt64_info
       {
         /* Header. */
         int32               rec_type;
         int32               subtype;
         int32               size;
         int32               count;

         /* Data. */
         flt64               sysmis;
         flt64               highest;
         flt64               lowest;
       };

int32 rec_type;
     Record type.  Always set to 7.

int32 subtype;
     Record subtype.  Always set to 4.

int32 size;
     Size of each piece of data in the data part, in bytes.  Always set to 4.

int32 count;
     Number of pieces of data in the data part.  Always set to 3.

flt64 sysmis;
     The system missing value.

flt64 highest;
     The value used for HIGHEST in missing values.

flt64 lowest;
     The value used for LOWEST in missing values.

Auxiliary Variable Parameter Record
===================================

There must be no more than one auxiliary variable parameter record per system file.  This  record must follow the variable records and precede the dictionary termination record.

     struct sysfile_aux_var_parameter
       {
         /* Header. */
         int32               rec_type;
         int32               subtype;
         int32               size;
         int32               count;

         /* Data. */
         struct aux_params   aux_params[/* variable length */];
       };

int32 rec_type;
     Record type.  Always set to 7.

int32 subtype;
     Record subtype.  Always set to 11.

int32 size;
     The size  int32. Always set to 4.

int32 count;
     The total number of records in aux_params, multiplied by 3.

struct aux_params aux_params[];
     An array of struct aux_params.   The order of the elements corresponds to the order of the variables in the Variable Records.  No element corresponds to variable records that continue long string variables. The struct aux_params type is defined as follows:

          struct aux_params
            {
              int32 measure;
              int32 width;
              int32 alignment;
            };

     int32 measure
          The measurement type of the variable:

          1
               Nominal Scale

          2
               Ordinal Scale

          3
               Continuous Scale

     int32 width
          The width of the display column for the variable in characters.

     int32 alignment
          The alignment of the variable for display purposes:

          0
               Left aligned

          1
               Right aligned

          2
               Centre aligned

Long Variable Names Record
==========================

There must be no more than one long variable names record per system file.  This  record must follow the variable records and precede the dictionary termination record.

     struct sysfile_long_variable_names
       {
         /* Header. */
         int32               rec_type;
         int32               subtype;
         int32               size;
         int32               count;

         /* Data. */
         char                var_name_pairs[/* variable length */];
       };

int32 rec_type;
     Record type.  Always set to 7.

int32 subtype;
     Record subtype.  Always set to 13.

int32 size;
     The size of each element in the var_name_pairs member. Always set to 1.

int32 count;
     The total number of bytes in var_name_pairs.

char var_name_pairs[/* variable length */];
     A list of KEY--VALUE tuples, where KEY is the name of a variable, and VALUE is its long variable name. The KEY field is at most 8 bytes long and must match the name of a variable which appears in the variable record (see Variable Record). The VALUE field is at most 64 bytes long. The KEY and VALUE fields are separated by a = byte. Each tuple is separated by a byte whose value is 09.  There is no trailing separator following the last tuple. The total length is count bytes.

Very Long String Length Record
==============================

There must be no more than one very long string length record per system file.  This  record must follow the variable records and precede the dictionary termination record.

     struct sysfile_very_long_string_lengths
       {
         /* Header. */
         int32               rec_type;
         int32               subtype;
         int32               size;
         int32               count;

         /* Data. */
         char                string_lengths[/* variable length */];
       };

int32 rec_type;
     Record type.  Always set to 7.

int32 subtype;
     Record subtype.  Always set to 14.

int32 size;
     The size of each element in the string_lengths member. Always set to 1.

int32 count;
     The total number of bytes in string_lengths.

char string_lengths[/* variable length */];
     A list of KEY--VALUE tuples, where KEY is the name of a variable, and VALUE is its length. The KEY field is at most 8 bytes long and must match the name of a variable which appears in the variable record (see Variable Record). The VALUE field is exactly 5 bytes long. It is a zero-padded, ASCII-encoded string that is the length of the variable. The KEY and VALUE fields are separated by a = byte. Tuples are delimited by a two-byte sequence {00, 09}. After the last tuple, there may be a single byte 00, or {00, 09}. The total length is count bytes.

Miscellaneous Informational Records
===================================

Miscellaneous informational records must follow the variable records and precede the dictionary termination record.

Some specific types of miscellaneous informational records are documented here, but others are known to exist.  PSPP ignores unknown miscellaneous informational records when reading system files.

     struct sysfile_misc_info
       {
         /* Header. */
         int32               rec_type;
         int32               subtype;
         int32               size;
         int32               count;

         /* Data. */
         char                data[/* variable length */];
       };

int32 rec_type;
     Record type.  Always set to 7.

int32 subtype;
     Record subtype.  May take any value.  According to Aapi Hämäläinen, value 5 indicates a set of grouped variables and 6 indicates date info (probably related to USE).

int32 size;
     Size of each piece of data in the data part.  Should have the value 4 or 8, for int32 and flt64, respectively.

int32 count;
     Number of pieces of data in the data part.

char data[/* variable length */];
     Arbitrary data.  There must be size times count bytes of data.

Dictionary Termination Record
=============================

The dictionary termination record must follow all other records, except for the actual cases, which it must precede.  There must be exactly one dictionary termination record in every system file.

     struct sysfile_dict_term
       {
         int32               rec_type;
         int32               filler;
       };

int32 rec_type;
     Record type.  Always set to 999.

int32 filler;
     Ignored padding.  Should be set to 0.

Data Record
===========

Data records must follow all other records in the data file.  There must be at least one data record in every system file.

The format of data records varies depending on whether the data is compressed.  Regardless, the data is arranged in a series of 8-byte elements.

When data is not compressed, each element corresponds to the variable declared in the respective variable record (see Variable Record).  Numeric values are given in flt64 format; string values are literal characters string, padded on the right when necessary.

Compressed data is arranged in the following manner: the first 8-byte element in the data section is divided into a series of 1-byte command codes.  These codes have meanings as described below:

0
     Ignored.  If the program writing the system file accumulates compressed data in blocks of fixed length, 0 bytes can be used to pad out extra bytes remaining at the end of a fixed-size block.

1 through 251
     These values indicate that the corresponding numeric variable has the value (CODE - BIAS) for the case being read, where CODE is the value of the compression code and BIAS is the variable compression_bias from the file header.  For example, code 105 with bias 100.0 (the normal value) indicates a numeric variable of value 5.

252
     End of file.  This code may or may not appear at the end of the data stream.  PSPP always outputs this code but its use is not required.

253
     This value indicates that the numeric or string value is not compressible.  The value is stored in the 8-byte element following the current block of command bytes.  If this value appears twice in a block of command bytes, then it indicates the second element following the command bytes, and so on.

254
     Used to indicate a string value that is all spaces.

255
     Used to indicate the system-missing value.

When the end of the first 8-byte element of command bytes is reached, any blocks of non-compressible values are skipped, and the next element of command bytes is read and interpreted, until the end of the file is reached.