Automatically infer variables' measurement level from format and data.

[pspp] / doc / dev / system-file-format.texi

diff --git a/doc/dev/system-file-format.texi b/doc/dev/system-file-format.texi
index e52b9571166a25b477a8f6263f34d0960b9f0423..17f5c1450fa0be9e996e3d47b79c5de6e96c4e34 100644 (file)
--- a/doc/dev/system-file-format.texi
+++ b/doc/dev/system-file-format.texi
@@ -1,3 +1,13 @@
+@c PSPP - a program for statistical analysis.
+@c Copyright (C) 2019 Free Software Foundation, Inc.
+@c Permission is granted to copy, distribute and/or modify this document
+@c under the terms of the GNU Free Documentation License, Version 1.3
+@c or any later version published by the Free Software Foundation;
+@c with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts.
+@c A copy of the license is included in the section entitled "GNU
+@c Free Documentation License".
+@c
+

 @node System File Format
 @appendix System File Format
 
 @node System File Format
 @appendix System File Format
 


@@ -6,7 +16,7 @@ that describes how they may be interpreted.  This chapter describes
 the format of a system file.
 
 System files use four data types: 8-bit characters, 32-bit integers,
 the format of a system file.
 
 System files use four data types: 8-bit characters, 32-bit integers,

-64-bit integers, 
+64-bit integers,

 and 64-bit floating points, called here @code{char}, @code{int32},
 @code{int64}, and
 @code{flt64}, respectively.  Data is not necessarily aligned on a word
 and 64-bit floating points, called here @code{char}, @code{int32},
 @code{int64}, and
 @code{flt64}, respectively.  Data is not necessarily aligned on a word


@@ -222,6 +232,12 @@ pspp 0.1.4 - sparc-sun-solaris2.5.2}.  The string is truncated if it
 would be longer than 60 characters; otherwise it is padded on the right
 with spaces.
 
 would be longer than 60 characters; otherwise it is padded on the right
 with spaces.
 

+The product name field allow readers to behave differently based on
+quirks in the way that particular software writes system files.
+@xref{Value Labels Records}, for the detail of the quirk that the PSPP
+system file reader tolerates in files written by ReadStat, which has
+@code{https://github.com/WizardMac/ReadStat} in @code{prod_name}.
+

 @anchor{layout_code}
 @item int32 layout_code;
 Normally set to 2, although a few system files have been spotted in
 @anchor{layout_code}
 @item int32 layout_code;
 Normally set to 2, although a few system files have been spotted in


@@ -310,10 +326,10 @@ so readers should take care to parse dummy variable records in the
 same way as other variable records.
 
 @anchor{Dictionary Index}
 same way as other variable records.
 
 @anchor{Dictionary Index}

-The @dfn{dictionary index} of a variable is its offset in the set of
+The @dfn{dictionary index} of a variable is a 1-based offset in the set of

 variable records, including dummy variable records for long string
 variable records, including dummy variable records for long string

-variables.  The first variable record has a dictionary index of 0, the
-second has a dictionary index of 1, and so on.
+variables.  The first variable record has a dictionary index of 1, the
+second has a dictionary index of 2, and so on.

 
 The system file format does not directly support string variables
 wider than 255 bytes.  Such very long string variables are represented
 
 The system file format does not directly support string variables
 wider than 255 bytes.  Such very long string variables are represented


@@ -508,6 +524,10 @@ Format types are defined as follows:
 @tab @code{EDATE}
 @item 39
 @tab @code{SDATE}
 @tab @code{EDATE}
 @item 39
 @tab @code{SDATE}

+@item 40
+@tab @code{MTIME}
+@item 41
+@tab @code{YMDHMS}

 @end multitable
 @end quotation
 
 @end multitable
 @end quotation
 


@@ -524,13 +544,21 @@ numeric and short string variables only.  Long string variables may
 have value labels, but their value labels are recorded using a
 different record type (@pxref{Long String Value Labels Record}).
 
 have value labels, but their value labels are recorded using a
 different record type (@pxref{Long String Value Labels Record}).
 

+ReadStat (@pxref{File Header Record}) writes value labels that label a
+single value more than once.  In more detail, it emits value labels
+whose values are longer than string variables' widths, that are
+identical in the actual width of the variable, e.g.@: labels for
+values @code{ABC123} and @code{ABC456} for a string variable with
+width 3.  For files written by this software, PSPP ignores such
+labels.
+

 The value label record has the following format:
 
 @example
 int32               rec_type;
 int32               label_count;
 
 The value label record has the following format:
 
 @example
 int32               rec_type;
 int32               label_count;
 

-/* @r{Repeated @code{label_cnt} times}. */
+/* @r{Repeated @code{n_label} times}. */

 char                value[8];
 char                label_len;
 char                label[];
 char                value[8];
 char                label_len;
 char                label[];


@@ -582,7 +610,7 @@ Number of variables that the associated value labels from the value
 label record are to be applied.
 
 @item int32 vars[];
 label record are to be applied.
 
 @item int32 vars[];

-A list of dictionary indexes of variables to which to apply the value
+A list of 1-based dictionary indexes of variables to which to apply the value

 labels (@pxref{Dictionary Index}).  There are @code{var_count}
 elements.
 
 labels (@pxref{Dictionary Index}).  There are @code{var_count}
 elements.
 


@@ -606,9 +634,7 @@ Record type.  Always set to 6.
 
 @item int32 n_lines;
 Number of lines of documents present.  This should be greater than
 
 @item int32 n_lines;
 Number of lines of documents present.  This should be greater than

-zero, but the system file writer that identifies itself as
-@url{https://github.com/WizardMac/ReadStat} writes document records
-with zero @code{n_lines}.
+zero, but ReadStats writes system files with zero @code{n_lines}.

 
 @item char lines[][80];
 Document lines.  The number of elements is defined by @code{n_lines}.
 
 @item char lines[][80];
 Document lines.  The number of elements is defined by @code{n_lines}.


@@ -975,18 +1001,24 @@ members are as follows:
 
 @table @code
 @item int32 measure;
 
 @table @code
 @item int32 measure;

-The measurement type of the variable:
+The measurement level of the variable:

 @table @asis
 @table @asis

+@item 0
+Unknown

 @item 1
 @item 1

-Nominal Scale
+Nominal

 @item 2
 @item 2

-Ordinal Scale
+Ordinal

 @item 3
 @item 3

-Continuous Scale
+Scale

 @end table
 
 @end table
 

-SPSS sometimes writes a @code{measure} of 0.  PSPP interprets this as
-nominal scale.
+An ``unknown'' @code{measure} of 0 means that the variable was created
+in some way that doesn't make the measurement level clear, e.g.@: with
+a @code{COMPUTE} transformation.  PSPP sets the measurement level the
+first time it reads the data using the rules documented in
+@ref{Measurement Level,,,pspp, PSPP Users Guide}, so this should
+rarely appear.

 
 @item int32 width;
 The width of the display column for the variable in characters.
 
 @item int32 width;
 The width of the display column for the variable in characters.


@@ -1492,7 +1524,8 @@ the following believed meanings:
 
 @table @asis
 @item 5
 
 @table @asis
 @item 5

-A set of grouped variables (according to Aapi H@"am@"al@"ainen).
+A named variable set for use in the GUI (according to Aapi
+H@"am@"al@"ainen).

 
 @item 6
 Date info, probably related to USE (according to Aapi H@"am@"al@"ainen).
 
 @item 6
 Date info, probably related to USE (according to Aapi H@"am@"al@"ainen).


@@ -1560,9 +1593,10 @@ value @var{code} - @var{bias}, where
 variable @code{bias} from the file header.  For example,
 code 105 with bias 100.0 (the normal value) indicates a numeric variable
 of value 5.
 variable @code{bias} from the file header.  For example,
 code 105 with bias 100.0 (the normal value) indicates a numeric variable
 of value 5.

-One file has been seen written by SPSS 14 that contained such a code
-in a @emph{string} field with the value 0 (after the bias is
-subtracted) as a way of encoding null bytes.
+
+A code of 0 (after subtracting the bias) in a string field encodes
+null bytes.  This is unusual, since a string field normally encodes
+text data, but it exists in real system files.

 
 @item 252
 End of file.  This code may or may not appear at the end of the data
 
 @item 252
 End of file.  This code may or may not appear at the end of the data


@@ -1624,7 +1658,7 @@ The number of bytes in the ZLIB data trailer.  This and the previous
 field sum to the size of the system file in bytes.
 @end table
 
 field sum to the size of the system file in bytes.
 @end table
 

-The data header is followed by @code{(ztrailer_ofs - 24) / 24} ZLIB
+The data header is followed by @code{(ztrailer_len - 24) / 24} ZLIB

 compressed data blocks.  Each ZLIB compressed data block begins with a
 ZLIB header as specified in RFC@tie{}1950, e.g.@: hex bytes @code{78
 01} (the only header yet observed in practice).  Each block
 compressed data blocks.  Each ZLIB compressed data block begins with a
 ZLIB header as specified in RFC@tie{}1950, e.g.@: hex bytes @code{78
 01} (the only header yet observed in practice).  Each block


@@ -1661,7 +1695,7 @@ been observed so far.
 
 @item int32 n_blocks;
 The number of ZLIB compressed data blocks, always exactly
 
 @item int32 n_blocks;
 The number of ZLIB compressed data blocks, always exactly

-@code{(ztrailer_ofs - 24) / 24}.
+@code{(ztrailer_len - 24) / 24}.

 @end table
 
 The fixed header is followed by @code{n_blocks} 24-byte ZLIB data
 @end table
 
 The fixed header is followed by @code{n_blocks} 24-byte ZLIB data