Merge remote branch 'origin/master' into import-gui

[pspp] / doc / dev / system-file-format.texi

diff --git a/doc/dev/system-file-format.texi b/doc/dev/system-file-format.texi
index 1309cb79195d7bcb04705b624e70c90abc251654..8315762cef52cb26ea20f1aa2757ca3d6f622554 100644 (file)
--- a/doc/dev/system-file-format.texi
+++ b/doc/dev/system-file-format.texi
@@ -60,7 +60,8 @@ if present.
 Document record, if present.
 
 @item
 Document record, if present.
 
 @item

-Any records not explicitly included in this list, in any order.
+Extension (type 7) records, in ascending numerical order of their
+subtypes.

 
 @item
 Dictionary termination record.
 
 @item
 Dictionary termination record.


@@ -114,7 +115,9 @@ char                padding[3];
 
 @table @code
 @item char rec_type[4];
 
 @table @code
 @item char rec_type[4];

-Record type code, set to @samp{$FL2}.
+Record type code, set to @samp{$FL2}, that is, either @code{24 46 4c
+32} if the file uses an ASCII-based character encoding, or @code{5b c6
+d3 f2} if the file uses an EBCDIC-based character encoding.

 
 @item char prod_name[60];
 Product identification string.  This always begins with the characters
 
 @item char prod_name[60];
 Product identification string.  This always begins with the characters


@@ -267,8 +270,10 @@ stops (@samp{.}).  The variable name is padded on the right with spaces.
 
 @item int32 label_len;
 This field is present only if @code{has_var_label} is set to 1.  It is
 
 @item int32 label_len;
 This field is present only if @code{has_var_label} is set to 1.  It is

-set to the length, in characters, of the variable label, which must be a
-number between 0 and 120.
+set to the length, in characters, of the variable label.  The
+documented maximum length varies from 120 to 255 based on SPSS
+version, but some files have been seen with longer labels.  PSPP
+accepts longer labels and truncates them to 255 bytes on input.

 
 @item char label[];
 This field is present only if @code{has_var_label} is set to 1.  It has
 
 @item char label[];
 This field is present only if @code{has_var_label} is set to 1.  It has


@@ -388,6 +393,11 @@ Format types are defined as follows:
 @end multitable
 @end quotation
 
 @end multitable
 @end quotation
 

+A few system files have been observed in the wild with invalid
+@code{write} fields, in particular with value 0.  Readers should
+probably treat invalid @code{print} or @code{write} fields as some
+default format.
+

 @node Value Labels Records
 @section Value Labels Records
 
 @node Value Labels Records
 @section Value Labels Records
 


@@ -426,7 +436,9 @@ in length.  Its type and width cannot be determined until the
 following value label variables record (see below) is read.
 
 @item char label_len;
 following value label variables record (see below) is read.
 
 @item char label_len;

-The label's length, in bytes.
+The label's length, in bytes.  The documented maximum length varies
+from 60 to 120 based on SPSS version.  PSPP supports value labels up
+to 255 bytes long.

 
 @item char label[];
 @code{label_len} bytes of the actual label, followed by up to 7 bytes
 
 @item char label[];
 @code{label_len} bytes of the actual label, followed by up to 7 bytes


@@ -545,14 +557,46 @@ Compression code.  Always set to 1.
 Machine endianness.  1 indicates big-endian, 2 indicates little-endian.
 
 @item int32 character_code;
 Machine endianness.  1 indicates big-endian, 2 indicates little-endian.
 
 @item int32 character_code;

-@anchor{character-code}
-Character code.  1 indicates EBCDIC, 2 indicates 7-bit ASCII, 3
-indicates 8-bit ASCII, 4 indicates DEC Kanji.
-Windows code page numbers are also valid.
-
-Experience has shown that in many files, this field is ignored or incorrect.
-For a more reliable indication of the file's character encoding
-see @ref{Character Encoding Record}.
+@anchor{character-code} Character code.  The following values have
+been actually observed in system files:
+
+@table @asis
+@item 1
+EBCDIC.
+
+@item 2
+7-bit ASCII.
+
+@item 1250
+The @code{windows-1250} code page for Central European and Eastern
+European languages.
+
+@item 1252
+The @code{windows-1252} code page for Western European languages.
+
+@item 28591
+ISO 8859-1.
+
+@item 65001
+UTF-8.
+@end table
+
+The following additional values are known to be defined:
+
+@table @asis
+@item 3
+8-bit ``ASCII''.
+
+@item 4
+DEC Kanji.
+@end table
+
+Other Windows code page numbers are known to be generally valid.
+
+Old versions of SPSS for Unix and Windows always wrote value 2 in this
+field, regardless of the encoding in use.  Newer versions also write
+the character encoding as a string (see @ref{Character Encoding
+Record}).

 @end table
 
 @node Machine Floating-Point Info Record
 @end table
 
 @node Machine Floating-Point Info Record


@@ -640,7 +684,8 @@ following:
 
 @itemize @bullet
 @item
 
 @itemize @bullet
 @item

-The set's name (an identifier that begins with @samp{$}).
+The set's name (an identifier that begins with @samp{$}), in mixed
+upper and lower case.

 
 @item
 An equals sign (@samp{=}).
 
 @item
 An equals sign (@samp{=}).


@@ -681,8 +726,8 @@ written if LABELSOURCE=VARLABEL was specified.
 A space.
 
 @item
 A space.
 
 @item

-The names of the variables in the set, each separated from the
-previous by a single space.
+The short names of the variables in the set, converted to lowercase,
+each separated from the previous by a single space.

 
 @item
 A line feed (byte 0x0a).
 
 @item
 A line feed (byte 0x0a).


@@ -770,8 +815,8 @@ Ordinal Scale
 Continuous Scale
 @end table
 
 Continuous Scale
 @end table
 

-SPSS 14 sometimes writes a @code{measure} of 0.  PSPP interprets this
-as nominal scale.
+SPSS 14 sometimes writes a @code{measure} of 0 for string variables.
+PSPP interprets this as nominal scale.

 
 @item int32 width;
 The width of the display column for the variable in characters.
 
 @item int32 width;
 The width of the display column for the variable in characters.


@@ -951,12 +996,29 @@ The size of each element in the @code{encoding} member. Always set to 1.
 The total number of bytes in @code{encoding}.
 
 @item char encoding[];
 The total number of bytes in @code{encoding}.
 
 @item char encoding[];

-The name of the character encoding.  Normally this will be an official IANA characterset name or alias.
+The name of the character encoding.  Normally this will be an official
+IANA character set name or alias.

 See @url{http://www.iana.org/assignments/character-sets}.
 See @url{http://www.iana.org/assignments/character-sets}.

+Character set names are not case-sensitive, but SPSS appears to write
+them in all-uppercase.

 @end table
 
 @end table
 

-This record is not present in files generated by older software.
-See also @ref{character-code}.
+This record is not present in files generated by older software.  See
+also the @code{character_code} field in the machine integer info
+record (@pxref{character-code}).
+
+When the character encoding record and the machine integer info record
+are both present, all system files observed in practice indicate the
+same character encoding, e.g.@: 1252 as @code{character_code} and
+@code{windows-1252} as @code{encoding}, 65001 and @code{UTF-8}, etc.
+
+If, for testing purposes, a file is crafted with different
+@code{character_code} and @code{encoding}, it seems that
+@code{character_code} controls the encoding for all strings in the
+system file before the dictionary termination record, including
+strings in data (e.g.@: string missing values), and @code{encoding}
+controls the encoding for strings following the dictionary termination
+record.

 
 @node Long String Value Labels Record
 @section Long String Value Labels Record
 
 @node Long String Value Labels Record
 @section Long String Value Labels Record


@@ -1083,7 +1145,8 @@ element.
 In record type 18, this field contains a sequence of one or more
 variable attribute sets.  If more than one variable attribute set is
 present, each one after the first is delimited from the previous by
 In record type 18, this field contains a sequence of one or more
 variable attribute sets.  If more than one variable attribute set is
 present, each one after the first is delimited from the previous by

-@code{/}.  Each variable attribute set consists of a variable name,
+@code{/}.  Each variable attribute set consists of a long
+variable name,

 followed by @code{:}, followed by an attribute set with the same
 syntax as on record type 17.
 
 followed by @code{:}, followed by an attribute set with the same
 syntax as on record type 17.