1 @c PSPP - a program for statistical analysis.
2 @c Copyright (C) 2019 Free Software Foundation, Inc.
3 @c Permission is granted to copy, distribute and/or modify this document
4 @c under the terms of the GNU Free Documentation License, Version 1.3
5 @c or any later version published by the Free Software Foundation;
6 @c with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts.
7 @c A copy of the license is included in the section entitled "GNU
8 @c Free Documentation License".
11 @node Portable File Format
12 @appendix Portable File Format
14 These days, most computers use the same internal data formats for
15 integer and floating-point data, if one ignores little differences like
16 big- versus little-endian byte ordering. However, occasionally it is
17 necessary to exchange data between systems with incompatible data
18 formats. This is what portable files are designed to do.
20 @strong{Please note:} This information is gleaned from examination of
21 ASCII-formatted portable files only, so some of it may be incorrect
22 for portable files formatted in EBCDIC or other character sets.
25 * Portable File Characters::
26 * Portable File Structure::
27 * Portable File Header::
28 * Version and Date Info Record::
29 * Identification Records::
30 * Variable Count Record::
32 * Case Weight Variable Record::
34 * Value Label Records::
35 * Portable File Document Record::
36 * Portable File Data::
39 @node Portable File Characters
40 @section Portable File Characters
42 Portable files are arranged as a series of lines of 80
43 characters each. Each line is terminated by a carriage-return,
44 line-feed sequence (``new-lines''). New-lines are only used to avoid
45 line length limits imposed by some OSes; they are not meaningful.
47 Most lines in portable files are exactly 80 characters long. The only
48 exception is a line that ends in one or more spaces, in which the
49 spaces may optionally be omitted. Thus, a portable file reader must
50 act as though a line shorter than 80 characters is padded to that
53 The file must be terminated with a @samp{Z} character. In addition, if
54 the final line in the file does not have exactly 80 characters, then it
55 is padded on the right with @samp{Z} characters. (The file contents may
56 be in any character set; the file contains a description of its own
57 character set, as explained in the next section. Therefore, the
58 @samp{Z} character is not necessarily an ASCII @samp{Z}.)
60 For the rest of the description of the portable file format, new-lines
61 and the trailing @samp{Z}s will be ignored, as if they did not exist,
62 because they are not an important part of understanding the file
65 @node Portable File Structure
66 @section Portable File Structure
68 Every portable file consists of the following records, in sequence:
76 Version and date info.
79 Product identification.
82 Author identification (optional).
85 Subproduct identification (optional).
91 Case weight variable (optional).
94 Variables. Each variable record may optionally be followed by a
95 missing value record and a variable label record.
98 Value labels (optional).
101 Documents (optional).
107 Most records are identified by a single-character tag code. The file
108 header and version info record do not have a tag.
110 Other than these single-character codes, there are three types of fields
111 in a portable file: floating-point, integer, and string. Floating-point
112 fields have the following format:
117 Zero or more leading spaces.
120 Optional asterisk (@samp{*}), which indicates a missing value. The
121 asterisk must be followed by a single character, generally a period
122 (@samp{.}), but it appears that other characters may also be possible.
123 This completes the specification of a missing value.
126 Optional minus sign (@samp{-}) to indicate a negative number.
129 A whole number, consisting of one or more base-30 digits: @samp{0}
130 through @samp{9} plus capital letters @samp{A} through @samp{T}.
133 Optional fraction, consisting of a radix point (@samp{.}) followed by
134 one or more base-30 digits.
137 Optional exponent, consisting of a plus or minus sign (@samp{+} or
138 @samp{-}) followed by one or more base-30 digits.
141 A forward slash (@samp{/}).
144 Integer fields take a form identical to floating-point fields, but they
145 may not contain a fraction.
147 String fields take the form of a integer field having value @var{n},
148 followed by exactly @var{n} characters, which are the string content.
150 @node Portable File Header
151 @section Portable File Header
153 Every portable file begins with a 464-byte header, consisting of a
154 200-byte collection of vanity splash strings, followed by a 256-byte
155 character set translation table, followed by an 8-byte tag string.
157 The 200-byte segment is divided into five 40-byte sections, each of
158 which represents the string @code{@var{charset} SPSS PORT FILE} in a
159 different character set encoding, where @var{charset} is the name of
160 the character set used in the file, e.g.@: @code{ASCII} or
161 @code{EBCDIC}. Each string is padded on the right with spaces in its
162 respective character set.
164 It appears that these strings exist only to inform those who might view
165 the file on a screen, and that they are not parsed by SPSS products.
166 Thus, they can be safely ignored. For those interested, the strings are
167 supposed to be in the following character sets, in the specified order:
168 EBCDIC, 7-bit ASCII, CDC 6-bit ASCII, 6-bit ASCII, Honeywell 6-bit
171 The 256-byte segment describes a mapping from the character set used in
172 the portable file to an arbitrary character set having characters at the
178 Control characters. Not important enough to describe in full here.
186 Digits @samp{0} through @samp{9}.
190 Capital letters @samp{A} through @samp{Z}.
194 Lowercase letters @samp{a} through @samp{z}.
210 Symbols @code{&[]!$*);^-/}
214 Broken vertical pipe.
218 Symbols @code{,%_>}?@code{`:} @c @code{?} is an inverted question mark
222 British pound symbol.
226 Symbols @code{@@'="}.
230 Less than or equal symbol.
262 Lower left corner box draw.
266 Upper left corner box draw.
270 Greater than or equal symbol.
274 Superscript @samp{0} through @samp{9}.
278 Lower right corner box draw.
282 Upper right corner box draw.
294 Superscript @samp{(}.
298 Superscript @samp{)}.
302 Horizontal dagger (?).
306 Symbols @samp{@{@}\}.
313 Centered dot, or bullet.
320 Symbols that are not defined in a particular character set are set to
321 the same value as symbol 64; i.e., to @samp{0}.
323 The 8-byte tag string consists of the exact characters @code{SPSSPORT}
324 in the portable file's character set, which can be used to verify that
325 the file is indeed a portable file.
327 @node Version and Date Info Record
328 @section Version and Date Info Record
330 This record does not have a tag code. It has the following structure:
334 A single character identifying the file format version. The letter A
335 represents version 0, and so on.
338 An 8-character string field giving the file creation date in the format
342 A 6-character string field giving the file creation time in the format
346 @node Identification Records
347 @section Identification Records
349 The product identification record has tag code @samp{1}. It consists of
350 a single string field giving the name of the product that wrote the
353 The author identification record has tag code @samp{2}. It is
354 optional. If present, it consists of a single string field giving the
355 name of the person who caused the portable file to be written.
357 The subproduct identification record has tag code @samp{3}. It is
358 optional. If present, it consists of a single string field giving
359 additional information on the product that wrote the portable file.
361 @node Variable Count Record
362 @section Variable Count Record
364 The variable count record has tag code @samp{4}. It consists of a
365 single integer field giving the number of variables in the file
368 @node Precision Record
369 @section Precision Record
371 The precision record has tag code @samp{5}. It consists of a single
372 integer field specifying the maximum number of base-30 digits used in
375 @node Case Weight Variable Record
376 @section Case Weight Variable Record
378 The case weight variable record is optional. If it is present, it
379 indicates the variable used for weighting cases; if it is absent,
380 cases are unweighted. It has tag code @samp{6}. It consists of a
381 single string field that names the weighting variable.
383 @node Variable Records
384 @section Variable Records
386 Each variable record represents a single variable. Variable records
387 have tag code @samp{7}. They have the following structure:
392 Width (integer). This is 0 for a numeric variable, and a number between 1
393 and 255 for a string variable.
396 Name (string). 1--8 characters long. Must be in all capitals.
398 A few portable files that contain duplicate variable names have been
399 spotted in the wild. PSPP handles these by renaming the duplicates
400 with numeric extensions: @code{@var{var}_1}, @code{@var{var}_2}, and
404 Print format. This is a set of three integer fields:
409 Format type (@pxref{Variable Record}).
415 Number of decimal places. 1--40.
418 A few portable files with invalid format types or formats that are not
419 of the appropriate width for their variables have been spotted in the
420 wild. PSPP assigns a default F or A format to a variable with an
424 Write format. Same structure as the print format described above.
427 Each variable record can optionally be followed by a missing value
428 record, which has tag code @samp{8}. A missing value record has one
429 field, the missing value itself (a floating-point or string, as
430 appropriate). Up to three of these missing value records can be used.
432 There is also a record for missing value ranges, which has tag code
433 @samp{B}. It is followed by two fields representing the range, which
434 are floating-point or string as appropriate. If a missing value range
435 is present, it may be followed by a single missing value record.
437 Tag codes @samp{9} and @samp{A} represent @code{LO THRU @var{x}} and
438 @code{@var{x} THRU HI} ranges, respectively. Each is followed by a
439 single field representing @var{x}. If one of the ranges is present, it
440 may be followed by a single missing value record.
442 In addition, each variable record can optionally be followed by a
443 variable label record, which has tag code @samp{C}. A variable label
444 record has one field, the variable label itself (string).
446 @node Value Label Records
447 @section Value Label Records
449 Value label records have tag code @samp{D}. They have the following
454 Variable count (integer).
457 List of variables (strings). The variable count specifies the number in
458 the list. Variables are specified by their names. All variables must
459 be of the same type (numeric or string), but string variables do not
460 necessarily have the same width.
463 Label count (integer).
466 List of (value, label) tuples. The label count specifies the number of
467 tuples. Each tuple consists of a value, which is numeric or string as
468 appropriate to the variables, followed by a label (string).
471 A few portable files that specify duplicate value labels, that is, two
472 different labels for a single value of a single variable, have been
473 spotted in the wild. PSPP uses the last value label specified in
476 @node Portable File Document Record
477 @section Document Record
479 One document record may optionally follow the value label record. The
480 document record consists of tag code @samp{E}, following by the number
481 of document lines as an integer, followed by that number of strings,
482 each of which represents one document line. Document lines must be 80
483 bytes long or shorter.
485 @node Portable File Data
486 @section Portable File Data
488 The data record has tag code @samp{F}. There is only one tag for all
489 the data; thus, all the data must follow the dictionary. The data is
490 terminated by the end-of-file marker @samp{Z}, which is not valid as the
491 beginning of a data element.
493 Data elements are output in the same order as the variable records
494 describing them. String variables are output as string fields, and
495 numeric variables are output as floating-point fields.