-@section SPSS Viewer Format
@node SPSS Viewer Format
+@section SPSS Viewer Format
SPSS Viewer or @file{.spv} files, here called SPV files, are written
by SPSS 16 and later to represent the contents of its output editor.
The rest of the members in an SPV file's Zip archive fall into two
categories: structure and details. ``Structure'' member names begin
with @file{outputViewer@var{nnnnnnnnnn}}, where each @var{n} is a
-decimal digit, and end with @file{.xml}. Each of these members
-represents some kind of output item (a table, a heading, a block of
-text, etc.) or a group of them. The member whose output goes at the
-beginning of the document is numbered 0, the next member in the output
-is numbered 1, and so on.
+decimal digit, and end with @file{.xml}, and often include the string
+@file{_heading} in between. Each of these members represents some
+kind of output item (a table, a heading, a block of text, etc.) or a
+group of them. The member whose output goes at the beginning of the
+document is numbered 0, the next member in the output is numbered 1,
+and so on.
Structure members contain XML. This XML is sometimes self-contained,
but it often references other members in the Zip archive named as
follows:
@table @asis
-@item @file{*_table.xml} and @file{*_tableData.bin}
-@itemx @file{*_lightTableData.bin}
+@item @file{@var{prefix}_table.xml} and @file{@var{prefix}_tableData.bin}
+@itemx @file{@var{prefix}_lightTableData.bin}
The structure of a table plus its data. Older SPV files pair a
-@file{*_table.xml} file that describes the table's structure with a
-binary @file{*_tableData.bin} file that gives its data. Newer SPV
-files (the majority of those in the corpus) instead include a single
-@file{*_lightTableData.bin} file that incorporates both into a single
-binary format.
-
-@item @file{*_warning.xml} and @file{*_warningData.bin}
-@itemx @file{*_lightWarningData.bin}
+@file{@var{prefix}_table.xml} file that describes the table's
+structure with a binary @file{@var{prefix}_tableData.bin} file that
+gives its data. Newer SPV files (the majority of those in the corpus)
+instead include a single @file{@var{prefix}_lightTableData.bin} file
+that incorporates both into a single binary format.
+
+@item @file{@var{prefix}_warning.xml} and @file{@var{prefix}_warningData.bin}
+@itemx @file{@var{prefix}_lightWarningData.bin}
Same format used for tables, with a different name.
-@item @file{*_notes.xml} and @file{*_notesData.bin}
-@itemx @file{*_lightNotesData.bin}
+@item @file{@var{prefix}_notes.xml} and @file{@var{prefix}_notesData.bin}
+@itemx @file{@var{prefix}_lightNotesData.bin}
Same format used for tables, with a different name.
-@item @file{*_chartData.bin} and @file{*_chart.xml}
+@item @file{@var{prefix}_chartData.bin} and @file{@var{prefix}_chart.xml}
The structure of a chart plus its data. Charts do not have a
``light'' format.
-@item *_model.xml
-@itemx *_pmml.xml
-@itemx *_stats.xml
+@item @var{prefix}_model.xml
+@itemx @var{prefix}_pmml.xml
+@itemx @var{prefix}_stats.xml
Not yet investigated. The corpus contains only one example of each.
@end table
-The @file{*} in the names of these members is typically an 11-digit
-decimal number that increases for each item, tending to skip values.
-Older files use different naming convention, and the exact names do
-not appear to matter as long as they are unique.
+The @file{@var{prefix}} in the names of the detail members is
+typically an 11-digit decimal number that increases for each item,
+tending to skip values. Older SPV files use different naming
+conventions. Structure member refer to detail members by name, and so
+their exact names do not appear to matter as long as they are unique.
+
+@node SPV Structure Member Format
+@subsection Structure Member Format
+
+Structure members XML files claim conformance with a collection of XML
+Schemas. These schemas are distributed, under a nonfree license, with
+SPSS binaries. Fortunately, the schemas are not necessary to
+understand the structure members. To a degree, the schemas can even
+be deceptive because they document elements and attributes that are
+not in the corpus and lack documentation of elements and attributes
+that are commonly found in the corpus.
+
+Structure members use a different XML namespace for each schema, but
+these namespaces are not entirely consistent: in some SPV files, for
+example, the @code{viewer-tree} schema is associated with namespace
+@indicateurl{http://xml.spss.com/spss/viewer-tree} and in other with
+@indicateurl{http://xml.spss.com/spss/viewer/viewer-tree} (note the
+additional @file{viewer/} directory. In any case, the schema URIs are
+not resolvable to obtain the schemas themselves.
+
+One may ignore all of the above in interpreting a structure member.
+The actual XML has a simple and straightforward form that does not
+require a reader to take schemas or namespaces into account.
+
+@table @code
+@item heading
+Parent: Document root or @code{heading} @*
+Contents: [@code{pageSetup}] @code{label} [@code{container} | @code{heading}]*
+
+The root of a structure member is a @code{heading}, which represents a
+section of output beginning with a title (the @code{label}) and
+ordinarily followed by content containers or further nested
+(sub)-sections of output.
+
+The document root heading may also contain a @code{pageSetup} element.
+
+The following attributes have been observed on both document root and
+nested @code{heading} elements:
+
+@table @asis
+@item Optional attribute: @code{creator-version}
+The version of the software that created this SPV file. A string of
+the form @code{xxyyzzww} represents software version xx.yy.zz.ww,
+e.g.@: @code{21000001} is version 21.0.0.1. Trailing pairs of zeros
+are sometimes omitted, so that @code{21}, @code{210000}, and
+@code{21000000} are all version 21.0.0.0 (and the corpus contains all
+three of those forms).
+@end table
+
+The following attributes have been observed on document root
+@code{heading} elements only:
+
+@table @asis
+@item Optional attribute: @code{creator}
+The directory of the software that created this SPV file,
+e.g. @file{C:\PROGRA~1\IBM\SPSS\STATIS~1\22} or
+@file{/Applications/IBM/SPSS/Statistics/22/SPSSStatistics.app/Contents/Resources/Java/../../bin}.
+
+@item Optional attribute: @code{creation-date-time}
+The date and time at which the SPV file was written, in a
+locale-specific format, e.g. @code{Friday, May 16, 2014 6:47:37 PM
+PDT} or @code{lunedì 17 marzo 2014 3.15.48 CET} or even @code{Friday,
+December 5, 2014 5:00:19 o'clock PM EST}.
+
+@item Optional attribute: @code{lockReader}
+Whether a reader should be allowed to edit the output. The possible
+values are @code{true} and @code{false}, but the corpus only contains
+@code{false}.
+
+@item Optional attribute: @code{schemaLocation}
+This is actually an XML Namespace attribute. A reader may ignore it.
+@end table
+
+The following attributes have been observed only on nested
+@code{heading} elements:
+
+@table @asis
+@item Required attribute: @code{commandName}
+The locale-invariant name of the command that produced the output,
+e.g.@: @code{Frequencies}, @code{T-Test}, @code{Non Par Corr}.
+
+@item Optional attribute: @code{visibility}
+To what degree the output represented by the element is visible. The
+only observed value is @code{collapsed}.
+
+@item Optional attribute: @code{locale}
+The locale used for output, in Windows format, which is similar to the
+format used in Unix with the underscore replaced by a hyphen, e.g.@:
+@code{en-US}, @code{en-GB}, @code{el-GR}, @code{sr-Cryl-RS}.
+
+@item Optional attribute: @code{olang}
+The output language, e.g.@: @code{en}, @code{it}, @code{es},
+@code{de}, @code{pt-BR}.
+@end table
+
+@item label
+Parent: @code{heading} or @code{container} @*
+Contents: text
+
+Every @code{heading} and @code{container} holds a @code{label} as its
+first child. The root @code{heading} in a structure member always
+contains the string ``Output''. Otherwise, the text in @code{label}
+describes what it labels, often by naming the statistical procedure
+that was executed, e.g.@: ``Frequencies'' or ``T-Test''. Labels are
+often very generic, especially within a @code{container}, e.g.@:
+``Title'' or ``Warnings'' or ``Notes''. Label text is localized
+according to the output language, e.g. in Italian a frequency table
+procedure is labeled ``Frequenze''.
+
+The corpus contains one example of an empty label, one that contains
+no text.
+
+@item container
+Parent: @code{heading} @*
+Contents: @code{label} [@code{table} | @code{text}]
+
+A @code{container} serves to label a @code{table} or a @code{text}
+item.
+
+@table @asis
+@item Required attribute: @code{visibility}
+Either @code{visible} or @code{hidden}, this indicates whether the
+container's content is displayed.
+
+@item Optional attribute: @code{text-align}
+Presumably indicates the alignment of text within the container. The
+only observed value is @code{left}. Observed with nested @code{table}
+and @code{text} elements.
+
+@item Optional attribute: @code{width}
+The width of the container in the form @code{@var{n}px}, e.g.@:
+@code{1097px}.
+@end table
+
+@item text
+Parent: @code{container} @*
+Contents: @code{html}
+
+This @code{text} element is nested inside a @code{container}. There
+is a different @code{text} element that is nested inside a
+@code{pageParagraph}.
+
+@table @asis
+@item Required attribute: @code{type}
+One of @code{title}, @code{log}, or @code{text}.
+
+@item Optional attribute: @code{commandName}
+As on the @code{heading} element. For output not specific to a
+command, this is simply @code{log}. The corpus contains one example
+of where @code{commandName} is present but set to the empty string.
+
+@item Optional attribute: @code{creator-version}
+As on the @code{heading} element.
+@end table
+
+@item html
+Parent: @code{text} @*
+Contents: cdata
+
+The cdata contains an HTML document. In some cases, the document
+starts with @code{<html>} and ends with @code{</html}; in others the
+@code{html} element is implied. Generally the HTML includes a
+@code{head} element with a CSS stylesheet. The HTML body often begins
+with @code{<BR>}. The actual content ranges from trivial to simple:
+just discarding the CSS and tags yields readable results.
+
+@table @asis
+@item Required attribute: @code{lang}
+This always contains @code{en} in the corpus.
+@end table
+
+@item table
+Parent: @code{container} @*
+Contents: @code{tableStructure}
+
+@table @asis
+@item Required attribute: @code{commandName}
+As on the @code{heading} element.
+
+@item Required attribute: @code{type}
+One of @code{table}, @code{note}, or @code{warning}.
+
+@item Required attribute: @code{subType}
+The locale-invariant name for the particular kind of output that this
+table represents in the procedure. This can be the same as
+@code{commandName} e.g.@: @code{Frequencies}, or different, e.g.@:
+@code{Case Processing Summary}. Generic subtypes @code{Notes} and
+@code{Warnings} are often used.
+
+@item Required attribute: @code{tableId}
+A number that uniquely identifies the table within the SPV file,
+typically a large negative number such as @code{-4147135649387905023}.
+
+@item Optional attribute: @code{creator-version}
+As on the @code{heading} element. In the corpus, this is only present
+for version 21 and up and always includes all 8 digits.
+@end table
+
+@item tableStructure
+Parent: @code{table}
+Contents: @code{dataPath}
+
+@item dataPath
+Parent: @code{tableStructure}
+Contents: text
+
+Contains the name of the Zip member that holds the table details,
+e.g.@: @code{0000000001437_lightTableData.bin}.
+
+@item pageSetup
+Parent: @code{heading} @*
+Contents: @code{pageHeader} @code{pageFooter}
+
+@table @asis
+@item Required attribute: @code{initial-page-number}
+Always @code{1}.
+
+@item Optional attribute: @code{chart-size}
+Always @code{as-is} or a localization (!) of it (e.g.@: @code{dimensione
+attuale}, @code{Wie vorgegeben}).
+
+@item Optional attribute: @code{margin-left}
+@itemx Optional attribute: @code{margin-right}
+@itemx Optional attribute: @code{margin-top}
+@itemx Optional attribute: @code{margin-bottom}
+Margin sizes in the form @code{@var{size}in}, e.g.@: @code{0.25in}.
+
+@item Optional attribute: @code{paper-height}
+@itemx Optional attribute: @code{paper-width}
+Paper sizes in the form @code{@var{size}in}, e.g.@: @code{8.5in} by
+@code{11in} for letter paper or @code{8.267in} by @code{11.692in} for
+A4 paper.
+
+@item Optional attribute: @code{reference-orientation}
+Always @code{0deg}.
+
+@item Optional attribute: @code{space-after}
+Always @code{12pt}.
+@end table
+
+@item pageHeader
+@itemx pageFooter
+Parent: @code{pageSetup} @*
+Contents: @code{pageParagraph}*
+
+No attributes.
+
+@item pageParagraph
+Parent: @code{pageHeader} or @code{pageFooter} @*
+Contents: @code{text}
+
+Text to go at the top or bottom of a page, respectively.
+
+@item text
+Parent: @code{pageParagraph} @*
+Contents: [cdata]
+
+This @code{text} element is nested inside a @code{pageParagraph}. There
+is a different @code{text} element that is nested inside a
+@code{container}.
+
+The element is either empty, or contains cdata that holds almost-XHTML
+text: in the corpus, either an @code{html} or @code{p} element. It is
+@emph{almost}-XHTML because the @code{html} element designates the
+default namespace as
+@code{http://xml.spss.com/spss/viewer/viewer-tree} instead of an XHTML
+namespace.
+
+The cdata can contain substitution variables: @code{&[Page]} for the
+page number and @code{&[PageTitle]} for the page title.
+
+Typical contents (indented for clarity):
+
+@example
+<html xmlns="http://xml.spss.com/spss/viewer/viewer-tree">
+ <head></head>
+ <body>
+ <p style="text-align:right; margin-top: 0">Page &[Page]</p>
+ </body>
+</html>
+@end example
+
+@table @asis
+@item Required attribute: @code{type}
+Always @code{text}.
+@end table
+@end table
+
+@node SPV Light Detail Member Format
+@subsection Light Detail Member Format
+
+A ``light'' detail member @file{.bin} consists of a number of sections
+concatenated together, terminated by a byte 01:
+
+@example
+light-member := header title styles dimensions data 01
+@end example
+
+The first section is a 0x27-byte header:
+
+@example
+header := 01 00 version 01 (00 | 01) byte*21 00 00 table-id byte*4
+version := i1 | i3
+table-id := int
+@end example
+
+@code{header} includes @code{version}, a version number that affects
+the interpretation of some of the other data in the member. We will
+refer to ``version 1'' and ``version 3'' members later on. It also
+@code{table-id} is a binary version of @code{tableId} attribute in the
+structure member that refers to the detail member. For example, if
+@code{tableId} is @code{-4154297861994971133}, then @code{table-id}
+would be 0xdca00003. The meaning of the other variable parts of the
+header is not known.
+
+@example
+title := value 01? /* @r{localized title} */
+ value 01? 31 /* @r{subtype} */
+ value 01? 00? 58 /* @r{locale-invariant title} */
+ (31 value | 58) /* @r{caption} */
+ int[n] footnote*[n] /* @r{footnotes} */
+footnote := value (31 value | 58) byte*4
+@end example
+
+@example
+styles := 00 font*8
+ int[x1] byte*[x1]
+ int[x2] byte*[x2]
+ int[x3] byte*[x3]
+ int[x4] int*[x4]
+ string[encoding]
+ (i0 | i-1) (00 | 01) 00 (00 | 01)
+ int
+ byte[decimal] byte[grouping]
+ int[x5] string*[x5] /* @r{custom currency} */
+ int[x6] byte*[x6]
+@end example
+
+In every example in the corpus, @code{x1} is 240. The meaning of the
+bytes that follow it is unknown.
+
+In every example in the corpus, @code{x2} is 18 and the bytes that
+follow it are @code{00 00 00 01 00 00 00 00 00 00 00 00 00 02 00 00 00
+00}. The meaning of these bytes is unknown.
+
+In every example in the corpus for version 1, @code{x3} is 16 and the
+bytes that follow it are @code{00 00 00 01 00 00 00 01 00 00 00 00 01
+01 01 01}. In version 3, observed @code{x3} varies from 117 to 150 and
+the bytes that follow it vary somewhat and often include a readable
+text string, e.g. ``Default'' or ``Academic'', which appears to be the
+name of a ``TableLook''.
+
+Observed values of @code{x4} vary from 0 to 17. Out of 7060 examples
+in the corpus, it is nonzero only 36 times.
+
+@code{encoding} is a character encoding, usually a Windows code page
+such as @code{en_US.windows-1252} or @code{it_IT.windows-1252}. The
+encoding string is itself encoded in US-ASCII. The rest of the
+character strings in the file use this encoding.
+
+@code{decimal} is the decimal point character. The observed values
+are @samp{.} and @samp{,}.
+
+@code{grouping} is the grouping character. The observed values are
+@samp{,}, @samp{.}, @samp{'}, @samp{ }, and zero (presumably
+indicating that digits should not be grouped).
+
+@code{x5} is observed as either 0 or 5. When it is 5, the following
+strings are CCA through CCE format strings. Most commonly these are
+all @code{-,,,} but other strings occur.
+
+@example
+font := byte[index] 31 string[typeface]
+ 00 00
+ (10 | 20 | 40 | 50 | 70 | 80)[f1]
+ 41
+ (i0 | i1 | i2)[f2]
+ 00
+ (i0 | i2 | i64173)[f3]
+ (i0 | i1 | i2 | i3)[f4]
+ string[fgcolor] string[bgcolor]
+ i0 i0 00
+ (v3: int[f5] int[f6] int[f7] int[f8])
+@end example
+
+Each @code{font}, in order, represents the font style for a different
+element: title, caption, footnote, row labels, column labels, corner
+labels, data, and layers.
+
+@code{index} is the 1-based index of the @code{font}, i.e. 1 for the
+first @code{font}, through 8 for the final @code{font}.
+
+@code{typeface} is the string name of the font. In the corpus, this
+is @code{SansSerif} in over 99% of instances and @code{Times New
+Roman} in the rest.
+
+@code{fgcolor} and @code{bgcolor} are the foreground color and
+background color, respectively. In the corpus, these are always
+@code{#000000} and @code{#ffffff}, respectively.
+
+The meaning of the remaining data is unknown. It seems likely to
+include font sizes, horizontal and vertical alignment, attributes such
+as bold or italic, and margins. @code{f1} is @code{40} most of the
+time. @code{f2} is @code{i1} most of the time for the title and
+@code{i0} most of the time for other fonts.
+
+The table below lists the values observed in the corpus. When a cell
+contains a single value, then 99+% of the corpus contains that value.
+When a cell contains a pair of values, then the first value is seen in
+about two-third of the corpus and the second value in about the
+remaining one-third. In fonts that include multiple pairs, values are
+correlated, that is, for font 3, f5 = 24, f6 = 24, f7 = 2 appears
+about two-thirds of the time, as does the combination of f4 = 0, f6 =
+10 for font 7.
+
+@example
+font f1 f2 f3 f4 f5 f6 f7 f8
+
+ 1 40 1 0 0 8 10/11 1 8
+ 2 40 0 2 1 8 10/11 1 1
+ 3 40 0 2 1 24/11 24/ 8 2/3 4
+ 4 40 0 2 3 8 10/11 1 1
+ 5 40 0 0 1 8 10/11 1 4
+ 6 40 0 2 1 8 10/11 1 4
+ 7 40 0 64173 0/1 8 10/11 1 1
+ 8 40 0 2 3 8 10/11 1 4
+@end example
+
+@example
+dimensions := int[n-dims] dimension*[n-dims]
+dimension := value[name]
+ byte[d1]
+ (00 | 01 | 02)[d2]
+ (i0 | i2)[d3]
+ (00 | 01)[d4]
+ (00 | 01)[d5]
+ 01
+ int[d6]
+ int[n-categories] category*[n-categories]
+@end example
+
+@code{name} is the name of the dimension, e.g. @code{Variables},
+@code{Statistics}, or a variable name.
+
+@code{d1} is usually 0 but many other values have been observed.
+
+@code{d3} is 2 over 99% of the time.
+
+@code{d5} is 0 over 99% of the time.
+
+@code{d6} is either -1 or the 0-based index of the dimension, e.g.@: 0
+for the first dimension, 1 for the second, and so on. The latter is
+the case 98% of the time in the corpus.
+
+@example
+category := value[name] (terminal | group)
+terminal-category := 00 00 00 i2 int[index] i0
+@end example
+
+@code{name} is the name of the category (or group).
+
+@code{category} can represent a terminal category. In that case,
+@code{index} is a nonnegative integer less than @code{n-categories} in
+the @code{dimension} in which the @code{category} is nested (directly
+or indirectly).
+
+Alternatively, @code{category} can represent a @code{group} of nested
+categories:
+
+@example
+group := (00 | 01)[merge] 00 01 (i0 | i2)[data]
+ i-1 int[n-subcategories] category*[n-subcategories]
+@end example
+
+Ordinarily a group has some nested content, so that
+@code{n-subcategories} is positive, but a few instances of groups with
+@code{n-subcategories} 0 has been observed.
+
+If @code{merge} is 00, the most common value, then the group is really
+a distinct group that should be represented as such in the visual
+representation and user interface. If @code{merge} is 01, however,
+the categories in this group should be shown and treated as if they
+were direct children of the group's parent group (or if it has no
+parent group, then direct children of the dimension), and this group's
+name is irrelevant and should not be displayed. (Merged groups can be
+nested!)
+
+@code{data} appears to be i2 when all of the categories within a group
+are terminal categories that directly represent data values for a
+variable (e.g. in a frequency table or crosstabulation, a group of
+values in a variable being tabulated) and i0 otherwise, but this might
+be naive.
+
+@example
+data := int[layers] int[rows] int[columns] int*[n-dimensions]
+ int[n-data] datum*[n-data]
+@end example
+
+The values of @code{layers}, @code{rows}, and @code{columns} each
+specifies the number of dimensions represented in layers or rows or
+columns, respectively, and their values sum to the number of
+dimensions.
+
+The @code{n-dimensions} integers are a permutation of the 0-based
+dimension numbers. The first @code{layers} of them specify each of
+the dimensions represented by layers, the next @code{rows} of them
+specify the dimensions represented by rows, and the final
+@code{columns} of them specify the dimensions represented by columns.
+When there is more than one dimension of a given kind, the inner
+dimensions are given first.
+
+@example
+datum := int64[index] 00? value /* @r{version 1} */
+datum := int64[index] value /* @r{version 3} */
+@end example
+
+The format of a datum varies slightly from version 1 to version 3: in
+version 1 it allows for an extra optional 00 byte.
+
+A datum consists of an index and a value. Suppose there are @math{d}
+dimensions and dimension @math{i} for @math{0 \le i < d} has
+@math{n_i} categories. Consider the datum at coordinates @math{x_i}
+for @math{0 \le i < d}; note that @math{0 \le x_i < n_i}. Then the
+index is calculated by the following algorithm:
+
+@display
+let index = 0
+for each @math{i} from 0 to @math{d - 1}:
+ index = @math{n_i \times} index + @math{x_i}
+@end display
+
+For example, suppose there are 3 dimensions with 3, 4, and 5
+categories, respectively. The datum at coordinates (1, 2, 3) has
+index @math{5 \times (4 \times (3 \times 0 + 1) + 2) + 3 = 33}.
+
+@example
+value := 00? 00? 00? 00? raw-value
+raw-value := 01 opt-value int32[format] double
+ | 02 opt-value int32[format] double string[varname] string[vallab]
+ (01 | 02 | 03)
+ | 03 string[local] opt-value string[id] string[c] (00 | 01)
+ | 04 opt-value int32[format] string[vallab] string[varname]
+ (01 | 02 | 03) string[vallab]
+ | 05 opt-value string[varname] string[varlabel] (01 | 02 | 03)
+ | opt-value string[format] int32[n-substs] substitution*[n-substs]
+substitution := i0 value
+ | int32[x] value*[x + 1] /* @r{x > 0} */
+opt-value := 31 i0 (i0 | i1 string) opt-value-i0-v1 /* @r{version 1} */
+ | 31 i0 (i0 | i1 string) opt-value-i0-v3 /* @r{version 3} */
+ | 31 i1 int32[footnote-number] nested-string
+ | 31 i2 (00 | 02) 00 (i1 | i2 | i3) nested-string
+ | 31 i3 00 00 01 00 i2 nested-string
+ | 58
+opt-value-i0-v1 := 00 (i1 | i2) 00 00 int32 00 00
+opt-value-i0-v3 := count(counted-string
+ (58
+ | 31 01? 00? 00? 00? 01
+ string[fgcolor] string[bgcolor] string[typeface]
+ byte)
+ (58
+ | 31 i0 i0 i0 i0 01 00 (01 | 02 | 08)
+ 00 08 00 0a 00))
+
+nested-string := 00 00 count(counted-string 58 58)
+counted-string := count((i0 (58 | 31 string))?)
+@end example
projects / pspp / blobdiff