SPSS Viewer Format
==================

SPSS Viewer or .spv files, here called SPV files, are written by SPSS 16 and later to represent the contents of its output editor. This section documents the format.  This description is detailed enough to read SPV files, but it is probably not sufficient to write them.

An an aside, SPSS 15 and earlier versions use a completely different output format based on the Microsoft Compound Document Format.  This format is not documented.

An SPV file is a Zip archive that can be read with zipinfo and unzip and similar programs.  The final member in the Zip archive is a file named META-INF/MANIFEST.MF.  This structure makes SPV files resemble Java ``JAR'' files, but whereas a JAR manifest contains a sequence of colon-delimited key/value pairs, an SPV manifest contains the string allowPivoting=true, without a new-line.

The rest of the members in an SPV file's Zip archive fall into two categories: structure and details.  ``Structure'' member names begin with outputViewerNNNNNNNNNN, where each N is a decimal digit, and end with .xml, and often include the string _heading in between.  Each of these members represents some kind of output item (a table, a heading, a block of text, etc.) or a group of them.  The member whose output goes at the beginning of the document is numbered 0, the next member in the output is numbered 1, and so on.

Structure members contain XML.  This XML is sometimes self-contained, but it often references other members in the Zip archive named as follows:

PREFIX_table.xml and PREFIX_tableData.bin
PREFIX_lightTableData.bin
     The structure of a table plus its data.  Older SPV files pair a PREFIX_table.xml file that describes the table's structure with a binary PREFIX_tableData.bin file that gives its data.  Newer SPV files (the majority of those in the corpus) instead include a single PREFIX_lightTableData.bin file that incorporates both into a single binary format.

PREFIX_warning.xml and PREFIX_warningData.bin
PREFIX_lightWarningData.bin
     Same format used for tables, with a different name.

PREFIX_notes.xml and PREFIX_notesData.bin
PREFIX_lightNotesData.bin
     Same format used for tables, with a different name.

PREFIX_chartData.bin and PREFIX_chart.xml
     The structure of a chart plus its data.  Charts do not have a ``light'' format.

PREFIX_model.xml
PREFIX_pmml.xml
PREFIX_stats.xml
     Not yet investigated.  The corpus contains only one example of each.

The PREFIX in the names of the detail members is typically an 11-digit decimal number that increases for each item, tending to skip values.  Older SPV files use different naming conventions.  Structure member refer to detail members by name, and so their exact names do not appear to matter as long as they are unique.

Structure Member Format
-----------------------

Structure members XML files claim conformance with a collection of XML Schemas.  These schemas are distributed, under a nonfree license, with SPSS binaries.  Fortunately, the schemas are not necessary to understand the structure members.  To a degree, the schemas can even be deceptive because they document elements and attributes that are not in the corpus and lack documentation of elements and attributes that are commonly found in the corpus.

Structure members use a different XML namespace for each schema, but these namespaces are not entirely consistent: in some SPV files, for example, the viewer-tree schema is associated with namespace http://xml.spss.com/spss/viewer-tree and in other with http://xml.spss.com/spss/viewer/viewer-tree (note the additional viewer/ directory.  In any case, the schema URIs are not resolvable to obtain the schemas themselves.

One may ignore all of the above in interpreting a structure member. The actual XML has a simple and straightforward form that does not require a reader to take schemas or namespaces into account.

heading
     Parent: Document root or heading

     Contents: [pageSetup] label [container | heading]*

     The root of a structure member is a heading, which represents a section of output beginning with a title (the label) and ordinarily followed by content containers or further nested (sub)-sections of output.

     The document root heading may also contain a pageSetup element.

     The following attributes have been observed on both document root and nested heading elements:

     Optional attribute: creator-version
          The version of the software that created this SPV file.  A string of the form xxyyzzww represents software version xx.yy.zz.ww, e.g. 21000001 is version 21.0.0.1.  Trailing pairs of zeros are sometimes omitted, so that 21, 210000, and 21000000 are all version 21.0.0.0 (and the corpus contains all three of those forms).

     The following attributes have been observed on document root heading elements only:

     Optional attribute: creator
          The directory of the software that created this SPV file, e.g. C:\PROGRA~1\IBM\SPSS\STATIS~1\22 or /Applications/IBM/SPSS/Statistics/22/SPSSStatistics.app/Contents/Resources/Java/../../bin.

     Optional attribute: creation-date-time
          The date and time at which the SPV file was written, in a locale-specific format, e.g. Friday, May 16, 2014 6:47:37 PM PDT or lunedì 17 marzo 2014 3.15.48 CET or even Friday, December 5, 2014 5:00:19 o'clock PM EST.

     Optional attribute: lockReader
          Whether a reader should be allowed to edit the output.  The possible values are true and false, but the corpus only contains false.

     Optional attribute: schemaLocation
          This is actually an XML Namespace attribute.  A reader may ignore it.

     The following attributes have been observed only on nested heading elements:

     Required attribute: commandName
          The locale-invariant name of the command that produced the output, e.g. Frequencies, T-Test, Non Par Corr.

     Optional attribute: visibility
          To what degree the output represented by the element is visible.  The only observed value is collapsed.

     Optional attribute: locale
          The locale used for output, in Windows format, which is similar to the format used in Unix with the underscore replaced by a hyphen, e.g. en-US, en-GB, el-GR, sr-Cryl-RS.

     Optional attribute: olang
          The output language, e.g. en, it, es, de, pt-BR.

label
     Parent: heading or container

     Contents: text

     Every heading and container holds a label as its first child.  The root heading in a structure member always contains the string ``Output''.  Otherwise, the text in label describes what it labels, often by naming the statistical procedure that was executed, e.g. ``Frequencies'' or ``T-Test''.  Labels are often very generic, especially within a container, e.g. ``Title'' or ``Warnings'' or ``Notes''.  Label text is localized according to the output language, e.g. in Italian a frequency table procedure is labeled ``Frequenze''.

     The corpus contains one example of an empty label, one that contains no text.

container
     Parent: heading

     Contents: label [table | text]

     A container serves to label a table or a text item.

     Required attribute: visibility
          Either visible or hidden, this indicates whether the container's content is displayed.

     Optional attribute: text-align
          Presumably indicates the alignment of text within the container.  The only observed value is left.  Observed with nested table and text elements.

     Optional attribute: width
          The width of the container in the form Npx, e.g. 1097px.

text
     Parent: container

     Contents: html

     This text element is nested inside a container.  There is a different text element that is nested inside a pageParagraph.

     Required attribute: type
          One of title, log, or text.

     Optional attribute: commandName
          As on the heading element.  For output not specific to a command, this is simply log.  The corpus contains one example of where commandName is present but set to the empty string.

     Optional attribute: creator-version
          As on the heading element.

html
     Parent: text

     Contents: cdata

     The cdata contains an HTML document.  In some cases, the document starts with <html> and ends with </html; in others the html element is implied.  Generally the HTML includes a head element with a CSS stylesheet.  The HTML body often begins with <BR>.  The actual content ranges from trivial to simple: just discarding the CSS and tags yields readable results.

     Required attribute: lang
          This always contains en in the corpus.

table
     Parent: container

     Contents: tableStructure

     Required attribute: commandName
          As on the heading element.

     Required attribute: type
          One of table, note, or warning.

     Required attribute: subType
          The locale-invariant name for the particular kind of output that this table represents in the procedure.  This can be the same as commandName e.g. Frequencies, or different, e.g. Case Processing Summary.  Generic subtypes Notes and Warnings are often used.

     Required attribute: tableId
          A number that uniquely identifies the table within the SPV file, typically a large negative number such as -4147135649387905023.

     Optional attribute: creator-version
          As on the heading element.  In the corpus, this is only present for version 21 and up and always includes all 8 digits.

tableStructure
     Parent: table Contents: dataPath

dataPath
     Parent: tableStructure Contents: text

     Contains the name of the Zip member that holds the table details, e.g. 0000000001437_lightTableData.bin.

pageSetup
     Parent: heading

     Contents: pageHeader pageFooter

     Required attribute: initial-page-number
          Always 1.

     Optional attribute: chart-size
          Always as-is or a localization (!) of it (e.g. dimensione attuale, Wie vorgegeben).

     Optional attribute: margin-left
     Optional attribute: margin-right
     Optional attribute: margin-top
     Optional attribute: margin-bottom
          Margin sizes in the form SIZEin, e.g. 0.25in.

     Optional attribute: paper-height
     Optional attribute: paper-width
          Paper sizes in the form SIZEin, e.g. 8.5in by 11in for letter paper or 8.267in by 11.692in for A4 paper.

     Optional attribute: reference-orientation
          Always 0deg.

     Optional attribute: space-after
          Always 12pt.

pageHeader
pageFooter
     Parent: pageSetup

     Contents: pageParagraph*

     No attributes.

pageParagraph
     Parent: pageHeader or pageFooter

     Contents: text

     Text to go at the top or bottom of a page, respectively.

text
     Parent: pageParagraph

     Contents: [cdata]

     This text element is nested inside a pageParagraph.  There is a different text element that is nested inside a container.

     The element is either empty, or contains cdata that holds almost-XHTML text: in the corpus, either an html or p element.  It is almost-XHTML because the html element designates the default namespace as http://xml.spss.com/spss/viewer/viewer-tree instead of an XHTML namespace.

     The cdata can contain substitution variables: &[Page] for the page number and &[PageTitle] for the page title.

     Typical contents (indented for clarity):

          <html xmlns="http://xml.spss.com/spss/viewer/viewer-tree">
              <head></head>
              <body>
                  <p style="text-align:right; margin-top: 0">Page &[Page]</p>
              </body>
          </html>

     Required attribute: type
          Always text.

Light Detail Member Format
--------------------------

A ``light'' detail member .bin consists of a number of sections concatenated together, terminated by a byte 01:

     light-member := header title styles dimensions data 01

The first section is a 0x27-byte header:

     header := 01 00 version 01 (00 | 01) byte*21 00 00 table-id byte*4
     version := i1 | i3
     table-id := int

header includes version, a version number that affects the interpretation of some of the other data in the member.  We will refer to ``version 1'' and ``version 3'' members later on.  It also table-id is a binary version of tableId attribute in the structure member that refers to the detail member.  For example, if tableId is -4154297861994971133, then table-id would be 0xdca00003.  The meaning of the other variable parts of the header is not known.

     title := value 01?              /* localized title */
              value 01? 31           /* subtype */
              value 01? 00? 58       /* locale-invariant title */
              (31 value | 58)        /* caption */
              int[n] footnote*[n]    /* footnotes */
     footnote := value (31 value | 58) byte*4

     styles := 00 font*8
               int[x1] byte*[x1]
               int[x2] byte*[x2]
               int[x3] byte*[x3]
               int[x4] int*[x4]
               string[encoding]
               (i0 | i-1) (00 | 01) 00 (00 | 01)
               int
               byte[decimal] byte[grouping]
               int[x5] string*[x5]    /* custom currency */
               int[x6] byte*[x6]

In every example in the corpus, x1 is 240.  The meaning of the bytes that follow it is unknown.

In every example in the corpus, x2 is 18 and the bytes that follow it are 00 00 00 01 00 00 00 00 00 00 00 00 00 02 00 00 00 00.  The meaning of these bytes is unknown.

In every example in the corpus for version 1, x3 is 16 and the bytes that follow it are 00 00 00 01 00 00 00 01 00 00 00 00 01 01 01 01.  In version 3, observed x3 varies from 117 to 150 and the bytes that follow it vary somewhat and often include a readable text string, e.g. ``Default'' or ``Academic'', which appears to be the name of a ``TableLook''.

Observed values of x4 vary from 0 to 17.  Out of 7060 examples in the corpus, it is nonzero only 36 times.

encoding is a character encoding, usually a Windows code page such as en_US.windows-1252 or it_IT.windows-1252.  The encoding string is itself encoded in US-ASCII.  The rest of the character strings in the file use this encoding.

decimal is the decimal point character.  The observed values are . and ,.

grouping is the grouping character.  The observed values are ,, ., ',  , and zero (presumably indicating that digits should not be grouped).

x5 is observed as either 0 or 5.  When it is 5, the following strings are CCA through CCE format strings.  Most commonly these are all -,,, but other strings occur.

     font := byte[index] 31 string[typeface]
             00 00
             (10 | 20 | 40 | 50 | 70 | 80)[f1]
             41
             (i0 | i1 | i2)[f2]
             00
             (i0 | i2 | i64173)[f3]
             (i0 | i1 | i2 | i3)[f4]
             string[fgcolor] string[bgcolor]
             i0 i0 00
             (v3: int[f5] int[f6] int[f7] int[f8])

Each font, in order, represents the font style for a different element: title, caption, footnote, row labels, column labels, corner labels, data, and layers.

index is the 1-based index of the font, i.e. 1 for the first font, through 8 for the final font.

typeface is the string name of the font.  In the corpus, this is SansSerif in over 99% of instances and Times New Roman in the rest.

fgcolor and bgcolor are the foreground color and background color, respectively.  In the corpus, these are always #000000 and #ffffff, respectively.

The meaning of the remaining data is unknown.  It seems likely to include font sizes, horizontal and vertical alignment, attributes such as bold or italic, and margins.  f1 is 40 most of the time.  f2 is i1 most of the time for the title and i0 most of the time for other fonts.

The table below lists the values observed in the corpus.  When a cell contains a single value, then 99+% of the corpus contains that value. When a cell contains a pair of values, then the first value is seen in about two-third of the corpus and the second value in about the remaining one-third.  In fonts that include multiple pairs, values are correlated, that is, for font 3, f5 = 24, f6 = 24, f7 = 2 appears about two-thirds of the time, as does the combination of f4 = 0, f6 = 10 for font 7.

     font  f1  f2     f3   f4     f5    f6  f7  f8

        1  40   1      0    0      8 10/11   1   8
        2  40   0      2    1      8 10/11   1   1
        3  40   0      2    1  24/11 24/ 8 2/3   4
        4  40   0      2    3      8 10/11   1   1
        5  40   0      0    1      8 10/11   1   4
        6  40   0      2    1      8 10/11   1   4
        7  40   0  64173  0/1      8 10/11   1   1
        8  40   0      2    3      8 10/11   1   4

     dimensions := int[n-dims] dimension*[n-dims]
     dimension := value[name]
                  byte[d1]
                  (00 | 01 | 02)[d2]
                  (i0 | i2)[d3]
                  (00 | 01)[d4]
                  (00 | 01)[d5]
                  01
                  int[d6]
                  int[n-categories] category*[n-categories]

name is the name of the dimension, e.g. Variables, Statistics, or a variable name.

d1 is usually 0 but many other values have been observed.

d3 is 2 over 99% of the time.

d5 is 0 over 99% of the time.

d6 is either -1 or the 0-based index of the dimension, e.g. 0 for the first dimension, 1 for the second, and so on.  The latter is the case 98% of the time in the corpus.

     category := value[name] (terminal | group)
     terminal-category := 00 00 00 i2 int[index] i0

name is the name of the category (or group).

category can represent a terminal category.  In that case, index is a nonnegative integer less than n-categories in the dimension in which the category is nested (directly or indirectly).

Alternatively, category can represent a group of nested categories:

     group := (00 | 01)[merge] 00 01 (i0 | i2)[data]
              i-1 int[n-subcategories] category*[n-subcategories]

Ordinarily a group has some nested content, so that n-subcategories is positive, but a few instances of groups with n-subcategories 0 has been observed.

If merge is 00, the most common value, then the group is really a distinct group that should be represented as such in the visual representation and user interface.  If merge is 01, however, the categories in this group should be shown and treated as if they were direct children of the group's parent group (or if it has no parent group, then direct children of the dimension), and this group's name is irrelevant and should not be displayed.  (Merged groups can be nested!)

data appears to be i2 when all of the categories within a group are terminal categories that directly represent data values for a variable (e.g. in a frequency table or crosstabulation, a group of values in a variable being tabulated) and i0 otherwise, but this might be naive.

     data := int[layers] int[rows] int[columns] int*[n-dimensions]
             int[n-data] datum*[n-data]

The values of layers, rows, and columns each specifies the number of dimensions represented in layers or rows or columns, respectively, and their values sum to the number of dimensions.

The n-dimensions integers are a permutation of the 0-based dimension numbers.  The first layers of them specify each of the dimensions represented by layers, the next rows of them specify the dimensions represented by rows, and the final columns of them specify the dimensions represented by columns. When there is more than one dimension of a given kind, the inner dimensions are given first.

     datum := int64[index] 00? value      /* version 1 */
     datum := int64[index] value          /* version 3 */

The format of a datum varies slightly from version 1 to version 3: in version 1 it allows for an extra optional 00 byte.

A datum consists of an index and a value.  Suppose there are d dimensions and dimension i for 0 \le i < d has n_i categories.  Consider the datum at coordinates x_i for 0 \le i < d; note that 0 \le x_i < n_i.  Then the index is calculated by the following algorithm:

     let index = 0
     for each i from 0 to d - 1:
         index = n_i \times index + x_i

For example, suppose there are 3 dimensions with 3, 4, and 5 categories, respectively.  The datum at coordinates (1, 2, 3) has index 5 \times (4 \times (3 \times 0 + 1) + 2) + 3 = 33.

     value := 00? 00? 00? 00? raw-value
     raw-value := 01 opt-value int32[format] double
                | 02 opt-value int32[format] double string[varname] string[vallab]
                  (01 | 02 | 03)
                | 03 string[local] opt-value string[id] string[c] (00 | 01)
                | 04 opt-value int32[format] string[vallab] string[varname]
                  (01 | 02 | 03) string[vallab]
                | 05 opt-value string[varname] string[varlabel] (01 | 02 | 03)
                | opt-value string[format] int32[n-substs] substitution*[n-substs]
     substitution := i0 value
                   | int32[x] value*[x + 1]      /* x > 0 */
     opt-value := 31 i0 (i0 | i1 string) opt-value-i0-v1        /* version 1 */
                | 31 i0 (i0 | i1 string) opt-value-i0-v3        /* version 3 */
                | 31 i1 int32[footnote-number] nested-string
                | 31 i2 (00 | 02) 00 (i1 | i2 | i3) nested-string
                | 31 i3 00 00 01 00 i2 nested-string
                | 58
     opt-value-i0-v1 := 00 (i1 | i2) 00 00 int32 00 00
     opt-value-i0-v3 := count(counted-string
                              (58
                               | 31 01? 00? 00? 00? 01
                                 string[fgcolor] string[bgcolor] string[typeface]
                                 byte)
                              (58
                               | 31 i0 i0 i0 i0 01 00 (01 | 02 | 08)
                                 00 08 00 0a 00))

     nested-string := 00 00 count(counted-string 58 58)
     counted-string := count((i0 (58 | 31 string))?)