X-Git-Url: https://pintos-os.org/cgi-bin/gitweb.cgi?a=blobdiff_plain;f=spv-file-format.texi;h=38effad71ff460b2978ea65eea991eea7af992cd;hb=6a45803579c7f58e62668306240784beec7a611a;hp=5e95b53a9e1491acc15326e297c70c47a5a07ba8;hpb=e1ef9687ac5b24e223f33050bd4ccde69c7cf227;p=pspp

diff --git a/spv-file-format.texi b/spv-file-format.texi
index 5e95b53a9e..38effad71f 100644
--- a/spv-file-format.texi
+++ b/spv-file-format.texi
@@ -353,3 +353,296 @@ Typical contents (indented for clarity):
 Always @code{text}.
 @end table
 @end table
+
+@node SPV Light Detail Member Format
+@subsection Light Detail Member Format
+
+A ``light'' detail member @file{.bin} consists of a number of sections
+concatenated together, terminated by a byte 01:
+
+@example
+light-member := header title styles dimensions data 01
+@end example
+
+The first section is a 0x27-byte header:
+
+@example
+header := 01 00 version 01 (00 | 01) byte*21 00 00 table-id byte*4
+version := i1 | i3
+table-id := int
+@end example
+
+@code{header} includes @code{version}, a version number that affects
+the interpretation of some of the other data in the member.  We will
+refer to ``version 1'' and ``version 3'' members later on.  It also
+@code{table-id} is a binary version of @code{tableId} attribute in the
+structure member that refers to the detail member.  For example, if
+@code{tableId} is @code{-4154297861994971133}, then @code{table-id}
+would be 0xdca00003.  The meaning of the other variable parts of the
+header is not known.
+
+@example
+title := value 01?              /* @r{localized title} */
+         value 01? 31           /* @r{subtype} */
+         value 01? 00? 58       /* @r{locale-invariant title} */
+         (31 value | 58)        /* @r{caption} */
+         int[n] footnote*[n]    /* @r{footnotes} */
+footnote := value (31 value | 58) byte*4
+@end example
+
+@example
+styles := 00 font*8
+          int[x1] byte*[x1]
+          int[x2] byte*[x2]
+          int[x3] byte*[x3]
+          int[x4] int*[x4]
+          string[encoding]
+          (i0 | i-1) (00 | 01) 00 (00 | 01)
+          int
+          byte[decimal] byte[grouping]
+          int[n-ccs] string*[n-ccs]     /* @r{custom currency} */
+          styles2
+
+styles2 := i0                           /* @r{version 1} */
+styles2 := count(count(x5) count(x6))   /* @r{version 3} */
+x5 := byte*33 int[n] int*n
+x6 := 01 00 (03 | 04) 00 00 00
+      string[command] string[subcommand]
+      string[language] string[charset] string[locale]
+      (00 | 01) 00 (00 | 01) (00 | 01)
+      int
+      byte[decimal] byte[grouping]
+      byte*8 01
+      (string[dataset] string[datafile] i0 int i0)?
+      int[n-ccs] string*[n-ccs]
+      2e (00 | 01)
+@end example
+
+In every example in the corpus, @code{x1} is 240.  The meaning of the
+bytes that follow it is unknown.
+
+In every example in the corpus, @code{x2} is 18 and the bytes that
+follow it are @code{00 00 00 01 00 00 00 00 00 00 00 00 00 02 00 00 00
+00}.  The meaning of these bytes is unknown.
+
+In every example in the corpus for version 1, @code{x3} is 16 and the
+bytes that follow it are @code{00 00 00 01 00 00 00 01 00 00 00 00 01
+01 01 01}.  In version 3, observed @code{x3} varies from 117 to 150 and
+the bytes that follow it vary somewhat and often include a readable
+text string, e.g. ``Default'' or ``Academic'', which appears to be the
+name of a ``TableLook''.
+
+Observed values of @code{x4} vary from 0 to 17.  Out of 7060 examples
+in the corpus, it is nonzero only 36 times.
+
+@code{encoding} is a character encoding, usually a Windows code page
+such as @code{en_US.windows-1252} or @code{it_IT.windows-1252}.  The
+encoding string is itself encoded in US-ASCII.  The rest of the
+character strings in the file use this encoding.
+
+@code{decimal} is the decimal point character.  The observed values
+are @samp{.} and @samp{,}.
+
+@code{grouping} is the grouping character.  The observed values are
+@samp{,}, @samp{.}, @samp{'}, @samp{ }, and zero (presumably
+indicating that digits should not be grouped).
+
+@code{x5} is observed as either 0 or 5.  When it is 5, the following
+strings are CCA through CCE format strings.  Most commonly these are
+all @code{-,,,} but other strings occur.
+
+@example
+font := byte[index] 31 string[typeface]
+        00 00
+        (10 | 20 | 40 | 50 | 70 | 80)[f1]
+        41
+        (i0 | i1 | i2)[f2]
+        00
+        (i0 | i2 | i64173)[f3]
+        (i0 | i1 | i2 | i3)[f4]
+        string[fgcolor] string[bgcolor]
+        i0 i0 00
+        (v3: int[f5] int[f6] int[f7] int[f8])
+@end example
+
+Each @code{font}, in order, represents the font style for a different
+element: title, caption, footnote, row labels, column labels, corner
+labels, data, and layers.
+
+@code{index} is the 1-based index of the @code{font}, i.e. 1 for the
+first @code{font}, through 8 for the final @code{font}.
+
+@code{typeface} is the string name of the font.  In the corpus, this
+is @code{SansSerif} in over 99% of instances and @code{Times New
+Roman} in the rest.
+
+@code{fgcolor} and @code{bgcolor} are the foreground color and
+background color, respectively.  In the corpus, these are always
+@code{#000000} and @code{#ffffff}, respectively.
+
+The meaning of the remaining data is unknown.  It seems likely to
+include font sizes, horizontal and vertical alignment, attributes such
+as bold or italic, and margins.  @code{f1} is @code{40} most of the
+time.  @code{f2} is @code{i1} most of the time for the title and
+@code{i0} most of the time for other fonts.
+
+The table below lists the values observed in the corpus.  When a cell
+contains a single value, then 99+% of the corpus contains that value.
+When a cell contains a pair of values, then the first value is seen in
+about two-third of the corpus and the second value in about the
+remaining one-third.  In fonts that include multiple pairs, values are
+correlated, that is, for font 3, f5 = 24, f6 = 24, f7 = 2 appears
+about two-thirds of the time, as does the combination of f4 = 0, f6 =
+10 for font 7.
+
+@example
+font  f1  f2     f3   f4     f5    f6  f7  f8
+
+   1  40   1      0    0      8 10/11   1   8
+   2  40   0      2    1      8 10/11   1   1
+   3  40   0      2    1  24/11 24/ 8 2/3   4
+   4  40   0      2    3      8 10/11   1   1
+   5  40   0      0    1      8 10/11   1   4
+   6  40   0      2    1      8 10/11   1   4
+   7  40   0  64173  0/1      8 10/11   1   1
+   8  40   0      2    3      8 10/11   1   4
+@end example
+
+@example
+dimensions := int[n-dims] dimension*[n-dims]
+dimension := value[name]
+             byte[d1]
+             (00 | 01 | 02)[d2]
+             (i0 | i2)[d3]
+             (00 | 01)[d4]
+             (00 | 01)[d5]
+             01
+             int[d6]
+             int[n-categories] category*[n-categories]
+@end example
+
+@code{name} is the name of the dimension, e.g. @code{Variables},
+@code{Statistics}, or a variable name.
+
+@code{d1} is usually 0 but many other values have been observed.
+
+@code{d3} is 2 over 99% of the time.
+
+@code{d5} is 0 over 99% of the time.
+
+@code{d6} is either -1 or the 0-based index of the dimension, e.g.@: 0
+for the first dimension, 1 for the second, and so on.  The latter is
+the case 98% of the time in the corpus.
+
+@example
+category := value[name] (terminal | group)
+terminal-category := 00 00 00 i2 int[index] i0
+@end example
+
+@code{name} is the name of the category (or group).
+
+@code{category} can represent a terminal category.  In that case,
+@code{index} is a nonnegative integer less than @code{n-categories} in
+the @code{dimension} in which the @code{category} is nested (directly
+or indirectly).
+
+Alternatively, @code{category} can represent a @code{group} of nested
+categories:
+
+@example
+group := (00 | 01)[merge] 00 01 (i0 | i2)[data]
+         i-1 int[n-subcategories] category*[n-subcategories]
+@end example
+
+Ordinarily a group has some nested content, so that
+@code{n-subcategories} is positive, but a few instances of groups with
+@code{n-subcategories} 0 has been observed.
+
+If @code{merge} is 00, the most common value, then the group is really
+a distinct group that should be represented as such in the visual
+representation and user interface.  If @code{merge} is 01, however,
+the categories in this group should be shown and treated as if they
+were direct children of the group's parent group (or if it has no
+parent group, then direct children of the dimension), and this group's
+name is irrelevant and should not be displayed.  (Merged groups can be
+nested!)
+
+@code{data} appears to be i2 when all of the categories within a group
+are terminal categories that directly represent data values for a
+variable (e.g. in a frequency table or crosstabulation, a group of
+values in a variable being tabulated) and i0 otherwise, but this might
+be naive.
+
+@example
+data := int[layers] int[rows] int[columns] int*[n-dimensions]
+        int[n-data] datum*[n-data]
+@end example
+
+The values of @code{layers}, @code{rows}, and @code{columns} each
+specifies the number of dimensions represented in layers or rows or
+columns, respectively, and their values sum to the number of
+dimensions.
+
+The @code{n-dimensions} integers are a permutation of the 0-based
+dimension numbers.  The first @code{layers} of them specify each of
+the dimensions represented by layers, the next @code{rows} of them
+specify the dimensions represented by rows, and the final
+@code{columns} of them specify the dimensions represented by columns.
+When there is more than one dimension of a given kind, the inner
+dimensions are given first.
+
+@example
+datum := int64[index] 00? value      /* @r{version 1} */
+datum := int64[index] value          /* @r{version 3} */
+@end example
+
+The format of a datum varies slightly from version 1 to version 3: in
+version 1 it allows for an extra optional 00 byte.
+
+A datum consists of an index and a value.  Suppose there are @math{d}
+dimensions and dimension @math{i} for @math{0 \le i < d} has
+@math{n_i} categories.  Consider the datum at coordinates @math{x_i}
+for @math{0 \le i < d}; note that @math{0 \le x_i < n_i}.  Then the
+index is calculated by the following algorithm:
+
+@display
+let index = 0
+for each @math{i} from 0 to @math{d - 1}:
+    index = @math{n_i \times} index + @math{x_i}
+@end display
+
+For example, suppose there are 3 dimensions with 3, 4, and 5
+categories, respectively.  The datum at coordinates (1, 2, 3) has
+index @math{5 \times (4 \times (3 \times 0 + 1) + 2) + 3 = 33}.
+
+@example
+value := 00? 00? 00? 00? raw-value
+raw-value := 01 opt-value int32[format] double
+           | 02 opt-value int32[format] double string[varname] string[vallab]
+             (01 | 02 | 03)
+           | 03 string[local] opt-value string[id] string[c] (00 | 01)
+           | 04 opt-value int32[format] string[vallab] string[varname]
+             (01 | 02 | 03) string[vallab]
+           | 05 opt-value string[varname] string[varlabel] (01 | 02 | 03)
+           | opt-value string[format] int32[n-substs] substitution*[n-substs]
+substitution := i0 value
+              | int32[x] value*[x + 1]      /* @r{x > 0} */
+opt-value := 31 i0 (i0 | i1 string) opt-value-i0-v1        /* @r{version 1} */
+           | 31 i0 (i0 | i1 string) opt-value-i0-v3        /* @r{version 3} */
+           | 31 i1 int32[footnote-number] nested-string
+           | 31 i2 (00 | 02) 00 (i1 | i2 | i3) nested-string
+           | 31 i3 00 00 01 00 i2 nested-string
+           | 58
+opt-value-i0-v1 := 00 (i1 | i2) 00 00 int32 00 00
+opt-value-i0-v3 := count(counted-string
+                         (58
+                          | 31 01? 00? 00? 00? 01
+                            string[fgcolor] string[bgcolor] string[typeface]
+                            byte)
+                         (58
+                          | 31 i0 i0 i0 i0 01 00 (01 | 02 | 08)
+                            00 08 00 0a 00))
+
+nested-string := 00 00 count(counted-string 58 58)
+counted-string := count((i0 (58 | 31 string))?)
+@end example