1 Time-stamp: <2004-11-30 22:59:24 blp>
3 What Ben's working on now.
4 --------------------------
6 Workspace exhaustion heuristics.
8 Does SET work correctly?
10 Update q2c input format description.
12 Rewrite output subsystem, break into multiple processes.
14 CROSSTABS needs to be re-examined.
16 RANK, which is needed for the Wilcoxon signed-rank statistic, Mann-Whitney U,
17 Kruskal-Wallis on NPAR TESTS and for Spearman and the Johnkheere trend test (in
23 Make valgrind --leak-check=yes --show-reachable=yes work.
27 Add NOT_REACHED() macro.
29 Add compression to casefiles.
31 Expressions need to be able to abbreviate function names. XDATE.QUARTER
32 abbreviates to XDA.QUA, etc.
34 The expression tests need tests for XDATE and a few others, see
35 tests/xforms/expressions.sh comments for details.
37 Expressions need random distribution functions.
39 There needs to be another layer onto the lexer, which should probably be
40 entirely rewritten anyway. The lexer needs to read entire *commands* at a
41 time, not just a *line* at a time. It also needs to support arbitrary putback,
42 probably by just backing up the "current position" in the command buffer.
44 Scratch variables should not be available for use following TEMPORARY.
46 Details of N OF CASES, SAMPLE, FILTER, PROCESS IF, TEMPORARY, etc., need to be
47 checked against the documentation. See notes on these at end of file for a
50 Check our results against the NIST StRD benchmark results at
51 strd.itl.nist.gov/div898/strd
53 In debug mode hash table code should verify that collisions are reasonably low.
55 Use AFM files instead of Groff font files, and include AFMs for our default
56 fonts with the distribution.
58 Storage of value labels on disk is inefficient. Invent new data structure.
60 Add an output flag which would cause a page break if a table segment could fit
61 vertically on a page but it just happens to be positioned such that it won't.
63 Fix spanned joint cells, i.e., EDLEVEL on crosstabs.stat.
67 PostScript driver should emit thin lines, then thick lines, to optimize time
70 New functions? var_name_or_label(), tab_value_or_label()
72 Should be able to bottom-justify cells. It'll be expensive, though, by
73 requiring an extra metrics call.
75 Perhaps instead of the current lines we should define the following line types:
76 null, thin, thick, double. It might look pretty classy.
78 Perhaps thick table borders that are cut off by a page break should decay to
79 thin borders. (i.e., on a thick bordered table that's longer than one page,
80 but narrow, the bottom border would be thin on the first page, and the top and
81 bottom borders on middle pages.)
83 Support multi-line titles on tables. (For the first page only, presumably.)
85 Rewrite the convert_F() function in data-out.c to be nicer code.
87 In addition to searching the source directory, we should search the current
88 directory (for data files). (Yuck!)
90 Fix line-too-long problems in PostScript code, instead of covering them up.
91 setlinecap is *not* a proper solution.
95 Has glob.c been pared down enough?
97 Improve interactivity of output by allowing a `commit' function for a page.
98 This will also allow for infinite-length pages.
100 Implement thin single lines, should be pretty easy now.
102 SELECT IF should be moved before other transformations whenever possible. It
103 should only be impossible when one of the variables referred to in SELECT IF is
104 created or modified by a previous transformation.
106 The manual: add text, add index entries, add examples.
108 The inline file should be improved: There should be *real* detection of whether
109 it is used (in dfm.c:cmd_begin_data), not after-the-fact detection.
111 Figure out a stylesheet for messages displayed by PSPP: i.e., what quotation
112 marks around filenames, etc.
114 New SET subcommand: OUTPUT. i.e., SET OUTPUT="filename" to send output to that
115 file; SET OUTPUT="filename"(APPEND) to append to that file; SET OUTPUT=DEFAULT
116 to reset everything. There might be a better approach, though--think about it.
118 From Zvi Grauer <z.grauer@csuohio.edu> and <zvi@mail.ohio.net>:
120 1. design of experiments software, specifically Factorial, response surface
121 methodology and mixrture design.
123 These would be EXTREMELY USEFUL for chemists, engineeris, and anyone
124 involved in the production of chemicals or formulations.
126 2. Multidimensional Scaling analysis (for market analysis) -
128 3. Preference mapping software for market analysis
130 4. Hierarchical clustering (as well as partition clustering)
134 6. Categorical data analsys ?
139 In addition to an "infinite journal", we should keep a number of
140 individual-session journals, pspp.jnl-1 through pspp.jnl-X, renaming and
141 deleting as needed. All of the journals should have date/time comments.
143 Qualifiers for variables giving type--categorical, ordinal, ...
147 Consider consequences of xmalloc(), fail(), hcf() in interactive
149 a. Can we safely just use setjmp()/longjmp()?
150 b. Will that leak memory?
151 i. I don't think so: all procedure-created memory is either
152 garbage-collected or globally-accessible.
153 ii. But you never know... esp. w/o Checker.
154 c. Is this too early to worry? too late?
156 Need to implement a shared buffer for funny functions that require relatively
157 large permanent transient buffers (1024 bytes or so), that is, buffers that are
158 permanent in the sense that they probably shouldn't be deallocated but are only
159 used from time to time, buffers that can't be allocated on the stack because
160 they are of variable and unpredictable but usually relatively small (usually
161 line buffers). There are too many of these lurking around; can save a sizeable
162 amount of space at very little overhead and with very little effort by merging
165 Clever multiplatform GUI idea (due partly to John Williams): write a GUI in
166 Java where each statistical procedure dialog box could be downloaded from the
167 server independently. The statistical procedures would run on (the/a) server
168 and results would be reported through HTML tables viewed with the user's choice
169 of web browsers. Help could be implemented through the browser as well.
171 Design a plotting API, with scatterplots, line plots, pie charts, barcharts,
172 Pareto plots, etc., as subclasses of the plot superclass.
177 1. How to add an operator for use in PSPP expressions:
179 a. Add the operator to the enumerated type at the top of expr.h. If the
180 operator has arguments (i.e., it's not a terminal) then add it *before*
181 OP_TERMINAL; otherwise, add it *after* OP_TERMINAL. All these begin with OP_.
183 b. If the operator's a terminal then you'll want to design a structure to hold
184 its content. Add the structure to the union any_node. (You can also reuse one
185 of the prefab structures, of course.)
187 c. Now switch to expr-prs.c--the module for expression parsing. Insert the
188 operator somewhere in the precedence hierarchy.
190 (1) If you're adding a operator that is a function (like ACOS, ABS, etc.) then
191 add the function to functab in `void init_functab(void)'. Order is not
192 important here. The first element is the function name, like "ACOS". The
193 second is the operator enumerator you added in expr.h, like OP_ARCOS. The
194 third element is the C function to parse the PSPP function. The predefined
195 functions will probably suit your needs, but if not, you can write your own.
196 The fourth element is an argument to the parsing function; it's only used
197 currently by generic_str_func(), which handles a rather general syntax for
198 functions that return strings; see the comment at the beginning of its code for
201 (2) If you're adding an actual operator you'll have to put a function in
202 between two of the operators there already in functions `exprtype
203 parse_*(any_node **n)'. Each of these stores the tree for its result into *n,
204 and returns the result type, or EX_ERROR on error. Be sure to delete all the
205 allocated memory on error before returning.
207 d. Add the operator to the table `op_desc ops[OP_SENTINEL+1]' in expr-prs.c,
208 which has an entry for every operator. These entries *must* be in the same
209 order as they are in expr.h. The entries have the form `op(A,B,C,D)'. A is
210 the name of the operator as it should be printed in a postfix output format.
211 For example, the addition operator is printed as `plus'. B is a bitmapped set
214 * Set the 001 bit (OP_VAR_ARGS) if the operator takes a variable number of
215 arguments. If a function can take, say, two args or three args, but no other
216 numbers of args, this is a poor way to do it--instead implement the operator as
217 two separate operators, one with two args, the other with three. (The main
218 effect of this bit is to cause the number of arguments to be output to the
219 postfix form so that the expression evaluator can know how many args the
220 operator takes. It also causes the expression optimizer to calculate the
221 needed stack height differently, without referencing C.)
223 * Set the 002 bit (OP_MIN_ARGS) if the operator can take an optional `dotted
224 argument' that specified the minimum number of non-SYSMIS arguments in order to
225 have a non-SYSMIS result. For instance, MIN.3(e1,e2,e3,e4,e5) returns a
226 non-SYSMIS result only if at least 3 out of 5 of the expressions e1 to e5 are
229 Minargs are passed in the nonterm_node structure in `arg[]''s elements past
230 `n'--search expr-prs.c for the words `terrible crock' for an example of this.
232 Minargs are output to the postfix form. A default value is output if none was
233 specified by the user.
235 You can use minargs for anything you want--they're not limited to actually
236 describing a minimum number of valid arguments; that's just what they're most
239 * Set the 004 bit (OP_FMT_SPEC) if the operator has an argument that is a
240 format specifier. (This causes the format specifier to be output to the
241 postfix representation.)
243 Format specs are passed in the nonterm_node structure in the same way as
244 minargs, except that there are three args, in this order: type, width, # of
245 decimals--search expr-prs.c for the words `is a crock' for an example of this.
247 * Set the 010 bit (OP_ABSORB_MISS) if the operator can *ever* have a result of
248 other than SYSMIS when given one or more arguments of SYSMIS. Operators
249 lacking this bit and known to have a SYSMIS argument are short-circuited to
250 SYSMIS by the expression optimizer.
252 * If your operator doesn't fit easily into the existing categories,
253 congratulations, you get to write lots of code to adjust everything to cope
254 with this new operator. Are you really sure you want to do that?
256 C is the effect the operator has on stack height. Set this to `varies' if the
257 operator has a variable number of arguments. Otherwise this 1, minus the
258 number of arguments the operator has. (Since terminals have no arguments, they
259 have a value of +1 for this; other operators have a value of 0 or less.)
261 D is the number of items output to the postfix form after the operator proper.
262 This is 0, plus 1 if the operator has varargs, plus 1 if the operator has
263 minargs, plus 3 if the operator has a format spec. Note that minargs/varargs
264 can't coexist with a format spec on the same operator as currently coded. Some
265 terminals also have a nonzero value for this but don't fit into the above
268 e. Switch to expr-opt.c. Add code to evaluate_tree() to evaluate the
269 expression when all arguments are known to be constants. Pseudo-random
270 functions can't be evaluated even if their arguments are constants. If the
271 function can be optimized even if its arguments aren't all known constants, add
272 code to optimize_tree() to do it.
274 f. Switch to expr-evl.c. Add code to evaluate_expression() to evaluate the
275 expression. You must be absolutely certain that the code in evaluate_tree(),
276 optimize_tree(), and evaluate_expression() will always return the same results,
277 otherwise users will get inconsistent results, a Bad Thing. You must be
278 certain that even on boundary conditions users will get identical results, for
279 instance for the values 0, 1, -1, SYSMIS, or, for string functions, the null
280 string, 1-char strings, and 255-char strings.
282 g. Test the code. Write some test syntax files. Examine the output carefully.
284 MORE NOTES/IDEAS/BUGS
285 ---------------------
287 Sometimes very wide (or very tall) columns can occur in tables. What is a good
288 way to truncate them? It doesn't seem to cause problems for the ascii or
289 postscript drivers, but it's not good in the general case. Should they be
290 split somehow? (One way that wide columns can occur is through user request,
291 for instance through a wide PRINT request--try time-date.stat with a narrow
292 ascii page or with the postscript driver on letter size paper.)
294 NULs in input files break the products we're replacing: although it will input
295 them properly and display them properly as AHEX format, it truncates them in A
296 format. Also, string-manipulation functions such as CONCAT truncate their
297 results after the first NUL. This should simplify the result of PSPP design.
298 Perhaps those ugly a_string, b_string, ..., can all be eliminated.
300 From Moshe Braner <mbraner@nessie.vdh.state.vt.us>: An idea regarding MATCH
301 FILES, again getting BEYOND the state of SPSS: it always bothered me that if I
302 have a large data file and I want to match it to a small lookup table, via
303 MATCH FILES FILE= /TABLE= /BY key, I need to SORT the large file on key, do the
304 match, then (usually) re-sort back into the order I really want it. There is
305 no reason to do this, when the lookup table is small. Even a dumb sequential
306 search through the table, for every case in the big file, is better, in some
307 cases, than the sort. So here's my idea: first look at the /TABLE file, if it
308 is "small enough", read it into memory, and create an index (or hash table,
309 whatever) for it. Then read the /FILE and use the index to match to each case.
310 OTOH, if the /TABLE is too large, then do it the old way, complaining if either
311 file is not sorted on key.
313 ----------------------------------------------------------------------
314 Statistical procedures:
316 For each case we read from the input program:
318 1. Execute permanent transformations. If these drop the case, stop.
319 2. N OF CASES. If we have already written N cases, stop.
320 3. Write case to replacement active file.
321 4. Execute temporary transformations. If these drop the case, stop.
322 5. Post-TEMPORARY N OF CASES. If we have already analyzed N cases, stop.
323 6. FILTER, PROCESS IF. If these drop the case, stop.
324 7. Pass case to procedure.
328 LAG records cases in step 3.
330 AGGREGATE: When output goes to an external file, this is just an ordinary
331 procedure. When output goes to the active file, step 3 should be skipped,
332 because AGGREGATE creates its own case sink and writes to it in step 7. Also,
333 TEMPORARY has no effect and we just cancel it. Regardless of direction of
334 output, we should not implement AGGREGATE through a transformation because that
335 will fail to honor FILTER, PROCESS IF, N OF CASES.
337 ADD FILES: Essentially an input program. It silently cancels unclosed LOOPs
338 and DO IFs. If the active file is used for input, then runs EXECUTE (if there
339 are any transformations) and then steals vfm_source and encapsulates it. If
340 the active file is not used for input, then it cancels all the transformations
341 and deletes the original active file.
347 MATCH FILES: Similar to AGGREGATE. This is a procedure. When the active file
348 is used for input, it reads the active file; otherwise, it just cancels all the
349 transformations and deletes the original active file. Step 3 should be
350 skipped, because MATCH FILES creates its own case sink and writes to it in step
351 7. TEMPORARY is not allowed.
359 UPDATE: same as ADD FILES.
362 ----------------------------------------------------------------------
365 * Before TEMPORARY, limits number of cases sent to the sink.
367 * After TEMPORARY, limits number of cases sent to the procedure.
369 * Without TEMPORARY, those are the same cases, so it limits both.
373 * Sample is just a transformation. It has no special properties.
377 * Always selects cases sent to the procedure.
379 * No effect on cases sent to sink.
381 * Before TEMPORARY, selection is permanent. After TEMPORARY,
382 selection stops after a procedure.
386 * Always selects cases sent to the procedure.
388 * No effect on cases sent to sink.
390 * Always stops after a procedure.
394 * Ignored by AGGREGATE. Used when procedures write matrices.
396 * Always applies to the procedure.
398 * Before TEMPORARY, splitting is permanent. After TEMPORARY,
399 splitting stops after a procedure.
403 * TEMPORARY has no effect on AGGREGATE when output goes to the active file.
405 * SORT CASES, ADD FILES, RENAME VARIABLES, CASESTOVARS, VARSTOCASES,
406 COMPUTE with a lag function cannot be used after TEMPORARY.
408 * Cannot be used in DO IF...END IF or LOOP...END LOOP.
410 * FLIP ignores TEMPORARY. All transformations become permanent.
412 * MATCH FILES and UPDATE cannot be used after TEMPORARY if active
413 file is an input source.
415 * RENAME VARIABLES is invalid after TEMPORARY.
417 * WEIGHT, SPLIT FILE, N OF CASES, FILTER, PROCESS IF apply only to
418 the next procedure when used after TEMPORARY.
422 * Always applies to the procedure.
424 * Before TEMPORARY, weighting is permanent. After TEMPORARY,
425 weighting stops after a procedure.
428 -------------------------------------------------------------------------------