+2007-06-06 Ben Pfaff <blp@gnu.org>
+
+ Add casereaders and casewriters, the basis of the new data processing
+ implementation. A casereader is a uniform interface to reading cases
+ from a data source; a casewriter is a uniform interface to writing
+ cases to a data sink.
+
+ * automake.mk: Add new files.
+
+ * casereader-filter.c: New file.
+
+ * casereader-provider.h: New file.
+
+ * casereader-translator.c: New file.
+
+ * casereader.c: New file.
+
+ * casereader.h: New file.
+
+ * casewriter-provider.h: New file.
+
+ * casewriter-translator.c: New file.
+
+ * casewriter.c: New file.
+
+ * casewriter.h: New file.
+
+2007-06-06 Ben Pfaff <blp@gnu.org>
+
+ "casewindow" data structure that extends the deque (from libpspp)
+ of cases with the ability to dump cases to disk if we get too many
+ of them in memory.
+
+ * automake.mk: Add new files.
+
+ * casewindow.c: New file.
+
+ * casewindow.h: New file.
+
+2007-06-06 Ben Pfaff <blp@gnu.org>
+
+ sparse_cases data structure that augments a sparse_array of cases
+ with the ability to dump cases to disk if we get too many cases in
+ memory.
+
+ * automake.mk: Add new files.
+
+ * sparse-cases.c: New file.
+
+ * sparse-cases.h: New file.
+
+2007-06-06 Ben Pfaff <blp@gnu.org>
+
+ Adds a low-level on-disk case array data structure.
+
+ * automake.mk: Add new files.
+
+ * case-tmpfile.c: New file.
+
+ * case-tmpfile.h: New file.
+
+2007-06-06 Ben Pfaff <blp@gnu.org>
+
+ In a couple of places we calculate the maximum number of cases to
+ keep in memory based on the user-defined workspace. Enable
+ centralizing the calculation through a new function.
+
+ * settings.c (get_workspace_cases): New function.
+
+2007-06-06 Ben Pfaff <blp@gnu.org>
+
+ The casenumber type is defined in transformations.h, but case.h is
+ a more sensible place. Move it.
+
+ * case.h (CASENUMBER_MAX): New macro.
+ (typedef casenumber): Move here, from transformations.h.
+
+2007-06-03 Ben Pfaff <blp@gnu.org>
+
+ Slightly generalize case_to_values and case_from_values functions.
+
+ * case.c (case_to_values): Rename case_copy_out, change interface.
+ (case_from_values): Rename case_copy_in, change interface.
+
+ * fastfile.c (fastfilereader_get_next_case): Update caller.
+ (write_case_to_disk): Ditto.
+
+2007-06-02 Ben Pfaff <blp@gnu.org>
+
+ Clean up after a forgotten part of patch #5829.
+
+ * casedeque.h: Remove unused file.
+
+ * automake.mk: Remove casedeque.h from sources.
+
+2007-05-10 Jason Stover <jhs@math.gcsu.edu>
+
+ * category.c: Removed redundant #include
+
+2007-05-06 Ben Pfaff <blp@gnu.org>
+
+ Abstract the documents within a dictionary a little better.
+ Thanks to John Darrington for suggestion, initial version, and
+ review. Patch #5917.
+
+ * dictionary.c (struct dictionary): Change `documents' member from
+ char * to struct string.
+ (dict_clear): Destroy struct string.
+ (dict_get_documents): Convert struct string to char *.
+ (dict_set_documents): Set struct string. Pad to 80-character
+ multiple.
+ (dict_clear_documents): New function.
+ (dict_add_document_line): New function.
+ (dict_get_document_line_cnt): New function.
+ (dict_get_document_line): New function.
+
+ * dictionary.h (macro DOC_LINE_LENGTH): New macro.
+
+ * sys-file-reader.c (read_documents): Use new document functions.
+
+2007-04-19 John Darrington <john@darrington.wattle.id.au>
+
+ * sys-file-reader.c: When reading a system file which has no
+ long name table, automatically create one where the long names
+ are the lower case versions of the short names.
+
+2007-04-22 Ben Pfaff <blp@gnu.org>
+
+ * dictionary.c (dict_set_split_vars): dict_destroy expects that
+ dict_clear will free most data related to the dictionary.
+ dict_clear does a decent job, except that dict_set_split_vars on
+ some systems won't actually free the dict's "split" member.
+ Instead, it'll allocate a 1-byte region. Fix this.
+
+ * value.c (value_copy): New function.
+ (value_set_missing): Ditto.
+
+2007-04-22 John Darrington <john@darrington.wattle.id.au>
+
+ * Deleted existing category.h and moved cat-routines.h into
+ category.h Encapsulated struct cat_vals better.
+
+2007-04-19 John Darrington <john@darrington.wattle.id.au>
+
+ * sys-file-reader.c: When reading a system file which has no
+ long name table, automatically create one where the long names
+ are the lower case versions of the short names.
+
+2007-04-16 John Darrington <john@darrington.wattle.id.au>
+
+ * sys-file-reader.c: Some versions of Other Software seem to
+ produce system files with string variables' measure set to
+ zero. We'll assume these are supposed to be nominal variables.
+
+2007-03-30 Ben Pfaff <blp@gnu.org>
+
+ * procedure.c: Adapt to new deque data structure.
+
+Mon Feb 19 10:53:21 2007 John McCabe-Dansted <gmatht@gmail.com>
+ Ben Pfaff <blp@gnu.org>
+
+ * file-name.c: Mingw compatibility fixes.
+ (fn_search_path): Use ISSLASH instead of comparing against '/'
+ directly.
+ (fn_dir_name): Use dir_name from gnulib.
+ (fn_is_absolute): Use IS_ABSOLUTE_FILE_NAME from gnulib.
+ (fn_get_identity): Use GetFullPathName instead of canonicalize
+ from gnulib, because the latter does not fully support
+ Windows-style path names. Use this implementation based on the
+ detected presence of Windows instead of the absence of Unix, since
+ the new implementation is Windows-specific.
+ (fn_compare_file_identities): In Windows implementation, compare
+ names case-insensitively.
+
+Sun Feb 18 13:28:02 2007 Ben Pfaff <blp@gnu.org>
+
+ * make-file.c: Don't include mkstemp.h, because gnulib now causes
+ <stdlib.h> to have the same effect.
+
+Sun Feb 18 11:20:24 2007 Ben Pfaff <blp@gnu.org>
+
+ * por-file-reader.c: Add missing _() around messages.
+
+Sun Feb 11 20:44:13 2007 Ben Pfaff <blp@gnu.org>
+
+ * make-file.c: Include "mkstemp.h", without which linking on
+ mingw32 fails.
+
+Thu Feb 8 14:59:05 2007 Ben Pfaff <blp@gnu.org>
+
+ Reduce platform dependence.
+
+ * file-name.c (fn_tilde_expand): Removed, and removed calls to it.
+ Everywhere we using this, we really should have just depended on
+ the shell to expand tildes.
+ (fn_search_path): Simplify, given that we don't do tilde expansion
+ any longer.
+ (fn_normalize): Removed. Caller changed to use the canonicalize
+ module from gnulib.
+ (fn_get_cwd): Removed. Only user was fn_normalize.
+ (fn_is_absolute): Really only test for absolute names.
+ (fn_is_special): Use pipe files if HAVE_POPEN, not if we're in
+ unix.
+ (fn_readlink): Removed, as it was only used fn_normalize.
+ (fn_exists): Assume the stat function is available; gnulib does.
+ (fn_open): Use pipe files if HAVE_POPEN, not if we're in unix.
+
+Sat Feb 3 21:52:17 2007 Ben Pfaff <blp@gnu.org>
+
+ * dictionary.c (dict_create_vector_assert): New function.
+
+Wed Feb 7 21:25:15 2007 Ben Pfaff <blp@gnu.org>
+
+ * file-name.c (fn_normalize): Correct name of function
+ fn_is_special. Thanks to John McCabe-Dansted <gmatht@gmail.com>
+ for pointing this out.
+
+Thu Feb 1 16:53:37 2007 Ben Pfaff <blp@gnu.org>
+
+ We are using a single member in struct file_handle, the "name"
+ field, for more than one purpose. When it begins with '"', it's a
+ file name; otherwise, it's a token that can be used to identify
+ it. When that assertion fires, it's because we searched for the
+ name case-sensitively as a file name (so that there was no match),
+ and then we try to insert it case-insensitively as a token, which
+ fails because duplicates aren't allowed.
+
+ Solution: break the two purposes into two separate fields. This
+ fixes the problem and likely makes the code easier to read too.
+
+ Fixes bug #18922. Thanks to John Darrington for bug report and
+ review.
+
+ * file-handle-def.c (struct file_handle): New `id' member.
+ (fh_from_name): Rename fh_from_id. Update all callers.
+ (create_handle): New `id' parameter. Update all callers.
+ (fh_create_file): Ditto.
+ (fh_get_id): New function.
+
+Mon Jan 15 16:18:10 2007 Ben Pfaff <blp@gnu.org>
+
+ * case.c (case_is_null): Change return type to bool.
+
+Mon Jan 15 10:57:28 2007 Ben Pfaff <blp@gnu.org>
+
+ Add debugging code.
+
+ * case.c (case_clone) [DEBUGGING]: When debugging, don't use
+ reference counting to share data. This makes it easy for
+ valgrind, etc. to find accesses to cases that have been destroyed
+ but have been kept around by another user's ref-count. This often
+ happens when the data set is small enough to find in memory; if a
+ bigger data set that would overflow to disk were used, then data
+ corruption would occur.
+
+Mon Jan 15 10:55:18 2007 Ben Pfaff <blp@gnu.org>
+
+ Simplify code.
+
+ * case.c (case_unshare): Make it check internally whether the
+ ref_cnt is greater than 1, so that the callers don't have to.
+ Update callers not to check.
+
+Mon Jan 15 10:53:01 2007 Ben Pfaff <blp@gnu.org>
+
+ Before, I was thinking that I might want to get rid of reference
+ counting at some point. Now, I'm pretty sure that it's here to
+ stay. Thus, because we have to store the value_cnt anyway for
+ reference-counted cases, we might as well expose it to users.
+
+ * case.c (case_get_value_cnt): New function.
+ (case_resize): Drop OLD_CNT argument. Update all callers. Only
+ resize case if its size actually changed.
+
+ * casefile.c (casefile_append_xfer): Use case_get_value_cnt
+ instead of peeking inside struct case directly.
+ (casefile_append): Ditto.
+
+Mon Jan 15 10:50:22 2007 Ben Pfaff <blp@gnu.org>
+
+ Get rid of the inlines for the case functions, which made the
+ header file hard to read. (Also, in testing with "-O2 -DNDEBUG",
+ the inlines didn't speed up "make check" at all, which is not a
+ perfect benchmark but seems indicative.)
+
+ * case.c: Remove #ifdef DEBUGGING...#endif around many function
+ definitions. Remove some assertions on nonnull pointers that were
+ redundant with a pointer dereference soon after in the function.
+ Also:
+ (struct case_data): Move definition here from case.h.
+ (case_data): Ditto.
+ (case_num): Ditto.
+ (case_str): Ditto.
+ (case_data_wr): Ditto.
+
+Sun Jan 14 21:41:12 2007 Ben Pfaff <blp@gnu.org>
+
+ * automake.mk: Add casedeque.h to sources.
+
+ * casedeque.h: New file.
+
+ * procedure.c: (struct dataset) Change lag_count, lag_head,
+ lag_queue member into single struct casedeque member. Update all
+ users to use the casedeque instead.
+ (lag_case) Removed.
+
+Sun Jan 14 21:43:12 2007 Ben Pfaff <blp@gnu.org>
+
+ * procedure.c: Simplify lagged cases interface. Updated all
+ clients--well, the only client--to use the simplified interface.
+ (dataset_n_lag) Removed.
+ (dataset_set_n_lag) Removed.
+ (dataset_need_lag) New function.
+
+Tue Jan 9 07:20:05 WST 2007 John Darrington <john@darrington.wattle.id.au>
+
+ * dictionary.c procedure.c: More changes to ensure that callbacks occur
+ whenever appropriate, but only when the dataset/dictionary is in a
+ consistent state.
+
+Sun Jan 7 08:33:04 WST 2007 John Darrington <john@darrington.wattle.id.au>
+
+ * dictionary.c dictionary.h : Added callbacks for change of filter and
+ split variables. Refactored some code to ensure that callbacks get
+ invoked when appropriate.
+
+ * procedure.c (proc_cancel_temporary_transformations): Make sure that
+ replace_dict callback occurs when permanent_dict replaces the current
+ dictionary.
+
+Wed Jan 3 11:02:11 WST 2007 John Darrington <john@darrington.wattle.id.au>
+
+ * dictionary.c dictionary.h : Added callback for when the weight
+ variable of a dictionary changes.
+
+Mon Jan 1 10:36:26 WST 2007 John Darrington <john@darrington.wattle.id.au>
+
+ * dictionary.c dictionary.h : Added replace_source and replace_dict
+ callbacks, and functions to deal with them.
+
+Fri Dec 22 13:56:08 2006 Ben Pfaff <blp@gnu.org>
+
+ Simplify missing value handling.
+
+ * missing-values.h (enum mv_class): New type.
+ (enum mv_type): Moved definition into missing-values.c and renamed
+ each MV_* to MVT_*, to distinguish them from the exposed mv_class
+ enums. Updated all uses.
+ (struct missing_values): Changed type of `type' from `enum
+ mv_type' to `int' because the definition is no longer exposed.
+
+ * missing-values.c (mv_is_value_missing): Add new enum mv_class
+ parameter. Update all callers.
+ (mv_is_num_missing): Ditto.
+ (mv_is_str_missing): Ditto.
+ (mv_is_value_user_missing): Removed. Changed callers to use
+ mv_is_value_missing.
+ (mv_is_num_user_missing): Removed. Changed callers to use
+ mv_is_num_missing.
+ (mv_is_str_user_missing): Removed. Changed callers to use
+ mv_is_str_missing.
+ (mv_is_value_system_missing): Removed. Changed callers to use
+ mv_is_value_missing.
+ (mv_set_type): Removed. Changed callers to use mv_clear.
+ (mv_clear): New function.
+
+ * variable.c (var_is_value_missing): Add new enum mv_class
+ parameter. Update all callers.
+ (var_is_num_missing): Ditto.
+ (var_is_str_missing): Ditto.
+ (var_is_value_user_missing): Removed. Changed callers to use
+ var_is_value_missing.
+ (var_is_num_user_missing): Removed. Changed callers to use
+ var_is_num_missing.
+ (var_is_str_user_missing): Removed. Changed callers to use
+ var_is_str_missing.
+ (var_is_value_system_missing): Removed. Changed callers to use
+ var_is_value_missing.
+
+ * casefilter.c (struct casefilter): Use enum mv_class in place of
+ bool.
+ (casefilter_variable_missing): Adapt to new member.
+ (casefilter_create): Change signature to take enum mv_class,
+ update callers.
+
+Fri Dec 22 20:08:38 WST 2006 John Darrington <john@darrington.wattle.id.au>
+
+ * casefile-factory.h fastfile-factory.c fastfile-factory.h: New files.
+
+ * case-sink.c case-sink.h procedure.c procedure.h
+ storage-stream.c: Now uses the factory.
+
+Sat Dec 16 22:05:18 2006 Ben Pfaff <blp@gnu.org>
+
+ Make it possible to pull cases from the active file with a
+ function call, instead of requiring indirection through a callback
+ function.
+
+ * case-source.h (struct case_source_class): Change ->read function
+ to return a single case, instead of calling a callback function
+ for each case. Change ->destroy function to return an error
+ status.
+
+ * case-source.c (free_case_source): Pass along the value returned
+ by the case_source ->destroy function.
+
+ * procedure.c (struct write_case_data): Removed.
+ (struct dataset): Added some members to track procedure state.
+ (procedure): Optimize the trivial case at this level.
+ (internal_procedure): Re-implement in terms of proc_open,
+ proc_read, proc_close.
+ (proc_open) New function.
+ (proc_read) New function.
+ (proc_close) New function.
+ (write_case) Moved into proc_read.
+ (close_active_file) Moved closing of data source into proc_close.
+
+ * storage-source.c: Rewrote to conform with modified
+ case_source_class interface.
+
+ * transformations.c (trns_chain_execute): Added argument to allow
+ starting execution from an arbitrary transformation. Updated
+ callers.
+
+ * transformations.h (enum TRNS_NEXT_CASE) Renamed TRNS_END_CASE.
+
+Sat Dec 16 14:09:25 2006 Ben Pfaff <blp@gnu.org>
+
+ * sys-file-reader.c (read_display_parameters): Don't assume that
+ MEASURE_* and ALIGN_* have the same values found in system files.
+
+ * sys-file-writer.c (write_variable_display_parameters): Ditto.
+
+ * variable.h: Change MEASURE_NOMINAL, MEASURE_ORDINAL,
+ MEASURE_SCALE to be 0-based instead of 1-based. This also fixes
+ the value of n_MEASURES, which was off by 1 (at least from my
+ point of view).
+
+Sat Dec 16 12:17:34 WST 2006 John Darrington <john@darrington.wattle.id.au>
+
+ * dictionary.c dictionary.h vardict.h variable.c: Added optional
+ callbacks which are invoked when the dictionary or its
+ variables are changed.
+
+ * missing-values.c missing-values.h value-labels.c: Tidied up
+ consistency checks, and made some of them return false
+ instead of assert-failing.
+
+Wed Dec 13 19:30:11 2006 Ben Pfaff <blp@gnu.org>
+
+ * calendar.c (calendar_days_in_month): New function.
+
+Mon Dec 11 07:53:39 2006 Ben Pfaff <blp@gnu.org>
+
+ * value-labels.c (hash_int_val_lab): Only hash as many bytes as
+ the value label's width.
+
+Sun Dec 10 14:21:29 2006 Ben Pfaff <blp@gnu.org>
+
+ * sfm-private.h: Move contents into sys-file-writer.c, which is
+ the only remaining user. Removed Borland C++-specific directives.
+
+ * sys-file-reader.c: Clean up and rewrite entire file. The
+ rewritten version is simpler and better abstracted, and should be
+ easier to maintain and extend. It avoids using structures to read
+ file data, which is prone to padding variations among compilers.
+ It should also handle non-IEEE 754 system files, although I
+ haven't been able to find any. It has been tested against many
+ .sav files obtained from the Web and found to produce the same
+ results as the earlier version of the code, or in some cases
+ improved results. It is more tolerant of format variations found
+ in the wild.
+
+ * sys-file-reader.h (struct sfm_read_info): Removed `big_endian'
+ member, putting an enum integer_format in its place. New member
+ `float_format'. Changed `compressed' member to type bool.
+
+Sun Dec 10 13:48:53 2006 Ben Pfaff <blp@gnu.org>
+
+ * dictionary.c (dict_delete_consecutive_vars): New function.
+
+Sat Dec 9 20:08:25 2006 Ben Pfaff <blp@gnu.org>
+
+ * file-name.c (fn_search_path): Remove prefix arg that was unused
+ by any caller. Updated all callers.
+
+Sat Dec 9 20:04:22 2006 Ben Pfaff <blp@gnu.org>
+
+ * format.c (fmt_dollar_template): Use user's decimal point
+ character. Add assertion.
+
+Sat Dec 9 20:02:25 2006 Ben Pfaff <blp@gnu.org>
+
+ * format.c (fmt_dollar_template): New function, based on
+ dollar_format_template from var-type-dialog.c.
+
+Sat Dec 9 18:05:59 2006 Ben Pfaff <blp@gnu.org>
+
+ * data-out.c (output_scientific): Fix bad assumption that "buf" is
+ null-terminated.
+
+Sat Dec 9 17:23:23 2006 Ben Pfaff <blp@gnu.org>
+
+ Finish converting struct variable to an opaque type. In this
+ phase, we add remaining setter and getter functions, convert the
+ remaining PSPP code to use them, and do a bunch of cleanup. The
+ resulting changes are pervasive but mostly trivial, and only the
+ notable changes are logged.
+
+ * automake.mk (src_data_libdata_a_SOURCES): Add the new source
+ files.
+
+ * case.c (case_data): Renamed case_data_idx.
+ (case_num): Renamed case_num_idx.
+ (case_str): Renamed case_str_idx.
+ (case_data_rw): Renamed case_data_rw_idx.
+
+ * case.h (case_data): New function with old name and an interface
+ that takes a variable instead of an index, which is easier to
+ use. Updated all callers to use the new interface, or to use the
+ new *_idx function (see above).
+ (case_num): Ditto.
+ (case_str): Ditto.
+ (case_data_rw): Ditto.
+
+ * category.c (cat_stored_values_destroy): Changed interface to
+ take a struct cat_vals * instead of a struct variable *.
+
+ * dictionary.c (dict_clone): Use new vector_clone function.
+ (dict_clear) Use new var_destroy function.
+ (add_var) New function.
+ (dict_create_var) Rewrite in terms of dict_create_var_assert.
+ (dict_create_var_assert) Rewrite in terms of add_var.
+ (dict_clone_var) Rewrite in terms of dict_clone_var_assert.
+ (dict_clone_var_assert) Rewrite in terms of var_clone, add_var.
+ (dict_lookup_var) Use new var_create, var_destroy functions.
+ (dict_contains_var) Rewrite in terms of new vardict functionality.
+ (set_var_dict_index) New function.
+ (set_var_case_index) New function.
+ (reindex_vars) New function.
+ (dict_delete_var) Rewrite in terms of new vardict functionality.
+ (dict_reorder_var) Ditto.
+ (dict_reorder_vars) Ditto.
+ (rename_var) New function.
+ (dict_rename_var) Use rename_var.
+ (dict_rename_vars) Use pool to simplify code. Use rename_var.
+ (dict_get_compacted_idx_to_fv) Rename
+ dict_get_compacted_dict_index_to_case_index, update callers.
+ (dict_create_vector) Use new vector_create function.
+ (dict_clear_vectors) Use new vector_destroy function.
+ (set_var_short_name_suffix) Move here from variable.c, renamed
+ from var_set_short_name_suffix, make static, update caller.
+
+ * sys-file-private.c: New file.
+ (sfm_width_to_bytes) Moved here from variable.c, renamed from
+ width_to_bytes, update callers.
+
+ * sys-file-private.h: New file. Later it will supplant
+ sfm-private.h; for now it supplements it.
+ (macro MIN_VERY_LONG_STRING) New macro.
+ (macro EFFECTIVE_LONG_STRING_LENGTH) New macro, from value.h.
+
+ * sys-file-reader.c: Use MIN_VERY_LONG_STRING - 1 where
+ MAX_LONG_STRING was used before.
+
+ * sys-file-writer.c: Ditto.
+
+ * value-labels.c: Change the paradigm here to be that a null
+ pointer is OK for a struct val_labs * in most cases; it just
+ represents an empty set of value labels.
+ (val_labs_copy) A copy of a null set is a null set.
+ (val_labs_count) A null set has 0 labels.
+ (val_labs_replace) Change return type to void. Rewrite for
+ simplicity.
+ (val_labs_find) A null set does not contain the value.
+ (value_to_string) Moved to variable.c, renamed var_get_value_name,
+ transposed argument order, updated all callers.
+
+ * value.c: New file.
+ (value_dup) Moved here from variable.c.
+ (compare_values) Ditto.
+ (hash_value) Ditto.
+
+ * value.h: (macro MAX_SHORT_STRING) Rewrote for simplicity.
+ (macro MAX_LONG_STRING) Removed, because it was only interesting
+ for system files, not for general code.
+ (macro MAX_VERY_LONG_STRING) Ditto.
+ (macro EFFECTIVE_LONG_STRING_LENGTH) Moved to sys-file-private.h.
+ (macro MAX_ELEMS_PER_VALUE) Removed, as it was unused.
+
+ * vardict.h: New file, for an interface between variables and
+ their dictionaries.
+
+ * variable.c: A lot of functions were moved around, for better
+ organization.
+ (struct variable) Move definition here, from variable.h.
+ (var_type_adj) Removed--makes i18n hard.
+ (var_type_noun) Ditto.
+ (var_create) New function.
+ (var_clone) New function.
+ (var_destroy) New function.
+ (var_set_name) Assert that variable is not in a dictionary.
+ (compare_var_names) Rename compare_vars_by_name and fix a couple
+ of callers who thought the args were strings.
+ (hash_var_name) Rename hash_var_by_name.
+ (compare_var_ptr_names) Rename compare_var_ptrs_by_name.
+ (hash_var_ptr_name) Rename hash_var_ptr_by_name.
+ (var_is_very_long_string) Removed, because it was only interesting
+ to system file code.
+ (var_set_missing_values) Allow the argument to be the wrong width,
+ as long as we can resize it. Simplify callers who were doing the
+ resizing themselves.
+ (var_get_value_labels) New function.
+ (var_has_value_labels) New function.
+ (var_set_value_labels) New function.
+ (alloc_value_labels) New function.
+ (var_add_value_label) New function.
+ (var_replace_value_label) New function.
+ (var_clear_value_labels) New function.
+ (var_lookup_value_label) New function.
+ (var_get_value_name) Moved here from variable.c, renamed from
+ var_get_value_name, transposed argument order, updated all
+ callers.
+ (var_to_string) Moved here, from variable-label.c.
+ (var_set_leave) New function.
+ (var_get_leave) New function.
+ (var_must_leave) New function.
+ (var_set_short_name_suffix) Moved to dictionary.c, renamed
+ set_var_short_name_suffix.
+ (var_get_dict_index) New function.
+ (var_get_case_index) New function.
+ (var_get_obs_vals) New function.
+ (var_set_obs_vals) New function.
+ (var_has_obs_vals) New function.
+ (var_get_vardict) New function.
+ (var_set_vardict) New function.
+ (var_has_vardict) New function.
+ (var_clear_vardict) New function.
+ (value_dup) Moved to value.c.
+ (compare_values) Ditto.
+ (hash_value) Ditto.
+
+ * variable.h: (enum NUMERIC) Rename VAR_NUMERIC, update all users.
+ (enum ALPHA) Rename VAR_STRING, update all users.
+
+ * vector.c: New file.
+ (struct vector) Moved here, from variable.h.
+ (check_widths) New function.
+ (vector_create) New function.
+ (vector_clone) New function.
+ (vector_destroy) New function.
+ (vector_get_name) New function.
+ (vector_get_var) New function.
+ (vector_get_var_cnt) New function.
+ (compare_vector_ptrs_by_name) New function.
+
+ * vector.h: New file.
+
+Sun Dec 10 11:32:56 WST 2006 John Darrington <john@darrington.wattle.id.au>
+
+ * casefilter.c (casefilter_variable_missing): Avoided comparision of
+ string variables to SYSMIS. Thanks to Ben Pfaff for reporting this
+ problem.
+
+Sat Dec 9 07:18:03 WST 2006 John Darrington <john@darrington.wattle.id.au>
+
+ * value-labels.c (destroy_atoms): New function.
+ * value-labels.c (atom_create): Call destroy_atoms in atexit handler.
+
+Thu Dec 7 17:38:26 2006 Ben Pfaff <blp@gnu.org>
+
+ Thanks to Jason Stover for pointing out this problem.
+
+ * data-out.c (output_number): Use gsl_finite from GSL, which is
+ portable, instead of isfinite, which is not.
+ (power256) Ditto.
+
+Thu Dec 7 15:22:38 WST 2006 John Darrington <john@darrington.wattle.id.au>
+
+ * variable.c variable.h (value_dup): New function.
+
+Mon Dec 4 22:20:17 2006 Ben Pfaff <blp@gnu.org>
+
+ Start converting struct variable to an opaque type. In this
+ phase, we add a bunch of setter and getter functions and convert
+ most of the PSPP code to use them. The resulting changes are
+ pervasive but mostly trivial, and only the notable changes are
+ logged.
+
+ * format.c (fmt_equal): New function.
+
+ * variable.c (var_type_is_valid): New function.
+ (measure_is_valid) Moved here, from format.c.
+ (alignment_is_valid) Moved here, from format.c.
+ (var_get_name) New function.
+ (var_set_name) New function.
+ (width_to_type) New function.
+ (var_get_type) New function.
+ (var_get_width) New function.
+ (var_set_width) New function.
+ (var_is_numeric) New function.
+ (var_is_alpha) New function.
+ (var_is_short_string) New function.
+ (var_is_long_string) New function.
+ (var_is_very_long_string) New function.
+ (var_get_missing_values) New function.
+ (var_set_missing_values) New function.
+ (var_clear_missing_values) New function.
+ (var_has_missing_values) New function.
+ (var_is_value_missing) New function.
+ (var_is_num_missing) New function.
+ (var_is_str_missing) New function.
+ (var_is_value_user_missing) New function.
+ (var_is_num_user_missing) New function.
+ (var_is_str_user_missing) New function.
+ (var_is_value_system_missing) New function.
+ (var_get_print_format) New function.
+ (var_set_print_format) New function.
+ (var_get_write_format) New function.
+ (var_set_write_format) New function.
+ (var_set_both_formats) New function.
+ (var_get_label) New function.
+ (var_set_label) New function.
+ (var_clear_label) New function.
+ (var_has_label) New function.
+ (var_get_measure) New function.
+ (var_set_measure) New function.
+ (var_get_display_width) New function.
+ (var_set_display_width) New function.
+ (var_get_alignment) New function.
+ (var_set_alignment) New function.
+ (var_get_value_cnt) New function.
+ (var_get_leave) New function.
+ (var_get_short_name) New function.
+
+ * variable.h: (struct variable) Removed "type" and "nv" members;
+ they are now computed from "width" where needed.
+
+Mon Dec 4 21:38:40 2006 Ben Pfaff <blp@gnu.org>
+
+ * missing-values.c (mv_resize): Don't write beyond end of the
+ allocated buffer when resizing a long string.
+
+Sat Dec 2 16:28:32 2006 Ben Pfaff <blp@gnu.org>
+
+ Clean up identifier code: don't require identifier enumerations to
+ be in a particular order; make better use of string library;
+ expose less of the internals.
+
+ * identifier.c: (lex_skip_identifier) Rename lex_id_get_length,
+ change interface. Updated all callers.
+ (lex_id_match) Change interface to use struct substring, update
+ all callers.
+ (lex_id_match_len) Removed. Update callers to use lex_id_match.
+ (global array keywords[]) Make static, change form. Update all
+ users to use lex_id_name instead.
+ (lex_is_keyword) New function.
+ (lex_id_to_token) Change interface to use struct substring, update
+ all callers.
+ (lex_id_name) New function.
+
+ * identifier.h: (T_FIRST_KEYWORD) Removed. Changed users to call
+ lex_is_keyword instead.
+ (T_LAST_KEYWORD) Removed.
+ (T_N_KEYWORDS) Removed.
+
+Sat Nov 18 20:46:35 2006 Ben Pfaff <blp@gnu.org>
+
+ * format.c: (fmt_date_template) Distinguish characters for which a
+ space is output and any date delimiter is allowed on input, from
+ those for which a space is output and only a space is allowed on
+ input. The former is represented by X, the latter by a space.
+ Also, drop distinction between h and H, changing the former to the
+ latter.
+
+ * data-in.c: Completely rewrite internals to conform to SPSS input
+ formats as closely as possible.
+ (data_in) Changed external interface by replacing the structure
+ that was used as a single argument by a set of arguments. Updated
+ all callers.
+ (data_in_finite_line) Removed. Converted all callers to use plain
+ data_in.
+ (data_in_get_integer_format) New function.
+ (data_in_set_integer_format) New function.
+ (data_in_get_float_format) New function.
+ (data_in_set_float_format) New function.
+
+ * data-in.h: (enums DI_IGNORE_ERROR, DI_IMPLIED_DECIMALS) Removed.
+ (struct data_in) Removed.
+
+ * data-out.c: (output_date) Drop each component from the input as
+ it is output, to allow us to drop the distinction between h (a
+ count of hours) and H (the hour of day) template characters.
+ Also, handle new X template character.
+ (output_scientific) Follow more rational rule on when to drop
+ fraction introduced between SPSS 13 and 15. Updated test case to
+ match new behavior.
+
+Sat Nov 11 11:41:26 2006 Ben Pfaff <blp@gnu.org>
+
+ Fix buffer overflow reported by John Darrington.
+
+ * data-out.c (output_bcd_integer): In case of SYSMIS, etc.,
+ realize that DIGITS is a count of nibbles, not of bytes.
+
+Sat Nov 4 15:59:56 2006 Ben Pfaff <blp@gnu.org>
+
+ * calendar.c (calendar_offset_to_gregorian) Also return the
+ year-of-day. Change callers to new interface.
+
+ * data-out.c: Completely rewrite internals to conform to SPSS
+ output formats as completely as possible.
+ (data_out) Change interface to put input parameters before output
+ parameters, for consistency with the style I now prefer. Update
+ all callers.
+ (data_out_get_integer_format) New public function.
+ (data_out_set_integer_format) New public function.
+ (data_out_get_float_format) New public function.
+ (data_out_set_float_format) New public function.
+
+ * data-out.h: New file. Move prototype for data_out here, from
+ format.h.
+
+ * format.c: (fmt_step_width) Use equality comparison instead of
+ bitwise and, for clarity.
+ (fmt_is_string) Ditto.
+ (fmt_input_to_output) Fix categories that are translated to F
+ format.
+
+Sun Nov 5 08:29:34 WST 2006 John Darrington <john@darrington.wattle.id.au>
+
+ * casefilter.c casefilter.h (new files), casefile.c casefile.h
+ casefile-private.h: Added casefilter to assist commands with missing
+ values.
+
+Sat Nov 4 11:47:09 2006 Ben Pfaff <blp@gnu.org>
+
+ Implement SET ERRORS, SHOW ERRORS. Fixes bug #17609.
+
+ * settings.c: (route_errors_to_terminal) New variable.
+ (route_errors_to_listing) New variable.
+ (get_error_routing_to_terminal) New function.
+ (set_error_routing_to_terminal) New function.
+ (get_error_routing_to_listing) New function.
+ (set_error_routing_to_listing) New function.
+
+ * settings.h: (SET_ROUTE_* enums) Removed, because unused.
+
+Tue Oct 31 19:58:27 2006 Ben Pfaff <blp@gnu.org>
+
+ * format.c: Completely rewrite, to achieve better abstraction.
+ Rewrite all references to formats in other files.
+
+ * format.def: Rewrite and reorganize.
+
+ * settings.c: Move everything related to custom currency formats
+ into format.[ch], changing them in form, so as to group related
+ code and definitions better. Changed all references to use the
+ new functions.
+ (static var decimal) Removed.
+ (static var grouping) Removed.
+ (static var cc) Removed.
+ (get_decimal) Removed.
+ (set_decimal) Removed.
+ (get_grouping) Removed.
+ (set_grouping) Removed.
+ (get_cc) Removed.
+ (set_cc) Removed.
+
+ * settings.h: (macro CC_CNT) Removed.
+ (macro CC_WIDTH) Removed.
+ (struct custom_currency) Removed.
+
+Tue Oct 31 19:56:19 2006 Ben Pfaff <blp@gnu.org>
+
+ * data-in.c (data_in): Use switch statement instead of table, to
+ avoid dependence on the order of the FMT_* enums.
+
+Tue Oct 31 19:35:36 2006 Ben Pfaff <blp@gnu.org>
+
+ * data-out.c: (num_to_string) Removed, because it was dead code.
+
+Tue Oct 31 18:09:24 2006 Ben Pfaff <blp@gnu.org>
+
+ * data-in.c (parse_trailer): Fix error message.
+
+Sat Oct 28 11:56:50 2006 Ben Pfaff <blp@gnu.org>
+
+ * format.c (fmt_is_binary): New function.
+
+Thu Oct 19 22:59:56 WST 2006 John Darrington <john@darrington.wattle.id.au>
+
+ * procedure.c procedure.h: Encapsulated the static data into a single
+ struct.
+
+Sat Oct 14 16:56:44 2006 Ben Pfaff <blp@gnu.org>
+
+ * casefile.c (casereader_read_xfer): Always initialize the case,
+ even on an error condition.
+
+Wed Sep 27 09:37:49 WST 2006 John Darrington <john@darrington.wattle.id.au>
+
+ * procedure.c (case_limit_trns_proc): Fixed buglet which rendered the
+ entire function useless.
+
+Mon Sep 25 17:11:46 WST 2006 John Darrington <john@darrington.wattle.id.au>
+
+ * casefile-private.h casefile.c casefile.h fastfile.c: Created new
+ casereader method casereader_clone.
+
+ * procedure.c pransformations.h: Introduced new type casenum_t
+
+Thu Sep 21 07:00:30 2006 Ben Pfaff <blp@gnu.org>
+
+ * variable.c: (width_to_bytes) Rephrase code for clarify.
+
+Sun Jul 16 19:52:03 2006 Ben Pfaff <blp@gnu.org>
+
+ * format.c: (fmt_type_from_string) New function.
+ (fmt_to_string) Include decimals in output if the format has
+ decimals, even if the format type does not. This way, we can
+ accurately reproduce incorrect formats in user output.
+ (check_common_specifier) Make the check for a bad format type an
+ assertion, so we get bug reports if they show up. Fix message.
+ Check for decimal places with a format type that doesn't allow
+ them.
+ (check_input_specifier) Remove check for FMT_X, which has been
+ deleted.
+ (check_output_specifier) Ditto.
+
+ * format.def: Remove FMT_T, FMT_X, FMT_DESCEND, FMT_NEWREC.
+
+ * format.h: (macro FMT_TYPE_LEN_MAX) New macro.
+ (struct fmt_desc) Use FMT_TYPE_LEN_MAX in definition.
+ (enum fmt_parse_flags) Removed.
+
+Mon Jul 17 18:26:21 WST 2006 John Darrington <john@darrington.wattle.id.au>
+
+ * casefile.c casefile.h: Converted to an abstract base class.
+ * casefile-private.h fastfile.c fastfile.h: New files.
+ * automake.mk procedure.c scratch-writer.c storage-stream.c
+
+Wed Jul 12 21:02:26 2006 Ben Pfaff <blp@gnu.org>
+
+ * procedure.c (internal_procedure): Create sink_case with only as
+ many values as the compacted dictionary.
+
+Wed Jul 12 21:01:00 2006 Ben Pfaff <blp@gnu.org>
+
+ Remove "debugging" code that caused plenty of false positives and
+ no true positives.
+
+ * case.h (struct ccase): [DEBUGGING] Remove `this' member.
+
+ * case.c: Remove all references to `this' member.
+
+Thu Jul 6 19:09:53 2006 Ben Pfaff <blp@gnu.org>
+
+ Fix link error noted by Jason Stover.
+
+ * storage-stream.c: Include <assert.h>.
+
+Tue Jul 4 08:47:35 2006 Ben Pfaff <blp@gnu.org>
+
+ Fix bug #15766 (/KEEP subcommand on SAVE doesn't fully support
+ ALL) and additional underlying system file issues.
+
+ Thanks to John Darrington for review.
+
+ First problem: var_hash points to variables not owned by the
+ sys-file-reader, which the caller may free or modify. Use an
+ array of sfm_vars instead, as done earlier (e.g. CVS version
+ 1.12).
+
+ * sys-file-reader.c (struct sfm_reader): Remove var_hash, svars
+ members and remove all code that references it. Add vars, var_cnt
+ members. Remove fix_specials member, which was unused.
+ (struct sfm_var) Remove name member, which was unused.
+ (sfm_close_reader) Free vars member instead of var_hash.
+ (compare_var_shortnames) Removed.
+ (hash_var_shortname) Removed.
+ (sfm_open_reader) Fill out vars array.
+ (compare_var_index) Removed.
+ (sfm_read_case) Use vars instead of var_hash.
+
+ Second problem: we're confused about when we actually have very
+ long strings, causing us to choose incorrectly between slow path
+ and fast path in sfm_read_case.
+
+ * sys-file-reader.c: (sfm_open_reader) Only mark has_vls if we
+ have very long strings, not when we have long variable names,
+ which is an unrelated feature.
+
+Tue Jun 27 12:06:49 2006 Ben Pfaff <blp@gnu.org>
+
+ * variable.h: Move var_set and variable parsing declarations to
+ new header, src/language/lexer/variable-parser.h. Modified lots
+ of files to include the new header.
+
+Sun Jun 25 22:39:32 2006 Ben Pfaff <blp@gnu.org>
+
+ * value-labels.c (value_to_string): When there's no value label,
+ format the variable according to its print format, instead of
+ always effectively using A or F format.
+
+Mon Jun 19 18:05:42 WST 2006 John Darrington <john@darrington.wattle.id.au>
+
+ * casefile.c (casefile_get_random_reader): Nasty hack to get around
+ the mode assertion.
+
+ * format.c: Removed tortological assertion.
+
+Fri Jun 9 12:20:09 2006 Ben Pfaff <blp@gnu.org>
+
+ Reform string library.
+
+ * file-name.c (fn_interp_vars): Change interface to take a
+ substring as input. Updated all users.
+
+Fri Jun 9 12:11:24 2006 Ben Pfaff <blp@gnu.org>
+
+ * format.c (measure_is_valid): Really return false when m >=
+ n_MEASURES.
+
+Tue Jun 6 18:46:26 2006 Ben Pfaff <blp@gnu.org>
+
+ Implement random access to casefiles, for use in GUI.
+
+ * casefile.c: (struct casereader) Add `random', `file_ofs',
+ `buffer_ofs' members.
+ (casefile_get_random_reader) New function.
+ (read_open_file) Break part into new function
+ seek_and_fill_buffer().
+ (fill_buffer) Update buffer_ofs, file_ofs.
+ (casereader_seek) New function.
+
+Tue May 30 19:52:33 WST 2006 John Darrington <john@darrington.wattle.id.au>
+
+ * settings.c: Added call to i18n{done, init}.
+
+Tue May 9 21:09:17 2006 Ben Pfaff <blp@gnu.org>
+
+ * procedure.h: Add WARN_UNUSED_RESULT to procedure function
+ prototypes.
+
+Tue May 9 21:08:05 2006 Ben Pfaff <blp@gnu.org>
+
+ * casefile.c: Convert many uses of `int' to `bool'.
+
+Sat May 6 22:49:43 2006 Ben Pfaff <blp@gnu.org>
+
+ * transformations.c (trns_chain_destroy): Destroy chain's trns
+ member, to fix memory leak.
+
+Sat May 6 22:48:30 2006 Ben Pfaff <blp@gnu.org>
+
+ * storage-stream.c (storage_source_decapsulate): Destroy case
+ source to fix memory leak.
+
+Sat May 6 22:46:47 2006 Ben Pfaff <blp@gnu.org>
+
+ * scratch-reader.c (scratch_reader_read_case): Copy into existing
+ case passed as argument instead of initializing the argument as a
+ case. Fixes memory leak that showed up in
+ tests/command/aggregate.sh with scratch files.
+
+Sat May 6 22:45:55 2006 Ben Pfaff <blp@gnu.org>
+
+ * procedure.c (proc_done): Destroy default_dict, to fix memory
+ leak.
+
+Sat May 6 22:44:44 2006 Ben Pfaff <blp@gnu.org>
+
+ Simplify procedure_with_splits().
+
+ * procedure.c (struct split_aux_data): Removed case_count member.
+ (procedure_with_splits) Don't initialize case_count.
+ (split_procedure_case_func) Check whether prev_case is null
+ instead of case_count.
+ (split_procedure_end_func) Ditto.
+
+Sat May 6 22:42:23 2006 Ben Pfaff <blp@gnu.org>
+
+ * case.c (case_move): Do nothing if dst and src are the same
+ object.
+ (case_try_create) Merge two similar cases.
+ (case_copy) Unshare only if data must be actually copied.
+
+Sun May 7 10:04:06 WST 2006 John Darrington <john@darrington.wattle.id.au>
+
+ * data-in.c data-out.c dictionary.c sys-file-reader.c
+ sys-file-writer.c variable.c variable.h: Reworked very long string
+ support for better encapsulation.
+
+Sat May 6 19:02:00 2006 Ben Pfaff <blp@gnu.org>
+
+ * value-labels.c (val_labs_can_set_width): New function.
+ (val_labs_set_width) Clear labels if increasing width to long
+ string.
+ (val_labs_destroy) Remove unneeded test for null.
+
+Sat May 6 16:14:08 2006 Ben Pfaff <blp@gnu.org>
+
+ * value-labels.h: Remove unneeded dependency on variable.h.
+
+Sat May 6 15:58:36 2006 Ben Pfaff <blp@gnu.org>
+
+ Get rid of `char *c' member in union value, for cleanliness.
+
+ * value.h: (union value) Remove `c' member.
+
+Sat May 6 15:36:59 2006 Ben Pfaff <blp@gnu.org>
+
+ Make dictionary compacting functions a little more general.
+
+ * sys-file-writer.c (sfm_open_writer): Use
+ dict_compacting_would_change().
+ (does_dict_need_translation) Removed.
+
+Sat May 6 15:35:42 2006 Ben Pfaff <blp@gnu.org>
+
+ Make dictionary compacting functions a little more general.
+
+ * dictionary.c (dict_needs_compaction): Rename
+ dict_compacting_would_shrink(). Update all callers.
+ (dict_compacting_would_change) New function.
+
+Sat May 6 14:25:49 2006 Ben Pfaff <blp@gnu.org>
+
+ * sys-file-writer.c: (does_dict_need_translation) Fix bug:
+ inverted return value (!).
+
+Sat May 6 13:37:52 2006 Ben Pfaff <blp@gnu.org>
+
+ Continue reforming procedure execution.
+
+ * procedure.c: Search and replace "vfm" by "proc". Notably:
+ (static var vfm_source) Rename proc_source. Update all
+ references.
+ (static var vfm_sink) Rename proc_sink. Update all references.
+
Sat May 6 12:38:55 2006 Ben Pfaff <blp@gnu.org>
Continue reforming procedure execution. In this phase, remove