+Mon May 31 20:45:24 2004 Ben Pfaff <blp@gnu.org>
+
+ Fix memory leaks.
+
+ * data-list.c: (cmd_data_list) Free dls->delims on lossage.
+ (data_list_trns_free) Free dls->delims.
+
+ * t-test.q: (tts_custom_pairs) Free vars.
+ (ssbox_one_sample_init) Fix tab_vline() argument.
+ (ssbox_independent_samples_init) Ditto.
+ (trbox_paired_init) Ditto.
+ (trbox_one_sample_init) Ditto.
+
+Mon May 31 17:19:27 2004 Ben Pfaff <blp@gnu.org>
+
+ Generalize casefiles to the extent that we can use them for
+ sorting and other kinds of data transformations. Change cases to
+ be copy-on-write to improve memory efficiency in common cases.
+ Every access to a member of a `struct ccase' was changed to be a
+ call to a case_*() function, especially case_data(), case_num(),
+ case_str(), or case_data_rw(). Many instances of a local variable
+ named "case_num" were changed to "case_idx" as a consequence.
+ Many `struct ccase *' were changed to actual `struct ccase'
+ because of copying semantics of cases. In several places there
+ was a choice between updating debug code to work with the new ADTs
+ or just deleting it because it was useless; I chose to delete it.
+
+ * Makefile.am: (pspp_SOURCES) Add case.c, case.h.
+
+ * case.c: New file.
+
+ * case.h: New file.
+
+ * aggregate.c: (struct agr_proc) Change type of `sort' to
+ sort_criteria *. Add `break_vars', `break_var_cnt' members.
+ Rename `vars' to `agr_vars', all references updated. Change
+ `agr_case' to type `struct ccase'.
+ (cmd_aggregate) Deal with new members. Use case_create(),
+ sort_active_file_in_place(), sort_active_file_to_casefile().
+ (agr_destroy) Deal with new members.
+ (aggregate_single_case) Ditto.
+ (dump_aggregate_info) Ditto.
+ (initialize_aggregate_info) Ditto.
+ (agr_to_active_file) Ditto.
+ (presorted_agr_to_sysfile) Ditto.
+ (sort_agr_to_sysfile) Removed.
+
+ * alloc.c: (out_of_memory) Make non-static.
+
+ * alloc.h: Prototype out_of_memory().
+
+ * casefile.c: Switched from a linked list in-memory representation
+ to a two-level array-style representation. The linked list was
+ appropriate when we could stick a header onto cases, but that's no
+ longer the case. Also, the two-level array will allow for random
+ in-memory access in case that's ever wanted. Also added the
+ concept of a `destructive casereader', one that destroys cases in
+ the underlying casefile as they are read out.
+ (macro CASES_PER_BLOCK) New macro.
+ (struct casefile) New members `value_cnt', `case_list_size',
+ `case_acct_size', `being_destroyed', `cases'. Removed `head',
+ `tail'.
+ (struct casereader) Removed `cur'. Added `destructive', `c'.
+ (global var casefiles) Made static.
+ (static var case_bytes) New var.
+ (casefile_create) Takes a value count, not a case size in bytes,
+ to conform to the case interface. All callers updated. Deal with
+ new and removed members.
+ (casefile_destroy) Deal with new and removed members.
+ (casefile_sleep) New function.
+ (casefile_get_case_size) Removed.
+ (casefile_get_value_cnt) New function.
+ (casefile_append) Rewritten to deal with new and removed members.
+ (casefile_append_xfer) New function.
+ (write_case_to_disk) Use case_serialize().
+ (call_posix_fadvise) Removed because posix_fadvise64 segfaults.
+ Couldn't figure out why.
+ (casefile_to_disk) Don't call call_posix_fadvise. Rewritten to
+ deal with new and removed members.
+ (merge) Removed.
+ (merge_sort) Removed.
+ (casefile_sort) Removed.
+ (casefile_get_reader) Deal with new and removed members.
+ (casefile_get_destructive_reader) New function.
+ (reader_open_file) Make code more readable. Create case for
+ reader.
+ (casereader_get_casefile) New function.
+ (casereader_read) Deal with new and removed members. Now returns
+ a copy of the case, so that the caller is responsible for
+ destroying the returned case.
+ (casereader_read_xfer) New function.
+ (casereader_destroy) Destroy reader's case.
+ (test_casefile) Second arg is now a value count, all callers
+ updated. Now tests destructive readers too.
+ (get_random_case) Deal with new case ADT.
+ (write_random_case) Ditto.
+ (read_and_verify_random_case) Ditto.
+
+ * crosstabs.q: Remove debug code.
+
+ * descript.q: (calc_descriptives) Deal with new case, casefile
+ ADTs.
+
+ * dfm.c: (cmd_begin_data) There's no storage_source_class anymore.
+
+ * do-if.c: Remove unneeded header inclusion.
+
+ * expr-prs.c: Remove debug code.
+
+ * exprP.h: Remove debug code.
+
+ * flip.c: (flip_file) Use fseeko() if available.
+
+ * formats.c: Remove debug code.
+
+ * get.c: Remove debug code.
+ (struct mtf_file) Change `input' from `union value *' to `struct
+ ccase', all references updated.
+
+ * levene.c: (levene) Deal with new case, casefile ADTs.
+
+ * list.q: Remove debug code.
+
+ * loop.c: Remove debug code.
+
+ * matrix-data.c: Remove debug code.
+
+ * means.q: Remove debug code.
+
+ * mis-val.c: Remove debug code.
+
+ * pfm-read.c: Remove debug code.
+ (pfm_read_code) Change second arg from `union value *' to `struct
+ ccase *', all references updated.
+
+ * recode.c: (string_to_long) Make first arg const.
+ (convert_to_double) Ditto.
+
+ * repeat.c: Remove debug code.
+
+ * sample.c: Remove debug code.
+
+ * sfm-read.c: Remove debug code.
+ (sfm_read_case) Change second arg from `union value *' to `struct
+ ccase *'.
+
+ * sort.c: Redone in terms of casefiles.
+ (enum sort_direction) Moved here from sort.h.
+ (struct sort_criterion) New structure.
+ (struct sort_criteria) New structure.
+ (cmd_sort_cases) Rewritten.
+ (prepare_to_sort_active_file) New function.
+ (sort_active_file_in_place) New function.
+ (sort_active_file_to_casefile) New function.
+ (parse_sort) Renamed sort_parse_criteria(), rewritten & interface
+ changed, all callers updated.
+ (destroy_sort_cases_pgm) Renamed sort_destroy_criteria(),
+ rewritten & interface changed, all callers updated.
+ (sort_cases) Renamed sort_execute(), rewritten & interface
+ changed, all callers updated.
+ (struct internal_sort) Removed.
+ (do_internal_sort) Rewritten, interface changed.
+ (destroy_internal_sort) Removed.
+ (compare_case_dblptrs) Use sort_criteria instead of sort_case_pgm.
+ (struct initial_run) Removed; an initial run is now just a
+ casefile.
+ (compare_initial_runs) Rewritten.
+ (struct external_sort) Changed almost completely.
+ (do_external_sort) Rewritten, interface changed.
+ (destroy_external_sort) Rewritten.
+ [HAVE_MKDTEMP] (make_temp_dir) Removed.
+ [!HAVE_MKDTEMP] (do_mkdir) Removed.
+ [!HAVE_MKDTEMP] (make_temp_dir) Removed.
+ (init_external_sort) Removed.
+ (simulate_error) Removed.
+ (rmdir_temp_dir) Removed.
+ (get_temp_file_name) Removed.
+ (open_temp_file) Removed.
+ (close_temp_file) Removed.
+ (remove_temp_file) Removed.
+ (write_temp_file) Removed.
+ (read_temp_file) Removed.
+ (struct record_run) Change `record' from `struct case_lit *' to
+ `struct ccase'.
+ (struct initial_run_state) Remove `idx_to_fv', `free_list',
+ `file_idx', `output_file'. Add `run', casefile'. Change
+ `last_output' from `struct case_list *' to `struct ccase'.
+ (write_initial_runs) Change interface, rewrite.
+ (sort_sink_write) Renamed process_case(), changed interfaced,
+ rewrote.
+ (destroy_initial_run_state) Rewritten.
+ (allocate_cases) Rewritten.
+ (compare_record) Interface changed, rewritten.
+ (start_run) Rewritten.
+ (end_run) Rewritten.
+ (output_record) Rewritten.
+ (grab_case) Removed.
+ (release_case) Removed.
+ (struct merge_case) Change `cases' from double pointer to single
+ pointer.
+ (merge) Deal with new case and casefile ADTs.
+ (struct run) Removed.
+ (merge_once) Rewritten, interface changed.
+ (fill_run_buffer) Removed.
+ (sort_sink_make_source) Removed.
+ (sort_sink_class) Removed.
+ (struct sort_source_aux) Removed.
+ (sort_source_read_helper) Removed.
+ (sort_source_read) Removed.
+ (read_sort_output) Removed.
+ (read_internal_sort_output) Removed.
+ (read_external_sort_output) Removed.
+ (sort_source_destroy) Removed.
+ (sort_source_class) Removed.
+
+ * sort.h: (struct sort_cases_pgm) Removed.
+ (enum sort_direction) Moved to sort.c.
+
+ * t-test.q: (calculate) Deal with new case, casefile ADTs.
+
+ * tab.c: Remove debug code.
+
+ * var-labs.c: Remove debug code.
+
+ * var.h: (struct ccase) Removed.
+ (struct case_list) Removed.
+
+ * vars-atr.c: (discard_variables) Use free_case_source().
+
+ * vars-prs.c: (parse_vs_variable) Make arg const.
+ (parse_dict_variable) Ditto.
+ (parse_variables) Make struct dictionary * arg const.
+ (parse_var_set_vars) Make struct var_set * arg const.
+ (struct var_set) Add const to some of the function pointers' args.
+ (var_set_get_cnt) Make arg const.
+ (var_set_get_var) Make first arg const.
+ (var_set_lookup_var) Make first arg const.
+ (dict_var_set_get_cnt) Make arg const.
+ (dict_var_set_get_var) Make first arg const.
+ (dict_var_set_lookup_var) Make first arg const.
+ (var_set_create_from_dict) Make arg const. Add cast to aux
+ assignment.
+ (struct array_var_set) Add const to var member.
+ (array_var_set_get_cnt) Make arg const.
+ (array_var_set_get_var) Make first arg const.
+ (array_var_set_lookup_var) Make first arg const.
+ (var_set_create_from_array) Make first arg const. Insert cast.
+
+ * vfm.c: (struct write_case_data) Change trns_case, sink_case
+ members from `struct ccase *' to `struct ccase'.
+ (static var lag_queue) Change from double to single pointer.
+ (procedure) Optimize trivial case.
+ (internal_procedure) Deal with changed case, case_source ADTs.
+ (create_trns_case) Changed interface, rewrote.
+ (open_active_interface) Initialize modified lag queue.
+ (write_case) Deal with changed case ADT.
+ (lag_case) Deal with modified lag queue.
+ (close_active_file) Destroy modified lag queue.
+ Deal with changed case_source, case_sink ADTs.
+ (destroy_storage_stream_info) Make null arg into no-op.
+ (storage_sink_make_source) Set aux in created source.
+ (storage_source_read) Deal with changed case, casefile ADTs.
+ (storage_source_create) New function.
+ (lagged_case) Rewrite.
+ (free_case_source) New function.
+ (free_case_sink) Rewrite.
+ (struct split_aux_data) Changed prev_case from `struct ccase *' to
+ `struct ccase'.
+ (procedure_with_splits) Deal with changed prev_case.
+ (procedure_with_splits_callback) Ditto.
+ (multipass_split_aux_data) Changed prev_case from `struct ccase *' to
+ `struct ccase'.
+ (multipass_procedure_with_splits) Deal with changed prev_case.
+ (multipass_split_callback) Ditto.
+
+
+Mon May 31 17:19:06 2004 Ben Pfaff <blp@gnu.org>
+
+ The workspace idea didn't work out.
+
+ * Makefile.am: (pspp_SOURCES) Remove workspace.c, workspace.h.
+
+ * workspace.c: Removed.
+
+ * workspace.h: Removed.
+
+Sun May 30 18:35:19 2004 Ben Pfaff <blp@gnu.org>
+
+ Fully implement arbitrary delimiters on DATA LIST, extending the
+ half implementation that was already there.
+
+ * data-list.c: (struct data_list_pgm) Remove `delim', add
+ `delims', `delim_cnt'.
+ (cmd_data_list) Initialize new members. Parse delimiters and
+ clean up code a bit.
+ (cut_field) Extract fields with arbitrary delimiters. Also, fix
+ handling of leading commas.
+ (read_from_data_list_fixed) Expand tabs. Adapt to new DFM
+ interfaces.
+ (read_from_data_list_free) Adapt to new DFM interfaces.
+ (read_from_data_list_list) Ditto.
+ (repeating_data_trns_proc) Ditto.
+
+ * dfm.c: Split up reader and writer into separate code, because
+ they do different things. Use struct string instead of explicit
+ allocation code, for clarity.
+ (enum dfm_reader_flags) New enum.
+ (struct dfm_fhuser_ext) Removed.
+ (struct dfm_reader_ext) New.
+ (get_reader) New function, used by just about all the reader
+ functions.
+ (dfm_close) Removed.
+ (close_reader) New function.
+ (dfm_open_for_reading) Rewrite initialization of dfm_fhuser_ext.
+ (dfm_open_for_writing) Ditto.
+ (macro force_line_buffer_expansion) Removed.
+ (count_tabs) Removed.
+ (tabs_to_spaces) Removed.
+ (read_record) Deal with new dfm_reader_ext. Use struct string
+ functions. Don't convert tabs to spaces.
+ (dfm_eof) New function.
+ (dfm_get_record) Changed interface, rewrote.
+ (dfm_expand_tabs) New function.
+ (dfm_fwd_record) Renamed dfm_forward_record(), updated to new
+ dfm_reader_ext, rewritten.
+ (dfm_bkwd_record) Renamed dfm_reread_record(), updated to new
+ dfm_reader_ext, rewritten.
+ (dfm_set_record) Removed in favor of dfm_forward_columns().
+ (dfm_forward_columns) New function.
+ (dfm_get_cur_col) Renamed dfm_column_start, updated to new
+ dfm_reader_ext, rewritten.
+ (static var dfm_r_class) Use close_reader for the destructor.
+ (struct dfm_writer_ext) New.
+ (dfm_put_record) Updated to new dfm_writer_ext, rewritten. Uses
+ bounce buffer now instead of local allocation.
+ (close_writer) New function.
+ (static var dfm_writer_ext) Use close_writer for destructor.
+ (cmd_begin_data) Adapt to new dfm_reader_ext.
+
+ * file-handle.q: Add support for per-file tab width.
+ (struct private_file_handle) Add tab_width member.
+ (q2c specifications) Add tabwidth subcommand.
+ (cmd_file_handle) Put parsed tab width into private_file_handle.
+ (create_file_handle) Set default tab width.
+ (handle_get_tab_width) New function.
+
+ * file-type.c: (file_type_source_read) Adapt to new DFM interface.
+
+ * inpt-pgm.c: (reread_trns_proc) Ditto.
+
+ * matrix-data.c: (context) Ditto.
+ (another_token) Ditto.
+ (mget_token) Ditto.
+ (force_eol) Ditto.
+
+Sun May 30 18:33:59 2004 Ben Pfaff <blp@gnu.org>
+
+ * casefile.c: (casefile_destroy) Fix memory leak by freeing
+ cf->filename.
+ (casereader_destroy) Don't close file descriptor -1.
+
+ * recode.c: (cmd_recode) Fix memory leak.
+
+ * set.q: (q2c specifications) Fix typo in user message.
+
+ * str.c: (st_bare_pad_len_copy) Change memcpy to memmove to avoid
+ undefined behavior for overlapping arguments.
+
+Sun May 30 18:31:48 2004 Ben Pfaff <blp@gnu.org>
+
+ * casefile.c: valgrind doesn't implement posix_fadvise() yet, so
+ don't call it when we're running under valgrind.
+ (call_posix_fadvise) New function.
+ (casefile_to_disk) Use call_posix_fadvise().
+ (reader_open_file) Ditto.
+
+Sun May 30 18:20:12 2004 Ben Pfaff <blp@gnu.org>
+
+ Update our string ADTs, struct string and struct len_string. Get
+ rid of pool support, which was largely unused. Rename lots of
+ functions to have more obvious or consistent names.
+
+ * ascii.c: Get rid of ascii_pool. It was only used for string
+ allocations.
+ (ascii_open_global) Don't create ascii_pool.
+ (ascii_close_driver) Don't destroy ascii_pool.
+ (ascii_postopen_driver) Don't use pool.
+ (ascii_close_driver) Destroy strings manually.
+
+ * str.c: (ds_create) Remove pool argument, all references updated.
+ (ds_init) Ditto.
+ (ds_replace) Remove pool support, make more efficient when we
+ don't need to reallocate.
+ (ds_destroy) Remove pool support.
+ (ds_rpad) New function.
+ (ds_size) Renamed ds_capacity(), all references updated.
+ (ds_value) Renamed ds_c_str(), all references updated.
+ (ds_concat) Renamed ds_puts(), all references updated.
+ (ds_concat_buffer) Renamed ds_concat(), all references updated.
+ (ds_putchar) Renamed ds_putc(), all references updated.
+ (ds_getline) Renamed ds_gets(), all references updated.
+ (ls_create) Remove pool argument, all references updated.
+ (ls_create_buffer) Ditto.
+ (ls_destroy) Removed pool support.
+ (ls_value) Renamed ls_c_str(), all references updated.
+
+ * str.h: (ls_length) [__GNUC__] Add inline version.
+ (ls_c_str) [__GNUC__] Add inline version.
+ (ls_end) [__GNUC__] Add inline version.
+ (struct string) Remove pool member. Rename `size' to `capacity',
+ all references updated.
+
+ * tab.c: (text_format) Instead of using pool argument to
+ ls_create_buffer(), call pool_register() on allocated data.
+
+Mon Apr 26 22:40:07 2004 Ben Pfaff <blp@gnu.org>
+
+ We're abusing the current ASCII driver by telling it to allocate a
+ 9999-line, 9999-character page in the tests. This causes some
+ systems to curl up and die because it allocates 20 MB of
+ contiguous RAM. This change alleviates at least part of the
+ problem. It is mostly a stop-gap until the new output system is
+ ready.
+
+ * ascii.c: (struct line) New structure.
+ (struct ascii_driver_ext) Remove `page', `page_size', `line_len',
+ `line_len_size', `n_output' members. Add `lines', `lines_cap'.
+ (ascii_preopen_driver) Initialize new members, not old ones.
+ (ascii_close_driver) Destroy new members, not old ones.
+ (ascii_open_page) Allocate new members, not old ones.
+ (expand_line) Allocate room in line.
+ (draw_line) Use new members.
+ (ascii_line_horz) Ditto.
+ (ascii_line_vert) Ditto.
+ (ascii_line_intersection) Ditto.
+ (text_draw) Ditto.
+ (output_lines) Ditto.
+ (ascii_close_page) Ditto.
+
+Sun Apr 25 23:40:15 2004 Ben Pfaff <blp@gnu.org>
+
+ * matrix.c: Dead code. Removed.
+
+ * matrix.h: Dead code. Removed.
+
+Fri Apr 16 23:59:51 2004 Ben Pfaff <blp@gnu.org>
+
+ Contrary to what I'd always understood, there is an efficient
+ algorithm for deletion from a hash table populated via linear
+ probing. This change implements it.
+
+ * hash.c: (hsh_rehash) Probe in increasing order.
+ (hsh_probe) Ditto.
+ (locate_matching_entry) Ditto.
+ (hsh_delete) Use Knuth's Algorithm 6.4R for deletion.
+
+Tue Apr 13 19:24:15 2004 Ben Pfaff <blp@gnu.org>
+
+ * moments.c (calc_moments): Adjust calculation of kurtosis to
+ avoid subtracting huge numbers from huge numbers, on Michael
+ Kiefte's advice.
+
+Sun Apr 11 14:22:12 2004 Ben Pfaff <blp@gnu.org>
+
+ Rework moments routines for improved numerical stability based on
+ Michael Kiefte's advice. Any bugs or remaining numerical problems
+ are still mine though.
+
+ There is now a struct moments1 for use with one-pass moments. It
+ uses a provisional means algorithm as an attempt to improve
+ accuracy of higher moments. The older struct moments now only
+ handles two-pass moments.
+
+ * aggregate.c: Use moments1 instead moments.
+
+ * descript.c: Revert previous change, which is no longer needed
+ due to the moments revision.
+
+ * moments.c: (calc_moments) New function for calculating variance,
+ skewness, kurtosis.
+ (moments_pass_one) Only accumulate weights bigger than zero.
+ (moments_calculate) Allow calculating the mean on pass one, others
+ require pass two. Implement in terms of calc_moments().
+ (struct moments1) New structure.
+ (init_moments1) New function.
+ (moments1_clear) Ditto.
+ (moments1_create) Ditto.
+ (moments1_add) Ditto.
+ (moments1_calculate) Ditto.
+ (moments1_destroy) Ditto.
+ (cmd_debug_moments) Deal with `struct moments' or `struct
+ moments1' as requested by user.
+
+Sun Apr 11 14:21:55 2004 Ben Pfaff <blp@gnu.org>
+
+ * Makefile.am (pspp_SOURCES): Remove debug.c.
+
+ * debug.c: Removed. It was empty anyway.
+
Fri Apr 9 20:04:49 2004 Ben Pfaff <blp@gnu.org>
* descript.c (calc_descriptives): Fix assert failure when only
projects / pspp-builds.git / blobdiff