Ben Pfaff [Sun, 13 Mar 2022 19:34:36 +0000 (12:34 -0700)]
Switch from recursive to non-recursive use of Gnulib.
This allows for better use of dependencies in Makefiles. It causes a new
warning from Automake, which I hope to fix through a change to Gnulib
sometime soon. I'm consulting the other Gnulib maintainers on that:
https://lists.gnu.org/archive/html/bug-gnulib/2022-03/msg00044.html
Ben Pfaff [Sun, 13 Mar 2022 19:12:53 +0000 (12:12 -0700)]
Gnulib needs Automake 1.14 or later.
Ben Pfaff [Sun, 13 Mar 2022 18:41:50 +0000 (11:41 -0700)]
tests: Update testsuite results to accomodate new Gnulib.
It seems that libunistring now does line wrapping slightly differently,
perhaps due to Unicode updates.
Ben Pfaff [Sun, 13 Mar 2022 18:32:58 +0000 (11:32 -0700)]
Update to newest gnulib.
This required a dependency on Automake 1.11 or later. It also required
dropping C2x compatibility warnings because Gnulib now triggers them. The
Gnulib tempname module requires linking against $(LIB_GETRANDOM), so this
adds that in each case where the build links against libgl.
Ben Pfaff [Sun, 13 Mar 2022 18:30:49 +0000 (11:30 -0700)]
spv-light-decoder: Avoid GCC 11.x warning in decode_spvlb_halign().
It seems to be wrong, but GCC can't tell and it's hard to blame it.
Ben Pfaff [Sun, 13 Mar 2022 17:55:30 +0000 (10:55 -0700)]
expressions: Fix value of $TIME variable and add a test.
Thanks to Frans Houweling for reporting this bug.
Bug #62003.
Ben Pfaff [Thu, 13 Jan 2022 05:50:54 +0000 (21:50 -0800)]
psppire-dialog-action-crosstabs: Do not add PIVOT or NOPIVOT to command.
The CROSSTABS command does not support /FORMAT=PIVOT or /FORMAT=NOPIVOT.
I can't find any evidence that it ever did. Perhaps q2c ignored keywords
it didn't recognize? SPSS does not document support for any such format
keywords, so it's better to remove them from the emitted syntax than to
add support to CROSSTABS.
Thanks to Alexandra Younger for reporting the issue.
Ben Pfaff [Sat, 1 Jan 2022 19:10:05 +0000 (11:10 -0800)]
lexer: New functions for parsing real numbers in specified ranges.
These will be more useful in the future.
Ben Pfaff [Sat, 1 Jan 2022 19:09:49 +0000 (11:09 -0800)]
format-parser: Improve comment.
Ben Pfaff [Sat, 1 Jan 2022 19:01:00 +0000 (11:01 -0800)]
FREQUENCIES: Fix percentiles and median calculation for multiple variables.
The calculations here were using the same space each time for
calculating percentiles and median, but different areas are needed. This
commit fixes the problem.
Bug #60728.
Thanks to elias tsolis for reporting the problem.
Ben Pfaff [Sat, 1 Jan 2022 18:35:17 +0000 (10:35 -0800)]
FREQUENCIES: Remove unused pool.
Ben Pfaff [Sat, 1 Jan 2022 18:34:56 +0000 (10:34 -0800)]
FREQUENCIES: Remove unused struct members.
John Darrington [Sat, 1 Jan 2022 08:56:21 +0000 (09:56 +0100)]
Fixed misleading results in the Kruskal-Wallis test
Fixed misleading results if the lower bound was in fact higher
than the upper bound.
Ben Pfaff [Fri, 31 Dec 2021 19:06:48 +0000 (11:06 -0800)]
Replace more uses of 'cnt' by 'n'.
This continues the work begun in commit
339f1956cc72 ("treewide: Replace
<name>_cnt by n_<name>s and <name>_cap by allocated_<name>.") by replacing
more uses of 'cnt' by 'n', which I think is a better choice these days.
Friedrich Beckmann [Fri, 31 Dec 2021 17:40:43 +0000 (18:40 +0100)]
windows cross build: Add example build instructions in documentation
I added a description of the required steps to do a windows cross
build.
Ben Pfaff [Fri, 31 Dec 2021 05:35:13 +0000 (21:35 -0800)]
moments: Spell out "variance" ("var" is easily confused with "variable").
Ben Pfaff [Thu, 30 Dec 2021 21:58:12 +0000 (13:58 -0800)]
Change how checking for missing values works.
It's more flexible to have the functions that check for missing values
return what kind of missing value was found (or 0 if none), than to have
to pass in the kind of missing values to check for.
Ben Pfaff [Sun, 19 Dec 2021 23:41:50 +0000 (15:41 -0800)]
expressions: Major work to improve error messages.
This makes the error messages related to expressions much better, by being
more precise about the location of the problem. It also fixes a number of
documentation and compatibility issues encountered along the way, as well
as redoing the testsuite for expressions to make it easier to understand
and to work on.
Ben Pfaff [Sat, 25 Dec 2021 22:12:39 +0000 (14:12 -0800)]
expressions: Fix built sources.
The dependencies were incomplete for the built sources, which meant that
build errors often came up if one built particular targets, e.g. for
"make src/ui/terminal/pspp". This fixes the problem.
Ben Pfaff [Fri, 17 Dec 2021 02:09:14 +0000 (18:09 -0800)]
expressions: Simplify type declarations for nodes.
The widespread availability of C99 anonymous structs and unions helps to
simplify the node structure.
This will make it a lot easier to add a new member to the expression nodes
for source locators in an upcoming commit.
Ben Pfaff [Thu, 16 Dec 2021 05:00:32 +0000 (21:00 -0800)]
Convert all Perl build tools to Python and remove Perl build dependency.
Python is much more commonly used these days and easier for beginners
to pick up.
Ben Pfaff [Thu, 16 Dec 2021 22:08:06 +0000 (14:08 -0800)]
num-out-cmp.pl: Remove unused script.
Ben Pfaff [Thu, 16 Dec 2021 04:20:48 +0000 (20:20 -0800)]
MATRIX: Update EVAL test to avoid failure on i386.
The i386 buildbot produced different precision for the eigenvalues
(integers rather than almost-integers), which isn't a significant
difference. This commit should avoid the problem.
Ben Pfaff [Thu, 16 Dec 2021 03:22:46 +0000 (19:22 -0800)]
matrix: Avoid gsl_linalg_LU_invx() because it was new in GSL 2.6.
Debian only added GSL 2.6 in the current stable (bullseye) but one of the
CI systems is using oldstable (buster), thus failures.
Thanks to Friedrich Beckmann for reporting the problem.
Ben Pfaff [Thu, 16 Dec 2021 02:56:45 +0000 (18:56 -0800)]
perl-module: Skip memory test if Address Sanitizer is in use.
Otherwise, Address Sanitizer produces unexpected memory behavior and
causes this test to fail even though there is no real leak.
Ben Pfaff [Thu, 16 Dec 2021 02:55:44 +0000 (18:55 -0800)]
perl-module: Fix build errors and ignore warnings.
This allows the module to build again, for me.
Ben Pfaff [Thu, 16 Dec 2021 02:54:47 +0000 (18:54 -0800)]
configure: Check for Perl modules needed to build PSPP's Perl module.
Fedora doesn't have Config::Perl::V by default and its absence caused
mysterious build failures.
Ben Pfaff [Tue, 14 Dec 2021 05:06:23 +0000 (21:06 -0800)]
expressions: Rewrite code generator in Python.
This doesn't change the code that's generated at all. It's preparation
for doing that later.
Ben Pfaff [Thu, 16 Dec 2021 01:11:32 +0000 (17:11 -0800)]
Fix build requirements by correctly noting that Python is required.
Building PSPP has required Python since support for the SPV format was
added, because the SPV parser-generators use Python.
Only Python 3+ can reasonably be supported, because support for Python 2.x
ended in April 2020.
Ben Pfaff [Fri, 10 Dec 2021 06:23:59 +0000 (22:23 -0800)]
Clean up how transformations work.
Transformations had a messy system that included "finalizers" that needed
to be called before a chain could be executed and the need to keep track
of relative offsets for control flow. This commit reworks and simplifies
the whole thing.
Also adds some INPUT PROGRAM, LOOP, and DO IF tests.
Ben Pfaff [Thu, 9 Dec 2021 23:03:45 +0000 (15:03 -0800)]
treewide: Replace <name>_cnt by n_<name>s and <name>_cap by allocated_<name>.
I used to think that _cnt was a good suffix for a count representing the
number of items in an array, but I haven't used that convention in a long
time and it looks wrong now. This changes these to an n_ prefix instead.
Similarly, but in far fewer cases, replace <name>_cap for the allocated
capacity of an array by allocated_<name>.
This doesn't change the name of ref_cnt members used as reference counts.
Those are pure counts, not the length of anything.
This doesn't change the Perl library because that's an external API.
Ben Pfaff [Mon, 27 Sep 2021 05:35:44 +0000 (22:35 -0700)]
Implement the MATRIX command.
Ben Pfaff [Mon, 6 Dec 2021 06:03:31 +0000 (22:03 -0800)]
message: Introduce underlining for error message regions.
This will allow for the upcoming matrix language implementation
to give better error messages that clearly show the erroneous
part of the source.
Ben Pfaff [Thu, 25 Nov 2021 01:00:57 +0000 (17:00 -0800)]
message: Intern file names in msg_location to make them cheaper to copy.
Ben Pfaff [Tue, 23 Nov 2021 03:16:48 +0000 (19:16 -0800)]
lexer: Allow for negative lookahead within a command.
This allows to getting the location of prior tokens within a command,
which makes it easier to find the end of a group of tokens that was just
parsed by referring to the token at offset -1.
Ben Pfaff [Mon, 22 Nov 2021 20:46:19 +0000 (12:46 -0800)]
lexer: Drop 'line_pos' from lex_token and lex_source.
It can now be calculated by reading backward in the source buffer.
Ben Pfaff [Mon, 22 Nov 2021 19:45:48 +0000 (11:45 -0800)]
lexer: Keep entire source file in memory.
Previously, the lexer tried to keep only part of each source file in
memory, the part that wasn't fully parsed yet. With this commit, the
lexer holds the entire file in memory, even parts that are no longer
needed. This should make it easier to produce better error messages.
Ben Pfaff [Mon, 6 Dec 2021 05:21:19 +0000 (21:21 -0800)]
pivot-table: New function pivot_value_new_variable__().
Ben Pfaff [Mon, 6 Dec 2021 05:20:53 +0000 (21:20 -0800)]
u8-line: Add definition of an initializer.
Ben Pfaff [Mon, 6 Dec 2021 05:20:42 +0000 (21:20 -0800)]
string-array: New functions for comparing string arrays.
Ben Pfaff [Mon, 6 Dec 2021 05:20:00 +0000 (21:20 -0800)]
variable-parser: New functions for parsing syntax without a dictionary.
This will acquire its first user in an upcoming commit.
Ben Pfaff [Mon, 6 Dec 2021 05:19:12 +0000 (21:19 -0800)]
expressions: Simplify function name parsing.
Ben Pfaff [Mon, 6 Dec 2021 05:18:22 +0000 (21:18 -0800)]
distributions: New module for probability distribution functions.
These functions are currently just used in expressions, but in an upcoming
commit they will also be used in matrices, so this commit makes them more
widely available.
Ben Pfaff [Mon, 6 Dec 2021 05:15:02 +0000 (21:15 -0800)]
data-writer: New function dfm_put_record_utf8().
This will have another user in the upcoming support for the matrix
language.
Ben Pfaff [Mon, 6 Dec 2021 05:14:08 +0000 (21:14 -0800)]
lexer: Add tokens for '{', '}', ':', ';' for use in the matrix language.
Ben Pfaff [Mon, 6 Dec 2021 05:13:00 +0000 (21:13 -0800)]
file-handle-def: New function fh_equal().
This will have its first user in an upcoming commit.
Ben Pfaff [Mon, 6 Dec 2021 04:02:37 +0000 (20:02 -0800)]
lexer: Factor out functions for counting columns.
These will have additional upcoming users.
Ben Pfaff [Mon, 6 Dec 2021 04:01:01 +0000 (20:01 -0800)]
lexer: New lex_at_phrase(), lex_get_n() functions.
These will have their first users in upcoming commits.
Ben Pfaff [Mon, 6 Dec 2021 17:02:55 +0000 (09:02 -0800)]
lexer: Issue error message in forgotten case in lex_force_int_range().
Found by inspection.
Ben Pfaff [Mon, 6 Dec 2021 17:02:30 +0000 (09:02 -0800)]
lexer: Be consistent across 32/64 bit in lex_force_int_range().
The exact error message that this function reported varied between 32-
and 64-bit platforms for invalid integers that require more than 32 bits
but no more than 64 bits, since "long int" has a different range between
those platforms. This commit fixes the problem.
This issue caused the test "testing lexer crash due to overflow" to fail
on 32-bit platforms.
Thanks to Friedrich Beckmann for reporting the problem.
Ben Pfaff [Mon, 6 Dec 2021 05:11:40 +0000 (21:11 -0800)]
doc: Typo fixes, wording and formatting improvements.
Ben Pfaff [Tue, 30 Nov 2021 14:54:48 +0000 (06:54 -0800)]
segment: Add comment about zero-length segments.
Ben Pfaff [Sat, 6 Nov 2021 21:15:29 +0000 (14:15 -0700)]
format: Make fmt_check() easier to translate.
Ben Pfaff [Mon, 6 Dec 2021 05:12:17 +0000 (21:12 -0800)]
any-writer: Add comment.
Ben Pfaff [Mon, 6 Dec 2021 05:16:29 +0000 (21:16 -0800)]
driver: New function output_log_nocopy().
Ben Pfaff [Mon, 6 Dec 2021 05:21:41 +0000 (21:21 -0800)]
MATRIX DATA: Add test for factors and splits together.
Ben Pfaff [Mon, 6 Dec 2021 05:18:42 +0000 (21:18 -0800)]
expressions: Fix definitions of IDF.T1G and IDF.T2G.
Ben Pfaff [Sat, 9 Oct 2021 17:20:28 +0000 (10:20 -0700)]
dataset: Fix memory leak destroying a dataset that has a permanent_dict.
Found by Address Sanitizer.
Ben Pfaff [Sat, 9 Oct 2021 17:13:34 +0000 (10:13 -0700)]
pivot-table-test: Fix memory leak when table is not displayed.
Found by Address Sanitizer.
Ben Pfaff [Sat, 9 Oct 2021 17:10:35 +0000 (10:10 -0700)]
lexer: Fix memory leak merging tokens only some of which come from macros.
Found by Address Sanitizer.
Ben Pfaff [Sat, 9 Oct 2021 16:41:01 +0000 (09:41 -0700)]
macro: Fix memory leaks in error cases parsing function arguments.
Found by Address Sanitizer.
Ben Pfaff [Sat, 9 Oct 2021 16:35:41 +0000 (09:35 -0700)]
lexer: Fix memory leak when macro expands as empty.
Found by Address Sanitizer.
Ben Pfaff [Sat, 9 Oct 2021 16:32:02 +0000 (09:32 -0700)]
macro: Fix memory leak expanding !DO loop over list.
Found by Address Sanitizer.
Ben Pfaff [Sat, 9 Oct 2021 16:23:52 +0000 (09:23 -0700)]
macro: Fix memory leak with keyword "enclose" arguments.
The memory for the argument was being allocated two places, which caused
the first-allocated block to be leaked.
Found by Address Sanitizer.
Ben Pfaff [Sat, 9 Oct 2021 16:13:40 +0000 (09:13 -0700)]
segment: Fix read past end of buffer when input ends in '-'.
Thanks to John Darrington for reporting this.
Found by Address Sanitizer.
Ben Pfaff [Wed, 6 Oct 2021 05:15:28 +0000 (22:15 -0700)]
expressions: Parse multiple sets of parentheses for grouping together.
Fuzzers are fond of driving expression parsers to failure by exhausting
the stack in trivial ways. This defeats the simplest attempts by
lining up thousands of left parentheses in a row.
I am a bit curious whether the fuzzer will now invent something more
sophisticated, such as nested function calls or non-empty expressions like
1+(1+(1+(1+(1+....
This fixes bug #61286.
Thanks to Irfan Ariq for reporting the bug.
Ben Pfaff [Wed, 6 Oct 2021 04:57:37 +0000 (21:57 -0700)]
DATA LIST: Fix assertion when RECORDS given twice with decreasing value.
Fixes bug #61285.
Thanks to Irfan Ariq for reporting this bug.
Ben Pfaff [Tue, 5 Oct 2021 16:19:26 +0000 (09:19 -0700)]
dictionary: Allow dict_set_documents() argument to reference old documents.
merge_dictionary() in combine-file.c includes the old documents in the new
ones by just copying pointers. dict_set_documents() didn't handle this
properly. This fixes the problem.
Fixes bug #61258.
Thanks to Irfan Ariq for reporting the problem.
Ben Pfaff [Tue, 5 Oct 2021 15:47:46 +0000 (08:47 -0700)]
encoding-guesser: Avoid reading uninitialized data for zero-length files.
Found while investigating bug #61254.
Thanks to Irfan Ariq for reporting this bug.
Ben Pfaff [Tue, 5 Oct 2021 06:14:32 +0000 (23:14 -0700)]
segment: Fix 1-byte read past initialized data when file ends in CR.
Fixes bug #61253.
Thanks to Irfan Ariq for reporting this bug.
John Darrington [Sun, 3 Oct 2021 06:03:44 +0000 (08:03 +0200)]
Remove some unnecessary gettext macro definitions
John Darrington [Sat, 2 Oct 2021 19:16:55 +0000 (21:16 +0200)]
Fix compiler warning
John Darrington [Sat, 2 Oct 2021 16:33:21 +0000 (18:33 +0200)]
Fix memory leak in tokenize_string_segment
John Darrington [Sat, 2 Oct 2021 14:51:37 +0000 (16:51 +0200)]
Fix memory leak in MCONVERT
John Darrington [Sat, 2 Oct 2021 14:35:42 +0000 (16:35 +0200)]
Fix memory leak upon failure to create matrix reader
John Darrington [Sat, 2 Oct 2021 14:31:14 +0000 (16:31 +0200)]
More ASAN_OPTIONS
John Darrington [Sat, 2 Oct 2021 13:32:40 +0000 (15:32 +0200)]
Use XCALLOC / XZALLOC macros where reasonable
John Darrington [Sat, 2 Oct 2021 04:48:57 +0000 (06:48 +0200)]
Fix possible segfault when running RANK with bad syntax
* src/language/stats/rank.c (): Initialise the vars member
Fixes bug #61257
John Darrington [Sat, 2 Oct 2021 04:47:31 +0000 (06:47 +0200)]
Replace numerous instances of xzalloc with XZALLOC
John Darrington [Sun, 26 Sep 2021 18:26:45 +0000 (20:26 +0200)]
Matrix readers - fix memory leaks
* src/language/data-io/matrix-reader.c (matrix_reader_destroy): free members cvar, svars and fvars
John Darrington [Sun, 26 Sep 2021 18:39:38 +0000 (20:39 +0200)]
Ignore sanitizer ODR warnings, during testing with -fsanitize=address
* tests/libpspp/sparse-xarray-test.at: Set detect_odr_violation=0
Ben Pfaff [Mon, 27 Sep 2021 05:35:33 +0000 (22:35 -0700)]
Implement the MCONVERT command.
Ben Pfaff [Sun, 26 Sep 2021 18:06:45 +0000 (11:06 -0700)]
lexer: Fix use-after-free error in lex_source_get_lookahead().
This code used local variable 'out' as if its value stayed the same from
one iteration of the loop to the next, but in fact its scope meant that
it became indeterminate on each new iteration. This commit fixes the
problem by moving its declaration to an outer scope.
Thanks to John Darrington for reporting the problem.
John Darrington [Sun, 26 Sep 2021 14:53:38 +0000 (16:53 +0200)]
Avoid numerical problems with missing weights on non-linear cases.
* src/math/order-stats.c (order_stats_accumulate_idx): Ignore cases
with missing weight values.
John Darrington [Sat, 25 Sep 2021 16:59:52 +0000 (18:59 +0200)]
Fix possible incorrect assertion when creating unique casereaders.
* src/data/casereader-translator.c (uniquify): Force dir to be an element of {0, 1, -1}
John Darrington [Sat, 25 Sep 2021 09:41:04 +0000 (11:41 +0200)]
Remove unused module src/math/extrema
* src/math/extrema.c: Delete
* src/math/extrema.h: Delete
* src/math/automake.mk: Remove entries for src/math/extrema.c and
src/math/extrema.h
John Darrington [Sat, 11 Sep 2021 05:39:29 +0000 (07:39 +0200)]
Options dialog: add checkbox for startup tips
Ben Pfaff [Mon, 6 Sep 2021 17:07:41 +0000 (10:07 -0700)]
sys-file-encoding: Automatically generate the .c file at build time.
Suggested by John Darrington.
John Darrington [Sun, 5 Sep 2021 11:16:33 +0000 (13:16 +0200)]
Windows/build-dependencies: Add -fstack-protector flags
For some reason this flag seems to be necessary with the latest
x86_64_w64-mingw32 toolchain.
John Darrington [Sun, 5 Sep 2021 11:01:25 +0000 (13:01 +0200)]
Windows/build-dependencies: Use correct logical OR operator
John Darrington [Sun, 5 Sep 2021 10:59:49 +0000 (12:59 +0200)]
Windows/build-dependencies: New flag --no-clean
John Darrington [Sat, 4 Sep 2021 07:36:55 +0000 (09:36 +0200)]
Add some missing #include directives
* src/language/stats/crosstabs.c: Add missing #include directive.
* src/libpspp/pool.c: Add missing #include directive.
Ben Pfaff [Fri, 3 Sep 2021 05:15:53 +0000 (22:15 -0700)]
MATRIX DATA: Fully implement.
This command had a partial implementation for correlation matrices that
left out some of the language features. This commit adds those features.
Ben Pfaff [Fri, 3 Sep 2021 02:59:23 +0000 (19:59 -0700)]
case: Introduce new functions for numbers and substrings in cases.
Use the case_num_*() functions everywhere in the tree for clarity and
brevity.
Ben Pfaff [Fri, 3 Sep 2021 01:45:39 +0000 (18:45 -0700)]
sys-file-encoding: Put the buffer-read-only declaration at the very top.
This accidentally got pushed down when license notices were added to a
bunch of files en masse. It only works if it's at the top.
Ben Pfaff [Sat, 31 Jul 2021 02:48:08 +0000 (19:48 -0700)]
Remove unneeded Emacs declarations that say that a .c file is in C.
These were there because they were formerly .q files.
Ben Pfaff [Thu, 2 Sep 2021 16:23:11 +0000 (09:23 -0700)]
DEFINE: Properly support redefining a macro.
Redefining a macro didn't work in simple cases because the macro name was
being expanded in the DEFINE command.
Thanks to Frans Houweling for reporting the bug.
Ben Pfaff [Wed, 1 Sep 2021 17:02:26 +0000 (10:02 -0700)]
macro: Fix crash for !QUOTE of a quoted string.
This revealed that the tests didn't include anything for !QUOTE and
!UNQUOTE, even though the manual had lots of examples. I added a test for
these and for !NULL, which had also been forgotten.
Thanks to Frans Houweling for reporting the bug.
John Darrington [Sun, 29 Aug 2021 17:45:38 +0000 (19:45 +0200)]
Avoid GtkCritical on startup
John Darrington [Sat, 28 Aug 2021 08:50:32 +0000 (10:50 +0200)]
TeX tests: Use the shell instead of wc to test for maximum line length.
When testing for maximum line length use the shell rather than
relying on wc -L : The -L flag is not present on some systems.
tests/tex.at: Remove dependence on wc
configure.ac: Remove test for wc -L
Fixes bug #59859