MATRIX: Fix test for very large matrix.

MATRIX: Fix test for very large matrix.

I kept getting processes killed as out-of-memory for the "MATRIX - very
large matrices" test.  Somehow, GSL was really trying to allocate a
matrix with 230,000,000,000 elements, which would require 1.84 PB of
memory, which I don't have.

This makes the matrix allocator report out-of-memory if a sequence would
have more than about 2 billion elements, fixing my OOM killer problem.