lexer: Add support for embedded \0 bytes and missing trailing new-line.

lexer: Add support for embedded \0 bytes and missing trailing new-line.

The lexer, at a low level, has not supported \0 bytes in the input stream
because it used such a byte as the end-of-input indicator.  This caused
some awkwardness for the higher-level lexer, which had to remove and flag
\0 bytes as it read them.  This caused a bug in the higher-level lexer,
which raised an error for each \0 byte it removed but did so when the
lexer was in an intermediate state, which could read uninitialized data.

This commit fixes the problem by adding support for \0 bytes to the
low-level lexer (segmenter).  At the same time, it adds support for input
that doesn't end in a new-line character.

Bug #54664.
Thanks to Tianxiao Gu for reporting this bug.