segment: Add support for segmenting "snippets".
authorBen Pfaff <blp@cs.stanford.edu>
Sun, 13 Jun 2021 20:01:30 +0000 (13:01 -0700)
committerBen Pfaff <blp@cs.stanford.edu>
Sun, 13 Jun 2021 20:01:30 +0000 (13:01 -0700)
commitaaa693a2c7f2ab9e11504b50184018d188efdc0b
tree69eac4b063a9448f5173691892202726695379c6
parentc0bffcdd876481bbf21ac9ed7510b93c2c41decf
segment: Add support for segmenting "snippets".

Otherwise, segmenting a little bit of syntax like "-123" will produce
T_ENDCMD, T_POS_NUM.
src/language/control/repeat.c
src/language/lexer/lexer.c
src/language/lexer/scan.c
src/language/lexer/segment.c
src/language/lexer/segment.h
tests/language/lexer/segment-test.c