segment: New library for low-level phase of lexical syntax analysis.
authorBen Pfaff <blp@cs.stanford.edu>
Sat, 19 Mar 2011 23:30:55 +0000 (16:30 -0700)
committerBen Pfaff <blp@cs.stanford.edu>
Sun, 20 Mar 2011 16:43:44 +0000 (09:43 -0700)
This library provides for a low-level part of lexical analysis for
PSPP syntax, which I call "segmentation".  Segmentation accepts a
stream of UTF-8 bytes as input.  It outputs a label (a segment type)
for each byte or contiguous sequence of bytes in the input.

The following commit will implement the high-level phase of lexical
analysis, called "scanning", that converts a sequence of segments into
PSPP tokens.


No differences found