segment: New library for low-level phase of lexical syntax analysis.
authorBen Pfaff <blp@cs.stanford.edu>
Sat, 19 Mar 2011 23:30:55 +0000 (16:30 -0700)
committerBen Pfaff <blp@cs.stanford.edu>
Sun, 20 Mar 2011 16:43:44 +0000 (09:43 -0700)
commit75a467ed2d32e1adb0c24cf89676cfb48845be98
tree75eb007e50d4325189e977193458e47f64f17c77
parentd3e294c031bb767336435d2f0048994103fcd47a
segment: New library for low-level phase of lexical syntax analysis.

This library provides for a low-level part of lexical analysis for
PSPP syntax, which I call "segmentation".  Segmentation accepts a
stream of UTF-8 bytes as input.  It outputs a label (a segment type)
for each byte or contiguous sequence of bytes in the input.

The following commit will implement the high-level phase of lexical
analysis, called "scanning", that converts a sequence of segments into
PSPP tokens.
Smake
src/language/lexer/automake.mk
src/language/lexer/segment.c [new file with mode: 0644]
src/language/lexer/segment.h [new file with mode: 0644]
src/libpspp/automake.mk
src/libpspp/prompt.c [new file with mode: 0644]
src/libpspp/prompt.h [new file with mode: 0644]
tests/automake.mk
tests/language/lexer/segment-test.c [new file with mode: 0644]
tests/language/lexer/segment.at [new file with mode: 0644]