scan: Introduce string_lexer for simple tokenizing of a string.
authorBen Pfaff <blp@cs.stanford.edu>
Sat, 12 Jan 2013 19:10:08 +0000 (11:10 -0800)
committerBen Pfaff <blp@cs.stanford.edu>
Sat, 12 Jan 2013 19:10:08 +0000 (11:10 -0800)
commit35140ad55d6ce3cbce7d9ffc4b894598f603cd42
tree85a0b67d22da7d1a66277c67e3a0a667da04b325
parent1a22a807e81d565483d7bada25c463e91f1164cc
scan: Introduce string_lexer for simple tokenizing of a string.

The following commit will introduce a user outside of the tests.
src/language/lexer/scan.c
src/language/lexer/scan.h
tests/language/lexer/scan-test.c