lexer: New type enum token_type.

lexer: New type enum token_type.

It seems useful to have a type that indicates that a value is a token.
Furthermore, being able to enumerate all of the possible tokens in a
straightforward way seems worthwhile.  It also makes it possible for
GCC to check "switch" statements on token types and to build arrays
indexed by token.