segment: Add comment about zero-length segments.

diff --git a/src/language/lexer/segment.c b/src/language/lexer/segment.c
index e5d0af235213443ada5b2df9994e78eeaae97373..05a9f57236f8b30fc0e26931ab93beaba3b4c715 100644 (file)
--- a/src/language/lexer/segment.c
+++ b/src/language/lexer/segment.c
@@ -1893,6 +1893,9 @@ segmenter_get_mode (const struct segmenter *s)
    bytes as part of INPUT, because they have (figuratively) been consumed by
    the segmenter.
 
+   Segments can have zero length, including segment types SEG_END,
+   SEG_SEPARATE_COMMANDS, SEG_START_DOCUMENT, SEG_INLINE_DATA, and SEG_SPACES.
+
    Failure occurs only if the segment type of the N bytes in INPUT cannot yet
    be determined.  In this case segmenter_push() returns -1.  If more input is
    available, the caller should obtain some more, then call again with a larger