lexer: Reimplement for better testability and internationalization.

[pspp-builds.git] / doc / utilities.texi

diff --git a/doc/utilities.texi b/doc/utilities.texi
index 9f57a76739b43c0a7921b96df18bf66c32905e26..2cf95a3107839634d4b360ea1ce4431b236bf5ea 100644 (file)
--- a/doc/utilities.texi
+++ b/doc/utilities.texi
@@ -10,6 +10,7 @@ encountered in the input.
 
 @menu
 * ADD DOCUMENT::                Add documentary text to the active file.
 
 @menu
 * ADD DOCUMENT::                Add documentary text to the active file.

+* CACHE::                       Ignored for compatibility.

 * CD::                          Change the current directory.
 * COMMENT::                     Document your syntax file.
 * DOCUMENT::                    Document the active file.
 * CD::                          Change the current directory.
 * COMMENT::                     Document your syntax file.
 * DOCUMENT::                    Document the active file.


@@ -52,6 +53,16 @@ DOCUMENTS}.
 Each line of documentary text must be enclosed in quotation marks, and 
 may not be more than 80 bytes long. @xref{DOCUMENT}.
 
 Each line of documentary text must be enclosed in quotation marks, and 
 may not be more than 80 bytes long. @xref{DOCUMENT}.
 

+@node CACHE
+@section CACHE
+@vindex CACHE
+
+@display
+CACHE.
+@end display
+
+This command is accepted, for compatibility, but it has no effect.
+

 @node CD
 @section CD
 @vindex CD
 @node CD
 @section CD
 @vindex CD


@@ -231,7 +242,7 @@ subshell.
 @vindex INCLUDE
 
 @display
 @vindex INCLUDE
 
 @display

-        INCLUDE [FILE=]'file-name'.
+        INCLUDE [FILE=]'file-name' [ENCODING='encoding'].

 @end display
 
 @cmd{INCLUDE} causes the PSPP command processor to read an
 @end display
 
 @cmd{INCLUDE} causes the PSPP command processor to read an


@@ -242,19 +253,11 @@ stop and no more commands will be processed.
 Include files may be nested to any depth, up to the limit of available
 memory.
 
 Include files may be nested to any depth, up to the limit of available
 memory.
 

+The @cmd{INSERT} command (@pxref{INSERT}) is a more flexible
+alternative to @cmd{INCLUDE}.  An INCLUDE command acts the same as
+INSERT with ERROR=STOP CD=NO SYNTAX=BATCH specified.

 
 

-The @cmd{INSERT} command (@pxref{INSERT}) may be used instead of
-@cmd{INCLUDE} if you require more flexible options.
-The syntax 
-@example
-INCLUDE FILE=@var{file-name}.
-@end example
-@noindent 
-functions identically to 
-@example
-INSERT FILE=@var{file-name} ERROR=STOP CD=NO SYNTAX=BATCH.
-@end example
-
+The optional ENCODING subcommand has the same meaning as on INSERT.

 
 @node INSERT
 @section INSERT
 
 @node INSERT
 @section INSERT


@@ -264,7 +267,8 @@ INSERT FILE=@var{file-name} ERROR=STOP CD=NO SYNTAX=BATCH.
      INSERT [FILE=]'file-name'
         [CD=@{NO,YES@}]
         [ERROR=@{CONTINUE,STOP@}]
      INSERT [FILE=]'file-name'
         [CD=@{NO,YES@}]
         [ERROR=@{CONTINUE,STOP@}]

-        [SYNTAX=@{BATCH,INTERACTIVE@}].
+        [SYNTAX=@{BATCH,INTERACTIVE@}]
+        [ENCODING='encoding'].

 @end display
 
 @cmd{INSERT} is similar to @cmd{INCLUDE} (@pxref{INCLUDE}) 
 @end display
 
 @cmd{INSERT} is similar to @cmd{INCLUDE} (@pxref{INCLUDE}) 


@@ -292,6 +296,37 @@ the included file must conform to interactive syntax
 conventions. @xref{Syntax Variants}.
 The default setting is @samp{SYNTAX=BATCH}.
 
 conventions. @xref{Syntax Variants}.
 The default setting is @samp{SYNTAX=BATCH}.
 

+ENCODING optionally specifies the character set used by the included
+file.  Its argument, which is not case-sensitive, must be in one of
+the following forms:
+
+@table @asis
+@item @code{Locale}
+The encoding used by the system locale, or as overridden by the SET
+LOCALE command (@pxref{SET}).  On Unix systems, environment variables,
+e.g.@: @env{LANG} or @env{LC_ALL}, determine the system locale.
+
+@item IANA character set name
+One of the character set names listed by IANA at
+@uref{http://www.iana.org/assignments/character-sets}.  Some examples
+are @code{ASCII} (United States), @code{ISO-8859-1} (western Europe),
+@code{EUC-JP} (Japan), and @code{windows-1252} (Windows).  Not all
+systems support all character sets.
+
+@item @code{Auto}
+@item @code{Auto,@var{encoding}}
+Automatically detects whether a syntax file is encoded in
+@var{encoding} or in a Unicode encoding such as UTF-8, UTF-16, or
+UTF-32.  The @var{encoding} may be an IANA character set name or
+@code{Locale} (the default).  Only ASCII compatible encodings can
+automatically be distinguished from UTF-8 (the most common locale
+encodings are all ASCII-compatible).
+@end table
+
+When ENCODING is not specified, the default is taken from the
+@option{--syntax-encoding} command option, if it was specified, and
+otherwise it is @code{Auto}.
+

 @node PERMISSIONS
 @section PERMISSIONS
 @vindex PERMISSIONS
 @node PERMISSIONS
 @section PERMISSIONS
 @vindex PERMISSIONS


@@ -347,19 +382,13 @@ SET
         /RIB=@{NATIVE,MSBFIRST,LSBFIRST,VAX@}
         /RRB=@{NATIVE,ISL,ISB,IDL,IDB,VF,VD,VG,ZS,ZL@}
 
         /RIB=@{NATIVE,MSBFIRST,LSBFIRST,VAX@}
         /RRB=@{NATIVE,ISL,ISB,IDL,IDB,VF,VD,VG,ZS,ZL@}
 

-(program input)
-        /ENDCMD='.'
-        /NULLINE=@{ON,OFF@}
-

 (interaction)
 (interaction)

-        /CPROMPT='cprompt_string'
-        /DPROMPT='dprompt_string'

         /MXERRS=max_errs
         /MXWARNS=max_warnings
         /MXERRS=max_errs
         /MXWARNS=max_warnings

-        /PROMPT='prompt'

         /WORKSPACE=workspace_size
 
         /WORKSPACE=workspace_size
 

-(program execution)
+(syntax execution)
+        /LOCALE='locale'

         /MEXPAND=@{ON,OFF@}
         /MITERATE=max_iterations
         /MNEST=max_nest
         /MEXPAND=@{ON,OFF@}
         /MITERATE=max_iterations
         /MNEST=max_nest


@@ -519,30 +548,10 @@ formats are only for use with very old input files.
 The default is NATIVE.
 @end table
 
 The default is NATIVE.
 @end table
 

-Program input subcommands affect the way that programs are parsed when
-they are typed interactively or run from a command file.  They are
-
-@table @asis
-@item ENDCMD
-This is a single character indicating the end of a command.  The default
-is @samp{.}.  Don't change this.
-
-@item NULLINE
-Whether a blank line is interpreted as ending the current command.  The
-default is ON.
-@end table
-

 Interaction subcommands affect the way that PSPP interacts with an
 online user.  The interaction subcommands are
 
 @table @asis
 Interaction subcommands affect the way that PSPP interacts with an
 online user.  The interaction subcommands are
 
 @table @asis

-@item CPROMPT
-The command continuation prompt.  The default is @samp{    > }.
-
-@item DPROMPT
-Prompt used when expecting data input within @cmd{BEGIN DATA} (@pxref{BEGIN
-DATA}).  The default is @samp{data> }.
-

 @item MXERRS
 The maximum number of errors before PSPP halts processing of the current
 command file.  The default is 50.
 @item MXERRS
 The maximum number of errors before PSPP halts processing of the current
 command file.  The default is 50.


@@ -554,15 +563,22 @@ The special value of zero means that all warning situations should be ignored.
 No warnings will be issued, except a single initial warning advising the user
 that warnings will not be given.
 The default value is 100.
 No warnings will be issued, except a single initial warning advising the user
 that warnings will not be given.
 The default value is 100.

-
-@item PROMPT
-The command prompt.  The default is @samp{PSPP> }.

 @end table
 
 @end table
 

-Program execution subcommands control the way that PSPP commands
-execute.  The program execution subcommands are
+Syntax execution subcommands control the way that PSPP commands
+execute.  The syntax execution subcommands are

 
 @table @asis
 
 @table @asis

+@item LOCALE
+Overrides the system locale for the purpose of reading and writing
+syntax and data files.  The argument should be a locale name in the
+general form @code{language_country.encoding}, where @code{language}
+and @code{country} are 2-character language and country abbreviations,
+respectively, and @code{encoding} is an IANA character set name.
+Example locales are @code{en_US.UTF-8} (UTF-8 encoded English as
+spoken in the United States) and @code{ja_JP.EUC-JP} (EUC-JP encoded
+Japanese as spoken in Japan).
+

 @item MEXPAND
 @itemx MITERATE
 @itemx MNEST
 @item MEXPAND
 @itemx MITERATE
 @itemx MNEST


@@ -796,7 +812,6 @@ SHOW
         [CCE]
         [COPYING]
         [DECIMALS]
         [CCE]
         [COPYING]
         [DECIMALS]

-        [ENDCMD]

         [FORMAT]
         [LENGTH]
         [MXERRS]
         [FORMAT]
         [LENGTH]
         [MXERRS]