Invoking gnulib-tool
********************

The gnulib-tool command is the recommended way to import Gnulib modules.  It is possible to borrow Gnulib modules in a package without using gnulib-tool, relying only on the meta-information stored in the modules/* files, but with a growing number of modules this becomes tedious.  gnulib-tool simplifies the management of source files, Makefile.ams and configure.ac in packages incorporating Gnulib modules.

gnulib-tool is not installed in a standard directory that is contained in the PATH variable.  It needs to be run directly in the directory that contains the Gnulib source code.  You can do this either by specifying the absolute filename of gnulib-tool, or you can also use a symbolic link from a place inside your PATH to the gnulib-tool file of your preferred and most up-to-date Gnulib checkout, like this:

     $ ln -s $HOME/gnu/src/gnulib.git/gnulib-tool $HOME/bin/gnulib-tool

Run gnulib-tool --help for information.  To get familiar with gnulib-tool without affecting your sources, you can also try some commands with the option --dry-run; then gnulib-tool will only report which actions it would perform in a real run without changing anything.

Finding modules
===============

There are three ways of finding the names of Gnulib modules that you can use in your package:

  * You have the complete module list, sorted according to categories, in http://www.gnu.org/software/gnulib/MODULES.html.

  * If you are looking for a particular POSIX header or function replacement, look in the chapters Header File Substitutes and Function Substitutes.  For headers and functions that are provided by Glibc but not standardized by POSIX, look in the chapters Glibc Header File Substitutes and Glibc Function Substitutes.

  * If you have already found the source file in Gnulib and are looking for the module that contains this source file, you can use the command gnulib-tool --find FILENAME.

Initial import
==============

Gnulib assumes that your project uses Autoconf.  When using Gnulib, you will need to have Autoconf and Automake among your build tools.  Note that while the use of Automake in your project's top level directory is an easy way to fulfil the Makefile conventions of the GNU coding standards, Gnulib does not require it.  But when you use Gnulib, Automake will be used at least in a subdirectory of your project.

Invoking gnulib-tool --import will copy source files, create a Makefile.am to build them, generate a file gnulib-comp.m4 with Autoconf M4 macro declarations used by configure.ac, and generate a file gnulib-cache.m4 containing the cached specification of how Gnulib is used.

Our example will be a library that uses Autoconf, Automake and Libtool.  It calls strdup, and you wish to use gnulib to make the package portable to C89 and C99 (which don't have strdup).

     ~/src/libfoo$ gnulib-tool --import strdup
     Module list with included dependencies:
       absolute-header
       extensions
       strdup
       string
     File list:
       lib/dummy.c
       lib/strdup.c
       lib/string.in.h
       m4/absolute-header.m4
       m4/extensions.m4
       m4/gnulib-common.m4
       m4/strdup.m4
       m4/string_h.m4
     Creating directory ./lib
     Creating directory ./m4
     Copying file lib/dummy.c
     Copying file lib/strdup.c
     Copying file lib/string.in.h
     Copying file m4/absolute-header.m4
     Copying file m4/extensions.m4
     Copying file m4/gnulib-common.m4
     Copying file m4/gnulib-tool.m4
     Copying file m4/strdup.m4
     Copying file m4/string_h.m4
     Creating lib/Makefile.am
     Creating m4/gnulib-cache.m4
     Creating m4/gnulib-comp.m4
     Finished.

     You may need to add #include directives for the following .h files.
       #include <string.h>

     Don't forget to
       - add "lib/Makefile" to AC_CONFIG_FILES in ./configure.ac,
       - mention "lib" in SUBDIRS in Makefile.am,
       - mention "-I m4" in ACLOCAL_AMFLAGS in Makefile.am,
       - invoke gl_EARLY in ./configure.ac, right after AC_PROG_CC,
       - invoke gl_INIT in ./configure.ac.
     ~/src/libfoo$

By default, the source code is copied into lib/ and the M4 macros in m4/.  You can override these paths by using --source-base=DIRECTORY and --m4-base=DIRECTORY.  Some modules also provide other files necessary for building.  These files are copied into the directory specified by AC_CONFIG_AUX_DIR in configure.ac or by the --aux-dir=DIRECTORY option.  If neither is specified, the current directory is assumed.

gnulib-tool can make symbolic links instead of copying the source files.  The option to specify for this is --symlink, or -s for short.  This can be useful to save a few kilobytes of disk space.  But it is likely to introduce bugs when gnulib is updated; it is more reliable to use gnulib-tool --update (see below) to update to newer versions of gnulib.  Furthermore it requires extra effort to create self-contained tarballs, and it may disturb some mechanism the maintainer applies to the sources.  For these reasons, this option is generally discouraged.

gnulib-tool will overwrite any pre-existing files, in particular Makefile.am.  It is also possible to separate the generated Makefile.am content (for building the gnulib library) into a separate file, say gnulib.mk, that can be included by your handwritten Makefile.am, but this is a more advanced use of gnulib-tool.

Consequently, it is a good idea to choose directories that are not already used by your projects, to separate gnulib imported files from your own files.  This approach is also useful if you want to avoid conflicts between other tools (e.g., gettextize that also copy M4 files into your package.  Simon Josefsson successfully uses a source base of gl/, and a M4 base of gl/m4/, in several packages.

After the --import option on the command line comes the list of Gnulib modules that you want to incorporate in your package.  The names of the modules coincide with the filenames in Gnulib's modules/ directory.

Some Gnulib modules depend on other Gnulib modules.  gnulib-tool will automatically add the needed modules as well; you need not list them explicitly.  gnulib-tool will also memorize which dependent modules it has added, so that when someday a dependency is dropped, the implicitly added module is dropped as well (unless you have explicitly requested that module).

If you want to cut a dependency, i.e., not add a module although one of your requested modules depends on it, you may use the option --avoid=MODULE to do so.  Multiple uses of this option are possible.  Of course, you will then need to implement the same interface as the removed module.

A few manual steps are required to finish the initial import. gnulib-tool printed a summary of these steps.

First, you must ensure Autoconf can find the macro definitions in gnulib-comp.m4.  Use the ACLOCAL_AMFLAGS specifier in your top-level Makefile.am file, as in:

     ACLOCAL_AMFLAGS = -I m4

You are now ready to call the M4 macros in gnulib-comp.m4 from configure.ac.  The macro gl_EARLY must be called as soon as possible after verifying that the C compiler is working. Typically, this is immediately after AC_PROG_CC, as in:

     ...
     AC_PROG_CC
     gl_EARLY
     ...

The core part of the gnulib checks are done by the macro gl_INIT.  Place it further down in the file, typically where you normally check for header files or functions.  It must come after other checks which may affect the compiler invocation, such as AC_MINIX.  For example:

     ...
     # For gnulib.
     gl_INIT
     ...

gl_INIT will in turn call the macros related with the gnulib functions, be it specific gnulib macros, like gl_FUNC_ALLOCA or autoconf or automake macros like AC_FUNC_ALLOCA or AM_FUNC_GETLINE.  So there is no need to call those macros yourself when you use the corresponding gnulib modules.

You must also make sure that the gnulib library is built.  Add the Makefile in the gnulib source base directory to AC_CONFIG_FILES, as in:

     AC_CONFIG_FILES(... lib/Makefile ...)

You must also make sure that make will recurse into the gnulib directory.  To achieve this, add the gnulib source base directory to a SUBDIRS Makefile.am statement, as in:

     SUBDIRS = lib

or if you, more likely, already have a few entries in SUBDIRS, you can add something like:

     SUBDIRS += lib

Finally, you have to add compiler and linker flags in the appropriate source directories, so that you can make use of the gnulib library. Since some modules (getopt, for example) may copy files into the build directory, top_builddir/lib is needed as well as top_srcdir/lib.  For example:

     ...
     AM_CPPFLAGS = -I$(top_builddir)/lib -I$(top_srcdir)/lib
     ...
     LDADD = lib/libgnu.a
     ...

Don't forget to #include the various header files.  In this example, you would need to make sure that #include <string.h> is evaluated when compiling all source code files, that want to make use of strdup.

In the usual case where Autoconf is creating a config.h file, you should include config.h first, before any other include file.  That way, for example, if config.h defines restrict to be the empty string on a pre-C99 host, or a macro like _FILE_OFFSET_BITS that affects the layout of data structures, the definition is consistent for all include files. Also, on some platforms macros like _FILE_OFFSET_BITS and _GNU_SOURCE may be ineffective, or may have only a limited effect, if defined after the first system header file is included.

Finally, note that you can not use AC_LIBOBJ or AC_REPLACE_FUNCS in your configure.ac and expect the resulting object files to be automatically added to lib/libgnu.a. This is because your AC_LIBOBJ and AC_REPLACE_FUNCS invocations from configure.ac augment a variable @LIBOBJS@ (and/or @LTLIBOBJS@ if using Libtool), whereas lib/libgnu.a is built from the contents of a different variable, usually @gl_LIBOBJS@ (or @gl_LTLIBOBJS@ if using Libtool).

Modified imports
================

You can at any moment decide to use Gnulib differently than the last time.

There are two ways to change how Gnulib is used.  Which one you'll use, depends on where you keep track of options and module names that you pass to gnulib-tool.

  * If you store the options and module names in a file under your own control, such as autogen.sh, bootstrap, bootstrap.conf, or similar, simply invoke gnulib-tool again, with modified options and more or fewer module names.

  * gnulib-tool remembers which modules were used last time.  If you want to rely on gnulib-tool's own memory of the last used options and module names, you can use the commands gnulib-tool --add-import and gnulib-tool --remove-import.

     So, if you only want to use more Gnulib modules, simply invoke gnulib-tool --add-import NEW-MODULES.  The list of modules that you pass after --add-import is added to the previous list of modules.

     Similarly, if you want to use fewer Gnulib modules, simply invoke gnulib-tool --remove-import UNNEEDED-MODULES.  The list of modules that you pass after --remove-import is removed from the previous list of modules.  Note that if a module is then still needed as dependency of other modules, it will be used nevertheless. If you want to really not use a module any more, regardless of whether other modules may need it, you need to use the --avoid option.

     For other changes, such as different choices of --lib, --source-base or --aux-dir, the normal way is to modify manually the file gnulib-cache.m4 in the M4 macros directory, then launch gnulib-tool --add-import.

     The only change for which this doesn't work is a change of the --m4-base directory.  Because, when you pass a different value of --m4-base, gnulib-tool will not find the previous gnulib-cache.m4 file any more.  A possible solution is to manually copy the gnulib-cache.m4 into the new M4 macro directory.

     In the gnulib-cache.m4 file, the macros have the following meaning:

     gl_MODULES
          The argument is a space separated list of the requested modules, not including dependencies.

     gl_AVOID
          The argument is a space separated list of modules that should not be used, even if they occur as dependencies.  Corresponds to the --avoid command line argument.

     gl_SOURCE_BASE
          The argument is the relative file name of the directory containing the gnulib source files (mostly *.c and *.h files).  Corresponds to the --source-base command line argument.

     gl_M4_BASE
          The argument is the relative file name of the directory containing the gnulib M4 macros (*.m4 files).  Corresponds to the --m4-base command line argument.

     gl_TESTS_BASE
          The argument is the relative file name of the directory containing the gnulib unit test files.  Corresponds to the --tests-base command line argument.

     gl_LIB
          The argument is the name of the library to be created.  Corresponds to the --lib command line argument.

     gl_LGPL
          The presence of this macro without arguments corresponds to the --lgpl command line argument.  The presence of this macro with an argument (whose value must be 2 or 3) corresponds to the --lgpl=ARG command line argument.

     gl_LIBTOOL
          The presence of this macro corresponds to the --libtool command line argument and to the absence of the --no-libtool command line argument. It takes no arguments.

     gl_MACRO_PREFIX
          The argument is the prefix to use for macros in the gnulib-comp.m4 file.  Corresponds to the --macro-prefix command line argument.

Simple update
=============

When you want to update to a more recent version of Gnulib, without changing the list of modules or other parameters, a simple call does it:

     $ gnulib-tool --add-import

This will create, update or remove files, as needed.

Note: From time to time, changes are made in Gnulib that are not backward compatible.  When updating to a more recent Gnulib, you should consult Gnulib's NEWS file to check whether the incompatible changes affect your project.

Changing your sources for use with Gnulib
=========================================

Gnulib contains some header file overrides.  This means that when building on systems with deficient header files in /usr/include/, it may create files named string.h, stdlib.h, stdint.h or similar in the build directory.  In the other source directories of your package you will usually pass -I options to the compiler, so that these Gnulib substitutes are visible and take precedence over the files in /usr/include/.

These Gnulib substitute header files rely on <config.h> being already included.  Furthermore <config.h> must be the first include in every compilation unit.  This means that to all your source files and likely also to all your tests source files you need to add an #include <config.h> at the top.  Which source files are affected? Exactly those whose compilation includes a -I option that refers to the Gnulib library directory.

This is annoying, but inevitable: On many systems, <config.h> is used to set system dependent flags (such as _GNU_SOURCE on GNU systems), and these flags have no effect after any system header file has been included.

Caveat: gettextize and autopoint users
======================================

The programs gettextize and autopoint, part of GNU gettext, import or update the internationalization infrastructure. Some of this infrastructure, namely ca. 20 autoconf macro files and the config.rpath file, is also contained in Gnulib and may be imported by gnulib-tool.  The use of gettextize or autopoint will therefore overwrite some of the files that gnulib-tool has imported, and vice versa.

Avoiding to use gettextize (manually, as package maintainer) or autopoint (as part of a script like autoreconf or autogen.sh) is not the solution: These programs also import the infrastructure in the po/ and optionally in the intl/ directory.

The copies of the conflicting files in Gnulib are more up-to-date than the copies brought in by gettextize and autopoint.  When a new gettext release is made, the copies of the files in Gnulib will be updated immediately.

The solution is therefore:

 1. When you run gettextize, always use the gettextize from the matching GNU gettext release.  For the most recent Gnulib checkout, this is the newest release found on http://ftp.gnu.org/gnu/gettext/.  For an older Gnulib snapshot, it is the release that was the most recent release at the time the Gnulib snapshot was taken.  Then, after gettextize, invoke gnulib-tool.

 2. When a script of yours run autopoint, invoke gnulib-tool afterwards.

 3. If you get an error message like *** error: gettext infrastructure mismatch: using a Makefile.in.in from gettext version ... but the autoconf macros are from gettext version ..., it means that a new GNU gettext release was made, and its autoconf macros were integrated into Gnulib and now mismatch the po/ infrastructure. In this case, fetch and install the new GNU gettext release and run gettextize followed by gnulib-tool.

 4. When you invoke autoreconf after gnulib-tool, make sure to not invoke autopoint a second time, by setting the AUTOPOINT environment variable, like this:

          $ env AUTOPOINT=true autoreconf --install

Handling Gnulib's own message translations
==========================================

Gnulib provides some functions that emit translatable messages using GNU gettext.  The gnulib domain at the Translation Project (http://translationproject.org/) collects translations of these messages, which you should incorporate into your own programs.

There are two basic ways to achieve this.  The first, and older, method is to list all the source files you use from Gnulib in your own po/POTFILES.in file.  This will cause all the relevant translatable strings to be included in your POT file.  When you send this POT file to the Translation Project, translators will normally fill in the translations of the Gnulib strings from their ``translation memory'', and send you back updated PO files.

However, this process is error-prone: you might forget to list some source files, or the translator might not be using a translation memory and provide a different translation than another translator, or the translation might not be kept in sync between Gnulib and your package. It is also slow and causes substantial extra work, because a human translator must be in the loop for each language and you will need to incorporate their work on request.

For these reasons, a new method was designed and is now recommended.  If you pass the --po-base=DIRECTORY and --po-domain=DOMAIN options to gnulib-tool, then gnulib-tool will create a separate directory with its own POTFILES.in, and fetch current translations directly from the Translation Project (using rsync or wget, whichever is available). The POT file in this directory will be called DOMAIN-gnulib.pot, depending on the DOMAIN you gave to the --po-domain option (typically the same as the package name). This causes these translations to reside in a separate message domain, so that they do not clash either with the translations for the main part of your package nor with those of other packages on the system that use possibly different versions of Gnulib. When you use these options, the functions in Gnulib are built in such a way that they will always use this domain regardless of the default domain set by textdomain.

In order to use this method, you must -- in each program that might use Gnulib code -- add an extra line to the part of the program that initializes locale-dependent behavior.  Where you would normally write something like:

       setlocale (LC_ALL, "");
       bindtextdomain (PACKAGE, LOCALEDIR);
       textdomain (PACKAGE);

you should add an additional bindtextdomain call to inform gettext of where the MO files for the extra message domain may be found:

       bindtextdomain (PACKAGE "-gnulib", LOCALEDIR);

(This example assumes that the DOMAIN that you specified to gnulib-tool is the same as the value of the PACKAGE preprocessor macro.)

Since you do not change the textdomain call, the default message domain for your program remains the same and your own use of gettext functions will not be affected.

Issues with Version Control Systems
===================================

If a project stores its source files in a version control system (VCS), such as CVS, Subversion, or Git, one needs to decide which files to commit.

In principle, all files created by gnulib-tool, except gnulib-cache.m4, can be treated like generated source files, like for example a parser.c file generated from parser.y.  Alternatively, they can be considered source files and updated manually.

Here are the three different approaches in common use.  Each has its place, and you should use whichever best suits your particular project and development methods.

 1. In projects which commit all source files, whether generated or not, into their VCS, the gnulib-tool generated files should all be committed.  In this case, you should pass the option --no-vc-files to gnulib-tool, which avoids alteration of VCS-related files such as .cvsignore.

     Gnulib also contains files generated by make (and removed by make clean), using information determined by configure.  For a Gnulib source file of the form lib/foo.in.h, the corresponding lib/foo.h is such a make-generated file.  These should not be checked into the VCS, but instead added to .cvsignore or equivalent.

 2. In projects which customarily omit from their VCS all files that are generated from other source files, none of these files and directories are added into the VCS.  As described in Modified imports, there are two ways to keep track of options and module names that are passed to gnulib-tool.  The command for restoring the omitted files depends on it:

       * If they are stored in a file other than gnulib-cache.m4, such as autogen.sh, bootstrap, bootstrap.conf, or similar, the restoration command is the entire gnulib-tool ... --import ... invocation with all options and module names.

       * If the project relies on gnulib-tool's memory of the last used options and module names, then the file gnulib-cache.m4 in the M4 macros directory must be added to the VCS, and the restoration command is:

               $ gnulib-tool --update

          The --update option operates much like the --add-import option, but it does not offer the possibility to change the way Gnulib is used.  Also it does not report in the ChangeLogs the files that it had to add because they were missing.

     Gnulib includes the file build-aux/bootstrap to aid a developer in using this setup.  Furthermore, in projects that use git for version control, it is possible to use a git submodule containing the precise commit of the gnulib repository, so that each developer running bootstrap will get the same version of all gnulib-provided files.  The location of the submodule can be chosen to fit the package's needs; here's how to initially create the submodule in the directory .gnulib:

          $ dir=.gnulib
          $ git submodule add -- git://git.sv.gnu.org/gnulib.git $dir
          $ git config alias.syncsub "submodule foreach git pull origin master"

     Thereafter, bootstrap can run this command to update the submodule to the recorded checkout level:

          git submodule update --init $dir

     and a developer can use this sequence to update to a newer version of gnulib:

          $ git syncsub
          $ git add $dir
          $ ./bootstrap

 3. Some projects take a ``middle road'': they do commit Gnulib source files as in the first approach, but they do not commit other derived files, such as a Makefile.in generated by Automake.  This increases the size and complexity of the repository, but can help occasional contributors by not requiring them to have a full Gnulib checkout to do a build, and all developers by ensuring that all developers are working with the same version of Gnulib in the repository.  It also supports multiple Gnulib instances within a project.  It remains important not to commit the make-generated files, as described above.

Bundling the unit tests of the Gnulib modules
=============================================

You can bundle the unit tests of the Gnulib modules together with your package, through the --with-tests option.  Together with --with-tests, you also specify the directory for these tests through the --tests-base option.  Of course, you need to add this directory to the SUBDIRS variable in the Makefile.am of the parent directory.

The advantage of having the unit tests bundled is that when your program has a problem on a particular platform, running the unit tests may help determine quickly if the problem is on Gnulib's side or on your package's side.  Also, it helps verifying Gnulib's portability, of course.

The unit tests will be compiled and run when the user runs make check. When the user runs only make, the unit tests will not be compiled.

In the SUBDIRS variable, it is useful to put the Gnulib tests directory after the directory containing the other tests, not before:

     SUBDIRS = gnulib-lib src man tests gnulib-tests

This will ensure that on platforms where there are test failures in either directory, users will see and report the failures from the tests of your program.

Note: In packages which use more than one invocation of gnulib-tool in the scope of the same configure.ac, you cannot use --with-tests.  You will have to use a separate configure.ac in this case.