Invoking gnulib-tool
********************

The gnulib-tool command is the recommended way to import Gnulib modules.  It is possible to borrow Gnulib modules in a package without using gnulib-tool, relying only on the meta-information stored in the modules/* files, but with a growing number of modules this becomes tedious.  gnulib-tool simplifies the management of source files, Makefile.ams and configure.ac in packages incorporating Gnulib modules.

Run gnulib-tool --help for information.  To get familiar with gnulib-tool without affecting your sources, you can also try some commands with the option --dry-run; then gnulib-tool will only report which actions it would perform in a real run without changing anything.

Initial import
==============

Gnulib assumes your project uses Autoconf and Automake.  Invoking gnulib-tool --import will copy source files, create a Makefile.am to build them, generate a file gnulib-comp.m4 with Autoconf M4 macro declarations used by configure.ac, and generate a file gnulib-cache.m4 containing the cached specification of how Gnulib is used.

Our example will be a library that uses Autoconf, Automake and Libtool.  It calls strdup, and you wish to use gnulib to make the package portable to C89 and C99 (which don't have strdup).

     ~/src/libfoo$ gnulib-tool --import strdup
     Module list with included dependencies:
       absolute-header
       extensions
       strdup
       string
     File list:
       lib/dummy.c
       lib/strdup.c
       lib/string.in.h
       m4/absolute-header.m4
       m4/extensions.m4
       m4/gnulib-common.m4
       m4/strdup.m4
       m4/string_h.m4
     Creating directory ./lib
     Creating directory ./m4
     Copying file lib/dummy.c
     Copying file lib/strdup.c
     Copying file lib/string.in.h
     Copying file m4/absolute-header.m4
     Copying file m4/extensions.m4
     Copying file m4/gnulib-common.m4
     Copying file m4/gnulib-tool.m4
     Copying file m4/strdup.m4
     Copying file m4/string_h.m4
     Creating lib/Makefile.am
     Creating m4/gnulib-cache.m4
     Creating m4/gnulib-comp.m4
     Finished.

     You may need to add #include directives for the following .h files.
       #include <string.h>

     Don't forget to
       - add "lib/Makefile" to AC_CONFIG_FILES in ./configure.ac,
       - mention "lib" in SUBDIRS in Makefile.am,
       - mention "-I m4" in ACLOCAL_AMFLAGS in Makefile.am,
       - invoke gl_EARLY in ./configure.ac, right after AC_PROG_CC,
       - invoke gl_INIT in ./configure.ac.
     ~/src/libfoo$

By default, the source code is copied into lib/ and the M4 macros in m4/.  You can override these paths by using --source-base=DIRECTORY and --m4-base=DIRECTORY.  Some modules also provide other files necessary for building. These files are copied into the directory specified by AC_CONFIG_AUX_DIR in configure.ac or by the --aux-dir=DIRECTORY option.  If neither is specified, the current directory is assumed.

gnulib-tool can make symbolic links instead of copying the source files.  The option to specify for this is --symlink, or -s for short.  This can be useful to save a few kilobytes of disk space.  But it is likely to introduce bugs when gnulib is updated; it is more reliable to use gnulib-tool --update (see below) to update to newer versions of gnulib.  Furthermore it requires extra effort to create self-contained tarballs, and it may disturb some mechanism the maintainer applies to the sources.  For these reasons, this option is generally discouraged.

gnulib-tool will overwrite any pre-existing files, in particular Makefile.am.  Unfortunately, separating the generated Makefile.am content (for building the gnulib library) into a separate file, say gnulib.mk, that could be included by your handwritten Makefile.am is not possible, due to how variable assignments are handled by Automake.

Consequently, it is a good idea to choose directories that are not already used by your projects, to separate gnulib imported files from your own files.  This approach is also useful if you want to avoid conflicts between other tools (e.g., gettextize that also copy M4 files into your package.  Simon Josefsson successfully uses a source base of gl/, and a M4 base of gl/m4/, in several packages.

After the --import option on the command line comes the list of Gnulib modules that you want to incorporate in your package.  The names of the modules coincide with the filenames in Gnulib's modules/ directory.

Some Gnulib modules depend on other Gnulib modules.  gnulib-tool will automatically add the needed modules as well; you need not list them explicitly.  gnulib-tool will also memorize which dependent modules it has added, so that when someday a dependency is dropped, the implicitly added module is dropped as well (unless you have explicitly requested that module).

If you want to cut a dependency, i.e., not add a module although one of your requested modules depends on it, you may use the option --avoid=MODULE to do so.  Multiple uses of this option are possible.  Of course, you will then need to implement the same interface as the removed module.

A few manual steps are required to finish the initial import. gnulib-tool printed a summary of these steps.

First, you must ensure Autoconf can find the macro definitions in gnulib-comp.m4.  Use the ACLOCAL_AMFLAGS specifier in your top-level Makefile.am file, as in:

     ACLOCAL_AMFLAGS = -I m4

You are now ready to call the M4 macros in gnulib-comp.m4 from configure.ac.  The macro gl_EARLY must be called as soon as possible after verifying that the C compiler is working. Typically, this is immediately after AC_PROG_CC, as in:

     ...
     AC_PROG_CC
     gl_EARLY
     ...

The core part of the gnulib checks are done by the macro gl_INIT.  Place it further down in the file, typically where you normally check for header files or functions.  It must come after other checks which may affect the compiler invocation, such as AC_MINIX.  For example:

     ...
     # For gnulib.
     gl_INIT
     ...

gl_INIT will in turn call the macros related with the gnulib functions, be it specific gnulib macros, like gl_FUNC_ALLOCA or autoconf or automake macros like AC_FUNC_ALLOCA or AM_FUNC_GETLINE.  So there is no need to call those macros yourself when you use the corresponding gnulib modules.

You must also make sure that the gnulib library is built.  Add the Makefile in the gnulib source base directory to AC_CONFIG_FILES, as in:

     AC_CONFIG_FILES(... lib/Makefile ...)

You must also make sure that make will recurse into the gnulib directory.  To achieve this, add the gnulib source base directory to a SUBDIRS Makefile.am statement, as in:

     SUBDIRS = lib

or if you, more likely, already have a few entries in SUBDIRS, you can add something like:

     SUBDIRS += lib

Finally, you have to add compiler and linker flags in the appropriate source directories, so that you can make use of the gnulib library. Since some modules (getopt, for example) may copy files into the build directory, top_builddir/lib is needed as well as top_srcdir/lib.  For example:

     ...
     AM_CPPFLAGS = -I$(top_builddir)/lib -I$(top_srcdir)/lib
     ...
     LDADD = lib/libgnu.a
     ...

Don't forget to #include the various header files.  In this example, you would need to make sure that #include <string.h> is evaluated when compiling all source code files, that want to make use of strdup.

In the usual case where Autoconf is creating a config.h file, you should include config.h first, before any other include file.  That way, for example, if config.h defines restrict to be the empty string on a pre-C99 host, or a macro like _FILE_OFFSET_BITS that affects the layout of data structures, the definition is consistent for all include files. Also, on some platforms macros like _FILE_OFFSET_BITS and _GNU_SOURCE may be ineffective, or may have only a limited effect, if defined after the first system header file is included.

Finally, note that you can not use AC_LIBOBJ or AC_REPLACE_FUNCS in your configure.ac and expect the resulting object files to be automatically added to lib/libgnu.a. This is because your AC_LIBOBJ and AC_REPLACE_FUNCS invocations from configure.ac augment a variable @LIBOBJS@ (and/or @LTLIBOBJS@ if using Libtool), whereas lib/libgnu.a is built from the contents of a different variable, usually @gl_LIBOBJS@ (or @gl_LTLIBOBJS@ if using Libtool).

Modified imports
================

You can at any moment decide to use Gnulib differently than the last time.

If you only want to use more Gnulib modules, simply invoke gnulib-tool --import NEW-MODULES.  gnulib-tool remembers which modules were used last time.  The list of modules that you pass after --import is added to the previous list of modules.

For most changes, such as added or removed modules, or even different choices of --lib, --source-base or --aux-dir, there are two ways to perform the change.

The standard way is to modify manually the file gnulib-cache.m4 in the M4 macros directory, then launch gnulib-tool --import.

The other way is to call gnulib-tool again, with the changed command-line options.  Note that this doesn't let you remove modules, because as you just learned, the list of modules is always cumulated. Also this way is often impractical, because you don't remember the way you invoked gnulib-tool last time.

The only change for which this doesn't work is a change of the --m4-base directory.  Because, when you pass a different value of --m4-base, gnulib-tool will not find the previous gnulib-cache.m4 file any more... A possible solution is to manually copy the gnulib-cache.m4 into the new M4 macro directory.

In the gnulib-cache.m4, the macros have the following meaning:

gl_MODULES
     The argument is a space separated list of the requested modules, not including dependencies.

gl_AVOID
     The argument is a space separated list of modules that should not be used, even if they occur as dependencies.  Corresponds to the --avoid command line argument.

gl_SOURCE_BASE
     The argument is the relative file name of the directory containing the gnulib source files (mostly *.c and *.h files).  Corresponds to the --source-base command line argument.

gl_M4_BASE
     The argument is the relative file name of the directory containing the gnulib M4 macros (*.m4 files).  Corresponds to the --m4-base command line argument.

gl_TESTS_BASE
     The argument is the relative file name of the directory containing the gnulib unit test files.  Corresponds to the --tests-base command line argument.

gl_LIB
     The argument is the name of the library to be created.  Corresponds to the --lib command line argument.

gl_LGPL
     The presence of this macro without arguments corresponds to the --lgpl command line argument.  The presence of this macro with an argument (whose value must be 2 or 3) corresponds to the --lgpl=ARG command line argument.

gl_LIBTOOL
     The presence of this macro corresponds to the --libtool command line argument and to the absence of the --no-libtool command line argument. It takes no arguments.

gl_MACRO_PREFIX
     The argument is the prefix to use for macros in the gnulib-comp.m4 file.  Corresponds to the --macro-prefix command line argument.

Simple update
=============

When you want to update to a more recent version of Gnulib, without changing the list of modules or other parameters, a simple call does it:

     $ gnulib-tool --import

This will create, update or remove files, as needed.

Note: From time to time, changes are made in Gnulib that are not backward compatible.  When updating to a more recent Gnulib, you should consult Gnulib's NEWS file to check whether the incompatible changes affect your project.

Changing your sources for use with Gnulib
=========================================

Gnulib contains some header file overrides.  This means that when building on systems with deficient header files in /usr/include/, it may create files named string.h, stdlib.h, stdint.h or similar in the build directory.  In the other source directories of your package you will usually pass -I options to the compiler, so that these Gnulib substitutes are visible and take precedence over the files in /usr/include/.

These Gnulib substitute header files rely on <config.h> being already included.  Furthermore <config.h> must be the first include in every compilation unit.  This means that to all your source files and likely also to all your tests source files you need to add an #include <config.h> at the top.  Which source files are affected? Exactly those whose compilation includes a -I option that refers to the Gnulib library directory.

This is annoying, but inevitable: On many systems, <config.h> is used to set system dependent flags (such as _GNU_SOURCE on GNU systems), and these flags have no effect after any system header file has been included.

Issues with Version Control Systems
===================================

If a project stores its source files in a version control system (VCS), such as CVS, SVN, or Git, one needs to decide which files to commit.

All files created by gnulib-tool, except gnulib-cache.m4, should be treated like generated source files, like for example a parser.c file is generated from parser.y.

  * In projects which commit all source files, whether generated or not, into their VCS, the gnulib-tool generated files should all be committed.

     Gnulib also contains files generated by make (and removed by make clean), using information determined by configure They should not be checked into the VCS, but instead added to .cvsignore.  When you have a Gnulib source file of the form lib/foo_.h, the corresponding lib/foo.h is such a file.

  * In projects which customarily omit from their VCS all files that are generated from other source files, all these files and directories would not be added into the VCS.  The only file that must be added to the VCS is gnulib-cache.m4 in the M4 macros directory.  Also, the script for restoring files not in the VCS, customarily called autogen.sh or bootstrap.sh, will typically contain the statement for restoring the omitted files:

          $ gnulib-tool --update

     The --update option operates much like the --import option, but it does not offer the possibility to change the way Gnulib is used. Also it does not report in the ChangeLogs the files that it had to add because they were missing.