[Freeipa-devel] Generating files from .in template files

[John Dennis]

Traditionally when using autotools it's job of the configure script to 
generate files from .in template files. The reason it's done this way is 
because configure has to probe the system it's running on to discover 
the correct substitutions to be used in the .in templates.

In our top level Makefile we generate several files from .in templates 
under the "version-update" target. Most of this logic applies to 
inserting version information into specific files. In the context of 
release specific data this is probably a valid methodology although 
other projects have incorporated the version munging into configure. For 
version strings I don't think there is necessarily a correct approach 
(e.g. part of configure or external to configure). Technically updating 
the version strings is part of "make dist" step, i.e. the act of 
creating the distribution, which is independent of configure.

However, any template file that depends on the target system where 
configure is being run should be created by configure. Why? Because 
that's the design of autotools and how people expect things will work in 
an autotools environment. Another reason is you might mistakenly use a 
generated file on a target it was not configured for. When generated 
files are not created by configure the likely hood of that mistake 
increases because the distinction between a "distribution" file and a 
file designed to be generated on the target is lost (in this case the 
.in template is the distribution file).

Currently we create ipapython/services.py from  ipapython/services.py.in 
in the top level Makefile, not from configure. I believe that's a 
mistake,  ipapython/services.py is a file that depends on the target 
it's being built on and should be produced by configure. Also, it's 
generation is part of the 'version-update' target in the top level 
Makefile, but generating  ipapython/services.py is completely divorced 
from updating version information for the distribution snapshot, it 
logically doesn't belong there.

FWIW, I discovered this issue when trying to fix the build for the i18n 
validation which is now run with the lint check (because it too has 
dependencies on files generated by configure).

Some of our test/validation utilities (e.g. makeapi, make-lint, etc) 
locate all our source files. But if our set of source files is not 
complete until we generate them (or not syntactically complete until 
processed) then those validation steps will be incomplete or erroneous.

We shouldn't run the validation code until AFTER we've generated all our 
source files from their respective template files. Which means we 
shouldn't run them until after configure has executed, which means 
making bootstrap-autogen a dependency of the validation targets.

The only reason it's worked so far is because everything is dependent on 
updating the version information which as a side-effect is generating 
other source code files.

The problem manifests itself in other ways as well. Here is a more 
general explanation:

Many of our tools depend on knowing our set of source files. Some of our 
tools locate them by searching, some have a hardcoded enumerated list, 
and some derive the list by interrogating git.

Anything that depends on git fails after a distribution tarball is 
created because the git repo does not exist in the distribution. 
Anything that depends on search is probably O.K. when building in an 
antiseptic environment known to be clean (i.e. building an RPM) but is 
likely to fail in a development environment that picks up cruft along 
the way. Hardcoded lists fail when developers forget to update them.

I discovered this problem when I tried to move the i18n validation 
checks into the normal build process. It fails because it uses git to 
generate the list of files to validate. Perhaps a better approach would 
be to generate the file list using git and include the file list in the 
distribution.

Anyway, all of this is to say we need to be careful how and when we 
generate files as well as our dependencies on those generated files as 
well as what's in the distribution, where the build is occurring, and 
how, when and what we choose to validate.

-- 
John Dennis <jdennis at redhat.com>

Looking to carve out IT costs?
www.redhat.com/carveoutcosts/