[Freeipa-devel] Generating files from .in template files

[Petr Viktorin]

On 04/16/2012 10:38 PM, John Dennis wrote:
> Traditionally when using autotools it's job of the configure script to
> generate files from .in template files. The reason it's done this way is
> because configure has to probe the system it's running on to discover
> the correct substitutions to be used in the .in templates.
>
> In our top level Makefile we generate several files from .in templates
> under the "version-update" target. Most of this logic applies to
> inserting version information into specific files. In the context of
> release specific data this is probably a valid methodology although
> other projects have incorporated the version munging into configure. For
> version strings I don't think there is necessarily a correct approach
> (e.g. part of configure or external to configure). Technically updating
> the version strings is part of "make dist" step, i.e. the act of
> creating the distribution, which is independent of configure.
>
> However, any template file that depends on the target system where
> configure is being run should be created by configure. Why? Because
> that's the design of autotools and how people expect things will work in
> an autotools environment. Another reason is you might mistakenly use a
> generated file on a target it was not configured for. When generated
> files are not created by configure the likely hood of that mistake
> increases because the distinction between a "distribution" file and a
> file designed to be generated on the target is lost (in this case the
> .in template is the distribution file).
>
> Currently we create ipapython/services.py from ipapython/services.py.in
> in the top level Makefile, not from configure. I believe that's a
> mistake, ipapython/services.py is a file that depends on the target it's
> being built on and should be produced by configure. Also, it's
> generation is part of the 'version-update' target in the top level
> Makefile, but generating ipapython/services.py is completely divorced
> from updating version information for the distribution snapshot, it
> logically doesn't belong there.
>
> FWIW, I discovered this issue when trying to fix the build for the i18n
> validation which is now run with the lint check (because it too has
> dependencies on files generated by configure).
>
> Some of our test/validation utilities (e.g. makeapi, make-lint, etc)
> locate all our source files. But if our set of source files is not
> complete until we generate them (or not syntactically complete until
> processed) then those validation steps will be incomplete or erroneous.
>
> We shouldn't run the validation code until AFTER we've generated all our
> source files from their respective template files. Which means we
> shouldn't run them until after configure has executed, which means
> making bootstrap-autogen a dependency of the validation targets.
>
> The only reason it's worked so far is because everything is dependent on
> updating the version information which as a side-effect is generating
> other source code files.
>
> The problem manifests itself in other ways as well. Here is a more
> general explanation:
>
> Many of our tools depend on knowing our set of source files. Some of our
> tools locate them by searching, some have a hardcoded enumerated list,
> and some derive the list by interrogating git.
>
> Anything that depends on git fails after a distribution tarball is
> created because the git repo does not exist in the distribution.
> Anything that depends on search is probably O.K. when building in an
> antiseptic environment known to be clean (i.e. building an RPM) but is
> likely to fail in a development environment that picks up cruft along
> the way. Hardcoded lists fail when developers forget to update them.
>
> I discovered this problem when I tried to move the i18n validation
> checks into the normal build process. It fails because it uses git to
> generate the list of files to validate. Perhaps a better approach would
> be to generate the file list using git and include the file list in the
> distribution.
>
> Anyway, all of this is to say we need to be careful how and when we
> generate files as well as our dependencies on those generated files as
> well as what's in the distribution, where the build is occurring, and
> how, when and what we choose to validate.
>

I'll add my own gripe: we only run autogen.sh in a directory (install, 
ipa-client, daemons) if there is no Makefile in that directory. This 
means that if e.g. install/Makefile.in changes, make doesn't rebuild 
install/Makefile. Is that standard autoconf practice?

-- 
Petr³