plague 0.4 config file formats...

[Dan Williams]

On Tue, 2005-10-18 at 22:21 -0400, Chris Weyl wrote:
> Ok!  So, after a couple... intense;)... hours of hacking at my plague
> config files, here's what I've come up with for server/builder/target.
>  This is the result of both looking at the default configs that are
> generated, looking at the code, and repetitively starting the
> server/builder and waiting for it to complain when it doesn't find
> something it expects :)

Yeah, we should really package up default config files that show how to
do stuff.  I think Dennis Gilmore has done so for Sparc.

> However, it still doesn't build from CVS correctly.  So I include
> these files here in the hopes the error of my ways will be shown to
> me, and also so others may glean what they will from them :)  (server
> names have been replaced, so as to protect the innocent.)  To make
> things a little simpler while figuring it all out, I only have one
> target cfg defined:  fedora-3-i386-core.

Hmm, any errors in the logs or the emails that it sends when the build
fails?

> One generic question beforehand...  what's the appropriate way to
> specify multiple values on a line?  Whitespace separating the values,
> or a ', '?

Should be a ' ' since we're splitting the string on that character to
get the different values.

> Question:  Did I get the aliases/cvs_alias correct?  (Note that I
> setup the cvs repo to mirror that of fedora-extras, and to leverage
> the same Makefile.common, etc., system.)  I tried a couple for
> cvs_alias, to no avail.

The CVS alias causes the server to look for a certain directory rather
than the one the user specified.  So, if you enqueue a package for
fedora-4-core, obviously the name of the CVS directory in Fedora CVS
that corresponds here isn't fedora-4-core.  It's actually "FC-4".  So
the CVS alias allows you to specify an alternate naming for the target's
directory on the CVS server.

So rather than:
/cvs/foobar-package/fedora-4-core

the server would look, given a cvs_alias for this target, for:
/cvs/foobar-package/FC-4

> At this point, I can successfully enqueue package builds by cvstag
> (make tag ; make plague), but the build always fails, within seconds,
> without any error logging by either the builder or the server.

Something that might help is to turn on debugging info
in /usr/share/plague/server/PackageJob.py, near the top.  You'll see a
line that says "DEBUG = False".  Change that to "DEBUG = True" (note the
capital T) and you should get prints of all the commands the server is
running to check stuff out from CVS.

Here's a short walk through the CVS code in plague, hopefully that will
help us figure out what's going on for you...

    def _stage_checkout(self):
	<---snip--->

        # Set up CVS environment
        env_args = "CVSROOT='%s'" % self._target_cfg.get_str("CVS", "cvs_root")
        cvs_rsh = self._target_cfg.get_str("CVS", "cvs_rsh")
        if len(cvs_rsh) > 0:
            env_args = "%s CVS_RSH='%s'" % (env_args, cvs_rsh)

        # Checkout the module

<<< construct the checkout command.  first 'cd' to our temporary directory where
<<< the checkout will be put, then execute 'cvs' to check out the package.
<<< In Fedora CVS, each package is a module, under that module is contained the
<<< stuff for every release.  IE, 'plague' is the module, and the plague module
<<< has FC-3, FC-4, devel subdirectories.  So just checking out 'plague' module
<<< should be enough to get us the directory we want for our target.

        cmd = 'cd %s; %s %s co -r %s %s' % (self.checkout_tmpdir, env_args, CVS_CMD,
                self._source, self.package)
        debugprint("%d: Running %s" % (self.uid, cmd))
        s, o = commands.getstatusoutput(cmd)
        if s != 0:
            err_msg = "Error: could not check out %s from %s - output was:\n\n" \
                            "%s" % (self._source, self._target_str, o)
        else:

<<< The 'common' directory contains the base makefile with most of the magic.
<<< We make sure to check that directory out specifically (it's also part of
<<< the package's module) because we need it to do 'make srpm' later.

            # Just in case the 'common' directory didn't come along for the ride,
            # get it from CVS
            pkg_path = os.path.join(self.checkout_tmpdir, self.package)
            if not os.path.exists(os.path.join(pkg_path, "common")):
                cmd = 'cd %s; %s %s co common' % (pkg_path, env_args, CVS_CMD)
                debugprint("%d: Running %s" % (self.uid, cmd))
                s, o = commands.getstatusoutput(cmd)
                if s != 0:
                    err_msg = "Error: could not check out common directory - " \
                            "output was:\n\n%s" % (self._source, self._target_str, o)

	<---snip--->

    def _stage_make_srpm(self):
        # Map our target to the CVS target alias, since CVS may have
        # different target names than we expose
        cvs_target = self._target_dict['target']
        cvs_alias = self._target_cfg.get_str("Aliases", "cvs_alias")
        if len(cvs_alias) > 0:
            cvs_target = cvs_alias

<<< cd to the checkout directory for this target, for example
<<< /tmp/32-plague-0.4-3252352323552/plague/FC-4 .  Then run 'make srpm' in
<<< that directory.

        self.srpm_path = None
        srpm_dir = os.path.join(self.checkout_tmpdir, self.package, cvs_target)
        if not os.path.exists(srpm_dir):
            msg = "Error: could not find path %s for %s." % (srpm_dir, self._source)
            raise PrepError(msg)

        cmd = 'cd %s; %s srpm' % (srpm_dir, MAKE_CMD)
        debugprint("%d: Running %s in %s" % (self.uid, cmd, srpm_dir))
        s, o = commands.getstatusoutput(cmd)
        if s != 0:
            # Don't include download progress lines in output
            lines = o.split('\n')
            output_lines = []
            for line in lines:
                if line.find('..........') == -1 and len(line) > 0:
                    output_lines.append(line)
            o = string.join(output_lines, '\n')
            msg = "Error: could not make srpm for %s - output was:\n\n%s" % (self._source, o)
            raise PrepError(msg)
        
        srpmpath = None

<<< This relies on the output of 'make srpm' to tell us in its output
<<< where it put the SRPM it just made.  There's some magic here, namely
<<< we look for the line starting with "Wrote:" and assume the rest is the path
<<< to the SRPM

        for line in o.split("\n"):
            if line.startswith("Wrote:"):
                line.replace("\n", "")
                (garbage, path) = line.split(':')
                srpmpath = path.strip()
                break
        if not srpmpath:
            msg = "Error: could not find srpm for %s - output was:\n\n%s" % (self._source, o)
            raise PrepError(msg)

        self.srpm_path = srpmpath