build system glue scripts and requirements

[Jeremy Katz]

On Fri, 2005-03-18 at 01:52 -0500, seth vidal wrote:
>  So right now I think things with mach and yum are working for building
> fedora extras. The packages _seem_ like they're coming out right and
> things seem functional enough. 

Yep, has seemed pretty reasonable to me in the poking at it I've done so
far.  mschwendt's idea of running a lot of already built packages
through to see how they fare isn't a bad one.  I'll try to set up one of
my test boxes to do this over the course of this week (hopefully
tomorrow).  You don't have to worry about doing that right now :-)

> The second part that I need help and
> input on is the glue scripts and requirements for having automatic
> triggering of builds for packagers.
> 
> The questions I have:
>   1. if this is meant to run on the red hat boxes in the PHX coloc, what
> does that infrastructure look like? What features does it have? Can we
> assume all the build boxes have access to the cvs tree? Do we need to
> worry about pushing srpms around?

I can't answer authoritatively on what the set up is, but I have a
pretty good idea.  As it stands right now, the build boxes are all
x86_64 boxes with reasonable amounts of RAM and disk space.  They should
be able to access CVS (if not, that can be fixed) with the theory that
you want the buildsystem to be given an explicit tag to build and then
you check out off the tag, make the src.rpm and go.

We _do_ need to worry about how the binary packages end up.  I'm not
entirely sure what's best here.  Internally, we use writing to directory
trees that look like
  release/package-N/V-R/A
(N is name, V is version, R is release, A is arch)

Then, the trees which end up on the FTP site are composed out of these
directory trees.  This has the nice feature of making the inheritance of
builds from older releases a bit easier. 

What is slightly more complicated with a scheme like this is how do you
update the repodata after a build completes quickly with the new package
info.  Some of the "createrepo should be able to be done incrementally"
people will probably come back out of the woodwork.

>   2. How do folks want packagers to send notices about builds? Just a
> cvs tag? A webpage? A gpg-signed email with specific content? A custom
> xmpp/jabber-client to send a custom message to a listening build client
> across an xmpp infrastructure? :)

Magic occurring at cvs tag time is less than ideal.  People like it from
the "oh, it looks simple" perspective, but there's a lot of other
metadata that you sometimes need.  Having it be a web form that you give
all the appropriate info to seems reasonable to me.  Or XML-RPC.  Or
something like that.  Either gives you the ability to have a relatively
simple makefile target for a 'make build' with the appropriate info you
need to set[1].  This still all works off of a tag being made [2]

Then, after the build, we probably want to kick off a "build complete"
mail at least to the originator of the build.  Failed builds should
similarly get a mail and the build logs need to be able to be easily
accessible.  

One other thing that springs to mind is the question of what arches to
build packages for and whether a specific arch failing should block the
build.  My opinion, which matches how Core gets built, is that 
  * Packages get built for all Extras arches.  ExcludeArch/ExclusiveArch
can be used for the (rare) things which need otherwise
  * Build failures on one arch block all arches.  Otherwise, some arches
will fall far behind and things like prereqs get really painful

>   3. What things am I missing or not understanding about what is needed
> from the build system? The requirements I've been working under
> are/were:
>     - self hosting on Fedora Core
>     - not crazy
>   What else do I need to think about? 

Security is probably one.  Although "not crazy" probably covers
that.  :)

Otherwise, I can't think of big things which need thinking about.
Perhaps easy setup such that it can be included in the kickstart configs
for the build machines without any real difficulty.  But how things look
now doesn't seem bad for that.  

And the bit that we talked about during LinuxWorld on how we want to
make it possible and easy for developers to download the buildsystem and
get it going on their own workstation for test builds.  That then also
enables other third party repositories (there won't be only one :-)  to
use it and get some consistency.

>   4. Who else is interested in working on this and getting things
> progressing more? The yum changes to mach are just a hackjob to get a
> problem solved for the short term. However, I'd like to continue down
> this general line of development. so <buffy>Where do we go from
> here?</buffy>

I'm interested... let me see how much time I can commit.  I think where
we go from here is mostly trying to get some of the glue pieces
working.  

Jeremy

[1] Even if this requires writing a (simple-ish) python app to run for
kicking it off
[2] Yes, I know that the tags aren't being done right now.  I'll get
that added to Makefile.common tonight or in the morning.