From symbiont at berlios.de Sat Oct 1 01:23:18 2005 From: symbiont at berlios.de (Jeff Pitman) Date: Sat, 1 Oct 2005 09:23:18 +0800 Subject: Cyclic Deps in Mock Message-ID: <200510010923.19145.symbiont@berlios.de> Anyone deal with builds between to SRPMS that require each other? I am working on "aspell" and "aspell-en". aspell Requires aspell-en. aspell-en BuildRequires aspell. Fun, eh? I had to work around with a: $ rpm -ivh --nodeps aspell*rpm in the chroot. -- -jeff From ivazquez at ivazquez.net Sat Oct 1 01:29:51 2005 From: ivazquez at ivazquez.net (Ignacio Vazquez-Abrams) Date: Fri, 30 Sep 2005 21:29:51 -0400 Subject: Cyclic Deps in Mock In-Reply-To: <200510010923.19145.symbiont@berlios.de> References: <200510010923.19145.symbiont@berlios.de> Message-ID: <1128130191.9641.3.camel@ignacio.lan> On Sat, 2005-10-01 at 09:23 +0800, Jeff Pitman wrote: > aspell Requires aspell-en. aspell-en BuildRequires aspell. Does aspell actually *require* aspell-en or does it just provide a convenient starting point? -- Ignacio Vazquez-Abrams http://fedora.ivazquez.net/ gpg --keyserver hkp://subkeys.pgp.net --recv-key 38028b72 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From symbiont at berlios.de Sat Oct 1 01:37:19 2005 From: symbiont at berlios.de (Jeff Pitman) Date: Sat, 1 Oct 2005 09:37:19 +0800 Subject: Cyclic Deps in Mock In-Reply-To: <1128130191.9641.3.camel@ignacio.lan> References: <200510010923.19145.symbiont@berlios.de> <1128130191.9641.3.camel@ignacio.lan> Message-ID: <200510010937.19404.symbiont@berlios.de> On Saturday 01 October 2005 09:29, Ignacio Vazquez-Abrams wrote: > On Sat, 2005-10-01 at 09:23 +0800, Jeff Pitman wrote: > > aspell Requires aspell-en. ?aspell-en BuildRequires aspell. ? > > Does aspell actually *require* aspell-en or does it just provide a > convenient starting point? === Patch2: aspell-0.60.3-long_gettext.patch Patch3: aspell-0.60.3-install_info.patch Requires: aspell-en <------- It requires it. BuildRoot: %{_tmppath}/%{name}-%{version}-root BuildRequires: gettext, ncurses-devel === -- -jeff From ivazquez at ivazquez.net Sat Oct 1 01:51:43 2005 From: ivazquez at ivazquez.net (Ignacio Vazquez-Abrams) Date: Fri, 30 Sep 2005 21:51:43 -0400 Subject: Cyclic Deps in Mock In-Reply-To: <200510010937.19404.symbiont@berlios.de> References: <200510010923.19145.symbiont@berlios.de> <1128130191.9641.3.camel@ignacio.lan> <200510010937.19404.symbiont@berlios.de> Message-ID: <1128131503.9641.5.camel@ignacio.lan> On Sat, 2005-10-01 at 09:37 +0800, Jeff Pitman wrote: > On Saturday 01 October 2005 09:29, Ignacio Vazquez-Abrams wrote: > > On Sat, 2005-10-01 at 09:23 +0800, Jeff Pitman wrote: > > > aspell Requires aspell-en. aspell-en BuildRequires aspell. > > > > Does aspell actually *require* aspell-en or does it just provide a > > convenient starting point? > > === > Patch2: aspell-0.60.3-long_gettext.patch > Patch3: aspell-0.60.3-install_info.patch > Requires: aspell-en <------- It requires it. > BuildRoot: %{_tmppath}/%{name}-%{version}-root > BuildRequires: gettext, ncurses-devel > === No, that says that it's in the Requires for the package. There's a fundamental difference between being in Requires and being required for normal operation. -- Ignacio Vazquez-Abrams http://fedora.ivazquez.net/ gpg --keyserver hkp://subkeys.pgp.net --recv-key 38028b72 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From symbiont at berlios.de Sat Oct 1 01:56:13 2005 From: symbiont at berlios.de (Jeff Pitman) Date: Sat, 1 Oct 2005 09:56:13 +0800 Subject: Cyclic Deps in Mock In-Reply-To: <1128131503.9641.5.camel@ignacio.lan> References: <200510010923.19145.symbiont@berlios.de> <200510010937.19404.symbiont@berlios.de> <1128131503.9641.5.camel@ignacio.lan> Message-ID: <200510010956.14377.symbiont@berlios.de> On Saturday 01 October 2005 09:51, Ignacio Vazquez-Abrams wrote: > No, that says that it's in the Requires for the package. There's a > fundamental difference between being in Requires and being required > for normal operation. I have no intention of changing aspell from fedora devel. Any other ideas? -- -jeff From skvidal at phy.duke.edu Fri Oct 7 05:42:12 2005 From: skvidal at phy.duke.edu (seth vidal) Date: Fri, 07 Oct 2005 01:42:12 -0400 Subject: Once again: kernel-modules in the buildsystem In-Reply-To: <1127940624.24429.78.camel@dcbw.boston.redhat.com> References: <1127913625.3098.12.camel@localhost.localdomain> <1127940624.24429.78.camel@dcbw.boston.redhat.com> Message-ID: <1128663732.22419.54.camel@cutter> On Wed, 2005-09-28 at 16:50 -0400, Dan Williams wrote: > On Wed, 2005-09-28 at 15:20 +0200, Thorsten Leemhuis wrote: > > Dan, if something like that goes into mock -- what would be the best way > > to make this new mock-option usable in plague for kernel-modules in > > fedora-extras? Should plague automatically build the modules for all > > kernels available (the one that was shipped and those currently in > > updates-released)? Or only for the newest kernel available? Or should > > the packager say something like "make plague KERNEL=put-kernel-ver-here" > > when requesting build of a fdr-extras-kernel-module-pkg? > > I recognize the need for this, there are a few questions that need > answering here though. First, methods. > > 1) have plague automatically build for all kernels > > Requires a large amount of work with yum to essentially list all > kernel-* packages in a given repository, then queue up builds for them. > That's a chunk of work. > > 2) have plague build for latest kernel > > Still requires work to ask yum about a certain repository to get latest > kernel package. Slightly easier, but not very much so. Given that we > are not guaranteed that the kernel-* package will be in the server's > repository (it might be in the upstream "official" mirror instead) we > pretty much have to use yum. > > 3) have the packager do it > > Pretty much the status quo right now, not much of a win IMHO. > > > In any case, you're going to have to get some form of argument passing > through Seth and into mock. I don't like the "allow packager to pass > random args" thing, since that's a direct impact on the security of the > build system. In the short term though, limiting such things to > automatic rebuilds of kernel modules seems sane. > > Ideally, yes, plague should be doing some depsolving on the server side. > That's going to take a while to build in. In the mean time, we could > build in some kernel-module specific stuff, I'm not against that. Most > likely post 0.4.x work (1). > > Some nice things that could go along with depsolving in the server: > > - Store mock configs on the server and push them out to builders > on-demand. Bonus: we use same yum configs on server & builders. > > - Don't build packages until their deps are satisfied, waiting a max of > 8 or 10 hours before failing the package. > > Dan > > (1) That doesn't mean "a year from now" and work could start on that > fairly soon after 0.4.x comes out, which is "in a few weeks" sorry for being away for so long. 1. passing in kernel args is fine - but we're going to need to add a way to vet them out to make sure someone doesn't play silly buggers with the command line args passed to rpmbuild (ex: rpmbuild --define 'kernelver=something'; rm -rf /) as a dumb example. That's an obviously malicious example but I'm more concerned about the just silly mistakes that could hose up the buildsystem. 2. I don't like the idea of arbitrary defines being passed in for the buildsystem - it makes for hard-to-replicate builds, I think. I know a number of folks agree with me about that, too. With regard to the kernels I think we could reasonably write a couple of simple yum-utils that plague could use to query the chroot to find out what kernel to build for. It wouldn't need to run as root once the chroot was setup b/c the metadata would all be there. So maybe this is a middle point but: mock --defines=/path/to/some/file --other-mock-options-here so then plague could just provide a defines file and mock would read those in and add them to the rpm defines in the .rpmmacros in the chrootbuilders homedir. Or does that sound silly? -sv From dcbw at redhat.com Fri Oct 7 10:29:34 2005 From: dcbw at redhat.com (Dan Williams) Date: Fri, 07 Oct 2005 06:29:34 -0400 Subject: Once again: kernel-modules in the buildsystem In-Reply-To: <1128663732.22419.54.camel@cutter> References: <1127913625.3098.12.camel@localhost.localdomain> <1127940624.24429.78.camel@dcbw.boston.redhat.com> <1128663732.22419.54.camel@cutter> Message-ID: <1128680975.2781.1.camel@localhost.localdomain> On Fri, 2005-10-07 at 01:42 -0400, seth vidal wrote: > So maybe this is a middle point but: > > mock --defines=/path/to/some/file --other-mock-options-here > > so then plague could just provide a defines file and mock would read > those in and add them to the rpm defines in the .rpmmacros in the > chrootbuilders homedir. > > Or does that sound silly? This was really the only thing I could think of to get around the malicious defines thing too... We avoid a _ton_ of command line quoting/scrubbing doing it this way. Dan From fedora at leemhuis.info Fri Oct 7 13:26:13 2005 From: fedora at leemhuis.info (Thorsten Leemhuis) Date: Fri, 07 Oct 2005 15:26:13 +0200 Subject: Once again: kernel-modules in the buildsystem In-Reply-To: <1128663732.22419.54.camel@cutter> References: <1127913625.3098.12.camel@localhost.localdomain> <1127940624.24429.78.camel@dcbw.boston.redhat.com> <1128663732.22419.54.camel@cutter> Message-ID: <1128691573.3020.20.camel@localhost.localdomain> Am Freitag, den 07.10.2005, 01:42 -0400 schrieb seth vidal: > On Wed, 2005-09-28 at 16:50 -0400, Dan Williams wrote: > > On Wed, 2005-09-28 at 15:20 +0200, Thorsten Leemhuis wrote: > > > Dan, if something like that goes into mock -- what would be the best way > > > to make this new mock-option usable in plague for kernel-modules in > > > fedora-extras? Should plague automatically build the modules for all > > > kernels available (the one that was shipped and those currently in > > > updates-released)? Or only for the newest kernel available? Or should > > > the packager say something like "make plague KERNEL=put-kernel-ver-here" > > > when requesting build of a fdr-extras-kernel-module-pkg? > > > > I recognize the need for this, there are a few questions that need > > answering here though. First, methods. > > > > 1) have plague automatically build for all kernels > > > > Requires a large amount of work with yum to essentially list all > > kernel-* packages in a given repository, then queue up builds for them. > > That's a chunk of work. > > > > 2) have plague build for latest kernel > > > > Still requires work to ask yum about a certain repository to get latest > > kernel package. Slightly easier, but not very much so. Given that we > > are not guaranteed that the kernel-* package will be in the server's > > repository (it might be in the upstream "official" mirror instead) we > > pretty much have to use yum. > > > > 3) have the packager do it > > > > Pretty much the status quo right now, not much of a win IMHO. > > > > > > In any case, you're going to have to get some form of argument passing > > through Seth and into mock. I don't like the "allow packager to pass > > random args" thing, since that's a direct impact on the security of the > > build system. In the short term though, limiting such things to > > automatic rebuilds of kernel modules seems sane. > > > > Ideally, yes, plague should be doing some depsolving on the server side. > > That's going to take a while to build in. In the mean time, we could > > build in some kernel-module specific stuff, I'm not against that. Most > > likely post 0.4.x work (1). > > > > Some nice things that could go along with depsolving in the server: > > > > - Store mock configs on the server and push them out to builders > > on-demand. Bonus: we use same yum configs on server & builders. > > > > - Don't build packages until their deps are satisfied, waiting a max of > > 8 or 10 hours before failing the package. > > > > Dan > > > > (1) That doesn't mean "a year from now" and work could start on that > > fairly soon after 0.4.x comes out, which is "in a few weeks" > > sorry for being away for so long. np > 1. passing in kernel args is fine - but we're going to need to add a way > to vet them out to make sure someone doesn't play silly buggers with the > command line args passed to rpmbuild (ex: rpmbuild --define > 'kernelver=something'; rm -rf /) as a dumb example. That's an obviously > malicious example but I'm more concerned about the just silly mistakes > that could hose up the buildsystem. Sound reasonable > 2. I don't like the idea of arbitrary defines being passed in for the > buildsystem - it makes for hard-to-replicate builds, I think. Is there really a big difference if the define is passed via the command line or via a defines file when it comes to "hard-to-replicate builds"? > I know a > number of folks agree with me about that, too. With regard to the > kernels I think we could reasonably write a couple of simple yum-utils > that plague could use to query the chroot to find out what kernel to > build for. Questions 1 and 2 from the initial mail (quoted above) remain here: Build only for the latest kernel, for the initial shipped and the latest kernel or for all available? If one of the last two we might need a mechanism to exclude some kernels if they are know to fail build (or plague needs to ignore if a build fails). And what about kernels from updates-testing? Should we build for those, too, and but them in extras/testing while that kernel is in updates-testing? > It wouldn't need to run as root once the chroot was setup b/c > the metadata would all be there. > > So maybe this is a middle point but: > > mock --defines=/path/to/some/file --other-mock-options-here > > so then plague could just provide a defines file and mock would read > those in and add them to the rpm defines in the .rpmmacros in the > chrootbuilders homedir. > > Or does that sound silly? If you and Dan think doing it this way it's okay for me -- I don't really care how exactly the defines gets to the rpmbuild call. -- Thorsten Leemhuis From symbiont at berlios.de Sat Oct 8 03:55:40 2005 From: symbiont at berlios.de (Jeff Pitman) Date: Sat, 8 Oct 2005 11:55:40 +0800 Subject: Mock Patch to Enable Realtime Output of logs Message-ID: <200510081155.41263.symbiont@berlios.de> Hi: This patch allows one to run "tail -f" on the build.log. It mostly re-uses existing variables and things like resultdir, etc. work properly. Anyway, I hope to get this committed as this is a massive improvement from what mock had inherited. (Multiple mount/umount in root.log, but I don't think that many people care. It's the simplest patch without taking in account of corner cases, etc.) -- -jeff -------------- next part -------------- A non-text attachment was scrubbed... Name: mock.py-pipe.patch Type: text/x-diff Size: 4869 bytes Desc: not available URL: From dcbw at redhat.com Mon Oct 10 13:51:20 2005 From: dcbw at redhat.com (Dan Williams) Date: Mon, 10 Oct 2005 09:51:20 -0400 Subject: plague 0.4 Message-ID: <1128952280.9096.24.camel@localhost.localdomain> Hi, So 0.4/HEAD has built about 500 packages so far without huge problems on my Sparc cluster. Dennis has been building Fedora Extras for Sparc for a while with 0.4/HEAD as well. I think it's pretty solid right now, so the time has come to think about updating the Extras buildsystem to 0.4. One of the larger improvements has been the ability to use real databases, like Postgres, as the job database rather than SQLite. We've all seen the issues with SQLite, which doesn't play very well in a multithreaded situation. Some of that can painfully fixed, but the fact remains that SQLite just doesn't have the features that we really want out of it (ie, row locking). SQLite can still be used in a smaller buildsystem though, where it works just fine and doesn't require any setup. Seth: Jeremy thought you'd be quite averse to using Postgres... Not sure what your feelings here really are. I've noticed quite a big improvement by switching to Postgres when rebuilding an entire distro, namely in the buildsystem not falling over due to database contention. If there's not much else to fix up, I'll push out a plague 0.4 release this week and we can work on migrating the current Extras boxes to it. Dan From skvidal at phy.duke.edu Mon Oct 10 14:00:18 2005 From: skvidal at phy.duke.edu (seth vidal) Date: Mon, 10 Oct 2005 10:00:18 -0400 Subject: plague 0.4 In-Reply-To: <1128952280.9096.24.camel@localhost.localdomain> References: <1128952280.9096.24.camel@localhost.localdomain> Message-ID: <1128952818.9847.6.camel@cutter> On Mon, 2005-10-10 at 09:51 -0400, Dan Williams wrote: > Hi, > > So 0.4/HEAD has built about 500 packages so far without huge problems on > my Sparc cluster. Dennis has been building Fedora Extras for Sparc for > a while with 0.4/HEAD as well. I think it's pretty solid right now, so > the time has come to think about updating the Extras buildsystem to 0.4. > > One of the larger improvements has been the ability to use real > databases, like Postgres, as the job database rather than SQLite. We've > all seen the issues with SQLite, which doesn't play very well in a > multithreaded situation. Some of that can painfully fixed, but the fact > remains that SQLite just doesn't have the features that we really want > out of it (ie, row locking). SQLite can still be used in a smaller > buildsystem though, where it works just fine and doesn't require any > setup. > > Seth: Jeremy thought you'd be quite averse to using Postgres... Not > sure what your feelings here really are. I've noticed quite a big > improvement by switching to Postgres when rebuilding an entire distro, > namely in the buildsystem not falling over due to database contention. I'm averse to all change. Everyone knows that. From where will the postgres database need to be accessed? There shouldn't be any reason I can think of why anything outside of the local machine would need access to the db. Does plague make the databases/tables on the fly? I'm curious how much other infrastructure, backups and maintenance will be required for the postgres system, mostly. other than that, no - I've no objection to postgres. -sv From dcbw at redhat.com Mon Oct 10 15:10:56 2005 From: dcbw at redhat.com (Dan Williams) Date: Mon, 10 Oct 2005 11:10:56 -0400 Subject: plague 0.4 In-Reply-To: <1128952818.9847.6.camel@cutter> References: <1128952280.9096.24.camel@localhost.localdomain> <1128952818.9847.6.camel@cutter> Message-ID: <1128957057.18590.19.camel@localhost.localdomain> On Mon, 2005-10-10 at 10:00 -0400, seth vidal wrote: > On Mon, 2005-10-10 at 09:51 -0400, Dan Williams wrote: > > Seth: Jeremy thought you'd be quite averse to using Postgres... Not > > sure what your feelings here really are. I've noticed quite a big > > improvement by switching to Postgres when rebuilding an entire distro, > > namely in the buildsystem not falling over due to database contention. > > I'm averse to all change. Everyone knows that. From where will the > postgres database need to be accessed? There shouldn't be any reason I > can think of why anything outside of the local machine would need access > to the db. Correct, only the machine running the plague server needs access to the database. So of course that can be either user-level access (ie, only 'root' can access the job database since that's what the server runs as) or host-level access (ie, only 127.0.0.1 can access the job database). > Does plague make the databases/tables on the fly? I'm curious how much > other infrastructure, backups and maintenance will be required for the > postgres system, mostly. Yes, same as before. As long as you've at least created the database (using something like postgres 'createdb' utility) then plague will take care of setting up the tables automatically, as with SQLite. Postgres install would go like this: 1) yum install postgresql postgresql-server postgresql-python 2) Modify your /var/lib/pgsql/data/pg_hba.conf file to allow local users to connect from 127.0.0.1, mapping local username to database username. Note that this lets any local user connect, so DB permissions have to reject modification attempts from users other than 'root' and the Postgres admin user. Obviously permissions can be tightened, exercise for the reader, blah blah blah. # Allow IPv4 local connections: host all all 127.0.0.1/32 trust sameuser 3) sudo -u postgres createdb fe-build-db 4) sudo -u postgres createuser --no-adduser --no-createdb root 5) /sbin/service postgresql start 6) Modify your /etc/plague/server/plague-server.cfg file: [Database] engine = pgdb [pgdb Engine] host = localhost database = fe-build-db user = root password = 7) start plague. It will yell if something is wrong. I wouldn't think we'd need backups that much right now. The DB is pretty much just a history of what people built what where. It's not that bad if the buildsystem starts fresh when it comes back after its machine melts down. Maintenance is essentially making sure that the postgresql service starts up before plague does. If postgres goes down, things need to be kicked all around like happens now when the sqlite db locks. Note that sqlite is still required for the User DB. Since that's not really modified all that often, its quite acceptable for it to still be an sqlite database since we won't run into locking issues there. At least, we haven't yet. Dan From chris.weyl at gmail.com Wed Oct 12 14:38:07 2005 From: chris.weyl at gmail.com (Chris Weyl) Date: Wed, 12 Oct 2005 10:38:07 -0400 Subject: Plague & CVS... Message-ID: <7dd7ab490510120738n6d974e29x107a2242897e5a15@mail.gmail.com> Hi all! I'm working on using plague as the buildsystem for a project I'm working on right now. Its distributed approach and use of CVS makes it ideal, given the number of people and packages we're working with right now. However, while I have the buildsystem itself (mostly ;)) setup (server, builder, clients), I'm having trouble implementing the CVS integration. It looks as if that half of the buildsystem hasn't been released yet? Plague seems to depend on specific tags being generated, as well as a specific lookaside-cache system. While I've spent a few hours digging through how the f-e CVS repository is setup, I'm still somewhat confused as to: * what tags are generated when * how the tags are generated and used * where the lookaside / cvs maintenance code all is * what license the lookaside / cvs / "common" module code is under, and if I can use it w/o impacting my project's packages licenses I'd like to be able to use the CVS integration; it would be a great boon to the reliability and usability of plague for my project. Even if I just had a clearer understanding of how plague interacts with CVS & lookaside I could hopefully rig something... But ideally the complementary code could just be used. Hopefully, I'm just pulling a "Monday" and all this is out there somewhere. Finally, but certainly not lastly, a major thank you to all who have worked on this and contributed time and code... Plauge is excellent, and I can't wait to begin using it in "production". :) Thanks- -Chris From chris.weyl at gmail.com Wed Oct 12 17:02:38 2005 From: chris.weyl at gmail.com (Chris Weyl) Date: Wed, 12 Oct 2005 13:02:38 -0400 Subject: plague server / createrepos error? In-Reply-To: <7dd7ab490510120900j37c24233kb7f4241b86cad71b@mail.gmail.com> References: <7dd7ab490510120900j37c24233kb7f4241b86cad71b@mail.gmail.com> Message-ID: <7dd7ab490510121002n625a52c6l758e9780b55b7bf0@mail.gmail.com> On 10/12/05, Chris Weyl wrote: > Hey all-- > > A quick question. After building a srpm (successfully), I get: To answer my own post :), upgrading to createrepo 4.3 resolves. Bugzilla bug opened: https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=170531 -Chris From chris.weyl at gmail.com Wed Oct 12 16:00:16 2005 From: chris.weyl at gmail.com (Chris Weyl) Date: Wed, 12 Oct 2005 12:00:16 -0400 Subject: plague server / createrepos error? Message-ID: <7dd7ab490510120900j37c24233kb7f4241b86cad71b@mail.gmail.com> Hey all-- A quick question. After building a srpm (successfully), I get: 3 (meanwhile/i386): Build result files - [ 'meanwhile-0.4.0-0.i386.rpm', 'meanwhile-0.4.0-0.src.rpm', 'build.log', 'root.log', 'meanwhile-debuginfo-0.4.0-0.i386.rpm', 'builder.log', 'mockconfig.log', 'meanwhile-devel-0.4.0-0.i386.rpm' ] Repo 'fc3': updating repository metadata... Error: createrepo failed with exit status 256! Output: 'Options Error: option -c not recognized. createrepo [options] directory-of-packages Options: -u, --baseurl = optional base url location for all files -x, --exclude = files globs to exclude, can be specified multiple times -q, --quiet = run quietly -g, --groupfile to point to for group information (precreated) -v, --verbose = run verbosely -s, --checksum = md5 or sha - select type of checksum to use (default: sha) -h, --help = show this help -V, --version = output version -p, --pretty = output xml files in pretty format. ' Repo 'fc3': Done updating. 3 (meanwhile): Job finished. Poking around google, it looks like the -c is a checksum cache option[1], but the current level of createrepo in fc4 doesn't have it. What's the best way to work around this? Fudge Repo.py and chip out the checksum cache until an updated createrepo package is released, or find an updated createrepos package out there somewhere? It sounds like the cache option "makes a huge difference in repo creation time", so I'd far prefer to use it. Thanks:) -Chris [1] https://lists.dulug.duke.edu/pipermail/rpm-metadata/2005-July/000532.html From dcbw at redhat.com Fri Oct 14 14:14:10 2005 From: dcbw at redhat.com (Dan Williams) Date: Fri, 14 Oct 2005 10:14:10 -0400 Subject: Plague & CVS... In-Reply-To: <7dd7ab490510120738n6d974e29x107a2242897e5a15@mail.gmail.com> References: <7dd7ab490510120738n6d974e29x107a2242897e5a15@mail.gmail.com> Message-ID: <1129299250.2569.42.camel@dhcp83-40.boston.redhat.com> On Wed, 2005-10-12 at 10:38 -0400, Chris Weyl wrote: > However, while I have the buildsystem itself (mostly ;)) setup > (server, builder, clients), I'm having trouble implementing the CVS > integration. It looks as if that half of the buildsystem hasn't been > released yet? The current CVS code is quite tailored for the Fedora Core/Extras CVS repository setup. Here you have a layout like this: /cvsroot/plague/FC-3/ /cvsroot/plague/FC-4/ /cvsroot/plague/devel/ in other words, you have ////. The CVS code is set up to expect that right now. Furthermore, once the path to the package is determined, there is a set of commands to take the files in that directory above and make an SRPM for them. Mainly, we run the command 'make srpm' in the above directory and hopefully we get an SRPM out the other end, which is fed to the build slaves. > * what tags are generated when They are generated by packagers, who must tag the files in CVS before submitting a build. The build always uses the CVS tag given by the packager in the enqueue request. The packager runs 'make tag' which constructs a tag based on the RPM's NVR (name, version, release), then runs 'make plague' which executes plague-client to enqueue the package. If you forget to 'make tag' then the build will fail and you'll get a nasty email. > * how the tags are generated and used There's some magic in the 'common' directory's Makefile that extracts the tag from the specfile based on NVR of the RPM. > * where the lookaside / cvs maintenance code all is The lookaside cache of sources is essentially an HTTP server. wget is used to grab the files based on the 'sources' file in the packages directory. Most of the "real" code is in the common module for each package. > * what license the lookaside / cvs / "common" module code is under, > and if I can use it w/o impacting my project's packages licenses I believe you can do so, though I'm not a lawyer. The license of the code in the Makefiles and such should not impact the license of your source and/or product, since you do not link any of the buildsystem/infrastructure code into your sources (hopefully). If the license of the infrastructure is under the GPL (which plague is) then your obligation is simply to make changes to the infrastructure and plague code available if you modify them. > I'd like to be able to use the CVS integration; it would be a great > boon to the reliability and usability of plague for my project. Even > if I just had a clearer understanding of how plague interacts with CVS > & lookaside I could hopefully rig something... But ideally the > complementary code could just be used. Hopefully, I'm just pulling a > "Monday" and all this is out there somewhere. In all fairness, the general idea is to make more "pluggable" backends for the RCS stuff, so that you could potentially add subversion/git/arch/your-favorite-RCS-system-here support. All that plague _really_ needs out of the RCS backend is the path to an SRPM on the local file system, and it doesn't care where it gets that SRPM path from. At a minimum, the inputs to the RCS backend would generally be target specification (distro, repo, target) as well as the user-determined 'source' field. This 'source' field can be anything; current Fedora Extras uses it for CVS tags. It's specified by the user when the enqueue the package. What the backend does with this information isn't really a concern of plague as long as the path to the SRPM comes out the other end. So, if you've currently got a certain layout for CVS, I'd like to know more to figure out how to generalize the current CVS code in plague. In the end you'll have to do a bit of tweaking to get your own CVS stuff to work, if you choose not to adopt the infrastructure that Fedora CVS uses. But it shouldn't be very hard at all. I just need a bit more information first. > Finally, but certainly not lastly, a major thank you to all who have > worked on this and contributed time and code... Plauge is excellent, > and I can't wait to begin using it in "production". :) Glad it works, hope you're using 0.4/HEAD too :) Dan From dcbw at redhat.com Mon Oct 17 02:27:28 2005 From: dcbw at redhat.com (Dan Williams) Date: Sun, 16 Oct 2005 22:27:28 -0400 Subject: Extras build system maintenance Monday 10/17 Message-ID: <1129516048.16144.14.camel@localhost.localdomain> Hi all, We'll be switching to the latest build system code on Monday morning. This should resolve most of the random kicks that have been needed lately due to locking issues with sqlite. It will also improve stability of the build slaves themselves. I expect to bring the build system down around 10 or 11 AM US Eastern Time. It will likely be down until mid-afternoon, probably about 3 or 4 hours total. When everything is back up and working, we'll send out a mail saying so. When it comes back up, you'll need to upgrade your plague-client and plague-common packages with yum. Thanks for the patience, Dan From dcbw at redhat.com Mon Oct 17 18:08:57 2005 From: dcbw at redhat.com (Dan Williams) Date: Mon, 17 Oct 2005 14:08:57 -0400 Subject: Extras build system back up Message-ID: <1129572538.28887.1.camel@dhcp83-40.boston.redhat.com> Hi, We're back up and using the latest plague code. yum update your plague-client and you should be good to go. Dan From dcbw at redhat.com Mon Oct 17 21:15:51 2005 From: dcbw at redhat.com (Dan Williams) Date: Mon, 17 Oct 2005 17:15:51 -0400 Subject: Extras build system back up In-Reply-To: <20051017201619.GB30904@shell0.pdx.osdl.net> References: <1129572538.28887.1.camel@dhcp83-40.boston.redhat.com> <20051017201619.GB30904@shell0.pdx.osdl.net> Message-ID: <1129583751.1418.3.camel@dhcp83-40.boston.redhat.com> On Mon, 2005-10-17 at 13:16 -0700, Chris Wright wrote: > * Dan Williams (dcbw at redhat.com) wrote: > > We're back up and using the latest plague code. yum update your > > plague-client and you should be good to go. > > Anything else that needs to be done? > > w at vas devel]$ plague-client build git-core git-core-0_99_8d-1_fc5 devel > Traceback (most recent call last): > File "/usr/bin/plague-client", line 426, in ? > cli.dispatch(cmd, sys.argv[2:]) > File "/usr/bin/plague-client", line 372, in dispatch > func(args) > File "/usr/bin/plague-client", line 172, in _cmd_build > self._enqueue(is_srpm, package, source, target_alias) > File "/usr/bin/plague-client", line 197, in _enqueue > if self._cfg.get_bool("Server", "allow_uploads"): > File "/usr/lib/python2.3/site-packages/plague/BaseConfig.py", line 51, in get_bool > opt = self.get_option(section, name) > File "/usr/lib/python2.3/site-packages/plague/BaseConfig.py", line 45, in get_option > raise ConfigError("Config file %s does not have option: %s/%s" % (self._filename, section, name)) > plague.BaseConfig.ConfigError: Config file /home/chrisw/.plague-client.cfg does not have option: Server/allow_uploads > [chrisw at vas devel]$ vim /home/chrisw/.plague-client.cfg > [chrisw at vas devel]$ plague-client build git-core git-core-0_99_8d-1_fc5 devel > Error connecting to build server: '' As I see you've figured out, it should be fixed now. Server-side issue with some database fields being way too small. They are larger now :) Dan From chris.weyl at gmail.com Mon Oct 17 23:40:02 2005 From: chris.weyl at gmail.com (Chris Weyl) Date: Mon, 17 Oct 2005 19:40:02 -0400 Subject: Plague & CVS... In-Reply-To: <1129299250.2569.42.camel@dhcp83-40.boston.redhat.com> References: <7dd7ab490510120738n6d974e29x107a2242897e5a15@mail.gmail.com> <1129299250.2569.42.camel@dhcp83-40.boston.redhat.com> Message-ID: <7dd7ab490510171640l6def9dccp4b7b19d0252b62fd@mail.gmail.com> On 10/14/05, Dan Williams wrote: > The current CVS code is quite tailored for the Fedora Core/Extras CVS > repository setup. Here you have a layout like this: > > /cvsroot/plague/FC-3/ > /cvsroot/plague/FC-4/ > /cvsroot/plague/devel/ > > in other words, you have ////. The CVS code is > set up to expect that right now. > > Furthermore, once the path to the package is determined, there is a set > of commands to take the files in that directory above and make an SRPM > for them. Mainly, we run the command 'make srpm' in the above directory > and hopefully we get an SRPM out the other end, which is fed to the > build slaves. Gotcha. I'm starting to make sense of Makefile.common, etc now. [...snip...] > > * where the lookaside / cvs maintenance code all is > > The lookaside cache of sources is essentially an HTTP server. wget is > used to grab the files based on the 'sources' file in the packages > directory. Most of the "real" code is in the common module for each > package. Gotcha. I essentially faked this out, so for read operations the implementation I'm using operates identically to the one f-e is using. > > * what license the lookaside / cvs / "common" module code is under, > > and if I can use it w/o impacting my project's packages licenses > > I believe you can do so, though I'm not a lawyer. The license of the > code in the Makefiles and such should not impact the license of your > source and/or product, since you do not link any of the > buildsystem/infrastructure code into your sources (hopefully). If the > license of the infrastructure is under the GPL (which plague is) then > your obligation is simply to make changes to the infrastructure and > plague code available if you modify them. After my last cvs update, I see it's under the BSD license. Cool :) But you're right -- Makefile.common et al don't actually become part of the package, they just help it to be built. > So, if you've currently got a certain layout for CVS, I'd like to know > more to figure out how to generalize the current CVS code in plague. In > the end you'll have to do a bit of tweaking to get your own CVS stuff to > work, if you choose not to adopt the infrastructure that Fedora CVS > uses. But it shouldn't be very hard at all. I just need a bit more > information first. Gotcha. I think I've decided to adopt the approach f-e CVS uses (why make life harder for myself?), and tweak Makefile.common/et-al as necessary. I've already adapted most of what is necessary to our environment, and built out a substitute lookaside cache. > Glad it works, hope you're using 0.4/HEAD too :) I am now! Of course, it presents new questions, with regards to the config files. (Figures, I get my head around 0.3 and 0.4 presents itself ;) ) plague-server built me a new default config file, but I'm not entirely sure where I should be translating my old CONFIG.py to. Specifically, I see options to "use_cvs", but not sure where I should be defining my cvs root to; targets are now in other, per-target files, but aside from where those files are supposed to go, I'm not sure what their content should be. Is there something/where I can look at to figure this out? Even just a slightly more fleshed out plague-server.cfg and an example of a target config file would do wonders. Thanks again:) -Chris From dcbw at redhat.com Tue Oct 18 01:56:52 2005 From: dcbw at redhat.com (Dan Williams) Date: Mon, 17 Oct 2005 21:56:52 -0400 Subject: Extras build system back up In-Reply-To: <20051017212801.GJ5856@shell0.pdx.osdl.net> References: <1129572538.28887.1.camel@dhcp83-40.boston.redhat.com> <20051017201619.GB30904@shell0.pdx.osdl.net> <1129583751.1418.3.camel@dhcp83-40.boston.redhat.com> <20051017212801.GJ5856@shell0.pdx.osdl.net> Message-ID: <1129600612.8182.1.camel@localhost.localdomain> On Mon, 2005-10-17 at 14:28 -0700, Chris Wright wrote: > * Dan Williams (dcbw at redhat.com) wrote: > > As I see you've figured out, it should be fixed now. Server-side issue > > with some database fields being way too small. They are larger now :) > > Yes, thanks. Is > > [Server] > ... > allow_uploads = True > > the recommended change to config? > > I notice plague list once creates config with > > allow_uploads = no It's a feature that requires server-side support too, which is currently disabled on the Extras server since we build from CVS. Therefore, for Extras, it has no effect either way. What this option does, if you've enabled server-side support though, is to allow users to upload SRPMs to the build server via 'scp', providing you have an account there. Dan From dcbw at redhat.com Tue Oct 18 16:22:45 2005 From: dcbw at redhat.com (Dan Williams) Date: Tue, 18 Oct 2005 12:22:45 -0400 Subject: Extras build system back up In-Reply-To: <1129603219.5390.237.camel@mccallum.corsepiu.local> References: <1129572538.28887.1.camel@dhcp83-40.boston.redhat.com> <20051017201619.GB30904@shell0.pdx.osdl.net> <1129583751.1418.3.camel@dhcp83-40.boston.redhat.com> <20051017212801.GJ5856@shell0.pdx.osdl.net> <1129600612.8182.1.camel@localhost.localdomain> <1129603219.5390.237.camel@mccallum.corsepiu.local> Message-ID: <1129652565.9685.3.camel@localhost.localdomain> On Tue, 2005-10-18 at 04:40 +0200, Ralf Corsepius wrote: > On Mon, 2005-10-17 at 21:56 -0400, Dan Williams wrote: > > On Mon, 2005-10-17 at 14:28 -0700, Chris Wright wrote: > > > * Dan Williams (dcbw at redhat.com) wrote: > > > > As I see you've figured out, it should be fixed now. Server-side issue > > > > with some database fields being way too small. They are larger now :) > > > > > > Yes, thanks. Is > > > > > > [Server] > > > ... > > > allow_uploads = True > > > > > > the recommended change to config? > > > > > > I notice plague list once creates config with > > > > > > allow_uploads = no > > > > It's a feature that requires server-side support too, which is currently > > disabled on the Extras server since we build from CVS. Therefore, for > > Extras, it has no effect either way. > > Well, apparently it does have an effect: It has an no effect _as long as its there_, once you've added it to the config file. Dan From chris.weyl at gmail.com Wed Oct 19 02:21:53 2005 From: chris.weyl at gmail.com (Chris Weyl) Date: Tue, 18 Oct 2005 22:21:53 -0400 Subject: plague 0.4 config file formats... Message-ID: <7dd7ab490510181921w6b9332cbxf2946047b3ba6f78@mail.gmail.com> Ok! So, after a couple... intense;)... hours of hacking at my plague config files, here's what I've come up with for server/builder/target. This is the result of both looking at the default configs that are generated, looking at the code, and repetitively starting the server/builder and waiting for it to complain when it doesn't find something it expects :) However, it still doesn't build from CVS correctly. So I include these files here in the hopes the error of my ways will be shown to me, and also so others may glean what they will from them :) (server names have been replaced, so as to protect the innocent.) To make things a little simpler while figuring it all out, I only have one target cfg defined: fedora-3-i386-core. One generic question beforehand... what's the appropriate way to specify multiple values on a line? Whitespace separating the values, or a ', '? plague-server.cfg ------------------------------------------------------------------ [Database] engine = pgdb [Directories] repo_dir = /var/plague/repodir server_work_dir = /var/plague/rpmbuild target_configs_dir = /etc/plague/targets tmpdir = /tmp [CVS] use_cvs = yes cvs_root = :pserver:anonymous at our.cvs.server:/cvsroot/cvsrepo [General] hostname = server debug = yes [SSL] server_key_and_cert = /etc/plague/server/certs/server_key_and_cert.pem ca_cert = /etc/plague/server/certs/buildsystem_ca_cert.pem [UI] use_ssl = yes log_url = http://server/logs/ guest_allowed = yes port = 8887 client_ca_cert = /etc/plague/server/certs/users_ca_cert.pem [pgdb Engine] host = localhost password = user = root database = buildsys [sqlite Engine] timeout = 3 database = /etc/plague/server/jobdb [Builders] use_ssl = yes # builders = 127.0.0.1:8888 builders = server:8888 [Email] success_emails = buildsys at mail.addy email_from = buildsys at server admin_emails = buildsys at mail.addy [Arches] base_arches = i386 --------end file---------------------------------------------------------- Question: what's the purpose of guest_allowed? I'm thinking I should disable it... plague-builder.cfg -------------------------------------------------- [SSL] builder_key_and_cert_dir = /etc/plague/builder/certs use_ssl = yes ca_cert = /etc/plague/builder/certs/buildsystem_ca_cert.pem [Directories] target_configs_dir = /etc/plague/targets builder_work_dir = /tmp/builder_work [Network] fileserver_port = 8889 xmlrpc_port = 8888 hostname = server [General] debug = yes builder_cmd = /usr/bin/mock builder_user = plague-builder -----end file----------------------------------------------------- targets/fedora-4-i386-core.cfg -------------------------- [General] distro = fedora target = fc4 basearch = i386 repo = core mock_config = fedora-4-i386-core repo_script = [Arches] base_arches = i386 [Aliases] user_aliases = 4 fc4 FC-4 [CVS] use_cvs = yes cvs_root = :pserver:anonymous at our.cvs.server:/cvsroot/cvsrepo cvs_rsh = [Aliases] cvs_alias = ------end file------------------------------------------------------ Question: Did I get the aliases/cvs_alias correct? (Note that I setup the cvs repo to mirror that of fedora-extras, and to leverage the same Makefile.common, etc., system.) I tried a couple for cvs_alias, to no avail. At this point, I can successfully enqueue package builds by cvstag (make tag ; make plague), but the build always fails, within seconds, without any error logging by either the builder or the server. Again, if there's something I should be reading, etc, to figure this all out, just smack me and send me on my way :) -Chris From chris.weyl at gmail.com Wed Oct 19 03:44:20 2005 From: chris.weyl at gmail.com (Chris Weyl) Date: Tue, 18 Oct 2005 23:44:20 -0400 Subject: plague 0.4 config file formats... In-Reply-To: <7dd7ab490510181921w6b9332cbxf2946047b3ba6f78@mail.gmail.com> References: <7dd7ab490510181921w6b9332cbxf2946047b3ba6f78@mail.gmail.com> Message-ID: <7dd7ab490510182044t5c617fd8wa6e59d1f56d5875d@mail.gmail.com> Well, in reply to myself.... Setting cvs_alias in the target config file to the directory to expect the branch in (e.g. FC-4), the builder started up. However, now jobs hang in prep/in-progress, with the server generating this error: 1 (package): Restarting 'package-3_01-2_fc4' on target 'fedora-fc4-core' Exception in thread Thread-7: Traceback (most recent call last): File "/usr/lib/python2.4/threading.py", line 442, in __bootstrap self.run() File "/usr/share/plague/server/PackageJob.py", line 82, in run self._pkg_job.process() File "/usr/share/plague/server/PackageJob.py", line 509, in process if func(): File "/usr/share/plague/server/PackageJob.py", line 377, in _stage_prep (self.archjobs, pkg_arches, allowed_arches) = self.arch_handling(hdr) File "/usr/share/plague/server/PackageJob.py", line 185, in arch_handling addl_arches = self._target_cfg.addl_pkg_arches()[self.name] TypeError: list indices must be integers Thoughts? Thanks:) -Chris From skvidal at phy.duke.edu Wed Oct 19 06:59:19 2005 From: skvidal at phy.duke.edu (seth vidal) Date: Wed, 19 Oct 2005 02:59:19 -0400 Subject: Mock Patch to Enable Realtime Output of logs In-Reply-To: <200510081155.41263.symbiont@berlios.de> References: <200510081155.41263.symbiont@berlios.de> Message-ID: <1129705159.30517.90.camel@cutter> On Sat, 2005-10-08 at 11:55 +0800, Jeff Pitman wrote: > Hi: > > This patch allows one to run "tail -f" on the build.log. It mostly > re-uses existing variables and things like resultdir, etc. work > properly. > > Anyway, I hope to get this committed as this is a massive improvement > from what mock had inherited. (Multiple mount/umount in root.log, but I > don't think that many people care. It's the simplest patch without > taking in account of corner cases, etc.) so commit it. :) I believe you have commit access to mock. I think a lot of folks would like this and I don't think it will break plague's use in anyway. -sv From symbiont at berlios.de Wed Oct 19 07:12:12 2005 From: symbiont at berlios.de (Jeff Pitman) Date: Wed, 19 Oct 2005 15:12:12 +0800 Subject: Mock Patch to Enable Realtime Output of logs In-Reply-To: <1129705159.30517.90.camel@cutter> References: <200510081155.41263.symbiont@berlios.de> <1129705159.30517.90.camel@cutter> Message-ID: <200510191512.12596.symbiont@berlios.de> On Wednesday 19 October 2005 14:59, seth vidal wrote: > so commit it. :) > > I believe you have commit access to mock. I do?! When did that happen? :P There is one final problem that this patch does not cover. That is, when initializing the chroot and the log directory is not available yet. There's a bit of a buffering job done and then dumped to the log after the directory is created. I need to use a lil imagination to make this happen. Currently, I got around this by dropping what is logged during the preliminary init stage. (Not really sure how much is logged, yet. Probably just some directory creates, etc.) Anyway, more polish is needed before commit. -- -jeff From skvidal at phy.duke.edu Wed Oct 19 07:18:58 2005 From: skvidal at phy.duke.edu (seth vidal) Date: Wed, 19 Oct 2005 03:18:58 -0400 Subject: Mock Patch to Enable Realtime Output of logs In-Reply-To: <200510191512.12596.symbiont@berlios.de> References: <200510081155.41263.symbiont@berlios.de> <1129705159.30517.90.camel@cutter> <200510191512.12596.symbiont@berlios.de> Message-ID: <1129706338.30517.99.camel@cutter> On Wed, 2005-10-19 at 15:12 +0800, Jeff Pitman wrote: > On Wednesday 19 October 2005 14:59, seth vidal wrote: > > so commit it. :) > > > > I believe you have commit access to mock. > > I do?! When did that happen? :P I think you do - try it out - check it out from fedora cvs and try a commit :) If you don't have commit access I'll be glad to request it. > > There is one final problem that this patch does not cover. That is, when > initializing the chroot and the log directory is not available yet. > There's a bit of a buffering job done and then dumped to the log after > the directory is created. > > I need to use a lil imagination to make this happen. Currently, I got > around this by dropping what is logged during the preliminary init > stage. (Not really sure how much is logged, yet. Probably just some > directory creates, etc.) Anyway, more polish is needed before commit. Thanks, -sv From dcbw at redhat.com Wed Oct 19 15:15:21 2005 From: dcbw at redhat.com (Dan Williams) Date: Wed, 19 Oct 2005 11:15:21 -0400 Subject: plague 0.4 config file formats... In-Reply-To: <7dd7ab490510181921w6b9332cbxf2946047b3ba6f78@mail.gmail.com> References: <7dd7ab490510181921w6b9332cbxf2946047b3ba6f78@mail.gmail.com> Message-ID: <1129734921.31458.24.camel@localhost.localdomain> On Tue, 2005-10-18 at 22:21 -0400, Chris Weyl wrote: > Ok! So, after a couple... intense;)... hours of hacking at my plague > config files, here's what I've come up with for server/builder/target. > This is the result of both looking at the default configs that are > generated, looking at the code, and repetitively starting the > server/builder and waiting for it to complain when it doesn't find > something it expects :) Yeah, we should really package up default config files that show how to do stuff. I think Dennis Gilmore has done so for Sparc. > However, it still doesn't build from CVS correctly. So I include > these files here in the hopes the error of my ways will be shown to > me, and also so others may glean what they will from them :) (server > names have been replaced, so as to protect the innocent.) To make > things a little simpler while figuring it all out, I only have one > target cfg defined: fedora-3-i386-core. Hmm, any errors in the logs or the emails that it sends when the build fails? > One generic question beforehand... what's the appropriate way to > specify multiple values on a line? Whitespace separating the values, > or a ', '? Should be a ' ' since we're splitting the string on that character to get the different values. > Question: Did I get the aliases/cvs_alias correct? (Note that I > setup the cvs repo to mirror that of fedora-extras, and to leverage > the same Makefile.common, etc., system.) I tried a couple for > cvs_alias, to no avail. The CVS alias causes the server to look for a certain directory rather than the one the user specified. So, if you enqueue a package for fedora-4-core, obviously the name of the CVS directory in Fedora CVS that corresponds here isn't fedora-4-core. It's actually "FC-4". So the CVS alias allows you to specify an alternate naming for the target's directory on the CVS server. So rather than: /cvs/foobar-package/fedora-4-core the server would look, given a cvs_alias for this target, for: /cvs/foobar-package/FC-4 > At this point, I can successfully enqueue package builds by cvstag > (make tag ; make plague), but the build always fails, within seconds, > without any error logging by either the builder or the server. Something that might help is to turn on debugging info in /usr/share/plague/server/PackageJob.py, near the top. You'll see a line that says "DEBUG = False". Change that to "DEBUG = True" (note the capital T) and you should get prints of all the commands the server is running to check stuff out from CVS. Here's a short walk through the CVS code in plague, hopefully that will help us figure out what's going on for you... def _stage_checkout(self): <---snip---> # Set up CVS environment env_args = "CVSROOT='%s'" % self._target_cfg.get_str("CVS", "cvs_root") cvs_rsh = self._target_cfg.get_str("CVS", "cvs_rsh") if len(cvs_rsh) > 0: env_args = "%s CVS_RSH='%s'" % (env_args, cvs_rsh) # Checkout the module <<< construct the checkout command. first 'cd' to our temporary directory where <<< the checkout will be put, then execute 'cvs' to check out the package. <<< In Fedora CVS, each package is a module, under that module is contained the <<< stuff for every release. IE, 'plague' is the module, and the plague module <<< has FC-3, FC-4, devel subdirectories. So just checking out 'plague' module <<< should be enough to get us the directory we want for our target. cmd = 'cd %s; %s %s co -r %s %s' % (self.checkout_tmpdir, env_args, CVS_CMD, self._source, self.package) debugprint("%d: Running %s" % (self.uid, cmd)) s, o = commands.getstatusoutput(cmd) if s != 0: err_msg = "Error: could not check out %s from %s - output was:\n\n" \ "%s" % (self._source, self._target_str, o) else: <<< The 'common' directory contains the base makefile with most of the magic. <<< We make sure to check that directory out specifically (it's also part of <<< the package's module) because we need it to do 'make srpm' later. # Just in case the 'common' directory didn't come along for the ride, # get it from CVS pkg_path = os.path.join(self.checkout_tmpdir, self.package) if not os.path.exists(os.path.join(pkg_path, "common")): cmd = 'cd %s; %s %s co common' % (pkg_path, env_args, CVS_CMD) debugprint("%d: Running %s" % (self.uid, cmd)) s, o = commands.getstatusoutput(cmd) if s != 0: err_msg = "Error: could not check out common directory - " \ "output was:\n\n%s" % (self._source, self._target_str, o) <---snip---> def _stage_make_srpm(self): # Map our target to the CVS target alias, since CVS may have # different target names than we expose cvs_target = self._target_dict['target'] cvs_alias = self._target_cfg.get_str("Aliases", "cvs_alias") if len(cvs_alias) > 0: cvs_target = cvs_alias <<< cd to the checkout directory for this target, for example <<< /tmp/32-plague-0.4-3252352323552/plague/FC-4 . Then run 'make srpm' in <<< that directory. self.srpm_path = None srpm_dir = os.path.join(self.checkout_tmpdir, self.package, cvs_target) if not os.path.exists(srpm_dir): msg = "Error: could not find path %s for %s." % (srpm_dir, self._source) raise PrepError(msg) cmd = 'cd %s; %s srpm' % (srpm_dir, MAKE_CMD) debugprint("%d: Running %s in %s" % (self.uid, cmd, srpm_dir)) s, o = commands.getstatusoutput(cmd) if s != 0: # Don't include download progress lines in output lines = o.split('\n') output_lines = [] for line in lines: if line.find('..........') == -1 and len(line) > 0: output_lines.append(line) o = string.join(output_lines, '\n') msg = "Error: could not make srpm for %s - output was:\n\n%s" % (self._source, o) raise PrepError(msg) srpmpath = None <<< This relies on the output of 'make srpm' to tell us in its output <<< where it put the SRPM it just made. There's some magic here, namely <<< we look for the line starting with "Wrote:" and assume the rest is the path <<< to the SRPM for line in o.split("\n"): if line.startswith("Wrote:"): line.replace("\n", "") (garbage, path) = line.split(':') srpmpath = path.strip() break if not srpmpath: msg = "Error: could not find srpm for %s - output was:\n\n%s" % (self._source, o) raise PrepError(msg) self.srpm_path = srpmpath From chris.weyl at gmail.com Wed Oct 19 22:00:03 2005 From: chris.weyl at gmail.com (Chris Weyl) Date: Wed, 19 Oct 2005 18:00:03 -0400 Subject: plague 0.4 config file formats... In-Reply-To: <1129734921.31458.24.camel@localhost.localdomain> References: <7dd7ab490510181921w6b9332cbxf2946047b3ba6f78@mail.gmail.com> <1129734921.31458.24.camel@localhost.localdomain> Message-ID: <7dd7ab490510191500q63dac537w867b58efa97ee7f4@mail.gmail.com> On 10/19/05, Dan Williams wrote: > On Tue, 2005-10-18 at 22:21 -0400, Chris Weyl wrote: > > Ok! So, after a couple... intense;)... hours of hacking at my plague > > config files, here's what I've come up with for server/builder/target. > > This is the result of both looking at the default configs that are > > generated, looking at the code, and repetitively starting the > > server/builder and waiting for it to complain when it doesn't find > > something it expects :) > > Yeah, we should really package up default config files that show how to > do stuff. I think Dennis Gilmore has done so for Sparc. I think that's an excellent idea; I know I've been guessing alot. > > However, it still doesn't build from CVS correctly. So I include > > these files here in the hopes the error of my ways will be shown to > > me, and also so others may glean what they will from them :) (server > > names have been replaced, so as to protect the innocent.) To make > > things a little simpler while figuring it all out, I only have one > > target cfg defined: fedora-3-i386-core. > > Hmm, any errors in the logs or the emails that it sends when the build > fails? Hmm, I wasn't reading those -- didn't realize they contained information not logged. So I looked at the last one, saw that it was trying to build from a path that doesn't exist in CVS, corrected cvs_alias (per your comments) and that fixed that problem. > Something that might help is to turn on debugging info > in /usr/share/plague/server/PackageJob.py, near the top. You'll see a > line that says "DEBUG = False". Change that to "DEBUG = True" (note the > capital T) and you should get prints of all the commands the server is > running to check stuff out from CVS. That did help a touch. Now, when building from a CVS tag, it tells me that it's changing to the checked out directory, and running "make srpm" in it. Both these commands work (that is, as root I can execute them w/o problem), but the next thing logged to plague-server.log is: ===================== Exception in thread Thread-4: Traceback (most recent call last): File "/usr/lib/python2.4/threading.py", line 442, in __bootstrap self.run() File "/usr/share/plague/server/PackageJob.py", line 82, in run self._pkg_job.process() File "/usr/share/plague/server/PackageJob.py", line 509, in process if func(): File "/usr/share/plague/server/PackageJob.py", line 377, in _stage_prep (self.archjobs, pkg_arches, allowed_arches) = self.arch_handling(hdr) File "/usr/share/plague/server/PackageJob.py", line 185, in arch_handling addl_arches = self._target_cfg.addl_pkg_arches()[self.name] TypeError: list indices must be integers ===================== I'm thinking I have something wrong (or missing?) in a config file somewhere...? > Here's a short walk through the CVS code in plague, hopefully that will > help us figure out what's going on for you... [...snip...] > cmd = 'cd %s; %s srpm' % (srpm_dir, MAKE_CMD) > debugprint("%d: Running %s in %s" % (self.uid, cmd, srpm_dir)) > s, o = commands.getstatusoutput(cmd) > if s != 0: > # Don't include download progress lines in output > lines = o.split('\n') > output_lines = [] > for line in lines: > if line.find('..........') == -1 and len(line) > 0: > output_lines.append(line) > o = string.join(output_lines, '\n') > msg = "Error: could not make srpm for %s - output was:\n\n%s" % (self._source, o) > raise PrepError(msg) > > srpmpath = None Hmmmmm..... It would seem to be somewhere in here that the exception is being thrown. I think; not sure if the above messages point elsewhere.. -Chris From dcbw at redhat.com Thu Oct 20 13:41:02 2005 From: dcbw at redhat.com (Dan Williams) Date: Thu, 20 Oct 2005 09:41:02 -0400 Subject: plague 0.4 config file formats... In-Reply-To: <7dd7ab490510191500q63dac537w867b58efa97ee7f4@mail.gmail.com> References: <7dd7ab490510181921w6b9332cbxf2946047b3ba6f78@mail.gmail.com> <1129734921.31458.24.camel@localhost.localdomain> <7dd7ab490510191500q63dac537w867b58efa97ee7f4@mail.gmail.com> Message-ID: <1129815662.2559.4.camel@dhcp83-40.boston.redhat.com> On Wed, 2005-10-19 at 18:00 -0400, Chris Weyl wrote: > ===================== > Exception in thread Thread-4: > Traceback (most recent call last): > File "/usr/lib/python2.4/threading.py", line 442, in __bootstrap > self.run() > File "/usr/share/plague/server/PackageJob.py", line 82, in run > self._pkg_job.process() > File "/usr/share/plague/server/PackageJob.py", line 509, in process > if func(): > File "/usr/share/plague/server/PackageJob.py", line 377, in _stage_prep > (self.archjobs, pkg_arches, allowed_arches) = self.arch_handling(hdr) > File "/usr/share/plague/server/PackageJob.py", line 185, in arch_handling > addl_arches = self._target_cfg.addl_pkg_arches()[self.name] > TypeError: list indices must be integers > ===================== So I pushed new plague packages yesterday to fix this. If you want to fix it right now, you can modify /usr/share/plague/server/Config.py as such (in pseudo diff format, or course): def addl_pkg_arches(self): if not self._config.has_section("Additional Package Arches"): - return [] + return {} items = self._config.items("Additional Package Arches") ie, just change the [] -> {} and it should work there. Or add a [Additional Package Arches] section to your target's config file. Or update to latest plague RPMs. Dan From chris.weyl at gmail.com Thu Oct 20 21:26:22 2005 From: chris.weyl at gmail.com (Chris Weyl) Date: Thu, 20 Oct 2005 17:26:22 -0400 Subject: plague 0.4 config file formats... In-Reply-To: <1129815662.2559.4.camel@dhcp83-40.boston.redhat.com> References: <7dd7ab490510181921w6b9332cbxf2946047b3ba6f78@mail.gmail.com> <1129734921.31458.24.camel@localhost.localdomain> <7dd7ab490510191500q63dac537w867b58efa97ee7f4@mail.gmail.com> <1129815662.2559.4.camel@dhcp83-40.boston.redhat.com> Message-ID: <7dd7ab490510201426w4e56e520kadc7e8f3baccc888@mail.gmail.com> On 10/20/05, Dan Williams wrote: > On Wed, 2005-10-19 at 18:00 -0400, Chris Weyl wrote: [...snip...] > > So I pushed new plague packages yesterday to fix this. If you want to > fix it right now, you can modify /usr/share/plague/server/Config.py as > such (in pseudo diff format, or course): > > def addl_pkg_arches(self): > if not self._config.has_section("Additional Package Arches"): > - return [] > + return {} > items = self._config.items("Additional Package Arches") > > ie, just change the [] -> {} and it should work there. Or add a > [Additional Package Arches] section to your target's config file. Or > update to latest plague RPMs. Perfect, updated and it works nicely... It did complain about needing Arches/optional_arches in the target config file, so I added that (e.g. = i486 i586 i686 athlon noarch) and the complaints went away. Thanks again :) -Chris From chris.weyl at gmail.com Fri Oct 21 13:30:58 2005 From: chris.weyl at gmail.com (Chris Weyl) Date: Fri, 21 Oct 2005 09:30:58 -0400 Subject: plague-server && mysql Message-ID: <7dd7ab490510210630g4427e636x96b68a7c19ea7a9d@mail.gmail.com> Hey all-- I can't post a patch for various reasons beyond my control (sigh...), but updating plague to use a mysql backend was trivial. (Not that I don't like postgresql -- mysql is just easier for me in this environment.) 1) yum install MySQL-python 2) edit /usr/share/plague/server/DBManager.py: * copy the pgdbEngineClass definition verbatim, renaming it to 'mysqlEngineClass' * the correct type for mysql in get_uid_field_type is - "int(11) NOT NULL auto_increment PRIMARY KEY" * connect parameters are slightly different in _connect(): - host=host, db=database, user=user, passwd=password * add the new class to the db_engines array * add a try statement to importing the MySQLdb class 3) update your plague-server.cfg with a "[mysql Engine]" section And, of course, create your mysql db somewhere, etc, etc. Hope that helps someone! :) -Chris From chris.weyl at gmail.com Fri Oct 21 14:06:53 2005 From: chris.weyl at gmail.com (Chris Weyl) Date: Fri, 21 Oct 2005 10:06:53 -0400 Subject: builds seeming to hang in "add_to_repo/in-progress" Message-ID: <7dd7ab490510210706o717e90c8i5fae7836c234f866@mail.gmail.com> Hey Dan (et al) -- So now I can build from CVS, everything appears OK... Except packages (that don't fail) seem to hang in the "add_to_repo/in-progress" state. The log files from the build appear in the expected place. However, build.log is 0 bytes. The SRPM does not show up here. In plague-server.cfg, [General] repo_script = testing = False Trying to do a "plague-client finish #" on the jobids fails with the expected "Error: Job 9 must be either 'needsign' or 'failed' to finish it." Am I missing something? Or is something darker and more sinister at work here. -Chris From sheltren at cs.ucsb.edu Fri Oct 21 16:29:23 2005 From: sheltren at cs.ucsb.edu (Jeff Sheltren) Date: Fri, 21 Oct 2005 12:29:23 -0400 Subject: plague-server && mysql In-Reply-To: <7dd7ab490510210630g4427e636x96b68a7c19ea7a9d@mail.gmail.com> References: <7dd7ab490510210630g4427e636x96b68a7c19ea7a9d@mail.gmail.com> Message-ID: <1BC3A93C-7D83-4B10-82CF-737D42112AB2@cs.ucsb.edu> On Oct 21, 2005, at 9:30 AM, Chris Weyl wrote: > Hey all-- > > I can't post a patch for various reasons beyond my control (sigh...), > but updating plague to use a mysql backend was trivial. (Not that I > don't like postgresql -- mysql is just easier for me in this > environment.) > > 1) yum install MySQL-python > 2) edit /usr/share/plague/server/DBManager.py: > * copy the pgdbEngineClass definition verbatim, renaming it to > 'mysqlEngineClass' > * the correct type for mysql in get_uid_field_type is > - "int(11) NOT NULL auto_increment PRIMARY KEY" > * connect parameters are slightly different in _connect(): > - host=host, db=database, user=user, passwd=password > * add the new class to the db_engines array > * add a try statement to importing the MySQLdb class > 3) update your plague-server.cfg with a "[mysql Engine]" section > > And, of course, create your mysql db somewhere, etc, etc. > > Hope that helps someone! :) > > -Chris > Hey Chris, thanks, that is helpful for me as well. Any reason I can't submit a patch doing the above? :) -Jeff From chris.weyl at gmail.com Fri Oct 21 17:28:28 2005 From: chris.weyl at gmail.com (Chris Weyl) Date: Fri, 21 Oct 2005 13:28:28 -0400 Subject: plague-server && mysql In-Reply-To: <1BC3A93C-7D83-4B10-82CF-737D42112AB2@cs.ucsb.edu> References: <7dd7ab490510210630g4427e636x96b68a7c19ea7a9d@mail.gmail.com> <1BC3A93C-7D83-4B10-82CF-737D42112AB2@cs.ucsb.edu> Message-ID: <7dd7ab490510211028p3cb69ff0i142c1c5315ba37fc@mail.gmail.com> On 10/21/05, Jeff Sheltren wrote: > > Hey Chris, thanks, that is helpful for me as well. Any reason I > can't submit a patch doing the above? :) Not that I know of. Have at it! ;) -Chris From sheltren at cs.ucsb.edu Sun Oct 23 19:45:15 2005 From: sheltren at cs.ucsb.edu (Jeff Sheltren) Date: Sun, 23 Oct 2005 15:45:15 -0400 Subject: plague-server && mysql In-Reply-To: <7dd7ab490510211028p3cb69ff0i142c1c5315ba37fc@mail.gmail.com> References: <7dd7ab490510210630g4427e636x96b68a7c19ea7a9d@mail.gmail.com> <1BC3A93C-7D83-4B10-82CF-737D42112AB2@cs.ucsb.edu> <7dd7ab490510211028p3cb69ff0i142c1c5315ba37fc@mail.gmail.com> Message-ID: On Oct 21, 2005, at 1:28 PM, Chris Weyl wrote: > On 10/21/05, Jeff Sheltren wrote: > >> >> Hey Chris, thanks, that is helpful for me as well. Any reason I >> can't submit a patch doing the above? :) >> > > Not that I know of. Have at it! ;) > > -Chris > I'm attaching a patch to allow plague-server to use MySQL (by following Chris' instructions). I'm still working on getting my buildsys working, so I haven't tested this out with building packages, but I can say that at least the tables get created when the server first starts. It'd be nice if this could get added to CVS :) -Jeff -------------- next part -------------- A non-text attachment was scrubbed... Name: mysql-support.patch Type: application/octet-stream Size: 2782 bytes Desc: not available URL: From sheltren at cs.ucsb.edu Mon Oct 24 15:20:42 2005 From: sheltren at cs.ucsb.edu (Jeff Sheltren) Date: Mon, 24 Oct 2005 11:20:42 -0400 Subject: Docs & Examples Message-ID: OK, I got plague-(server/builder) working - yay! The README in CVS is pretty helpful, but I still had to do a lot of guesswork and reading through the code in order to figure out how to set it up. I think it would be very helpful to provide example config files (with comments explaining what each setting does, and a list of valid settings if applicable), and some more documentation. I'd be happy to help with this - anyone mind if I create some docs/examples subpages from the Plague wiki page? At least I can put up my config files (and try to comment in them a bit), show some example targets, document how I got the www scripts working with apache, etc. -Jeff From dcbw at redhat.com Mon Oct 24 15:25:50 2005 From: dcbw at redhat.com (Dan Williams) Date: Mon, 24 Oct 2005 11:25:50 -0400 Subject: Docs & Examples In-Reply-To: References: Message-ID: <1130167550.3113.38.camel@localhost.localdomain> On Mon, 2005-10-24 at 11:20 -0400, Jeff Sheltren wrote: > OK, I got plague-(server/builder) working - yay! The README in CVS > is pretty helpful, but I still had to do a lot of guesswork and > reading through the code in order to figure out how to set it up. I > think it would be very helpful to provide example config files (with > comments explaining what each setting does, and a list of valid > settings if applicable), and some more documentation. I'd be happy > to help with this - anyone mind if I create some docs/examples > subpages from the Plague wiki page? That would be AWESOME. > At least I can put up my config files (and try to comment in them a > bit), show some example targets, document how I got the www scripts > working with apache, etc. That would also be really great. I'd love to add a 'docs' directory and stick all this stuff in it. If you want to help, so much the better! Thanks, Dan From dcbw at redhat.com Mon Oct 24 15:32:51 2005 From: dcbw at redhat.com (Dan Williams) Date: Mon, 24 Oct 2005 11:32:51 -0400 Subject: plague-server && mysql In-Reply-To: References: <7dd7ab490510210630g4427e636x96b68a7c19ea7a9d@mail.gmail.com> <1BC3A93C-7D83-4B10-82CF-737D42112AB2@cs.ucsb.edu> <7dd7ab490510211028p3cb69ff0i142c1c5315ba37fc@mail.gmail.com> Message-ID: <1130167972.3113.40.camel@localhost.localdomain> On Sun, 2005-10-23 at 15:45 -0400, Jeff Sheltren wrote: > On Oct 21, 2005, at 1:28 PM, Chris Weyl wrote: > > > On 10/21/05, Jeff Sheltren wrote: > > > >> > >> Hey Chris, thanks, that is helpful for me as well. Any reason I > >> can't submit a patch doing the above? :) > >> > > > > Not that I know of. Have at it! ;) > > > > -Chris > > > > I'm attaching a patch to allow plague-server to use MySQL (by > following Chris' instructions). I'm still working on getting my > buildsys working, so I haven't tested this out with building > packages, but I can say that at least the tables get created when the > server first starts. > > It'd be nice if this could get added to CVS :) Committed, thanks! Dan From sheltren at cs.ucsb.edu Mon Oct 24 15:42:13 2005 From: sheltren at cs.ucsb.edu (Jeff Sheltren) Date: Mon, 24 Oct 2005 11:42:13 -0400 Subject: plague-server && mysql In-Reply-To: <1130167972.3113.40.camel@localhost.localdomain> References: <7dd7ab490510210630g4427e636x96b68a7c19ea7a9d@mail.gmail.com> <1BC3A93C-7D83-4B10-82CF-737D42112AB2@cs.ucsb.edu> <7dd7ab490510211028p3cb69ff0i142c1c5315ba37fc@mail.gmail.com> <1130167972.3113.40.camel@localhost.localdomain> Message-ID: On Oct 24, 2005, at 11:32 AM, Dan Williams wrote: > On Sun, 2005-10-23 at 15:45 -0400, Jeff Sheltren wrote: >> >> I'm attaching a patch to allow plague-server to use MySQL (by >> following Chris' instructions). I'm still working on getting my >> buildsys working, so I haven't tested this out with building >> packages, but I can say that at least the tables get created when the >> server first starts. >> >> It'd be nice if this could get added to CVS :) >> > > Committed, thanks! > > Dan > Cool, thanks. I was just looking and noticed that the user information is always stored in sqlite no matter what backend is used for the job information. I think it would be nice to have everything stored in the same place no matter what DB backend is chosen. If I can figure out a way to get User.py to use DBManager, would you consider importing that into CVS, or is there a reason that sqlite is used exclusively? Thanks, Jeff From ivazquez at ivazquez.net Mon Oct 24 16:05:33 2005 From: ivazquez at ivazquez.net (Ignacio Vazquez-Abrams) Date: Mon, 24 Oct 2005 12:05:33 -0400 Subject: Docs & Examples In-Reply-To: References: Message-ID: <1130169933.26809.21.camel@ignacio.lan> On Mon, 2005-10-24 at 11:20 -0400, Jeff Sheltren wrote: > OK, I got plague-(server/builder) working - yay! The README in CVS > is pretty helpful, but I still had to do a lot of guesswork and > reading through the code in order to figure out how to set it up. I > think it would be very helpful to provide example config files (with > comments explaining what each setting does, and a list of valid > settings if applicable), and some more documentation. I'd be happy > to help with this - anyone mind if I create some docs/examples > subpages from the Plague wiki page? > > At least I can put up my config files (and try to comment in them a > bit), show some example targets, document how I got the www scripts > working with apache, etc. I've been working on a "Building RPMs" module for the FDP, and one of the appendices is on plague. Unfortunately I haven't been able to get an awful lot of it done due to school, but if you want to contribute something for it I wouldn't mind. -- Ignacio Vazquez-Abrams http://fedora.ivazquez.net/ gpg --keyserver hkp://subkeys.pgp.net --recv-key 38028b72 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From dcbw at redhat.com Mon Oct 24 16:21:39 2005 From: dcbw at redhat.com (Dan Williams) Date: Mon, 24 Oct 2005 12:21:39 -0400 Subject: plague-server && mysql In-Reply-To: References: <7dd7ab490510210630g4427e636x96b68a7c19ea7a9d@mail.gmail.com> <1BC3A93C-7D83-4B10-82CF-737D42112AB2@cs.ucsb.edu> <7dd7ab490510211028p3cb69ff0i142c1c5315ba37fc@mail.gmail.com> <1130167972.3113.40.camel@localhost.localdomain> Message-ID: <1130170900.4687.1.camel@localhost.localdomain> On Mon, 2005-10-24 at 11:42 -0400, Jeff Sheltren wrote: > On Oct 24, 2005, at 11:32 AM, Dan Williams wrote: > > > On Sun, 2005-10-23 at 15:45 -0400, Jeff Sheltren wrote: > >> > >> I'm attaching a patch to allow plague-server to use MySQL (by > >> following Chris' instructions). I'm still working on getting my > >> buildsys working, so I haven't tested this out with building > >> packages, but I can say that at least the tables get created when the > >> server first starts. > >> > >> It'd be nice if this could get added to CVS :) > >> > > > > Committed, thanks! > > > > Dan > > > > Cool, thanks. I was just looking and noticed that the user > information is always stored in sqlite no matter what backend is used > for the job information. I think it would be nice to have everything > stored in the same place no matter what DB backend is chosen. If I > can figure out a way to get User.py to use DBManager, would you > consider importing that into CVS, or is there a reason that sqlite is > used exclusively? Not really, just the fact that having users be in sqlite isn't a problem since nobody really writes to the userdb that often. That's not the case with the job database. But we should most likely use the same DB for everything. So if you've got a patch, I'll take it. It should be fairly straightforward since we don't need any of the Unique ID crap (email addresses are the index key here). Dan From chris.weyl at gmail.com Mon Oct 24 17:55:58 2005 From: chris.weyl at gmail.com (Chris Weyl) Date: Mon, 24 Oct 2005 13:55:58 -0400 Subject: Docs & Examples In-Reply-To: <1130167550.3113.38.camel@localhost.localdomain> References: <1130167550.3113.38.camel@localhost.localdomain> Message-ID: <7dd7ab490510241055v5530dd74na9d6e709a04e44ab@mail.gmail.com> On 10/24/05, Dan Williams wrote: > On Mon, 2005-10-24 at 11:20 -0400, Jeff Sheltren wrote: > > OK, I got plague-(server/builder) working - yay! The README in CVS > > is pretty helpful, but I still had to do a lot of guesswork and > > reading through the code in order to figure out how to set it up. I > > think it would be very helpful to provide example config files (with > > comments explaining what each setting does, and a list of valid > > settings if applicable), and some more documentation. I'd be happy > > to help with this - anyone mind if I create some docs/examples > > subpages from the Plague wiki page? > > That would be AWESOME. I was just thinking about that over the weekend, actually... I'd like to help out with that too -- does creating/editing wiki pages there require any sort of special authorization, or can I just hit the "create profile" button on the wiki's login page? Just trying to not be spanked :) -Chris From sheltren at cs.ucsb.edu Mon Oct 24 18:15:49 2005 From: sheltren at cs.ucsb.edu (Jeff Sheltren) Date: Mon, 24 Oct 2005 14:15:49 -0400 Subject: Docs & Examples In-Reply-To: <7dd7ab490510241055v5530dd74na9d6e709a04e44ab@mail.gmail.com> References: <1130167550.3113.38.camel@localhost.localdomain> <7dd7ab490510241055v5530dd74na9d6e709a04e44ab@mail.gmail.com> Message-ID: <7D89E92F-8AD5-4C52-9FC1-7B6BCFB82F2F@cs.ucsb.edu> On Oct 24, 2005, at 1:55 PM, Chris Weyl wrote: > On 10/24/05, Dan Williams wrote: > >> On Mon, 2005-10-24 at 11:20 -0400, Jeff Sheltren wrote: >> >>> OK, I got plague-(server/builder) working - yay! The README in CVS >>> is pretty helpful, but I still had to do a lot of guesswork and >>> reading through the code in order to figure out how to set it up. I >>> think it would be very helpful to provide example config files (with >>> comments explaining what each setting does, and a list of valid >>> settings if applicable), and some more documentation. I'd be happy >>> to help with this - anyone mind if I create some docs/examples >>> subpages from the Plague wiki page? >>> >> >> That would be AWESOME. >> > > I was just thinking about that over the weekend, actually... I'd like > to help out with that too -- does creating/editing wiki pages there > require any sort of special authorization, or can I just hit the > "create profile" button on the wiki's login page? > > Just trying to not be spanked :) > > -Chris I've added the following pages to the wiki: Plague/Targets Plague/ServerConfig Plague/BuilderConfig Plague/WebSetup Dan, I'd appreciate if you could go over those and add/correct as needed, since I don't know what all the config options do. Once you've done that we can link to them from the main Plague page. Chris, I think you can just create an account and then be able to create new pages, but I may be wrong :) -Jeff From chris.weyl at gmail.com Mon Oct 24 18:16:37 2005 From: chris.weyl at gmail.com (Chris Weyl) Date: Mon, 24 Oct 2005 14:16:37 -0400 Subject: Docs & Examples In-Reply-To: <7dd7ab490510241055v5530dd74na9d6e709a04e44ab@mail.gmail.com> References: <1130167550.3113.38.camel@localhost.localdomain> <7dd7ab490510241055v5530dd74na9d6e709a04e44ab@mail.gmail.com> Message-ID: <7dd7ab490510241116i122b1af6ife8db46f647eddfe@mail.gmail.com> On 10/24/05, Chris Weyl wrote: > I was just thinking about that over the weekend, actually... I'd like > to help out with that too -- does creating/editing wiki pages there > require any sort of special authorization, or can I just hit the > "create profile" button on the wiki's login page? nm -- answered my own question. heh -Chris From chris.weyl at gmail.com Mon Oct 24 19:16:26 2005 From: chris.weyl at gmail.com (Chris Weyl) Date: Mon, 24 Oct 2005 15:16:26 -0400 Subject: Docs & Examples In-Reply-To: <7D89E92F-8AD5-4C52-9FC1-7B6BCFB82F2F@cs.ucsb.edu> References: <1130167550.3113.38.camel@localhost.localdomain> <7dd7ab490510241055v5530dd74na9d6e709a04e44ab@mail.gmail.com> <7D89E92F-8AD5-4C52-9FC1-7B6BCFB82F2F@cs.ucsb.edu> Message-ID: <7dd7ab490510241216k4d39de32w9c03d37c510c639b@mail.gmail.com> On 10/24/05, Jeff Sheltren wrote: > > I've added the following pages to the wiki: > Plague/Targets > Plague/ServerConfig > Plague/BuilderConfig > Plague/WebSetup Excellent :) I'll add what I've learned, and might also create Plague/CvsSetup if I'm feeling ambitious. It'll probably be more of a series of pointers/instructions, as to fully leverage the CVS interface they're either going to need to adapt their own repos to meet what plague expects, or fiddle around with pulling bits out of f-e CVS and replicating that framework (which is what I did). > Chris, I think you can just create an account and then be able to > create new pages, but I may be wrong :) I'm in :) -Chris From sheltren at cs.ucsb.edu Mon Oct 24 19:20:25 2005 From: sheltren at cs.ucsb.edu (Jeff Sheltren) Date: Mon, 24 Oct 2005 15:20:25 -0400 Subject: Docs & Examples In-Reply-To: <1130169933.26809.21.camel@ignacio.lan> References: <1130169933.26809.21.camel@ignacio.lan> Message-ID: On Oct 24, 2005, at 12:05 PM, Ignacio Vazquez-Abrams wrote: > > I've been working on a "Building RPMs" module for the FDP, and one of > the appendices is on plague. Unfortunately I haven't been able to > get an > awful lot of it done due to school, but if you want to contribute > something for it I wouldn't mind. > Ignacio, I just put stuff up on the fedoraproject wiki since Plague already had a page there, and it was easy to add more pages there... Feel free to use/link to anything there. -Jeff From dcbw at redhat.com Tue Oct 25 20:16:50 2005 From: dcbw at redhat.com (Dan Williams) Date: Tue, 25 Oct 2005 16:16:50 -0400 Subject: Mock Patch to Enable Realtime Output of logs In-Reply-To: <200510191512.12596.symbiont@berlios.de> References: <200510081155.41263.symbiont@berlios.de> <1129705159.30517.90.camel@cutter> <200510191512.12596.symbiont@berlios.de> Message-ID: <1130271410.16449.4.camel@localhost.localdomain> On Wed, 2005-10-19 at 15:12 +0800, Jeff Pitman wrote: > On Wednesday 19 October 2005 14:59, seth vidal wrote: > There is one final problem that this patch does not cover. That is, when > initializing the chroot and the log directory is not available yet. > There's a bit of a buffering job done and then dumped to the log after > the directory is created. Got a final patch yet? If you don't have commit privs I can do it. I'm seeing issues right now in the fedora build system like this: 100 21961 0.0 0.2 71848 5232 ? S 14:46 0:00 /usr/bin/python -tt /usr/bin/mock -r fedora-3-x86_64-core --arch x86_64 --resultdir=/mnt/build/builder_work/b798c880f2 100 22916 0.0 0.0 0 0 ? Z 14:48 0:00 [sh] [hammer1 ~]# strace -p 21961 Process 21961 attached - interrupt to quit read(153, Process 21961 detached [hammer1 ~]# which I think could be avoided by not using commands.getstatusoutput(). Not sure though. Any other thoughts? Dan From skvidal at phy.duke.edu Tue Oct 25 20:20:35 2005 From: skvidal at phy.duke.edu (seth vidal) Date: Tue, 25 Oct 2005 16:20:35 -0400 Subject: Mock Patch to Enable Realtime Output of logs In-Reply-To: <1130271410.16449.4.camel@localhost.localdomain> References: <200510081155.41263.symbiont@berlios.de> <1129705159.30517.90.camel@cutter> <200510191512.12596.symbiont@berlios.de> <1130271410.16449.4.camel@localhost.localdomain> Message-ID: <1130271635.18955.158.camel@cutter> On Tue, 2005-10-25 at 16:16 -0400, Dan Williams wrote: > On Wed, 2005-10-19 at 15:12 +0800, Jeff Pitman wrote: > > On Wednesday 19 October 2005 14:59, seth vidal wrote: > > There is one final problem that this patch does not cover. That is, when > > initializing the chroot and the log directory is not available yet. > > There's a bit of a buffering job done and then dumped to the log after > > the directory is created. > > Got a final patch yet? If you don't have commit privs I can do it. > > I'm seeing issues right now in the fedora build system like this: > > 100 21961 0.0 0.2 71848 5232 ? S 14:46 0:00 /usr/bin/python -tt /usr/bin/mock -r fedora-3-x86_64-core --arch x86_64 --resultdir=/mnt/build/builder_work/b798c880f2 > 100 22916 0.0 0.0 0 0 ? Z 14:48 0:00 [sh] > > [hammer1 ~]# strace -p 21961 > Process 21961 attached - interrupt to quit > read(153, > Process 21961 detached > [hammer1 ~]# > > which I think could be avoided by not using commands.getstatusoutput(). Not sure though. Any other thoughts? > why would they start happening now if they didn't happen before the plague update? -sv From dcbw at redhat.com Tue Oct 25 20:24:34 2005 From: dcbw at redhat.com (Dan Williams) Date: Tue, 25 Oct 2005 16:24:34 -0400 Subject: Mock Patch to Enable Realtime Output of logs In-Reply-To: <1130271635.18955.158.camel@cutter> References: <200510081155.41263.symbiont@berlios.de> <1129705159.30517.90.camel@cutter> <200510191512.12596.symbiont@berlios.de> <1130271410.16449.4.camel@localhost.localdomain> <1130271635.18955.158.camel@cutter> Message-ID: <1130271874.16449.5.camel@localhost.localdomain> On Tue, 2005-10-25 at 16:20 -0400, seth vidal wrote: > On Tue, 2005-10-25 at 16:16 -0400, Dan Williams wrote: > > On Wed, 2005-10-19 at 15:12 +0800, Jeff Pitman wrote: > > > On Wednesday 19 October 2005 14:59, seth vidal wrote: > > > There is one final problem that this patch does not cover. That is, when > > > initializing the chroot and the log directory is not available yet. > > > There's a bit of a buffering job done and then dumped to the log after > > > the directory is created. > > > > Got a final patch yet? If you don't have commit privs I can do it. > > > > I'm seeing issues right now in the fedora build system like this: > > > > 100 21961 0.0 0.2 71848 5232 ? S 14:46 0:00 /usr/bin/python -tt /usr/bin/mock -r fedora-3-x86_64-core --arch x86_64 --resultdir=/mnt/build/builder_work/b798c880f2 > > 100 22916 0.0 0.0 0 0 ? Z 14:48 0:00 [sh] > > > > [hammer1 ~]# strace -p 21961 > > Process 21961 attached - interrupt to quit > > read(153, > > Process 21961 detached > > [hammer1 ~]# > > > > which I think could be avoided by not using commands.getstatusoutput(). Not sure though. Any other thoughts? > > > > why would they start happening now if they didn't happen before the > plague update? They did, just not quite as reproducibly I think... It seems to hit it on FC3 builds with gobby (ie, job 261), and I know I've seen this before. Dan From sheltren at cs.ucsb.edu Wed Oct 26 10:48:58 2005 From: sheltren at cs.ucsb.edu (Jeff Sheltren) Date: Wed, 26 Oct 2005 06:48:58 -0400 Subject: plague-server dying Message-ID: I am having problems with plague-server dying on me. I am starting plague-server like: plague-server -c /etc/plague/server/plague-server.cfg -d -l /var/log/ plague/plague.log my.ip.address.here After some amount of time (hours? I'm not sure exactly, but I know it was up all most of the day yesterday and then died sometime in the night) the process disappears, and nothing is left in the log file except for the startup and build messages. Is there anything I can do to have it print out more information? I am running the server on an EL4 box, might that have something to do with it? plague-server is CVS from a few days ago (using mysql back-end patch). python version is python-2.3.4-14.1 I also have plague-builder running on the same box, and it has been running with no problem. Thanks, Jeff From sheltren at cs.ucsb.edu Wed Oct 26 11:42:10 2005 From: sheltren at cs.ucsb.edu (Jeff Sheltren) Date: Wed, 26 Oct 2005 07:42:10 -0400 Subject: plague-server dying In-Reply-To: References: Message-ID: On Oct 26, 2005, at 6:48 AM, Jeff Sheltren wrote: > I am having problems with plague-server dying on me. I am starting > plague-server like: > plague-server -c /etc/plague/server/plague-server.cfg -d -l /var/ > log/plague/plague.log my.ip.address.here > > After some amount of time (hours? I'm not sure exactly, but I know > it was up all most of the day yesterday and then died sometime in > the night) the process disappears, and nothing is left in the log > file except for the startup and build messages. > > Is there anything I can do to have it print out more information? > I am running the server on an EL4 box, might that have something to > do with it? > > plague-server is CVS from a few days ago (using mysql back-end patch). > python version is python-2.3.4-14.1 > > I also have plague-builder running on the same box, and it has been > running with no problem. > > Thanks, > Jeff > Died again this morning after about 30 minutes - no jobs were submitted or run in this time. The only access was via the web interface to check if it was up. I was using it with a remote mysql server, so I'm going to try it with mysql on localhost and see if that works any better. -Jeff From dcbw at redhat.com Wed Oct 26 12:02:56 2005 From: dcbw at redhat.com (Dan Williams) Date: Wed, 26 Oct 2005 08:02:56 -0400 Subject: plague-server dying In-Reply-To: References: Message-ID: <1130328177.2627.4.camel@localhost.localdomain> On Wed, 2005-10-26 at 07:42 -0400, Jeff Sheltren wrote: > On Oct 26, 2005, at 6:48 AM, Jeff Sheltren wrote: > > > I am having problems with plague-server dying on me. I am starting > > plague-server like: > > plague-server -c /etc/plague/server/plague-server.cfg -d -l /var/ > > log/plague/plague.log my.ip.address.here > > > > After some amount of time (hours? I'm not sure exactly, but I know > > it was up all most of the day yesterday and then died sometime in > > the night) the process disappears, and nothing is left in the log > > file except for the startup and build messages. > > > > Is there anything I can do to have it print out more information? > > I am running the server on an EL4 box, might that have something to > > do with it? > > > > plague-server is CVS from a few days ago (using mysql back-end patch). > > python version is python-2.3.4-14.1 > > > > I also have plague-builder running on the same box, and it has been > > running with no problem. > > > > Thanks, > > Jeff > > > > Died again this morning after about 30 minutes - no jobs were > submitted or run in this time. The only access was via the web > interface to check if it was up. I was using it with a remote mysql > server, so I'm going to try it with mysql on localhost and see if > that works any better. Hmm. Can you try to run the python process under gdb? Essentially: gdb /usr/bin/python set args /usr/bin/plague-server -c /etc/plague/server/plague-server.cfg ... run Then just let it run for a while, and see where it segfaults. It would also be helpful to install python-debuginfo and pyOpenSSL-debuginfo too so that the gdb backtrace is more useful. Come to think of it, are you using SSL in your buildsystem at all, and if so, have you patched your copy of pyOpenSSL? This is probably something that should be documented :) FC4 and rawhide are already patched, but RHEL4 is most likely not patched. Dan From sheltren at cs.ucsb.edu Wed Oct 26 12:09:56 2005 From: sheltren at cs.ucsb.edu (Jeff Sheltren) Date: Wed, 26 Oct 2005 08:09:56 -0400 Subject: plague-server dying In-Reply-To: <1130328177.2627.4.camel@localhost.localdomain> References: <1130328177.2627.4.camel@localhost.localdomain> Message-ID: On Oct 26, 2005, at 8:02 AM, Dan Williams wrote: > > Hmm. Can you try to run the python process under gdb? Essentially: > > gdb /usr/bin/python > set args /usr/bin/plague-server -c /etc/plague/server/plague- > server.cfg ... > run > > Then just let it run for a while, and see where it segfaults. It > would > also be helpful to install python-debuginfo and pyOpenSSL-debuginfo > too > so that the gdb backtrace is more useful. > > Come to think of it, are you using SSL in your buildsystem at all, and > if so, have you patched your copy of pyOpenSSL? This is probably > something that should be documented :) FC4 and rawhide are already > patched, but RHEL4 is most likely not patched. > > Dan > Yes, I am using SSL for builders and for users. I haven't touched pyOpenSSL - could you direct me to the needed patch(es)? I'll try running it under gdb to see what happens in the meantime. Thanks, Jeff From dcbw at redhat.com Wed Oct 26 13:52:46 2005 From: dcbw at redhat.com (Dan Williams) Date: Wed, 26 Oct 2005 09:52:46 -0400 Subject: plague-server dying In-Reply-To: References: <1130328177.2627.4.camel@localhost.localdomain> Message-ID: <1130334766.2391.6.camel@dhcp83-40.boston.redhat.com> On Wed, 2005-10-26 at 08:09 -0400, Jeff Sheltren wrote: > On Oct 26, 2005, at 8:02 AM, Dan Williams wrote: > > > > Hmm. Can you try to run the python process under gdb? Essentially: > > > > gdb /usr/bin/python > > set args /usr/bin/plague-server -c /etc/plague/server/plague- > > server.cfg ... > > run > > > > Then just let it run for a while, and see where it segfaults. It > > would > > also be helpful to install python-debuginfo and pyOpenSSL-debuginfo > > too > > so that the gdb backtrace is more useful. > > > > Come to think of it, are you using SSL in your buildsystem at all, and > > if so, have you patched your copy of pyOpenSSL? This is probably > > something that should be documented :) FC4 and rawhide are already > > patched, but RHEL4 is most likely not patched. > > > > Dan > > > > Yes, I am using SSL for builders and for users. I haven't touched > pyOpenSSL - could you direct me to the needed patch(es)? Ok, that's probably the issue. pyOpenSSL does not do proper Python locking and such for certificate verification callbacks, and because plague makes heavy use of threads, things fall over quite quickly. You can get the patch here: http://cvs.fedora.redhat.com/viewcvs/*checkout*/extras-buildsys/pyOpenSSL-threadsafe.patch?root=fedora And you can find an SRPM for RHEL4 with the patch included here: http://people.redhat.com/dcbw/pyOpenSSL/pyOpenSSL-0.6-1.p23.1.el4.src.rpm Should be able to just rebuild that SRPM (if you have any problems with that, don't hesitate to ask) and you'll be good to go. Dan From sheltren at cs.ucsb.edu Wed Oct 26 18:48:14 2005 From: sheltren at cs.ucsb.edu (Jeff Sheltren) Date: Wed, 26 Oct 2005 14:48:14 -0400 Subject: Patch to store user info in different DB engines Message-ID: <66104028-31AE-46CA-B74C-2ED87983F94C@cs.ucsb.edu> I'm attaching a patch which uses DBManager for interacting with the user database rather than always relying upon sqlite. This way, if you use pgsql or mysql for your database back-end, user information will be stored there as well. The problem is, this will screw up current installations which are using pgsql/mysql because the old code assumes sqlite is used for user info, so suddenly your user database will be empty... I guess the easiest way around this is to write a quick script to pull everything out of the sqlite database and put it into mysql/pgsql. Also, I'll look at updating the user-manager.py script. In fact, I guess it may be easiest to add export/import functions to that for migrating the user info. Let me know what you think. -Jeff -------------- next part -------------- A non-text attachment was scrubbed... Name: plague-0.4-mysql_user_support.patch Type: application/octet-stream Size: 3863 bytes Desc: not available URL: From sheltren at cs.ucsb.edu Wed Oct 26 19:20:21 2005 From: sheltren at cs.ucsb.edu (Jeff Sheltren) Date: Wed, 26 Oct 2005 15:20:21 -0400 Subject: Patch to store user info in different DB engines In-Reply-To: <66104028-31AE-46CA-B74C-2ED87983F94C@cs.ucsb.edu> References: <66104028-31AE-46CA-B74C-2ED87983F94C@cs.ucsb.edu> Message-ID: <3BF69E00-889E-4214-8CAF-621ADE624444@cs.ucsb.edu> On Oct 26, 2005, at 2:48 PM, Jeff Sheltren wrote: > I'm attaching a patch which uses DBManager for interacting with the > user database rather than always relying upon sqlite. This way, if > you use pgsql or mysql for your database back-end, user information > will be stored there as well. Whoops, I forgot to include the diff for DBManager.py in that file. I'm attaching an updated patch which also changes DBManager.py so that the users table will be created as needed. -Jeff -------------- next part -------------- A non-text attachment was scrubbed... Name: plague-0.4-mysql_user_support.patch Type: application/octet-stream Size: 4974 bytes Desc: not available URL: From sheltren at cs.ucsb.edu Thu Oct 27 11:50:09 2005 From: sheltren at cs.ucsb.edu (Jeff Sheltren) Date: Thu, 27 Oct 2005 07:50:09 -0400 Subject: Patch to honor configfile flag Message-ID: <3B2D523E-E85D-4DD2-9E78-D4134B8CE53A@cs.ucsb.edu> Currently the configfile /etc/plague/server/plague-server.cfg is hard coded in plague-server, and it will not use a different file even if specified with the -c option. This patch changes plague-server to use the config file specified with -c, and also sets a default of / etc/plague/server/plague-server.cfg so that you do not have to use the -c option if you want to use the default config location. -Jeff -------------- next part -------------- A non-text attachment was scrubbed... Name: plague-0.4-configfile.patch Type: application/octet-stream Size: 1517 bytes Desc: not available URL: From dcbw at redhat.com Thu Oct 27 14:33:14 2005 From: dcbw at redhat.com (Dan Williams) Date: Thu, 27 Oct 2005 10:33:14 -0400 Subject: Patch to honor configfile flag In-Reply-To: <3B2D523E-E85D-4DD2-9E78-D4134B8CE53A@cs.ucsb.edu> References: <3B2D523E-E85D-4DD2-9E78-D4134B8CE53A@cs.ucsb.edu> Message-ID: <1130423594.2963.3.camel@dhcp83-40.boston.redhat.com> On Thu, 2005-10-27 at 07:50 -0400, Jeff Sheltren wrote: > Currently the configfile /etc/plague/server/plague-server.cfg is hard > coded in plague-server, and it will not use a different file even if > specified with the -c option. This patch changes plague-server to > use the config file specified with -c, and also sets a default of / > etc/plague/server/plague-server.cfg so that you do not have to use > the -c option if you want to use the default config location. Committed, thanks! Dan From sheltren at cs.ucsb.edu Thu Oct 27 20:07:45 2005 From: sheltren at cs.ucsb.edu (Jeff Sheltren) Date: Thu, 27 Oct 2005 16:07:45 -0400 Subject: Patch to honor configfile flag In-Reply-To: <1130423594.2963.3.camel@dhcp83-40.boston.redhat.com> References: <3B2D523E-E85D-4DD2-9E78-D4134B8CE53A@cs.ucsb.edu> <1130423594.2963.3.camel@dhcp83-40.boston.redhat.com> Message-ID: <03B40DA5-A7EC-43BD-8B6B-DB55E1EE5E46@cs.ucsb.edu> On Oct 27, 2005, at 10:33 AM, Dan Williams wrote: > > Committed, thanks! > > Dan > You're welcome! I was busy today enjoying my birthday and playing beach volleyball, but I am about half done with the changes to the user-manager script for using DBManager. I should have it done tomorrow. -Jeff From sheltren at cs.ucsb.edu Fri Oct 28 18:34:25 2005 From: sheltren at cs.ucsb.edu (Jeff Sheltren) Date: Fri, 28 Oct 2005 14:34:25 -0400 Subject: Patch to store user info in different DB engines In-Reply-To: <3BF69E00-889E-4214-8CAF-621ADE624444@cs.ucsb.edu> References: <66104028-31AE-46CA-B74C-2ED87983F94C@cs.ucsb.edu> <3BF69E00-889E-4214-8CAF-621ADE624444@cs.ucsb.edu> Message-ID: <301E5BB8-6F02-4C51-BB35-B09408D709ED@cs.ucsb.edu> Here is a patch for user-manager.py to use DBManager for its database access. This allows user-manager.py to work with multiple db types. -Jeff -------------- next part -------------- A non-text attachment was scrubbed... Name: plague-0.4-user-manager.patch Type: application/octet-stream Size: 4008 bytes Desc: not available URL: From chris.weyl at gmail.com Fri Oct 28 20:41:22 2005 From: chris.weyl at gmail.com (Chris Weyl) Date: Fri, 28 Oct 2005 16:41:22 -0400 Subject: Patch to store user info in different DB engines In-Reply-To: <301E5BB8-6F02-4C51-BB35-B09408D709ED@cs.ucsb.edu> References: <66104028-31AE-46CA-B74C-2ED87983F94C@cs.ucsb.edu> <3BF69E00-889E-4214-8CAF-621ADE624444@cs.ucsb.edu> <301E5BB8-6F02-4C51-BB35-B09408D709ED@cs.ucsb.edu> Message-ID: <7dd7ab490510281341m50f6e14emb1235dd7134ea4fc@mail.gmail.com> On 10/28/05, Jeff Sheltren wrote: > Here is a patch for user-manager.py to use DBManager for its database > access. This allows user-manager.py to work with multiple db types. I haven't had a chance to take a look at the patch yet, but does it provide for a way to keep the userdb & jobdb separate? I'm thinking (and might be wrong here ) that if it's easy to do, it might be useful to allow for, e.g, jobdb in mysql but userdb in sqlite, especially in terms of upgrades. Maybe a flag, e.g. "job_db_engine" or somesuch in the server config, that if left out, defaults to the overall db being used. -Chris From sheltren at cs.ucsb.edu Fri Oct 28 22:56:09 2005 From: sheltren at cs.ucsb.edu (Jeff Sheltren) Date: Fri, 28 Oct 2005 18:56:09 -0400 Subject: Patch to store user info in different DB engines In-Reply-To: <7dd7ab490510281341m50f6e14emb1235dd7134ea4fc@mail.gmail.com> References: <66104028-31AE-46CA-B74C-2ED87983F94C@cs.ucsb.edu> <3BF69E00-889E-4214-8CAF-621ADE624444@cs.ucsb.edu> <301E5BB8-6F02-4C51-BB35-B09408D709ED@cs.ucsb.edu> <7dd7ab490510281341m50f6e14emb1235dd7134ea4fc@mail.gmail.com> Message-ID: <0D179712-BDB4-4CF0-A55D-EB9F28639061@cs.ucsb.edu> On Oct 28, 2005, at 4:41 PM, Chris Weyl wrote: > On 10/28/05, Jeff Sheltren wrote: > >> Here is a patch for user-manager.py to use DBManager for its database >> access. This allows user-manager.py to work with multiple db types. >> > > I haven't had a chance to take a look at the patch yet, but does it > provide for a way to keep the userdb & jobdb separate? I'm thinking > (and might be wrong here ) that if it's easy to do, it might be > useful to allow for, e.g, jobdb in mysql but userdb in sqlite, > especially in terms of upgrades. Maybe a flag, e.g. "job_db_engine" > or somesuch in the server config, that if left out, defaults to the > overall db being used. > > -Chris > No it doesn't. The idea was that all database information should be stored in the same location. Off the top of my head, I think that using separate database types would complicate DBManager.py quite a bit. Currently it just assumes that everything is in the same DB, so there would need to be extra checks to see which DB holds which information, and then use the appropriate DB for each case. In fact, this would probably mean using two DBManager objects, one for jobs and one for users (currently only one DBManager object is used by plague-server). I don't know if that is a bad or good thing, I just think it is pretty complicated compared to the current situation. I think the easier thing to do is to write a tool for extracting all user info from sqlite in order to import it into mysql/pgsql. I'll try to work on that this weekend. -Jeff From chris.weyl at gmail.com Fri Oct 28 23:08:52 2005 From: chris.weyl at gmail.com (Chris Weyl) Date: Fri, 28 Oct 2005 19:08:52 -0400 Subject: Patch to store user info in different DB engines In-Reply-To: <0D179712-BDB4-4CF0-A55D-EB9F28639061@cs.ucsb.edu> References: <66104028-31AE-46CA-B74C-2ED87983F94C@cs.ucsb.edu> <3BF69E00-889E-4214-8CAF-621ADE624444@cs.ucsb.edu> <301E5BB8-6F02-4C51-BB35-B09408D709ED@cs.ucsb.edu> <7dd7ab490510281341m50f6e14emb1235dd7134ea4fc@mail.gmail.com> <0D179712-BDB4-4CF0-A55D-EB9F28639061@cs.ucsb.edu> Message-ID: <7dd7ab490510281608k7b3c0026yacc95b2765dfd004@mail.gmail.com> On 10/28/05, Jeff Sheltren wrote: > No it doesn't. The idea was that all database information should be > stored in the same location. Off the top of my head, I think that > using separate database types would complicate DBManager.py quite a > bit. Currently it just assumes that everything is in the same DB, so > there would need to be extra checks to see which DB holds which > information, and then use the appropriate DB for each case. In fact, > this would probably mean using two DBManager objects, one for jobs > and one for users (currently only one DBManager object is used by > plague-server). I don't know if that is a bad or good thing, I just > think it is pretty complicated compared to the current situation. Sounds... messy. Hm. > I think the easier thing to do is to write a tool for extracting all > user info from sqlite in order to import it into mysql/pgsql. I'll > try to work on that this weekend. Gotcha. I think you're right... just trying to keep the future 0.4 -> 0.5 migration pain at a lower level than 0.3 -> 0.4. :) -Chris From sheltren at cs.ucsb.edu Fri Oct 28 23:39:27 2005 From: sheltren at cs.ucsb.edu (Jeff Sheltren) Date: Fri, 28 Oct 2005 19:39:27 -0400 Subject: Patch to store user info in different DB engines In-Reply-To: <7dd7ab490510281608k7b3c0026yacc95b2765dfd004@mail.gmail.com> References: <66104028-31AE-46CA-B74C-2ED87983F94C@cs.ucsb.edu> <3BF69E00-889E-4214-8CAF-621ADE624444@cs.ucsb.edu> <301E5BB8-6F02-4C51-BB35-B09408D709ED@cs.ucsb.edu> <7dd7ab490510281341m50f6e14emb1235dd7134ea4fc@mail.gmail.com> <0D179712-BDB4-4CF0-A55D-EB9F28639061@cs.ucsb.edu> <7dd7ab490510281608k7b3c0026yacc95b2765dfd004@mail.gmail.com> Message-ID: On Oct 28, 2005, at 7:08 PM, Chris Weyl wrote: > > Gotcha. I think you're right... just trying to keep the future 0.4 > -> 0.5 migration pain at a lower level than 0.3 -> 0.4. :) > > -Chris > Yeah, I agree with that; I hate to break stuff with a version upgrade. But personally I think that may be the best solution instead of creating a bunch of workarounds to keep older setups working. I guess that ultimately it's up to Dan - so, what do you think? :) -Jeff From kad at crystalowl.net Sun Oct 30 19:26:21 2005 From: kad at crystalowl.net (Alexandr Kanevskiy) Date: Sun, 30 Oct 2005 21:26:21 +0200 Subject: [patch] Patch to fix double call to "mock clean" in case, if mock will failed on "prep" stage. Message-ID: <1130700381.12218.9.camel@orava.homeip.net> Hi all. I found one small issue, that in case of mock will fail on prep stage (no space on device, or repository access failed) mock will return earlier, and in _status_prepping(), _watch_mock function will call _start_cleanup(). After that _start_cleanup will change next state to be "cleanup". But, this "if" below, will change next status to "building" regardless of previous error: if not self._mock_using_repo(): self._status = 'building' that meas scenario: _status_prepping _watch_mock mock clean _status_building _watch_mock mock clean Attached patch solves that. -- Alexandr Kanevskiy. -------------- next part -------------- A non-text attachment was scrubbed... Name: plague-doubleclean.patch Type: text/x-patch Size: 583 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From oliver at linux-kernel.at Mon Oct 31 12:52:59 2005 From: oliver at linux-kernel.at (Oliver Falk) Date: Mon, 31 Oct 2005 13:52:59 +0100 Subject: RFE: mock kill Message-ID: <436613AB.1040702@linux-kernel.at> Hi! I allready wrote Seth and he asked me to write it to the list - so here we go. I'd like to have a mock command 'kill'. If I run a mock-build process it runs all chroot command as root and if I want to stop mock from running as a normal user I can't.... So a 'mock kill -r --uniqueext kill' would be a great thing. :-) Best, Oliver From dcbw at redhat.com Mon Oct 31 17:02:07 2005 From: dcbw at redhat.com (Dan Williams) Date: Mon, 31 Oct 2005 12:02:07 -0500 Subject: [patch] Patch to fix double call to "mock clean" in case, if mock will failed on "prep" stage. In-Reply-To: <1130700381.12218.9.camel@orava.homeip.net> References: <1130700381.12218.9.camel@orava.homeip.net> Message-ID: <1130778128.2516.1.camel@dhcp83-40.boston.redhat.com> On Sun, 2005-10-30 at 21:26 +0200, Alexandr Kanevskiy wrote: > Hi all. > > I found one small issue, that in case of mock will fail on prep stage > (no space on device, or repository access failed) mock will return > earlier, and in _status_prepping(), _watch_mock function will call > _start_cleanup(). After that _start_cleanup will change next state to be > "cleanup". But, this "if" below, will change next status to "building" > regardless of previous error: > if not self._mock_using_repo(): > self._status = 'building' > > that meas scenario: > _status_prepping > _watch_mock > mock clean > _status_building > _watch_mock > mock clean > > Attached patch solves that. Good catch. Thanks! It's been committed to HEAD and STABLE_4. Dan From kad at crystalowl.net Mon Oct 31 18:58:59 2005 From: kad at crystalowl.net (Alexandr Kanevskiy) Date: Mon, 31 Oct 2005 20:58:59 +0200 Subject: [patch] Patch to fix double call to "mock clean" in case, if mock will failed on "prep" stage. In-Reply-To: <1130778128.2516.1.camel@dhcp83-40.boston.redhat.com> References: <1130700381.12218.9.camel@orava.homeip.net> <1130778128.2516.1.camel@dhcp83-40.boston.redhat.com> Message-ID: <1130785140.5724.12.camel@orava.homeip.net> ? ???, 31/10/2005 ? 12:02 -0500, Dan Williams ?????: > On Sun, 2005-10-30 at 21:26 +0200, Alexandr Kanevskiy wrote: > > Attached patch solves that. > > Good catch. Thanks! It's been committed to HEAD and STABLE_4. Welcome ;) Another findings from today looking at code: You have work_dir declared in global scope (line 847) and use it from get_url_for_file(), line 62. In that function will be redefining of global variable. But expected to be local variable assignment. Maybe rewrite that function to use _work_dir instead work_dir ? The same thing is for 'hostname', 'cfg','port' in get_url_for_file()... or maybe it will be better that global scope code as def main(): ..... and put at bottom: if __name__ == '__main__': main() ? More funny things with work_dir, are in BuilderMock.__init__(), line 102, and in BuilderMock.run(), line 438. I think, for BuilderMock class, all references to work_dir better to be rewritten as "self._work_dir" ? And few more findings in import section: import time - listed twice. SimpleXMLRPCServer, xmlrpclib imported, but not used. -- Alexandr Kanevskiy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From chrisw at osdl.org Mon Oct 17 20:16:32 2005 From: chrisw at osdl.org (Chris Wright) Date: Mon, 17 Oct 2005 20:16:32 -0000 Subject: Extras build system back up In-Reply-To: <1129572538.28887.1.camel@dhcp83-40.boston.redhat.com> References: <1129572538.28887.1.camel@dhcp83-40.boston.redhat.com> Message-ID: <20051017201619.GB30904@shell0.pdx.osdl.net> * Dan Williams (dcbw at redhat.com) wrote: > We're back up and using the latest plague code. yum update your > plague-client and you should be good to go. Anything else that needs to be done? w at vas devel]$ plague-client build git-core git-core-0_99_8d-1_fc5 devel Traceback (most recent call last): File "/usr/bin/plague-client", line 426, in ? cli.dispatch(cmd, sys.argv[2:]) File "/usr/bin/plague-client", line 372, in dispatch func(args) File "/usr/bin/plague-client", line 172, in _cmd_build self._enqueue(is_srpm, package, source, target_alias) File "/usr/bin/plague-client", line 197, in _enqueue if self._cfg.get_bool("Server", "allow_uploads"): File "/usr/lib/python2.3/site-packages/plague/BaseConfig.py", line 51, in get_bool opt = self.get_option(section, name) File "/usr/lib/python2.3/site-packages/plague/BaseConfig.py", line 45, in get_option raise ConfigError("Config file %s does not have option: %s/%s" % (self._filename, section, name)) plague.BaseConfig.ConfigError: Config file /home/chrisw/.plague-client.cfg does not have option: Server/allow_uploads [chrisw at vas devel]$ vim /home/chrisw/.plague-client.cfg [chrisw at vas devel]$ plague-client build git-core git-core-0_99_8d-1_fc5 devel Error connecting to build server: '' From chrisw at osdl.org Mon Oct 17 21:28:15 2005 From: chrisw at osdl.org (Chris Wright) Date: Mon, 17 Oct 2005 21:28:15 -0000 Subject: Extras build system back up In-Reply-To: <1129583751.1418.3.camel@dhcp83-40.boston.redhat.com> References: <1129572538.28887.1.camel@dhcp83-40.boston.redhat.com> <20051017201619.GB30904@shell0.pdx.osdl.net> <1129583751.1418.3.camel@dhcp83-40.boston.redhat.com> Message-ID: <20051017212801.GJ5856@shell0.pdx.osdl.net> * Dan Williams (dcbw at redhat.com) wrote: > As I see you've figured out, it should be fixed now. Server-side issue > with some database fields being way too small. They are larger now :) Yes, thanks. Is [Server] ... allow_uploads = True the recommended change to config? I notice plague list once creates config with allow_uploads = no thanks, -chris