From dcbw at redhat.com Tue Nov 1 14:50:44 2005 From: dcbw at redhat.com (Dan Williams) Date: Tue, 01 Nov 2005 09:50:44 -0500 Subject: Patch to store user info in different DB engines In-Reply-To: References: <66104028-31AE-46CA-B74C-2ED87983F94C@cs.ucsb.edu> <3BF69E00-889E-4214-8CAF-621ADE624444@cs.ucsb.edu> <301E5BB8-6F02-4C51-BB35-B09408D709ED@cs.ucsb.edu> <7dd7ab490510281341m50f6e14emb1235dd7134ea4fc@mail.gmail.com> <0D179712-BDB4-4CF0-A55D-EB9F28639061@cs.ucsb.edu> <7dd7ab490510281608k7b3c0026yacc95b2765dfd004@mail.gmail.com> Message-ID: <1130856645.2627.3.camel@localhost.localdomain> On Fri, 2005-10-28 at 19:39 -0400, Jeff Sheltren wrote: > On Oct 28, 2005, at 7:08 PM, Chris Weyl wrote: > > > > Gotcha. I think you're right... just trying to keep the future 0.4 > > -> 0.5 migration pain at a lower level than 0.3 -> 0.4. :) > > > > -Chris > > > > Yeah, I agree with that; I hate to break stuff with a version > upgrade. But personally I think that may be the best solution > instead of creating a bunch of workarounds to keep older setups > working. I guess that ultimately it's up to Dan - so, what do you > think? :) So here's my take... I think unified database is fine, but we do need to provide an easy migration path IMHO. Whether that's just a one-off tool that does transfer the users for you, or whatever, I think that would save people quite a bit of pain later. On the other hand, I don't think it would be _that_ much work to pull job and user data from different databases, but that strikes me as more clutter in the code for little obvious benefit... So my vote is for consolidating databases for 0.5, and providing a small tool to migrate for you. Anyone want to do that tool? :) Dan From sheltren at cs.ucsb.edu Tue Nov 1 18:43:22 2005 From: sheltren at cs.ucsb.edu (Jeff Sheltren) Date: Tue, 1 Nov 2005 14:43:22 -0400 Subject: Patch to store user info in different DB engines In-Reply-To: <1130856645.2627.3.camel@localhost.localdomain> References: <66104028-31AE-46CA-B74C-2ED87983F94C@cs.ucsb.edu> <3BF69E00-889E-4214-8CAF-621ADE624444@cs.ucsb.edu> <301E5BB8-6F02-4C51-BB35-B09408D709ED@cs.ucsb.edu> <7dd7ab490510281341m50f6e14emb1235dd7134ea4fc@mail.gmail.com> <0D179712-BDB4-4CF0-A55D-EB9F28639061@cs.ucsb.edu> <7dd7ab490510281608k7b3c0026yacc95b2765dfd004@mail.gmail.com> <1130856645.2627.3.camel@localhost.localdomain> Message-ID: <87FBC17A-0FE6-43CB-A7BE-5EE41D10F393@cs.ucsb.edu> On Nov 1, 2005, at 10:50 AM, Dan Williams wrote: > > So my vote is for consolidating databases for 0.5, and providing a > small > tool to migrate for you. Anyone want to do that tool? :) Sure, I'll work on that. -Jeff From sheltren at cs.ucsb.edu Wed Nov 2 13:57:07 2005 From: sheltren at cs.ucsb.edu (Jeff Sheltren) Date: Wed, 2 Nov 2005 09:57:07 -0400 Subject: Patch to store user info in different DB engines In-Reply-To: <87FBC17A-0FE6-43CB-A7BE-5EE41D10F393@cs.ucsb.edu> References: <66104028-31AE-46CA-B74C-2ED87983F94C@cs.ucsb.edu> <3BF69E00-889E-4214-8CAF-621ADE624444@cs.ucsb.edu> <301E5BB8-6F02-4C51-BB35-B09408D709ED@cs.ucsb.edu> <7dd7ab490510281341m50f6e14emb1235dd7134ea4fc@mail.gmail.com> <0D179712-BDB4-4CF0-A55D-EB9F28639061@cs.ucsb.edu> <7dd7ab490510281608k7b3c0026yacc95b2765dfd004@mail.gmail.com> <1130856645.2627.3.camel@localhost.localdomain> <87FBC17A-0FE6-43CB-A7BE-5EE41D10F393@cs.ucsb.edu> Message-ID: On Nov 1, 2005, at 2:43 PM, Jeff Sheltren wrote: > On Nov 1, 2005, at 10:50 AM, Dan Williams wrote: > >> >> So my vote is for consolidating databases for 0.5, and providing a >> small >> tool to migrate for you. Anyone want to do that tool? :) > > > Sure, I'll work on that. > > -Jeff OK, I don't know how pretty this is, I'm new to this whole python thing :) But I'm attaching a script for migrating the user data from sqlite to pgsql/mysql. I've tested it with MySQL and it seems to work great. Let me know if you notice any bugs or something that needs to be changed. Usage is like this: # ./plague-user-migration.py /etc/plague/server/userdb /etc/plague/ server/plague-server.cfg Using database engine mysql. Added user: sheltren at host.domain.com Added user: apache at host.domain.com Done! -Jeff -------------- next part -------------- A non-text attachment was scrubbed... Name: plague-user-migration.py Type: text/x-python-script Size: 4348 bytes Desc: not available URL: From sheltren at cs.ucsb.edu Thu Nov 3 11:30:49 2005 From: sheltren at cs.ucsb.edu (Jeff Sheltren) Date: Thu, 3 Nov 2005 07:30:49 -0400 Subject: Patch to store user info in different DB engines In-Reply-To: References: <66104028-31AE-46CA-B74C-2ED87983F94C@cs.ucsb.edu> <3BF69E00-889E-4214-8CAF-621ADE624444@cs.ucsb.edu> <301E5BB8-6F02-4C51-BB35-B09408D709ED@cs.ucsb.edu> <7dd7ab490510281341m50f6e14emb1235dd7134ea4fc@mail.gmail.com> <0D179712-BDB4-4CF0-A55D-EB9F28639061@cs.ucsb.edu> <7dd7ab490510281608k7b3c0026yacc95b2765dfd004@mail.gmail.com> <1130856645.2627.3.camel@localhost.localdomain> <87FBC17A-0FE6-43CB-A7BE-5EE41D10F393@cs.ucsb.edu> Message-ID: On Nov 2, 2005, at 9:57 AM, Jeff Sheltren wrote: > > I realized there are a couple of things that I didn't like, so I've made an updated script, located here: http://www.cs.ucsb.edu/~jeff/plague-user-migration.py Now it will not attempt to migrate from sqlite to sqlite :) Also, all errors are output to stderr - before some were going to stdout. -Jeff From chris.weyl at gmail.com Tue Nov 8 15:33:44 2005 From: chris.weyl at gmail.com (Chris Weyl) Date: Tue, 8 Nov 2005 10:33:44 -0500 Subject: Hrm, SSL issues? Message-ID: <7dd7ab490511080733o5ff4921ayda630cf7ef873485@mail.gmail.com> Ok, so this is probably something trivial, but I've been seeing these SSL errors. I don't know if it had to do with the recent plague updates or something else on this system (I'd imagine not, as it's dedicated to the buildsys), but it's rather perplexing. It seems to be referring to an expired cert, but after peeking at the certs all of them seem to expire in 2015. >From the client side: ----------------------------- [cweyl at twink FC-3]$ plague-client list Traceback (most recent call last): File "/usr/bin/plague-client", line 420, in ? cli = PlagueClient(os.path.expanduser(cfg_file)) File "/usr/bin/plague-client", line 85, in __init__ self._check_api_version(self._server) File "/usr/bin/plague-client", line 90, in _check_api_version server_ver = server.api_version() File "/usr/lib/python2.4/xmlrpclib.py", line 1096, in __call__ return self.__send(self.__name, args) File "/usr/lib/python2.4/xmlrpclib.py", line 1383, in __request verbose=self.__verbose File "/usr/lib/python2.4/xmlrpclib.py", line 1129, in request self.send_content(h, request_body) File "/usr/lib/python2.4/xmlrpclib.py", line 1243, in send_content connection.endheaders() File "/usr/lib/python2.4/httplib.py", line 794, in endheaders self._send_output() File "/usr/lib/python2.4/httplib.py", line 675, in _send_output self.send(msg) File "/usr/lib/python2.4/httplib.py", line 654, in send self.sock.sendall(str) File "/usr/lib/python2.4/site-packages/plague/SSLConnection.py", line 79, in sendall return self.__dict__["conn"].sendall(data, flags) OpenSSL.SSL.Error: [('SSL routines', 'SSL3_GET_SERVER_CERTIFICATE', 'certificate verify failed')] >From the server side: ------------------------------ Exception happened during processing of request from ('xx.xx.xx.xx', 59214) Traceback (most recent call last): File "/usr/lib/python2.4/SocketServer.py", line 463, in process_request_thread self.finish_request(request, client_address) File "/usr/lib/python2.4/SocketServer.py", line 254, in finish_request self.RequestHandlerClass(request, client_address, self) File "/usr/lib/python2.4/SocketServer.py", line 521, in __init__ self.handle() File "/usr/lib/python2.4/BaseHTTPServer.py", line 314, in handle self.handle_one_request() File "/usr/lib/python2.4/BaseHTTPServer.py", line 297, in handle_one_request self.raw_requestline = self.rfile.readline() File "/usr/lib/python2.4/socket.py", line 340, in readline data = self._sock.recv(self._rbufsize) File "/usr/lib/python2.4/site-packages/plague/SSLConnection.py", line 97, in recv return self.__dict__["conn"].recv(bufsize, flags) Error: [('SSL routines', 'SSL3_READ_BYTES', 'sslv3 alert certificate expired'), ('SSL routines', 'SSL3_READ_BYTES', 'ssl handshake failure')] ---------------------------------------- Any thoughts? Google isn't turning up anything that looks useful. -Chris From dcbw at redhat.com Tue Nov 8 16:38:38 2005 From: dcbw at redhat.com (Dan Williams) Date: Tue, 08 Nov 2005 11:38:38 -0500 Subject: Extras buildsystem down Message-ID: <1131467918.5308.2.camel@dhcp83-40.boston.redhat.com> Hi, All builders in the Extras buildsystem are currently unreachable by the build server, due to errors in firewall rules that we're currently fixing, plus prelink screwed up the PPC builder's pcre & grep, which are being fixed. Please hang on while we correct the situation. No requeueing of jobs should be necessary as they will automatically start when the builders come back up. Thanks! Dan From dcbw at redhat.com Tue Nov 8 16:41:41 2005 From: dcbw at redhat.com (Dan Williams) Date: Tue, 08 Nov 2005 11:41:41 -0500 Subject: Hrm, SSL issues? In-Reply-To: <7dd7ab490511080733o5ff4921ayda630cf7ef873485@mail.gmail.com> References: <7dd7ab490511080733o5ff4921ayda630cf7ef873485@mail.gmail.com> Message-ID: <1131468102.5308.6.camel@dhcp83-40.boston.redhat.com> On Tue, 2005-11-08 at 10:33 -0500, Chris Weyl wrote: > Ok, so this is probably something trivial, but I've been seeing these > SSL errors. I don't know if it had to do with the recent plague > updates or something else on this system (I'd imagine not, as it's > dedicated to the buildsys), but it's rather perplexing. It seems to > be referring to an expired cert, but after peeking at the certs all of > them seem to expire in 2015. First thing I'd try in this situation is using openssl to try to verify the certificates against their CA certificate. If the openssl verify fails, there's something in the certificate that's bad. Also make sure the CA certificate hasn't expired. Previous version of the plague certhelper.py utility incorrectly expired CA certificates after 30 days, which has been fixed. openssl verify [-CApath directory] cert.pem Something like ^^^^^ should do the trick, you may have to check on the exact arguments to use. Dan From chris.weyl at gmail.com Tue Nov 8 19:23:27 2005 From: chris.weyl at gmail.com (Chris Weyl) Date: Tue, 8 Nov 2005 14:23:27 -0500 Subject: Hrm, SSL issues? In-Reply-To: <1131468102.5308.6.camel@dhcp83-40.boston.redhat.com> References: <7dd7ab490511080733o5ff4921ayda630cf7ef873485@mail.gmail.com> <1131468102.5308.6.camel@dhcp83-40.boston.redhat.com> Message-ID: <7dd7ab490511081123h51cd5098s8e2d12cf79b4893@mail.gmail.com> On 11/8/05, Dan Williams wrote: > First thing I'd try in this situation is using openssl to try to verify > the certificates against their CA certificate. If the openssl verify > fails, there's something in the certificate that's bad. Also make sure > the CA certificate hasn't expired. > > Previous version of the plague certhelper.py utility incorrectly expired > CA certificates after 30 days, which has been fixed. Nuts. It looks like that's exactly what happened here... The individual certs claim to be good to 2015, but the CA certs are definitely expired: "error 10 at 0 depth lookup:certificate has expired". I don't suppose there's an easy fix for this? (Never too early in the week for wishful thinking.) Or is the fix to go and recreate the CA's, and reissue all new certs to everyone? > openssl verify [-CApath directory] cert.pem > > Something like ^^^^^ should do the trick, you may have to check on the > exact arguments to use. For the sake of others googling: openssl verify -CApath /etc/plague/ca_dir /etc/plague/ca_dir/buildsystem_ca_cert.pem does the trick. -Chris From dcbw at redhat.com Wed Nov 9 02:05:31 2005 From: dcbw at redhat.com (Dan Williams) Date: Tue, 08 Nov 2005 21:05:31 -0500 Subject: Hrm, SSL issues? In-Reply-To: <7dd7ab490511081123h51cd5098s8e2d12cf79b4893@mail.gmail.com> References: <7dd7ab490511080733o5ff4921ayda630cf7ef873485@mail.gmail.com> <1131468102.5308.6.camel@dhcp83-40.boston.redhat.com> <7dd7ab490511081123h51cd5098s8e2d12cf79b4893@mail.gmail.com> Message-ID: <1131501931.3020.12.camel@localhost.localdomain> On Tue, 2005-11-08 at 14:23 -0500, Chris Weyl wrote: > On 11/8/05, Dan Williams wrote: > > First thing I'd try in this situation is using openssl to try to verify > > the certificates against their CA certificate. If the openssl verify > > fails, there's something in the certificate that's bad. Also make sure > > the CA certificate hasn't expired. > > > > Previous version of the plague certhelper.py utility incorrectly expired > > CA certificates after 30 days, which has been fixed. > > Nuts. It looks like that's exactly what happened here... The > individual certs claim to be good to 2015, but the CA certs are > definitely expired: "error 10 at 0 depth lookup:certificate has > expired". > > I don't suppose there's an easy fix for this? (Never too early in the > week for wishful thinking.) Or is the fix to go and recreate the > CA's, and reissue all new certs to everyone? Unfortunately, I think that's the fix :( Sorry about that, it was my fault originally though I'll note that for whatever reason the line [CA_default] default_days = 3650 in the openssl conf file didn't actually make the CA certificate valid for 10 years, necessitating using the command-line option... go figure. Dan From katzj at redhat.com Thu Nov 10 20:26:17 2005 From: katzj at redhat.com (Jeremy Katz) Date: Thu, 10 Nov 2005 15:26:17 -0500 Subject: Extras buildsystem back up Message-ID: <1131654377.3533.99.camel@bree.local.net> The Fedora Extras buildsystem is now back up. Apologies for the downtime. There were some new firewall rules put into place which didn't play well as well as one of the build machines booting into an older, non-working kernel. Things should be back to normal now. Additionally, the problems with uploading new tarballs to CVS should be corrected. This was caused by a change in a mod_ssl errata Thanks for your patience, Jeremy From chris.weyl at gmail.com Wed Nov 16 22:43:37 2005 From: chris.weyl at gmail.com (Chris Weyl) Date: Wed, 16 Nov 2005 17:43:37 -0500 Subject: Best way to sign packages before adding to the repos? Message-ID: <7dd7ab490511161443h65fa7875v265a157ce6f18fc7@mail.gmail.com> Hey all-- Ok, so here's a question. What's the best way to sign a package(s) before adding it to the repository? I was thinking that the easiest way to do this would be while it's sitting in the "needsign/success" state, but it looks like it's actually being added to the repository before that state. Is there an easy way to do this? Or even a not-so-easy one? -Chris From dcbw at redhat.com Thu Nov 17 03:12:45 2005 From: dcbw at redhat.com (Dan Williams) Date: Wed, 16 Nov 2005 22:12:45 -0500 Subject: Best way to sign packages before adding to the repos? In-Reply-To: <7dd7ab490511161443h65fa7875v265a157ce6f18fc7@mail.gmail.com> References: <7dd7ab490511161443h65fa7875v265a157ce6f18fc7@mail.gmail.com> Message-ID: <1132197166.2636.16.camel@localhost.localdomain> On Wed, 2005-11-16 at 17:43 -0500, Chris Weyl wrote: > Hey all-- > > Ok, so here's a question. What's the best way to sign a package(s) > before adding it to the repository? I was thinking that the easiest > way to do this would be while it's sitting in the "needsign/success" > state, but it looks like it's actually being added to the repository > before that state. > > Is there an easy way to do this? Or even a not-so-easy one? The way Fedora Extras does it right now, there's a cron job or something that pushes the built RPMs to the real repository directory and signs them at that point. If you want to use the repository the build server dumps built RPMs into, then you can use the repo scripts feature of the build server. There's a config option in each target file on the build server which accepts a path to the repo script. That script is run after RPMs have been copied to the repo, and is given with 1 argument": the target string for the repo, which takes the form of something like "fedora-development-core" or "fedora-extras-4". You could probably do the signing from that script. Some issues to know about repo scripts: they are called every time packages are added to the repository. That means, if you want to do stuff only at certain times, like midnight, you have to do time tracking yourself to make sure your script is only called every 24 hours. Also, the build server blocks while the script runs, so it will kill the script after 1 hour of runtime to make sure that stuff doesn't block for too long. This part hasn't been too tested or fleshed out, so if you think of ideas for improvements, feel free to propose them. Dan From chris.weyl at gmail.com Fri Nov 18 15:08:31 2005 From: chris.weyl at gmail.com (Chris Weyl) Date: Fri, 18 Nov 2005 10:08:31 -0500 Subject: Best way to sign packages before adding to the repos? In-Reply-To: <1132197166.2636.16.camel@localhost.localdomain> References: <7dd7ab490511161443h65fa7875v265a157ce6f18fc7@mail.gmail.com> <1132197166.2636.16.camel@localhost.localdomain> Message-ID: <7dd7ab490511180708n45fa32bese78f7c823d8b923e@mail.gmail.com> On 11/16/05, Dan Williams wrote: > The way Fedora Extras does it right now, there's a cron job or something > that pushes the built RPMs to the real repository directory and signs > them at that point. > > If you want to use the repository the build server dumps built RPMs > into, then you can use the repo scripts feature of the build server. > There's a config option in each target file on the build server which > accepts a path to the repo script. That script is run after RPMs have > been copied to the repo, and is given with 1 argument": the target > string for the repo, which takes the form of something like > "fedora-development-core" or "fedora-extras-4". You could probably do > the signing from that script. Ok, that makes sense; it seems to catch it right before adding it to the repos. > Some issues to know about repo scripts: they are called every time > packages are added to the repository. That means, if you want to do > stuff only at certain times, like midnight, you have to do time tracking > yourself to make sure your script is only called every 24 hours. Also, > the build server blocks while the script runs, so it will kill the > script after 1 hour of runtime to make sure that stuff doesn't block for > too long. > > This part hasn't been too tested or fleshed out, so if you think of > ideas for improvements, feel free to propose them. Hmmm... One thing that comes to mind, is that it would be useful if as a second & third parameter, the repo_script took the name of the package, and version-revision-etc that had just been built. That could save time, either by avoiding any magic to figure out which one had just been completed, or by allowing brute-force approaches caused by not wanting to create said magic:) Another thing, if the packages have already been added to the local repos when the build state is "needsign", then what's the point of that step? If we go and sign them, then we need to play with the repos cache/etc to maintain a valid state; if we pull them out and push them to another (signed/etc) repos, then it's not needed; if we don't sign them it's not needed. IMO, it would be far more useful if packages held in this state weren't actually added to the repos until after manually "finished". -Chris From dcbw at redhat.com Fri Nov 18 16:26:13 2005 From: dcbw at redhat.com (Dan Williams) Date: Fri, 18 Nov 2005 11:26:13 -0500 Subject: Best way to sign packages before adding to the repos? In-Reply-To: <7dd7ab490511180708n45fa32bese78f7c823d8b923e@mail.gmail.com> References: <7dd7ab490511161443h65fa7875v265a157ce6f18fc7@mail.gmail.com> <1132197166.2636.16.camel@localhost.localdomain> <7dd7ab490511180708n45fa32bese78f7c823d8b923e@mail.gmail.com> Message-ID: <1132331173.5377.7.camel@dhcp83-87.boston.redhat.com> On Fri, 2005-11-18 at 10:08 -0500, Chris Weyl wrote: > On 11/16/05, Dan Williams wrote: > > The way Fedora Extras does it right now, there's a cron job or something > > that pushes the built RPMs to the real repository directory and signs > > them at that point. > > > > If you want to use the repository the build server dumps built RPMs > > into, then you can use the repo scripts feature of the build server. > > There's a config option in each target file on the build server which > > accepts a path to the repo script. That script is run after RPMs have > > been copied to the repo, and is given with 1 argument": the target > > string for the repo, which takes the form of something like > > "fedora-development-core" or "fedora-extras-4". You could probably do > > the signing from that script. > > Ok, that makes sense; it seems to catch it right before adding it to the repos. Hmm, it actually runs the scripts _after_ packages have been copied to the repository. So maybe it's not exactly what you want. > > Some issues to know about repo scripts: they are called every time > > packages are added to the repository. That means, if you want to do > > stuff only at certain times, like midnight, you have to do time tracking > > yourself to make sure your script is only called every 24 hours. Also, > > the build server blocks while the script runs, so it will kill the > > script after 1 hour of runtime to make sure that stuff doesn't block for > > too long. > > > > This part hasn't been too tested or fleshed out, so if you think of > > ideas for improvements, feel free to propose them. > > Hmmm... One thing that comes to mind, is that it would be useful if > as a second & third parameter, the repo_script took the name of the > package, and version-revision-etc that had just been built. That > could save time, either by avoiding any magic to figure out which one > had just been completed, or by allowing brute-force approaches caused > by not wanting to create said magic:) Well, you may be adding 20 packages to the repo, or just 1 at the point the script gets called. But this would be helpful, yes. Ideally I guess we'd give the target string as the first arg, then paths to all the RPMs that just got copied to the repo as the next 50000 arguments. > Another thing, if the packages have already been added to the local > repos when the build state is "needsign", then what's the point of > that step? If we go and sign them, then we need to play with the > repos cache/etc to maintain a valid state; if we pull them out and > push them to another (signed/etc) repos, then it's not needed; if we > don't sign them it's not needed. IMO, it would be far more useful if > packages held in this state weren't actually added to the repos until > after manually "finished". The problem here is that if packages are held until manually finished, then they are not available to builders. So if I want to build package B that depends on a new package A, I have to wait for you to sign package A and add it to the repo before I can build package B. That's the main reason to add packages to the repo ASAP, and also to run the repo scripts _after_ copying packages to the repodir and running createrepo. It's also probably why Fedora Extras doesn't use the build server's repository but requires manual runs of a push-script to sign & copy to the "official" mirrored repo. It's a balance of package maintainer's needs (quick building of deps) versus administrator needs (making sure repos are up to date and signed). Suggestions on how to better balance these two issues would be welcomed. In the longer term (plague 0.5) I'd like to put automatic depsolving into the build server, so that the server will wait like 8 hours for dependencies of a particular SRPM to be solved before failing the package. If during those 8 or 12 hours, the deps get solved, then the build server will allow the package to be built. This way, we let package maintainers queue up however many packages they want, and then walk away and not have to worry about stuff until they get mails about failures or successes. That's the end goal here. And it should solve your problem too, since it fixes the dependency issue that forces repo scripts to be run last. Dan From dcbw at redhat.com Fri Nov 18 16:58:08 2005 From: dcbw at redhat.com (Dan Williams) Date: Fri, 18 Nov 2005 11:58:08 -0500 Subject: Build system maintenance complete Message-ID: <1132333088.5377.15.camel@dhcp83-87.boston.redhat.com> Hi, I've updated and restarted the build system this morning to fix issues many of you may have noticed with jobs stuck in downloading/done state. Things should be flowing more smoothly now. The problem was caused by plague using blocking sockets for RPM downloads, which weren't playing well with SSL and firewall/iptables on the builders. We now use non-blocking sockets with timeouts for RPM downloads, and retry downloads up to 5 times before failing the job. If you notice your job sitting in downloading/done for more than 10 or 15 minutes (it's normal to be in that state for a short time), please email me the job # and I'll look at the logs to see what really went on and hopefully fix it. Thanks for the patience. Dan From chris.weyl at gmail.com Fri Nov 18 17:17:53 2005 From: chris.weyl at gmail.com (Chris Weyl) Date: Fri, 18 Nov 2005 12:17:53 -0500 Subject: mock configs for other flavors? Message-ID: <7dd7ab490511180917k79fb1cfctb61a836bbdcf49df@mail.gmail.com> Hey all-- Before I go about reinventing the wheel, does anyone have (or can point me to) sample mock config files for RHEL3/4? I'm also carrying on a rather spiritied conversation about using plague to build for other distros (e.g. mandrake) over here, and would love to be able to prove them wrong:) So, really, mock configs for any flavor/distro would be most appreciated... Thanks- -Chris From dcbw at redhat.com Fri Nov 18 17:25:53 2005 From: dcbw at redhat.com (Dan Williams) Date: Fri, 18 Nov 2005 12:25:53 -0500 Subject: mock configs for other flavors? In-Reply-To: <7dd7ab490511180917k79fb1cfctb61a836bbdcf49df@mail.gmail.com> References: <7dd7ab490511180917k79fb1cfctb61a836bbdcf49df@mail.gmail.com> Message-ID: <1132334753.5377.20.camel@dhcp83-87.boston.redhat.com> On Fri, 2005-11-18 at 12:17 -0500, Chris Weyl wrote: > Hey all-- > > Before I go about reinventing the wheel, does anyone have (or can > point me to) sample mock config files for RHEL3/4? > > I'm also carrying on a rather spiritied conversation about using > plague to build for other distros (e.g. mandrake) over here, and would > love to be able to prove them wrong:) So, really, mock configs for > any flavor/distro would be most appreciated... Hey, if patches are required to do this, that's fine. Some issues I'd see: 1) How do those other distros update packages? Do they use yum? If not, what kind of repository metadata do the update tools for those distros use? (ie, instead of calling createrepo on the server we'd need to call something else) 2) How do those distros rebuild packages? Do they use rpmbuild? If not, are rpmbuild-built packages sufficient for those distros? You should be able to designate certain builders as "mandrake" builders, for example, just by restricting that builder's target files. If that doesn't work, I'd be curious to know what the problems are. Dan From skvidal at phy.duke.edu Fri Nov 18 18:06:57 2005 From: skvidal at phy.duke.edu (seth vidal) Date: Fri, 18 Nov 2005 13:06:57 -0500 Subject: mock configs for other flavors? In-Reply-To: <7dd7ab490511180917k79fb1cfctb61a836bbdcf49df@mail.gmail.com> References: <7dd7ab490511180917k79fb1cfctb61a836bbdcf49df@mail.gmail.com> Message-ID: <1132337217.29189.26.camel@cutter> On Fri, 2005-11-18 at 12:17 -0500, Chris Weyl wrote: > Hey all-- > > Before I go about reinventing the wheel, does anyone have (or can > point me to) sample mock config files for RHEL3/4? I can give you configs but the problem isn't the configs - it's having the yum repos for rhel3 and 4. If you already have the repos then rhel3/4 mock configs are utterly trivial - just change the yum.conf section. > I'm also carrying on a rather spiritied conversation about using > plague to build for other distros (e.g. mandrake) over here, and would > love to be able to prove them wrong:) So, really, mock configs for > any flavor/distro would be most appreciated... I have mock configs for fc* and centos3 and 4. -sv From skvidal at phy.duke.edu Fri Nov 18 18:08:04 2005 From: skvidal at phy.duke.edu (seth vidal) Date: Fri, 18 Nov 2005 13:08:04 -0500 Subject: mock configs for other flavors? In-Reply-To: <1132334753.5377.20.camel@dhcp83-87.boston.redhat.com> References: <7dd7ab490511180917k79fb1cfctb61a836bbdcf49df@mail.gmail.com> <1132334753.5377.20.camel@dhcp83-87.boston.redhat.com> Message-ID: <1132337284.29189.29.camel@cutter> On Fri, 2005-11-18 at 12:25 -0500, Dan Williams wrote: > On Fri, 2005-11-18 at 12:17 -0500, Chris Weyl wrote: > > Hey all-- > > > > Before I go about reinventing the wheel, does anyone have (or can > > point me to) sample mock config files for RHEL3/4? > > > > I'm also carrying on a rather spiritied conversation about using > > plague to build for other distros (e.g. mandrake) over here, and would > > love to be able to prove them wrong:) So, really, mock configs for > > any flavor/distro would be most appreciated... > > Hey, if patches are required to do this, that's fine. Some issues I'd > see: > > 1) How do those other distros update packages? Do they use yum? If > not, what kind of repository metadata do the update tools for those > distros use? (ie, instead of calling createrepo on the server we'd need > to call something else) Well it shouldn't matter for the plague run. All yum is doing is making the chroot so as long as the repository for the other distros has the repodata made then it can work w/o any need for the distro to use yum. -sv From chris.weyl at gmail.com Fri Nov 18 20:13:09 2005 From: chris.weyl at gmail.com (Chris Weyl) Date: Fri, 18 Nov 2005 15:13:09 -0500 Subject: mock configs for other flavors? In-Reply-To: <1132337217.29189.26.camel@cutter> References: <7dd7ab490511180917k79fb1cfctb61a836bbdcf49df@mail.gmail.com> <1132337217.29189.26.camel@cutter> Message-ID: <7dd7ab490511181213k4953d632ibf1e39c94b5e3a69@mail.gmail.com> On 11/18/05, seth vidal wrote: > On Fri, 2005-11-18 at 12:17 -0500, Chris Weyl wrote: > > Hey all-- > > > > Before I go about reinventing the wheel, does anyone have (or can > > point me to) sample mock config files for RHEL3/4? > > I can give you configs but the problem isn't the configs - it's having > the yum repos for rhel3 and 4. If you already have the repos then > rhel3/4 mock configs are utterly trivial - just change the yum.conf > section. Excellent. That's the part I wasn't sure of; I do already have access to rhel 3/4 repos but wasn't sure if there was any additional configuration I needed to do from a mock perspective. (I know under mach at one point, we needed to change the runuser setting, for instance.) > > I'm also carrying on a rather spiritied conversation about using > > plague to build for other distros (e.g. mandrake) over here, and would > > love to be able to prove them wrong:) So, really, mock configs for > > any flavor/distro would be most appreciated... > > I have mock configs for fc* and centos3 and 4. Anything and everything is appreciated :) -Chris From symbiont at berlios.de Sat Nov 19 00:03:08 2005 From: symbiont at berlios.de (Jeff Pitman) Date: Sat, 19 Nov 2005 08:03:08 +0800 Subject: mock configs for other flavors? In-Reply-To: <7dd7ab490511181213k4953d632ibf1e39c94b5e3a69@mail.gmail.com> References: <7dd7ab490511180917k79fb1cfctb61a836bbdcf49df@mail.gmail.com> <1132337217.29189.26.camel@cutter> <7dd7ab490511181213k4953d632ibf1e39c94b5e3a69@mail.gmail.com> Message-ID: <6b9c17630511181603j69088a62q62989bac0dc5271c@mail.gmail.com> On 11/19/05, Chris Weyl wrote: > > Anything and everything is appreciated :) > https://opensvn.csie.org/traccgi/pyvault/trac.cgi/browser/scripts/config/ Check the *.cfg files. centos-3-i386.cfg will load general.cfg which loads pyv.cfg. This allows flexibility in changing between different repositories and sharing of configuration across different distributions. If you do not need that flexibility, just cut and paste what you need into one config. enjoy, -- -jeff -------------- next part -------------- An HTML attachment was scrubbed... URL: From dcbw at redhat.com Sun Nov 20 18:42:53 2005 From: dcbw at redhat.com (Dan Williams) Date: Sun, 20 Nov 2005 13:42:53 -0500 Subject: Patch to store user info in different DB engines In-Reply-To: <3BF69E00-889E-4214-8CAF-621ADE624444@cs.ucsb.edu> References: <66104028-31AE-46CA-B74C-2ED87983F94C@cs.ucsb.edu> <3BF69E00-889E-4214-8CAF-621ADE624444@cs.ucsb.edu> Message-ID: <1132512173.3035.0.camel@localhost.localdomain> On Wed, 2005-10-26 at 15:20 -0400, Jeff Sheltren wrote: > On Oct 26, 2005, at 2:48 PM, Jeff Sheltren wrote: > > > I'm attaching a patch which uses DBManager for interacting with the > > user database rather than always relying upon sqlite. This way, if > > you use pgsql or mysql for your database back-end, user information > > will be stored there as well. > > > Whoops, I forgot to include the diff for DBManager.py in that file. > I'm attaching an updated patch which also changes DBManager.py so > that the users table will be created as needed. Committed to HEAD, thanks! Dan From dcbw at redhat.com Sun Nov 20 18:43:14 2005 From: dcbw at redhat.com (Dan Williams) Date: Sun, 20 Nov 2005 13:43:14 -0500 Subject: Patch to store user info in different DB engines In-Reply-To: <301E5BB8-6F02-4C51-BB35-B09408D709ED@cs.ucsb.edu> References: <66104028-31AE-46CA-B74C-2ED87983F94C@cs.ucsb.edu> <3BF69E00-889E-4214-8CAF-621ADE624444@cs.ucsb.edu> <301E5BB8-6F02-4C51-BB35-B09408D709ED@cs.ucsb.edu> Message-ID: <1132512195.3035.2.camel@localhost.localdomain> On Fri, 2005-10-28 at 14:34 -0400, Jeff Sheltren wrote: > Here is a patch for user-manager.py to use DBManager for its database > access. This allows user-manager.py to work with multiple db types. Committed to HEAD, thanks! Dan From dcbw at redhat.com Sun Nov 20 18:43:33 2005 From: dcbw at redhat.com (Dan Williams) Date: Sun, 20 Nov 2005 13:43:33 -0500 Subject: Patch to store user info in different DB engines In-Reply-To: References: <66104028-31AE-46CA-B74C-2ED87983F94C@cs.ucsb.edu> <3BF69E00-889E-4214-8CAF-621ADE624444@cs.ucsb.edu> <301E5BB8-6F02-4C51-BB35-B09408D709ED@cs.ucsb.edu> <7dd7ab490510281341m50f6e14emb1235dd7134ea4fc@mail.gmail.com> <0D179712-BDB4-4CF0-A55D-EB9F28639061@cs.ucsb.edu> <7dd7ab490510281608k7b3c0026yacc95b2765dfd004@mail.gmail.com> <1130856645.2627.3.camel@localhost.localdomain> <87FBC17A-0FE6-43CB-A7BE-5EE41D10F393@cs.ucsb.edu> Message-ID: <1132512213.3035.4.camel@localhost.localdomain> On Thu, 2005-11-03 at 07:30 -0400, Jeff Sheltren wrote: > On Nov 2, 2005, at 9:57 AM, Jeff Sheltren wrote: > > > > > > > I realized there are a couple of things that I didn't like, so I've > made an updated script, located here: > > http://www.cs.ucsb.edu/~jeff/plague-user-migration.py > > Now it will not attempt to migrate from sqlite to sqlite :) Also, > all errors are output to stderr - before some were going to stdout. Added to HEAD, thanks! Dan From sheltren at cs.ucsb.edu Mon Nov 21 17:50:27 2005 From: sheltren at cs.ucsb.edu (Jeff Sheltren) Date: Mon, 21 Nov 2005 13:50:27 -0400 (AST) Subject: Patch to store user info in different DB engines In-Reply-To: <1132512195.3035.2.camel@localhost.localdomain> References: <66104028-31AE-46CA-B74C-2ED87983F94C@cs.ucsb.edu><3BF69E00-889E-4214-8CAF-621ADE624444@cs.ucsb.edu><301E5BB8-6F02-4C51-BB35-B09408D709ED@cs.ucsb.edu> <1132512195.3035.2.camel@localhost.localdomain> Message-ID: <7001.209.88.66.94.1132595427.squirrel@letters.cs.ucsb.edu> On Sun, November 20, 2005 2:43 pm, Dan Williams said: > On Fri, 2005-10-28 at 14:34 -0400, Jeff Sheltren wrote: >> Here is a patch for user-manager.py to use DBManager for its database >> access. This allows user-manager.py to work with multiple db types. > > Committed to HEAD, thanks! > > Dan > Cool, thanks for adding it! -Jeff From chris.weyl at gmail.com Tue Nov 22 20:39:34 2005 From: chris.weyl at gmail.com (Chris Weyl) Date: Tue, 22 Nov 2005 15:39:34 -0500 Subject: Plague and other distros... Message-ID: <7dd7ab490511221239p5dfa934by9ec921a44581ec57@mail.gmail.com> Hey all -- Just a quick update, for those that are curious :) RHEL -- This builds w/o issue under plague, mock configs are trivial. Two things: you'll need to create your own yum repso; in rhel 3, runuser needs to be set to '/bin/su'. Mandrake -- Again, yum repos need to be built (if anyone knows of any public repos with the base os and updates, I'd be much obliged). Ditch the "delta" rpms when generating, they'll just mess things up (for our purposes). I've been working with mandriva 2006.0. 'runuser' also needs to be set to '/bin/su' here. For whatever reason (that I haven't been fully able to deduce), I've found that the "checking for installed but unpackaged files" bit fails horribly under mdk. For mdk spec files (and problematic rhel ones), replacing that macro with this will help: %define __check_files /bin/sh -c "(export TMPDIR=%{_tmppath} ; cd %{_builddir}; /usr/lib/rpm/check-files %{buildroot})" CVS integration works nicely as well -- make plague, make tag, etc, all work nicely if you tweak CVSROOT/branches to include the rhel/mdk distros. There are some functions that are fedora-hardcoded in Makefile.common (e.g. "make mock"), but the basics work w/o tinkering. One issue I am noting is that at the top/bottom of build.log's from rhel, I often find 3-7 lines of: error: Macro % has illegal name (%define) error: Macro % has illegal name (%define) etc, etc I'm not entirely sure what's causing this. The build looks to complete successfully (with resulting rpms to show for it), but plague/mock claims it failed, so this is a bit of a problem. Anyways... for the curious :) Basic upshot is that I haven't found any changes that need to be made to plague proper, and 99% of it is finding/generating yum repositories and tweaking mock config files. -Chris From chris.weyl at gmail.com Tue Nov 22 21:16:13 2005 From: chris.weyl at gmail.com (Chris Weyl) Date: Tue, 22 Nov 2005 16:16:13 -0500 Subject: Plague and other distros... In-Reply-To: <7dd7ab490511221239p5dfa934by9ec921a44581ec57@mail.gmail.com> References: <7dd7ab490511221239p5dfa934by9ec921a44581ec57@mail.gmail.com> Message-ID: <7dd7ab490511221316g3d7e4b3co9e9030348c6ffe68@mail.gmail.com> On 11/22/05, Chris Weyl wrote: > I'm not entirely sure what's causing this. The build looks to > complete successfully (with resulting rpms to show for it), but > plague/mock claims it failed, so this is a bit of a problem. Hrm, scratch that. It's job.log claiming it failed; so that's plague thinking it failed, not mock (right?). -Chris From dcbw at redhat.com Tue Nov 22 21:46:43 2005 From: dcbw at redhat.com (Dan Williams) Date: Tue, 22 Nov 2005 16:46:43 -0500 Subject: Plague and other distros... In-Reply-To: <7dd7ab490511221316g3d7e4b3co9e9030348c6ffe68@mail.gmail.com> References: <7dd7ab490511221239p5dfa934by9ec921a44581ec57@mail.gmail.com> <7dd7ab490511221316g3d7e4b3co9e9030348c6ffe68@mail.gmail.com> Message-ID: <1132696003.12401.1.camel@dhcp83-87.boston.redhat.com> On Tue, 2005-11-22 at 16:16 -0500, Chris Weyl wrote: > On 11/22/05, Chris Weyl wrote: > > I'm not entirely sure what's causing this. The build looks to > > complete successfully (with resulting rpms to show for it), but > > plague/mock claims it failed, so this is a bit of a problem. > > Hrm, scratch that. It's job.log claiming it failed; so that's plague > thinking it failed, not mock (right?). It depends on what plague-builder got for the return code from mock. Plague-builder will also kill mock if mock doesn't write out its status file after 20s. You could put some stuff in plague-builder in the _watch_mock() function to see what mock returns. But if job.log claims that something failed, then yes that's plague-builder and not mock. Dan From dcbw at redhat.com Fri Nov 25 22:13:20 2005 From: dcbw at redhat.com (Dan Williams) Date: Fri, 25 Nov 2005 17:13:20 -0500 Subject: Patch to store user info in different DB engines In-Reply-To: <1132512213.3035.4.camel@localhost.localdomain> References: <66104028-31AE-46CA-B74C-2ED87983F94C@cs.ucsb.edu> <3BF69E00-889E-4214-8CAF-621ADE624444@cs.ucsb.edu> <301E5BB8-6F02-4C51-BB35-B09408D709ED@cs.ucsb.edu> <7dd7ab490510281341m50f6e14emb1235dd7134ea4fc@mail.gmail.com> <0D179712-BDB4-4CF0-A55D-EB9F28639061@cs.ucsb.edu> <7dd7ab490510281608k7b3c0026yacc95b2765dfd004@mail.gmail.com> <1130856645.2627.3.camel@localhost.localdomain> <87FBC17A-0FE6-43CB-A7BE-5EE41D10F393@cs.ucsb.edu> <1132512213.3035.4.camel@localhost.localdomain> Message-ID: <1132956800.5474.2.camel@localhost.localdomain> On Sun, 2005-11-20 at 13:43 -0500, Dan Williams wrote: > On Thu, 2005-11-03 at 07:30 -0400, Jeff Sheltren wrote: > > On Nov 2, 2005, at 9:57 AM, Jeff Sheltren wrote: > > > > > > > > > > > > I realized there are a couple of things that I didn't like, so I've > > made an updated script, located here: > > > > http://www.cs.ucsb.edu/~jeff/plague-user-migration.py > > > > Now it will not attempt to migrate from sqlite to sqlite :) Also, > > all errors are output to stderr - before some were going to stdout. > > Added to HEAD, thanks! Jeff, Could you retest the user migration script? It turns out that postgres doesn't like double-quotes in the INSERT INTO statement. Since all the rest of the stuff in BuildMaster.py uses single quotes in the SQL statements, I switched that bit of the migration script to use single-quotes too. Also, Postgres evidently uses "True" and "False" (probably case-insensitive) for fields of type BOOLEAN, so I added a conversion function for postgres to do that, and converted the %d format options in the INSERT INTO statement to %s instead. These shouldn't affect MySQL at all, but I just want to be sure... Thanks! Dan From chris.weyl at gmail.com Mon Nov 28 15:54:55 2005 From: chris.weyl at gmail.com (Chris Weyl) Date: Mon, 28 Nov 2005 10:54:55 -0500 Subject: package containing ChangeLog -- switch to -common? Message-ID: <7dd7ab490511280754m136a3b03o1d93af9cf0a20097@mail.gmail.com> Hey Dan, What do you think of moving the ChangeLog to the plague-common package, rather than the plague (server) package? That way one could upgrade the -common and -client packages on their workstation first and see what's changed, before upgrading the server. Thanks- -Chris From dcbw at redhat.com Mon Nov 28 17:01:03 2005 From: dcbw at redhat.com (Dan Williams) Date: Mon, 28 Nov 2005 12:01:03 -0500 Subject: package containing ChangeLog -- switch to -common? In-Reply-To: <7dd7ab490511280754m136a3b03o1d93af9cf0a20097@mail.gmail.com> References: <7dd7ab490511280754m136a3b03o1d93af9cf0a20097@mail.gmail.com> Message-ID: <1133197263.2857.0.camel@dhcp83-87.boston.redhat.com> On Mon, 2005-11-28 at 10:54 -0500, Chris Weyl wrote: > Hey Dan, > > What do you think of moving the ChangeLog to the plague-common > package, rather than the plague (server) package? That way one could > upgrade the -common and -client packages on their workstation first > and see what's changed, before upgrading the server. Sounds reasonable, will do. Dan From sheltren at cs.ucsb.edu Tue Nov 29 20:43:07 2005 From: sheltren at cs.ucsb.edu (Jeff Sheltren) Date: Tue, 29 Nov 2005 16:43:07 -0400 Subject: Patch to store user info in different DB engines In-Reply-To: <1132956800.5474.2.camel@localhost.localdomain> References: <66104028-31AE-46CA-B74C-2ED87983F94C@cs.ucsb.edu> <3BF69E00-889E-4214-8CAF-621ADE624444@cs.ucsb.edu> <301E5BB8-6F02-4C51-BB35-B09408D709ED@cs.ucsb.edu> <7dd7ab490510281341m50f6e14emb1235dd7134ea4fc@mail.gmail.com> <0D179712-BDB4-4CF0-A55D-EB9F28639061@cs.ucsb.edu> <7dd7ab490510281608k7b3c0026yacc95b2765dfd004@mail.gmail.com> <1130856645.2627.3.camel@localhost.localdomain> <87FBC17A-0FE6-43CB-A7BE-5EE41D10F393@cs.ucsb.edu> <1132512213.3035.4.camel@localhost.localdomain> <1132956800.5474.2.camel@localhost.localdomain> Message-ID: <16FF7F6D-30FB-412E-ABC3-FE556797487A@cs.ucsb.edu> On Nov 25, 2005, at 6:13 PM, Dan Williams wrote: > > Jeff, > > Could you retest the user migration script? It turns out that > postgres > doesn't like double-quotes in the INSERT INTO statement. Since all > the > rest of the stuff in BuildMaster.py uses single quotes in the SQL > statements, I switched that bit of the migration script to use > single-quotes too. > > Also, Postgres evidently uses "True" and "False" (probably > case-insensitive) for fields of type BOOLEAN, so I added a conversion > function for postgres to do that, and converted the %d format > options in > the INSERT INTO statement to %s instead. > > These shouldn't affect MySQL at all, but I just want to be sure... > > Thanks! > Dan Oops, guess I should have tested it against postgres also. Your new version works fine with mysql. -Jeff