From ivazquez at ivazquez.net Sat Jul 9 19:39:19 2005 From: ivazquez at ivazquez.net (Ignacio Vazquez-Abrams) Date: Sat, 09 Jul 2005 15:39:19 -0400 Subject: plague-builder initscript patch Message-ID: <1120937959.9522.44.camel@ignacio.lan> The attached patch adds initscript capability to plague-builder, among other things. The file daemonize.py was taken from the ASPN Python Cookbook at http://aspn.activestate.com/ASPN/Cookbook/Python. -- Ignacio Vazquez-Abrams http://fedora.ivazquez.net/ gpg --keyserver hkp://subkeys.pgp.net --recv-key 38028b72 -------------- next part -------------- A non-text attachment was scrubbed... Name: plague-builder-init.diff Type: text/x-patch Size: 11526 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From dcbw at redhat.com Sun Jul 10 04:49:19 2005 From: dcbw at redhat.com (Dan Williams) Date: Sun, 10 Jul 2005 00:49:19 -0400 (EDT) Subject: plague-builder initscript patch In-Reply-To: <1120937959.9522.44.camel@ignacio.lan> References: <1120937959.9522.44.camel@ignacio.lan> Message-ID: On Sat, 9 Jul 2005, Ignacio Vazquez-Abrams wrote: > The attached patch adds initscript capability to plague-builder, among > other things. > > The file daemonize.py was taken from the ASPN Python Cookbook at > http://aspn.activestate.com/ASPN/Cookbook/Python. Committed, thanks. Dan From dcbw at redhat.com Sun Jul 10 04:56:16 2005 From: dcbw at redhat.com (Dan Williams) Date: Sun, 10 Jul 2005 00:56:16 -0400 (EDT) Subject: New build system notes... Message-ID: Hi, So, a few quick notes. If you're setting it up, you'll need a couple things: Python 2.3 or 2.4 (consider issues with 2.2 bugs, I'll fix them) sqlite sqlite-python mock from CVS as of 07-01-2005 lighttpd (1) yum >= 2.2.1 pyOpenSSL (2) (1) This is technically optional, and use of lighttpd is off by default. However, the RPM currently requires lighttpd, but you can "--nodeps" the plague packages to ignore this. We should fix this. (2) IMPORTANT: You'll need to patch pyOpenSSL with the patch attached to this email, which enables threadsafety for pyOpenSSL. Without the patch, it _will_ not work and the build server will randomly crash. Dan -------------- next part -------------- --- pyOpenSSL-0.6/src/crypto/crypto.c.threadsafe 2005-07-09 21:53:17.000000000 -0400 +++ pyOpenSSL-0.6/src/crypto/crypto.c 2005-07-09 22:00:32.000000000 -0400 @@ -668,6 +668,71 @@ { NULL, NULL } }; + +/* Unashamedly stolen from http://curl.haxx.se/mail/lib-2005-05/0062.html */ + + +#define MUTEX_TYPE pthread_mutex_t +#define MUTEX_SETUP(x) pthread_mutex_init(&(x), NULL) +#define MUTEX_CLEANUP(x) pthread_mutex_destroy(&(x)) +#define MUTEX_LOCK(x) pthread_mutex_lock(&(x)) +#define MUTEX_UNLOCK(x) pthread_mutex_unlock(&(x)) +#define THREAD_ID pthread_self() + + +void handle_error(const char *file, int lineno, const char *msg) +{ + fprintf(stderr, "** %s:%i %s\n", file, lineno, msg); + ERR_print_errors_fp(stderr); +} + + +/* This array will store all of the mutexes available to OpenSSL. */ +static MUTEX_TYPE *mutex_buf = NULL; + + +static void locking_function(int mode, int n, const char * file, int line) +{ + if (mode & CRYPTO_LOCK) + MUTEX_LOCK(mutex_buf[n]); + else + MUTEX_UNLOCK(mutex_buf[n]); +} + +static unsigned long id_function(void) +{ + return ((unsigned long)THREAD_ID); +} + +int init_openssl_threads(void) +{ + int i; + + mutex_buf = (MUTEX_TYPE *)malloc(CRYPTO_num_locks() * sizeof(MUTEX_TYPE)); + if (!mutex_buf) + return 0; + for (i = 0; i < CRYPTO_num_locks(); i++) + MUTEX_SETUP(mutex_buf[i]); + CRYPTO_set_id_callback(id_function); + CRYPTO_set_locking_callback(locking_function); + return 1; +} + +int deinit_openssl_threads(void) +{ + int i; + + if (!mutex_buf) + return 0; + CRYPTO_set_id_callback(NULL); + CRYPTO_set_locking_callback(NULL); + for (i = 0; i < CRYPTO_num_locks(); i++) + MUTEX_CLEANUP(mutex_buf[i]); + free(mutex_buf); + mutex_buf = NULL; + return 1; +} + /* * Initialize crypto sub module * @@ -713,6 +775,8 @@ PyModule_AddIntConstant(module, "TYPE_DSA", crypto_TYPE_DSA); dict = PyModule_GetDict(module); + if (!init_openssl_threads()) + goto error; if (!init_crypto_x509(dict)) goto error; if (!init_crypto_x509name(dict)) From enrico.scholz at informatik.tu-chemnitz.de Sun Jul 10 14:30:01 2005 From: enrico.scholz at informatik.tu-chemnitz.de (Enrico Scholz) Date: Sun, 10 Jul 2005 16:30:01 +0200 Subject: Read-only / + !CAP_MKNOD support for mock Message-ID: <873bqmlonq.fsf@kosh.bigo.ensc.de> Hello, the patch which is available at http://ensc.de/fedora/mock-namespace.diff changes some things so that mock is a little bit more secure: * everything except /var/lib/mock can be read-only now; this is done by - avoiding modification of /etc/mtab* by using the '-n' switch for 'mount' - executing all mach operations in an own namespace; so the cleanup of mounts happens automatically without relying on /etc/mtab - workarounding the 'rpm --root'-touches-the-rpmdb-of-the-host bug; namespaces mentioned above make it possible to bind-mount the buildroot-rpmdb into the host * mock works with removed CAP_MKNOD capabilities; instead of, a precreated /dev template will be bind-mounted into the buildroot. Ideally, this precreated template is a mounted cramfs as it can not be modified but still allows the devices to work (this would not be the case e.g. with a read-only mounted ext3 fs) With these modifications, 'mock' can be used within VServers[1]. Please note that the patch above protects only the filesystem but not processes. So you will have to restart the buildsystem after each build (takes around 2-3 seconds with vservers and 1-2 minutes with regular hosts). Else, every hostile package can take control over subsequent builds. Footnotes: [1] http://linux-vservers.org -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 480 bytes Desc: not available URL: From ivazquez at ivazquez.net Mon Jul 11 02:08:09 2005 From: ivazquez at ivazquez.net (Ignacio Vazquez-Abrams) Date: Sun, 10 Jul 2005 22:08:09 -0400 Subject: Patch for plague-builder option parser change Message-ID: <1121047689.25730.2.camel@ignacio.lan> This fixes %post builder for the -a change. -- Ignacio Vazquez-Abrams http://fedora.ivazquez.net/ gpg --keyserver hkp://subkeys.pgp.net --recv-key 38028b72 -------------- next part -------------- A non-text attachment was scrubbed... Name: plague-builder-archsopt.diff Type: text/x-patch Size: 864 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From ivazquez at ivazquez.net Mon Jul 11 03:17:30 2005 From: ivazquez at ivazquez.net (Ignacio Vazquez-Abrams) Date: Sun, 10 Jul 2005 23:17:30 -0400 Subject: plague-server initscript and small plague-builder fixes Message-ID: <1121051850.25730.5.camel@ignacio.lan> Yet another patch... -- Ignacio Vazquez-Abrams http://fedora.ivazquez.net/ gpg --keyserver hkp://subkeys.pgp.net --recv-key 38028b72 -------------- next part -------------- A non-text attachment was scrubbed... Name: plague-server-init.diff Type: text/x-patch Size: 6178 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From ivazquez at ivazquez.net Mon Jul 11 03:51:45 2005 From: ivazquez at ivazquez.net (Ignacio Vazquez-Abrams) Date: Sun, 10 Jul 2005 23:51:45 -0400 Subject: Small plague patch Message-ID: <1121053905.25730.9.camel@ignacio.lan> Just a couple of silly mistakes on my part. Also, daemonize.py isn't playing nice with -l. I'll figure out what's wrong though. -- Ignacio Vazquez-Abrams http://fedora.ivazquez.net/ gpg --keyserver hkp://subkeys.pgp.net --recv-key 38028b72 -------------- next part -------------- A non-text attachment was scrubbed... Name: plague-brownpaperbag.diff Type: text/x-patch Size: 1400 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From ivazquez at ivazquez.net Mon Jul 11 05:06:36 2005 From: ivazquez at ivazquez.net (Ignacio Vazquez-Abrams) Date: Mon, 11 Jul 2005 01:06:36 -0400 Subject: Small plague patch In-Reply-To: <1121053905.25730.9.camel@ignacio.lan> References: <1121053905.25730.9.camel@ignacio.lan> Message-ID: <1121058396.25730.13.camel@ignacio.lan> On Sun, 2005-07-10 at 23:51 -0400, Ignacio Vazquez-Abrams wrote: > Also, daemonize.py isn't playing nice with -l. I'll figure out what's > wrong though. I take that back, apparently it has nothing at all to do with daemonize.py. I still haven't managed to track it down though. -- Ignacio Vazquez-Abrams http://fedora.ivazquez.net/ gpg --keyserver hkp://subkeys.pgp.net --recv-key 38028b72 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From dcbw at redhat.com Sat Jul 16 15:56:52 2005 From: dcbw at redhat.com (Dan Williams) Date: Sat, 16 Jul 2005 11:56:52 -0400 (EDT) Subject: plague enhancements... Message-ID: Hi, I've committed code to include the last 20 lines of the relevant build logs in the failure email. Should make it a bit easier at first glance to find out what went wrong. Second, the build system is aware of what targets (and what arches on those targets) the builder can actually do, which means we now deal with noarch packages correctly in a build system as complex as Extras, with lots of targets and lots of arches. This _does_ mean that you have to specify the same target names (ie, "development" not "devel") as is used in the mock config file names in /etc/mock. If you want to alias anything, do it in whatever you call plague-client with. I'm not really inclined to add target alias support to the server itself (other than the CVS target aliases where its actually necessary). Dan From dcbw at redhat.com Sat Jul 16 16:01:42 2005 From: dcbw at redhat.com (Dan Williams) Date: Sat, 16 Jul 2005 12:01:42 -0400 (EDT) Subject: mock --target question Message-ID: Seth, When mock builds an SRPM with "rpmbuild -ba --target %s" where %s is config_opts['target_arch'], that arch will always be "i386" or "x86_64" since it comes directly from the mock config file. So how does the build system build i486, i586, i686 packages then with mock? If the build system wants to have mock build an i586 package, the builder has to choose the "fedora-5-i386-core.cfg" mock config, which has target_arch == i386, which gets passed to rpmbuild's --target argument. So how would an i586 package ever come out of mock unless it specified ExclusiveArch: i586? Do we need to have another mock argument for --arch or am I missing some rpmbuild magic here? Dan From skvidal at phy.duke.edu Sat Jul 16 16:25:15 2005 From: skvidal at phy.duke.edu (seth vidal) Date: Sat, 16 Jul 2005 12:25:15 -0400 Subject: mock --target question In-Reply-To: References: Message-ID: <1121531115.20849.0.camel@cutter> On Sat, 2005-07-16 at 12:01 -0400, Dan Williams wrote: > Seth, > > When mock builds an SRPM with "rpmbuild -ba --target %s" where %s is > config_opts['target_arch'], that arch will always be "i386" or "x86_64" since it > comes directly from the mock config file. > > So how does the build system build i486, i586, i686 packages then with mock? If > the build system wants to have mock build an i586 package, the builder has to > choose the "fedora-5-i386-core.cfg" mock config, which has target_arch == i386, > which gets passed to rpmbuild's --target argument. So how would an i586 package > ever come out of mock unless it specified ExclusiveArch: i586? > > Do we need to have another mock argument for --arch or am I missing some > rpmbuild magic here? > mock has a --arch argument. -sv From ivazquez at ivazquez.net Tue Jul 19 20:46:17 2005 From: ivazquez at ivazquez.net (Ignacio Vazquez-Abrams) Date: Tue, 19 Jul 2005 16:46:17 -0400 Subject: New build system notes... In-Reply-To: References: Message-ID: <1121805977.28256.17.camel@ignacio.lan> On Sun, 2005-07-10 at 00:56 -0400, Dan Williams wrote: > (2) IMPORTANT: You'll need to patch pyOpenSSL with the patch attached to this > email, which enables threadsafety for pyOpenSSL. Without the patch, it _will_ > not work and the build server will randomly crash. I've built packages for FC4 with this patch which I've placed in my Alternatives repo. http://fedora.ivazquez.net/yum/4/i386/RPMS.alternatives/pyOpenSSL-0.6-1.p24.4.iva.1.i386.rpm http://fedora.ivazquez.net/yum/4/ppc/RPMS.alternatives/pyOpenSSL-0.6-1.p24.4.iva.1.ppc.rpm http://fedora.ivazquez.net/yum/4/x86_64/RPMS.alternatives/pyOpenSSL-0.6-1.p24.4.iva.1.x86_64.rpm -- Ignacio Vazquez-Abrams http://fedora.ivazquez.net/ gpg --keyserver hkp://subkeys.pgp.net --recv-key 38028b72 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From thias at spam.spam.spam.spam.spam.spam.spam.egg.and.spam.freshrpms.net Thu Jul 21 09:58:08 2005 From: thias at spam.spam.spam.spam.spam.spam.spam.egg.and.spam.freshrpms.net (Matthias Saou) Date: Thu, 21 Jul 2005 11:58:08 +0200 Subject: plague enhancements... In-Reply-To: References: Message-ID: <20050721115808.0f302e78@python2> Dan Williams wrote : > I've committed code to include the last 20 lines of the relevant build logs in > the failure email. Should make it a bit easier at first glance to find out what > went wrong. Would it be possible to also include "config.log" if it exists when a build fails? This could come in handy quite often, and has already been suggested as a useful possible enhancement. Thanks for all your great work! Matthias -- Clean custom Red Hat Linux rpm packages : http://freshrpms.net/ Fedora Core release 4 (Stentz) - Linux kernel 2.6.12-1.1398_FC4 Load : 0.46 0.49 0.54 From symbiont at berlios.de Tue Jul 26 00:33:10 2005 From: symbiont at berlios.de (Jeff Pitman) Date: Tue, 26 Jul 2005 08:33:10 +0800 Subject: permissions in mock Message-ID: <200507260833.10440.symbiont@berlios.de> Hi: Anyone else notice in the build.log that there are areas where permission is denied when doing stuff. The two most notable are 1. checking for unpackaged files, and 2. stripping binary libs. It has to do with the buildroot, but, I have not tracked it down yet. I'm using the CVS version of mock. -- -jeff From katzj at redhat.com Tue Jul 26 00:53:26 2005 From: katzj at redhat.com (Jeremy Katz) Date: Mon, 25 Jul 2005 20:53:26 -0400 Subject: permissions in mock In-Reply-To: <200507260833.10440.symbiont@berlios.de> References: <200507260833.10440.symbiont@berlios.de> Message-ID: <1122339206.3581.51.camel@bree.local.net> On Tue, 2005-07-26 at 08:33 +0800, Jeff Pitman wrote: > Anyone else notice in the build.log that there are areas where > permission is denied when doing stuff. The two most notable are 1. > checking for unpackaged files, and 2. stripping binary libs. It has to > do with the buildroot, but, I have not tracked it down yet. Can you send an example log? I don't remember seeing this when I was testing the SELinux stuff. Also, do you have SELinux enabled or disabled? Jeremy From jwboyer at jdub.homelinux.org Tue Jul 26 01:40:26 2005 From: jwboyer at jdub.homelinux.org (Josh Boyer) Date: Mon, 25 Jul 2005 20:40:26 -0500 Subject: SSL setup Message-ID: <1122342026.2999.5.camel@yoda.jdub.homelinux.org> Ok, either I'm a complete idiot (highly possible) or the instructions on setting up SSL in the README file of plague are a bit out of date. I generated the keys and CA as listed in the steps at the bottom of the README, but it seems that the builder expects the key and cert to be in one file instead of two separate ones. /usr/bin/plague-builder -c /etc/plague/builder/CONFIG.py /etc/plague/builder/certs/builder_key_and_cert.pem does not exist or is not readable. Is the README up to date? If not, could someone point out the highlights of what isn't so I can try to get this running again. I'm just doing a simple setup of 1 builder and 1 server on the same host. thx, josh From ivazquez at ivazquez.net Tue Jul 26 02:03:51 2005 From: ivazquez at ivazquez.net (Ignacio Vazquez-Abrams) Date: Mon, 25 Jul 2005 22:03:51 -0400 Subject: SSL setup In-Reply-To: <1122342026.2999.5.camel@yoda.jdub.homelinux.org> References: <1122342026.2999.5.camel@yoda.jdub.homelinux.org> Message-ID: <1122343431.3550.9.camel@ignacio.lan> On Mon, 2005-07-25 at 20:40 -0500, Josh Boyer wrote: > Ok, either I'm a complete idiot (highly possible) or the instructions on > setting up SSL in the README file of plague are a bit out of date. > > I generated the keys and CA as listed in the steps at the bottom of the > README, but it seems that the builder expects the key and cert to be in > one file instead of two separate ones. > > /usr/bin/plague-builder -c /etc/plague/builder/CONFIG.py > /etc/plague/builder/certs/builder_key_and_cert.pem does not exist or is > not readable. > > Is the README up to date? If not, could someone point out the > highlights of what isn't so I can try to get this running again. I'm > just doing a simple setup of 1 builder and 1 server on the same host. cat foo.key >> foo.cert -- Ignacio Vazquez-Abrams http://fedora.ivazquez.net/ gpg --keyserver hkp://subkeys.pgp.net --recv-key 38028b72 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From symbiont at berlios.de Tue Jul 26 07:13:38 2005 From: symbiont at berlios.de (Jeff Pitman) Date: Tue, 26 Jul 2005 15:13:38 +0800 Subject: permissions in mock In-Reply-To: <1122339206.3581.51.camel@bree.local.net> References: <200507260833.10440.symbiont@berlios.de> <1122339206.3581.51.camel@bree.local.net> Message-ID: <200507261513.39059.symbiont@berlios.de> On Tuesday 26 July 2005 08:53, Jeremy Katz wrote: > On Tue, 2005-07-26 at 08:33 +0800, Jeff Pitman wrote: > > Anyone else notice in the build.log that there are areas where > > permission is denied when doing stuff. The two most notable are 1. > > checking for unpackaged files, and 2. stripping binary libs. It > > has to do with the buildroot, but, I have not tracked it down yet. > > Can you send an example log? I don't remember seeing this when I was > testing the SELinux stuff. Also, do you have SELinux enabled or > disabled? [jeff at symbiont pyvault-build]$ selinuxenabled [jeff at symbiont pyvault-build]$ echo $? 1 Why do we document -256 as the return value? ;-) I'm actually using an old version of mock-helper. The one that doesn't have selinux changes, etc. But, I'm not running it so I didn't think it'd do anything. Anyway, I just complied the new one and did a rebuild. However, it still happens. Without fail I get this everytime: Checking for unpackaged file(s): /usr/lib/rpm/check-files /var/tmp/python22-2.2. 3-root find: cannot get current directory: Permission denied warning: Could not canonicalize hostname: symbiont.shacknet.nu I think the warning could be cleaned up by throwing something in /etc/hosts. -- -jeff From jwboyer at jdub.homelinux.org Sat Jul 30 15:08:09 2005 From: jwboyer at jdub.homelinux.org (Josh Boyer) Date: Sat, 30 Jul 2005 10:08:09 -0500 Subject: plague-client kill hangs Message-ID: <1122736090.4755.14.camel@yoda.jdub.homelinux.org> I tried killing a job on the buildsys recently and got some funny results. The job seemed to actually be killed, and I got an email stating so. However, the plague-client never returned to the command line. I'm not sure if it was waiting for a response from the server or what. Here's the backtrace I got when Ctrl-C'ing it: jwboyer at yoda FC-4]$ plague-client kill 218 Traceback (most recent call last): File "/usr/bin/plague-client", line 334, in ? kill(server, email, jobid) File "/usr/bin/plague-client", line 179, in kill (err, msg) = server.kill_job(email, jobid) File "/usr/lib/python2.4/xmlrpclib.py", line 1096, in __call__ return self.__send(self.__name, args) File "/usr/lib/python2.4/xmlrpclib.py", line 1383, in __request verbose=self.__verbose File "/usr/lib/python2.4/xmlrpclib.py", line 1131, in request errcode, errmsg, headers = h.getreply() File "/usr/lib/python2.4/httplib.py", line 1133, in getreply response = self._conn.getresponse() File "/usr/lib/python2.4/httplib.py", line 862, in getresponse response.begin() File "/usr/lib/python2.4/httplib.py", line 333, in begin version, status, reason = self._read_status() File "/usr/lib/python2.4/httplib.py", line 291, in _read_status line = self.fp.readline() File "/usr/lib/python2.4/socket.py", line 325, in readline data = recv(1) File "/usr/lib/python2.4/site-packages/plague/SSLConnection.py", line 74, in recv ret = self.__dict__["conn"].recv(bufsize) KeyboardInterrupt I'll be gone for the rest of today, but I can play around with stuff tonight and tomorrow if someone thinks of something. josh