From mikeb at redhat.com Mon Jan 5 18:20:42 2009 From: mikeb at redhat.com (Mike Bonnet) Date: Mon, 05 Jan 2009 13:20:42 -0500 Subject: Supporting EPEL Builds in Koji In-Reply-To: <48EA6382.2070000@redhat.com> References: <1215705751.30991.7.camel@burren.bos.redhat.com> <487F8762.6090007@redhat.com> <1216334914.16888.82.camel@burren.bos.redhat.com> <4880B8F4.9040701@redhat.com> <1218663337.16585.63.camel@maunalani.home> <48EA6382.2070000@redhat.com> Message-ID: <1231179642.22098.52.camel@localhost.localdomain> Picking up this thread again, sorry about the long delay. I'd like to come to consensus on the approach here, hammer out any remaining details at FUDCon this weekend, and hopefully get this implemented by the end of January. Time to really get rid of plague! On Mon, 2008-10-06 at 15:14 -0400, Mike McLean wrote: > Mike Bonnet wrote: > > On Fri, 2008-07-18 at 11:38 -0400, Mike McLean wrote: > >> Mike Bonnet wrote: > >>> On Thu, 2008-07-17 at 13:54 -0400, Mike McLean wrote: > >>>> If the remote_repo_url data is going to be inherited (and I tend to > >>>> think it should be), then I think it should be in a separate table. > ... > >>> I don't have any problem with this, though it does mean we'll need to > >>> duplicate quite a bit of the inheritance-walking code, > ... > >> Walking inheritance is just a matter of determining the inheritance > >> order and scanning data on the parent tags in sequence. > ... > > Sorry, I was referring to walking tag_inheritance. I'd rather have one > > place that walks the inheritance hierarchy and aggregates data from it, > > than two places that are doing almost the same thing. > > We're talking about inherently different data. External repos to be > merged in are quite different from builds in the system. Yes, I see the issue here. Since remote repos won't have their packages filtered out (by mergerepo) until after all packages in the local inheritance hierarchy are placed in the repo, they don't really follow the existing inheritance rules. Ok, you've convinced me. A separate table that stores a priority-ordered list of remote repos associated with each tag will probably be easier to manage. The lists will be aggregated when walking the tag hierarchy and passed to mergerepo in (priority, inheritance) order for proper filtering (based on srpm name, first match wins). > > Each tag has a set of builds associated with it. We walk the > > inheritance hierarchy, aggregating the builds from each tag in the > > hierarchy into a flat list, and then pass that list to createrepo. We > > would do essentially the same thing for external repos. When walking > > the hierarchy, if a tag has an external repo associated with it, we > > would append that repo url to a flat list, and pass that list to > > mergerepo. In both cases we're working with collections of packages > > that are associated with a tag, just in different formats. > > Sure, we can do this with one call to readFullInheritance, and traverse > both the build table and external repo table from the given order. Yes, that makes sense. > > In discussing this with Jesse, I think we want external repos to be > > inherited. This is probably the easiest way to deal with having > > multiple external repos getting pulled in to a single buildroot, which > > is essential for Fedora (think F9 GA and F9 Updates). > > > > The idea was that, by convention, we would have external-repo-only tags, > > with only a single external repo associated with it and no > > packages/builds associated. These external-repo-only tags could then be > > inserted into the build hierarchy where appropriate. An ordered list of > > external repos could then be constructed by performing the current > > depth-first search of the inheritance hierarchy. The ordered list would > > then be passed to mergerepo, which would ensure that packages in repos > > earlier in the list supersede packages (by srpm name) in repos later in > > the list. This would preserve the "first-match-wins" inheritance policy > > that Koji currently implements, and that admins expect. For example: > > > > dist-custom-build > > ??dist-custom > > ??dist-f9-updates-external > > ??dist-f9-ga-external > > > > would result mergerepo creating a single repo that would only contain > > packages from dist-f9-ga-external if they did not exist in the > > Koji-generated repo (dist-custom-build + dist-custom), > > dist-f9-updates-external, or the blacklist of blocked packages. This is > > consistent with how Koji package inheritance currently works, and I > > think is the most intuitive approach. > > It is similar, but different in potentially confusing ways. External > repos do not have build structure, so we can't really have the same sort > of inheritance behavior with a combination of external repo tags and > normal tags. > > We order the external repos in inheritance order, but ultimately those > repos are merged with the internal one in a way that does not honor > inheritance in the way that the admin might expect. > > Using tags to represent external repos fails intuition because external > repos are very much not like tags. When we get to supporting external > koji systems, we can do something like this, but for external repos the > "bolted-on" nature needs to be clear. This is why I'd prefer to have the > data a little more removed. Ok, we're agreed on this. > >> I see all that, and I'm almost convinced. The flipside is that by > >> default all the code will treat these external rpms the same as the > >> local ones, which will not be correct for a number of cases. > > > > Personally I'd prefer adding a few special cases to the existing code, > > rather than maintain a whole heap of almost-but-not-quite-the-same code > > to manage external rpms. I think that conceptually they're alike enough > > that the number of special cases will be minimal. > > I think I'm ok with using the rpminfo table. > > > I think that synthesizing builds for that sake of maintaining the > > not-null constraint is more pain than it's worth, and would make > > enforcing our nvr-uniqueness constraints (which we definitely want to do > > for local builds) more difficult. Having locally-built rpms always > > associated with a build, and external rpms not, makes sense to me. > > Ok, agreed. > > >> Also, I'm thinking we need to have some sort of rpm_origin table so that > >> all these references can be managed cleanly. > > > > That sounds reasonable to me. Note that we may end up with a lot of > > rows in this table, since we're allowing variable substitution in the > > external_repo_url (tag name and arch). But I don't see that as a > > problem. > > I'm thinking the only substitution we should support is arch. Anything > else sort of constitutes a different repo. > > If we use an origin table like this we can abstract out the arch. > Something like: > > create table external_repo ( > id SERIAL PRIMARY KEY, > name TEXT ); > create table external_repo_config ( > external_repo_id INTEGER NOT NULL REFERENCES external_repo (id), > url TEXT NOT NULL, > -- plus versioning fields > -- ... ); > > This way if upstream repo changes url scheme or moves to a different > host, you can keep some notion of connectedness. External rpms would > simply reference external_repo_id. Makes sense. So a tag would simply reference the external_repo_id as well, and the repo url would be set elsewhere (globally). The table storing the external repo info for tags would look like: create table tag_external_repos ( tag_id INTEGER NOT NULL REFERENCES tag(id), external_repo_id INTEGER NOT NULL REFERENCES external_repo(id), priority INTEGER NOT NULL, -- plus versioning fields UNIQUE (tag_id,priority,active) ); I like this, it keeps everything much more normalized. > >> In the same vein, what happens when an external repo has an nvra+sigmd5 > >> matching a /local/ rpm? Maybe it doesn't matter, though I guess > >> technically we want to record the origin properly when it gets into a > >> buildroot via external repo vs internal tag. > > > > Right, we would record the origin as the remote repo it came from (by > > parsing the merged repodata and looking at the baseurl). Right, and the origin can just be stored as a reference to the external_repo(id). > So where do we draw the line between code that we add to koji and code > that we add to createrepo (or some external merge-repo tool)? Koji would only be responsible for parsing the repodata and populating the database with the correct origin for any given rpm. mergerepo would be responsible for creating the repo and enforcing the filtering rules. > >>> However, we will already be parsing the remote repodata, which contains > >>> information like the srpm name for each rpm, so we could do something > >>> more sophisticated here. > >> -snipsnip- > >> ... > >>> The repomerge tool seems like it solves the problem better, and would be > >>> more useful in general. > >> If we're going to have our fingers in the repodata, we'll probably want > >> to have them in the merge too. Perhaps we can get createrepo and/or this > >> repomerge tool usefully libified? > > > > I was thinking we would probably just call out to the tool the way we do > > for createrepo, but I'm certainly not against using an API. I'm a > > little concerned about memory usage when doing the create/mergerepo > > in-process, since we know python and mod_python have garbage-collection > > issues, but that may be a "cross the bridge when we come to it" problem. > > Seth, is it feasible to provide an API to mergerepo that we could use > > directly? > > I don't think I even saw a reply from Seth on this. Where does the > mergerepo code stand now? It has been written by Seth, I just need to test it. The tool currently has command-line flags to do everything we need it to do (I believe) but we could also use it as an example to use the api directly. From notting at redhat.com Tue Jan 6 15:34:47 2009 From: notting at redhat.com (Bill Nottingham) Date: Tue, 6 Jan 2009 10:34:47 -0500 Subject: [PATCH] pungi: allow building a self-hosting distribution Message-ID: <1231256090-13946-1-git-send-email-notting@redhat.com> The attached patch series allows pungi to build 'self-hosting' trees by resolving all the build dependencies for a package set. It does not expand the package set for the 'assumed' build depedencies such as gcc; those are probably better specified in the kickstart file. As an example, with this, 'pungi --selfhosting' of a kickstart with just @Core yields over 1200 source packages, and over 500 source packages. bin/pungi.py | 7 +++++- pypungi/__init__.py | 59 +++++++++++++++++++++++++++++++++++++--------------- 2 files changed, 48 insertions(+), 18 deletions(-) From notting at redhat.com Tue Jan 6 15:34:48 2009 From: notting at redhat.com (Bill Nottingham) Date: Tue, 6 Jan 2009 10:34:48 -0500 Subject: [PATCH] Operate on source rpm package objects, not a list that is then turned into package objects. In-Reply-To: <1231256090-13946-1-git-send-email-notting@redhat.com> References: <1231256090-13946-1-git-send-email-notting@redhat.com> Message-ID: <1231256090-13946-2-git-send-email-notting@redhat.com> --- src/pypungi/__init__.py | 36 +++++++++++++++++++----------------- 1 files changed, 19 insertions(+), 17 deletions(-) diff --git a/src/pypungi/__init__.py b/src/pypungi/__init__.py index cc3928f..2590775 100644 --- a/src/pypungi/__init__.py +++ b/src/pypungi/__init__.py @@ -133,7 +133,7 @@ class Pungi(pypungi.PungiBase): self.ksparser = ksparser self.polist = [] - self.srpmlist = [] + self.srpmpolist = [] self.debuginfolist = [] self.resolved_deps = {} # list the deps we've already resolved, short circuit. @@ -396,15 +396,29 @@ class Pungi(pypungi.PungiBase): self.polist = final_pkgobjs.keys() self.logger.info('Finished gathering package objects.') + def getSRPMPo(self, po): + """Given a package object, get a package object for the + corresponding source rpm. Requires yum still configured + and a valid package object.""" + srpm = po.sourcerpm.split('.src.rpm')[0] + (sname, sver, srel) = srpm.rsplit('-', 2) + try: + srpmpo = self.ayum.pkgSack.searchNevra(name=sname, ver=sver, rel=srel, arch='src')[0] + return srpmpo + except IndexError: + print >> sys.stderr, "Error: Cannot find a source rpm for %s" % srpm + sys.exit(1) + def getSRPMList(self): """Cycle through the list of package objects and find the sourcerpm for them. Requires yum still configured and a list of package objects""" for po in self.polist: - srpm = po.sourcerpm.split('.src.rpm')[0] - if not srpm in self.srpmlist: - self.srpmlist.append(srpm) + srpmpo = self.getSRPMPo(po) + if not srpmpo in self.srpmpolist: + self.logger.info("Adding source package %s.%s" % (srpmpo.name, srpmpo.arch)) + self.srpmpolist.append(srpmpo) def getDebuginfoList(self): """Cycle through the list of package objects and find @@ -528,20 +542,8 @@ class Pungi(pypungi.PungiBase): """Cycle through the list of srpms and find the package objects for them, Then download them.""" - srpmpolist = [] - - for srpm in self.srpmlist: - (sname, sver, srel) = srpm.rsplit('-', 2) - try: - srpmpo = self.ayum.pkgSack.searchNevra(name=sname, ver=sver, rel=srel, arch='src')[0] - if not srpmpo in srpmpolist: - srpmpolist.append(srpmpo) - except IndexError: - print >> sys.stderr, "Error: Cannot find a source rpm for %s" % srpm - sys.exit(1) - # do the downloads - self._downloadPackageList(srpmpolist, os.path.join('source', 'SRPMS')) + self._downloadPackageList(self.srpmpolist, os.path.join('source', 'SRPMS')) def downloadDebuginfo(self): """Cycle through the list of debuginfo rpms and -- 1.6.0.6 From notting at redhat.com Tue Jan 6 15:34:49 2009 From: notting at redhat.com (Bill Nottingham) Date: Tue, 6 Jan 2009 10:34:49 -0500 Subject: [PATCH] Resolve package build dependencies. In-Reply-To: <1231256090-13946-2-git-send-email-notting@redhat.com> References: <1231256090-13946-1-git-send-email-notting@redhat.com> <1231256090-13946-2-git-send-email-notting@redhat.com> Message-ID: <1231256090-13946-3-git-send-email-notting@redhat.com> Since each package we add for build dependencies may add a new source rpm to our list, this needs to recurse. --- src/pypungi/__init__.py | 23 +++++++++++++++++++++++ 1 files changed, 23 insertions(+), 0 deletions(-) diff --git a/src/pypungi/__init__.py b/src/pypungi/__init__.py index 2590775..3271f26 100644 --- a/src/pypungi/__init__.py +++ b/src/pypungi/__init__.py @@ -420,6 +420,29 @@ class Pungi(pypungi.PungiBase): self.logger.info("Adding source package %s.%s" % (srpmpo.name, srpmpo.arch)) self.srpmpolist.append(srpmpo) + def resolvePackageBuildDeps(self): + """Make the package lists self hosting. Requires yum + still configured, a list of package objects, and a + a list of source rpms.""" + for srpm in self.srpmpolist: + self.ayum.tsInfo.addInstall(srpm) + deppass = 1 + checked_srpms = [] + while 1: + self.logger.info("Resolving build dependencies, pass %d" % (deppass)) + prev = list(self.ayum.tsInfo.getMembers()) + for srpm in self.srpmpolist[len(checked_srpms):]: + self.getPackageDeps(srpm) + for txmbr in self.ayum.tsInfo: + if txmbr.po.arch != 'src' and txmbr.po not in self.polist: + self.polist.append(txmbr.po) + # Now that we've resolved deps, refresh the source rpm list + checked_srpms = list(self.srpmpolist) + self.getSRPMList() + deppass = deppass + 1 + if len(prev) == len(self.ayum.tsInfo.getMembers()): + break + def getDebuginfoList(self): """Cycle through the list of package objects and find debuginfo rpms for them. Requires yum still -- 1.6.0.6 From notting at redhat.com Tue Jan 6 15:34:50 2009 From: notting at redhat.com (Bill Nottingham) Date: Tue, 6 Jan 2009 10:34:50 -0500 Subject: [PATCH] Wire up a commandline option for selfhosting support. In-Reply-To: <1231256090-13946-3-git-send-email-notting@redhat.com> References: <1231256090-13946-1-git-send-email-notting@redhat.com> <1231256090-13946-2-git-send-email-notting@redhat.com> <1231256090-13946-3-git-send-email-notting@redhat.com> Message-ID: <1231256090-13946-4-git-send-email-notting@redhat.com> --- src/bin/pungi.py | 7 ++++++- 1 files changed, 6 insertions(+), 1 deletions(-) diff --git a/src/bin/pungi.py b/src/bin/pungi.py index 7cc615c..9224713 100755 --- a/src/bin/pungi.py +++ b/src/bin/pungi.py @@ -86,13 +86,16 @@ def main(): mypungi._inityum() # initialize the yum object for things that need it if opts.do_all or opts.do_gather: mypungi.getPackageObjects() + if not opts.nosource or opts.selfhosting: + mypungi.getSRPMList() + if opts.selfhosting: + mypungi.resolvePackageBuildDeps() mypungi.downloadPackages() mypungi.makeCompsFile() if not opts.nodebuginfo: mypungi.getDebuginfoList() mypungi.downloadDebuginfo() if not opts.nosource: - mypungi.getSRPMList() mypungi.downloadSRPMs() if opts.do_all or opts.do_createrepo: @@ -153,6 +156,8 @@ if __name__ == '__main__': parser.add_option("--bugurl", dest="bugurl", type="string", action="callback", callback=set_config, callback_args=(config, ), help='the url for your bug system (defaults to http://bugzilla.redhat.com)') + parser.add_option("--selfhosting", action="store_true", dest="selfhosting", + help='build a self-hosting tree by following build dependencies (optional)') parser.add_option("--nosource", action="store_true", dest="nosource", help='disable gathering of source packages (optional)') parser.add_option("--nodebuginfo", action="store_true", dest="nodebuginfo", -- 1.6.0.6 From mikeb at redhat.com Tue Jan 6 20:21:56 2009 From: mikeb at redhat.com (Mike Bonnet) Date: Tue, 06 Jan 2009 15:21:56 -0500 Subject: Koji feature proposals Message-ID: <1231273316.15116.14.camel@burren.bos.redhat.com> I've just created tickets for a few Koji features that I've been wanting to implement for a while (as well as updated an old one), and I'm planning to devote some time to in the near future. If you have any comments on these features feel free to post to the tickets, or talk to me at FUDCon this weekend. Just figured people might want to see the direction that Koji is headed. The future is now! :) Thanks, Mike build the srpm in a chroot: https://fedorahosted.org/koji/ticket/103 support for building isos and disk images in Koji: https://fedorahosted.org/koji/ticket/122 koji callback system: https://fedorahosted.org/koji/ticket/123 drop the rpmfiles and rpmdeps tables: https://fedorahosted.org/koji/ticket/124 noarch subpackage support: https://fedorahosted.org/koji/ticket/125 From oliver at linux-kernel.at Tue Jan 6 20:51:24 2009 From: oliver at linux-kernel.at (Oliver Falk) Date: Tue, 06 Jan 2009 21:51:24 +0100 Subject: Koji feature proposals In-Reply-To: <1231273316.15116.14.camel@burren.bos.redhat.com> References: <1231273316.15116.14.camel@burren.bos.redhat.com> Message-ID: <4963C44C.6090805@linux-kernel.at> Hi Mike! Mike Bonnet schrieb: > I've just created tickets for a few Koji features that I've been wanting > to implement for a while (as well as updated an old one), and I'm > planning to devote some time to in the near future. If you have any > comments on these features feel free to post to the tickets, or talk to > me at FUDCon this weekend. Just figured people might want to see the > direction that Koji is headed. The future is now! :) [ ... ] > drop the rpmfiles and rpmdeps tables: https://fedorahosted.org/koji/ticket/124 -1 Only if you provide the same functionality using another approach :-) *I* do use it quite often; Find out which file belongs to which pkg(s). However, this was quite slow and now doesn't even seem to work in koji.fpo. :-( > noarch subpackage support: https://fedorahosted.org/koji/ticket/125 Duh? We do have already (at least one) packages that build arch-specific and noarch pkgs -> kernel or do we use some *hack* in the kernel.spec? And regarding your point: '... different arches build noarch subpackage with different contents'. Well, then it's definitly not *noarch*, is't it? :-) -of From mikeb at redhat.com Tue Jan 6 21:04:29 2009 From: mikeb at redhat.com (Mike Bonnet) Date: Tue, 06 Jan 2009 16:04:29 -0500 Subject: Koji feature proposals In-Reply-To: <4963C44C.6090805@linux-kernel.at> References: <1231273316.15116.14.camel@burren.bos.redhat.com> <4963C44C.6090805@linux-kernel.at> Message-ID: <1231275869.15116.19.camel@burren.bos.redhat.com> On Tue, 2009-01-06 at 21:51 +0100, Oliver Falk wrote: > Hi Mike! > > Mike Bonnet schrieb: > > I've just created tickets for a few Koji features that I've been wanting > > to implement for a while (as well as updated an old one), and I'm > > planning to devote some time to in the near future. If you have any > > comments on these features feel free to post to the tickets, or talk to > > me at FUDCon this weekend. Just figured people might want to see the > > direction that Koji is headed. The future is now! :) > > [ ... ] > > > drop the rpmfiles and rpmdeps tables: https://fedorahosted.org/koji/ticket/124 > > -1 > Only if you provide the same functionality using another approach :-) Yes, the plan is to query the information directly from the rpms rather than from the database. The content on the rpminfo page in the web UI should not change at all from the user perspective. > *I* do use it quite often; Find out which file belongs to which pkg(s). > > However, this was quite slow and now doesn't even seem to work in > koji.fpo. :-( Hmmm, it should be. In what way is it not working? > > noarch subpackage support: https://fedorahosted.org/koji/ticket/125 > > Duh? We do have already (at least one) packages that build arch-specific > and noarch pkgs -> kernel or do we use some *hack* in the kernel.spec? We use a hack in the kernel specfile and in the build system. The noarch subpackage support in rpm is much more generic and flexible, and we need to support it without build system hacks. > And regarding your point: '... different arches build noarch subpackage > with different contents'. Well, then it's definitly not *noarch*, is't > it? :-) True, but it's still possible, and we may need to check for this case and handle is appropriately (possibly by failing the build). From tibbs at math.uh.edu Tue Jan 6 21:21:27 2009 From: tibbs at math.uh.edu (Jason L Tibbitts III) Date: 06 Jan 2009 15:21:27 -0600 Subject: Koji feature proposals In-Reply-To: <4963C44C.6090805@linux-kernel.at> References: <1231273316.15116.14.camel@burren.bos.redhat.com> <4963C44C.6090805@linux-kernel.at> Message-ID: >>>>> "OF" == Oliver Falk writes: OF> And regarding your point: '... different arches build noarch OF> subpackage with different contents'. Well, then it's definitly not OF> *noarch*, is't it? :-) It is quite possible for the contents to differ by, say, date, or by timestamps being included in plain text output. Why would that render the output arch-specific? - J< From notting at redhat.com Tue Jan 6 21:35:47 2009 From: notting at redhat.com (Bill Nottingham) Date: Tue, 6 Jan 2009 16:35:47 -0500 Subject: [PATCH] pungi: allow building a self-hosting distribution In-Reply-To: <1231277751-23771-1-git-send-email-notting@redhat.com> References: <1231277751-23771-1-git-send-email-notting@redhat.com> Message-ID: <1231277751-23771-2-git-send-email-notting@redhat.com> GIT: Please enter your email below. GIT: Lines beginning in "GIT: " will be removed. GIT: Consider including an overall diffstat or table of contents GIT: for the patch you are writing. From notting at redhat.com Tue Jan 6 21:35:48 2009 From: notting at redhat.com (Bill Nottingham) Date: Tue, 6 Jan 2009 16:35:48 -0500 Subject: [PATCH] Remove obsolete code. In-Reply-To: <1231277751-23771-2-git-send-email-notting@redhat.com> References: <1231277751-23771-1-git-send-email-notting@redhat.com> <1231277751-23771-2-git-send-email-notting@redhat.com> Message-ID: <1231277751-23771-3-git-send-email-notting@redhat.com> --- src/pypungi/__init__.py | 2 -- 1 files changed, 0 insertions(+), 2 deletions(-) diff --git a/src/pypungi/__init__.py b/src/pypungi/__init__.py index 3271f26..bd57bf8 100644 --- a/src/pypungi/__init__.py +++ b/src/pypungi/__init__.py @@ -424,8 +424,6 @@ class Pungi(pypungi.PungiBase): """Make the package lists self hosting. Requires yum still configured, a list of package objects, and a a list of source rpms.""" - for srpm in self.srpmpolist: - self.ayum.tsInfo.addInstall(srpm) deppass = 1 checked_srpms = [] while 1: -- 1.6.0.6 From notting at redhat.com Tue Jan 6 21:35:49 2009 From: notting at redhat.com (Bill Nottingham) Date: Tue, 6 Jan 2009 16:35:49 -0500 Subject: [PATCH] Create dicts to map between source and binary packages. In-Reply-To: <1231277751-23771-3-git-send-email-notting@redhat.com> References: <1231277751-23771-1-git-send-email-notting@redhat.com> <1231277751-23771-2-git-send-email-notting@redhat.com> <1231277751-23771-3-git-send-email-notting@redhat.com> Message-ID: <1231277751-23771-4-git-send-email-notting@redhat.com> This avoids repeating the operation many times later if we do it on demand each time. --- src/pypungi/__init__.py | 32 ++++++++++++++++++++++++++------ 1 files changed, 26 insertions(+), 6 deletions(-) diff --git a/src/pypungi/__init__.py b/src/pypungi/__init__.py index bd57bf8..1d2734c 100644 --- a/src/pypungi/__init__.py +++ b/src/pypungi/__init__.py @@ -135,6 +135,9 @@ class Pungi(pypungi.PungiBase): self.polist = [] self.srpmpolist = [] self.debuginfolist = [] + self.srpms_build = [] + self.srpms_fulltree = [] + self.last_po = 0 self.resolved_deps = {} # list the deps we've already resolved, short circuit. def _inityum(self): @@ -409,33 +412,50 @@ class Pungi(pypungi.PungiBase): print >> sys.stderr, "Error: Cannot find a source rpm for %s" % srpm sys.exit(1) + def createSourceHashes(self): + """Create two dicts - one that maps binary POs to source POs, and + one that maps a single source PO to all binary POs it produces. + Requires yum still configured.""" + self.src_by_bin = {} + self.bin_by_src = {} + self.logger.info("Generating source <-> binary package mappings") + (dummy1, everything, dummy2) = yum.packages.parsePackages(self.ayum.pkgSack.returnPackages(), ['*']) + for po in everything: + if po.arch == 'src': + continue + srpmpo = self.getSRPMPo(po) + self.src_by_bin[po] = srpmpo + if self.bin_by_src.has_key(srpmpo): + self.bin_by_src[srpmpo].append(po) + else: + self.bin_by_src[srpmpo] = [po] + def getSRPMList(self): """Cycle through the list of package objects and find the sourcerpm for them. Requires yum still configured and a list of package objects""" - - for po in self.polist: - srpmpo = self.getSRPMPo(po) + for po in self.polist[self.last_po:]: + srpmpo = self.src_by_bin[po] if not srpmpo in self.srpmpolist: self.logger.info("Adding source package %s.%s" % (srpmpo.name, srpmpo.arch)) self.srpmpolist.append(srpmpo) + self.last_po = len(self.polist) def resolvePackageBuildDeps(self): """Make the package lists self hosting. Requires yum still configured, a list of package objects, and a a list of source rpms.""" deppass = 1 - checked_srpms = [] while 1: self.logger.info("Resolving build dependencies, pass %d" % (deppass)) prev = list(self.ayum.tsInfo.getMembers()) - for srpm in self.srpmpolist[len(checked_srpms):]: + for srpm in self.srpmpolist[len(self.srpms_build):]: self.getPackageDeps(srpm) for txmbr in self.ayum.tsInfo: if txmbr.po.arch != 'src' and txmbr.po not in self.polist: self.polist.append(txmbr.po) + self.srpms_build = list(self.srpmpolist) # Now that we've resolved deps, refresh the source rpm list - checked_srpms = list(self.srpmpolist) self.getSRPMList() deppass = deppass + 1 if len(prev) == len(self.ayum.tsInfo.getMembers()): -- 1.6.0.6 From notting at redhat.com Tue Jan 6 21:35:50 2009 From: notting at redhat.com (Bill Nottingham) Date: Tue, 6 Jan 2009 16:35:50 -0500 Subject: [PATCH] Add a method that completes the package set with all subpackages of currently used source rpms. In-Reply-To: <1231277751-23771-4-git-send-email-notting@redhat.com> References: <1231277751-23771-1-git-send-email-notting@redhat.com> <1231277751-23771-2-git-send-email-notting@redhat.com> <1231277751-23771-3-git-send-email-notting@redhat.com> <1231277751-23771-4-git-send-email-notting@redhat.com> Message-ID: <1231277751-23771-5-git-send-email-notting@redhat.com> In other places, this method could be called No Package Left Behind. --- src/pypungi/__init__.py | 26 ++++++++++++++++++++++++++ 1 files changed, 26 insertions(+), 0 deletions(-) diff --git a/src/pypungi/__init__.py b/src/pypungi/__init__.py index 1d2734c..01d7b90 100644 --- a/src/pypungi/__init__.py +++ b/src/pypungi/__init__.py @@ -461,6 +461,32 @@ class Pungi(pypungi.PungiBase): if len(prev) == len(self.ayum.tsInfo.getMembers()): break + def completePackageSet(self): + """Cycle through all package objects, and add any + that correspond to a source rpm that we are including. + Requires yum still configured and a list of package + objects.""" + thepass = 1 + while 1: + prevlen = len(self.srpmpolist) + self.logger.info("Completing package set, pass %d" % (thepass,)) + for srpm in self.srpmpolist[len(self.srpms_fulltree):]: + for po in self.bin_by_src[srpm]: + if po not in self.polist: + self.logger.info("Adding %s.%s to complete package set" % (po.name, po.arch)) + self.polist.append(po) + self.getPackageDeps(po) + for txmbr in self.ayum.tsInfo: + if txmbr.po.arch != 'src' and txmbr.po not in self.polist: + self.polist.append(txmbr.po) + self.srpms_fulltree = list(self.srpmpolist) + # Now that we've resolved deps, refresh the source rpm list + self.getSRPMList() + if len(self.srpmpolist) == prevlen: + self.logger.info("Completion finished in %d passes" % (thepass,)) + break + thepass = thepass + 1 + def getDebuginfoList(self): """Cycle through the list of package objects and find debuginfo rpms for them. Requires yum still -- 1.6.0.6 From notting at redhat.com Tue Jan 6 21:35:46 2009 From: notting at redhat.com (Bill Nottingham) Date: Tue, 6 Jan 2009 16:35:46 -0500 Subject: [PATCH] pungi: Implement building of 'full' trees Message-ID: <1231277751-23771-1-git-send-email-notting@redhat.com> The attached patches implement the building of 'full' trees - these are trees that contain all subpackages of any source RPMs used in the tree composition. This can be useful if the distribution you're building isn't going to have an 'Everything' tree like Fedora has. As an example, a compose of just 'basesystem' with both this option, and the selfhosting option, yields over 2600 binary packages. Patches apply on top of the prior patch series. Bill bin/pungi.py | 18 ++++++++++++++- pypungi/__init__.py | 60 +++++++++++++++++++++++++++++++++++++++++++++------- 2 files changed, 69 insertions(+), 9 deletions(-) From notting at redhat.com Tue Jan 6 21:35:51 2009 From: notting at redhat.com (Bill Nottingham) Date: Tue, 6 Jan 2009 16:35:51 -0500 Subject: [PATCH] Wire in support for composing 'full' trees with all subpackages. In-Reply-To: <1231277751-23771-5-git-send-email-notting@redhat.com> References: <1231277751-23771-1-git-send-email-notting@redhat.com> <1231277751-23771-2-git-send-email-notting@redhat.com> <1231277751-23771-3-git-send-email-notting@redhat.com> <1231277751-23771-4-git-send-email-notting@redhat.com> <1231277751-23771-5-git-send-email-notting@redhat.com> Message-ID: <1231277751-23771-6-git-send-email-notting@redhat.com> Since full trees and build-solved trees can affect each other, if we're doing both we need to loop between them until there are no new packages added. --- src/bin/pungi.py | 18 +++++++++++++++++- 1 files changed, 17 insertions(+), 1 deletions(-) diff --git a/src/bin/pungi.py b/src/bin/pungi.py index 9224713..eab54a6 100755 --- a/src/bin/pungi.py +++ b/src/bin/pungi.py @@ -86,10 +86,24 @@ def main(): mypungi._inityum() # initialize the yum object for things that need it if opts.do_all or opts.do_gather: mypungi.getPackageObjects() - if not opts.nosource or opts.selfhosting: + if not opts.nosource or opts.selfhosting or opts.fulltree: + mypungi.createSourceHashes() mypungi.getSRPMList() if opts.selfhosting: mypungi.resolvePackageBuildDeps() + if opts.fulltree: + mypungi.completePackageSet() + if opts.selfhosting and opts.fulltree: + # OUCH. + while 1: + plen = len(mypungi.srpmpolist) + mypungi.resolvePackageBuildDeps() + if plen == len(mypungi.srpmpolist): + break + plen = len(mypungi.srpmpolist) + mypungi.completePackageSet() + if plen == len(mypungi.srpmpolist): + break mypungi.downloadPackages() mypungi.makeCompsFile() if not opts.nodebuginfo: @@ -158,6 +172,8 @@ if __name__ == '__main__': help='the url for your bug system (defaults to http://bugzilla.redhat.com)') parser.add_option("--selfhosting", action="store_true", dest="selfhosting", help='build a self-hosting tree by following build dependencies (optional)') + parser.add_option("--fulltree", action="store_true", dest="fulltree", + help='build a tree that includes all packages built from corresponding source rpms (optional)') parser.add_option("--nosource", action="store_true", dest="nosource", help='disable gathering of source packages (optional)') parser.add_option("--nodebuginfo", action="store_true", dest="nodebuginfo", -- 1.6.0.6 From mikeb at redhat.com Tue Jan 6 21:40:43 2009 From: mikeb at redhat.com (Mike Bonnet) Date: Tue, 06 Jan 2009 16:40:43 -0500 Subject: Koji feature proposals In-Reply-To: References: <1231273316.15116.14.camel@burren.bos.redhat.com> <4963C44C.6090805@linux-kernel.at> Message-ID: <1231278043.15116.27.camel@burren.bos.redhat.com> On Tue, 2009-01-06 at 15:21 -0600, Jason L Tibbitts III wrote: > >>>>> "OF" == Oliver Falk writes: > > OF> And regarding your point: '... different arches build noarch > OF> subpackage with different contents'. Well, then it's definitly not > OF> *noarch*, is't it? :-) > > It is quite possible for the contents to differ by, say, date, or by > timestamps being included in plain text output. Why would that render > the output arch-specific? I'm not so much worried about that level of difference as I am of say different file lists from noarch rpms built on different hosts, or maybe different endianness of data files. There is some set of post-build checks we may want to run on these noarch subpackages to ensure they are in fact noarch, and that their content is sane. This noarch subpackage feature is new, and there are things Koji can and probably should be doing to make sure it's being used correctly. From tibbs at math.uh.edu Tue Jan 6 22:46:56 2009 From: tibbs at math.uh.edu (Jason L Tibbitts III) Date: 06 Jan 2009 16:46:56 -0600 Subject: Koji feature proposals In-Reply-To: <1231278043.15116.27.camel@burren.bos.redhat.com> References: <1231273316.15116.14.camel@burren.bos.redhat.com> <4963C44C.6090805@linux-kernel.at> <1231278043.15116.27.camel@burren.bos.redhat.com> Message-ID: >>>>> "MB" == Mike Bonnet writes: MB> There is some set of post-build checks we may want to run on these MB> noarch subpackages to ensure they are in fact noarch, and that MB> their content is sane. I think it would be sufficient to collect all of the noarch packages generated from the various arch builds, run rpmdiff -t on them, and fail the build if there is any output. That's a pretty strict test, but honestly I'd be concerned of any package that didn't pass it. - J< From oliver at linux-kernel.at Wed Jan 7 08:08:01 2009 From: oliver at linux-kernel.at (Oliver Falk) Date: Wed, 07 Jan 2009 09:08:01 +0100 Subject: Koji feature proposals In-Reply-To: <1231275869.15116.19.camel@burren.bos.redhat.com> References: <1231273316.15116.14.camel@burren.bos.redhat.com> <4963C44C.6090805@linux-kernel.at> <1231275869.15116.19.camel@burren.bos.redhat.com> Message-ID: <496462E1.4090500@linux-kernel.at> Mike Bonnet wrote: > On Tue, 2009-01-06 at 21:51 +0100, Oliver Falk wrote: >> Hi Mike! >> >> Mike Bonnet schrieb: >>> I've just created tickets for a few Koji features that I've been wanting >>> to implement for a while (as well as updated an old one), and I'm >>> planning to devote some time to in the near future. If you have any >>> comments on these features feel free to post to the tickets, or talk to >>> me at FUDCon this weekend. Just figured people might want to see the >>> direction that Koji is headed. The future is now! :) >> [ ... ] >> >>> drop the rpmfiles and rpmdeps tables: https://fedorahosted.org/koji/ticket/124 >> -1 >> Only if you provide the same functionality using another approach :-) > > Yes, the plan is to query the information directly from the rpms rather > than from the database. You'll likely need to cache the list of files somewhere. Just to mention it: Using yum metadata isn't enough, as you'll only query the latest pkgs... > The content on the rpminfo page in the web UI > should not change at all from the user perspective. Good. >> *I* do use it quite often; Find out which file belongs to which pkg(s). >> >> However, this was quite slow and now doesn't even seem to work in >> koji.fpo. :-( > > Hmmm, it should be. In what way is it not working? Query for Files (eg. /bin/ls): Mod_python error: "PythonHandler mod_python.publisher" Traceback (most recent call last): File "/usr/lib/python2.4/site-packages/mod_python/apache.py", line 299, in HandlerDispatch result = object(req) File "/usr/lib/python2.4/site-packages/mod_python/publisher.py", line 213, in handler published = publish_object(req, object) File "/usr/lib/python2.4/site-packages/mod_python/publisher.py", line 412, in publish_object return publish_object(req,util.apply_fs_data(object, req.form, req=req)) File "/usr/lib/python2.4/site-packages/mod_python/util.py", line 439, in apply_fs_data return object(**args) File "/usr/share/koji-web/scripts/index.py", line 1680, in search start=start, dataName='results', prefix='result', order=order) File "/usr/share/koji-web/lib/kojiweb/util.py", line 123, in paginateMethod totalRows = getattr(server, methodName)(*args, **kw) File "/usr/lib/python2.4/site-packages/koji/__init__.py", line 1133, in __call__ return self.__func(self.__name,args,opts) File "/usr/lib/python2.4/site-packages/koji/__init__.py", line 1378, in _callMethod raise err Fault: >>> noarch subpackage support: https://fedorahosted.org/koji/ticket/125 >> Duh? We do have already (at least one) packages that build arch-specific >> and noarch pkgs -> kernel or do we use some *hack* in the kernel.spec? > > We use a hack in the kernel specfile and in the build system. The > noarch subpackage support in rpm is much more generic and flexible, and > we need to support it without build system hacks. OK. I wasn't quite sure how it's working now... However, now that I know I do understand. >> And regarding your point: '... different arches build noarch subpackage >> with different contents'. Well, then it's definitly not *noarch*, is't >> it? :-) > > True, but it's still possible, and we may need to check for this case > and handle is appropriately (possibly by failing the build). Yet another post install section processing script (YAPISPS) :-) And yes, if it's not really noarch, it should fail. But shouldn't rpm itself check that? I mean, if someone writes a script to check that it should possibly go directly into rpm upstream sources... -of From oliver at linux-kernel.at Wed Jan 7 08:16:49 2009 From: oliver at linux-kernel.at (Oliver Falk) Date: Wed, 07 Jan 2009 09:16:49 +0100 Subject: Koji feature proposals In-Reply-To: References: <1231273316.15116.14.camel@burren.bos.redhat.com> <4963C44C.6090805@linux-kernel.at> Message-ID: <496464F1.1010909@linux-kernel.at> Jason L Tibbitts III wrote: >>>>>> "OF" == Oliver Falk writes: > > OF> And regarding your point: '... different arches build noarch > OF> subpackage with different contents'. Well, then it's definitly not > OF> *noarch*, is't it? :-) > > It is quite possible for the contents to differ by, say, date, or by > timestamps being included in plain text output. Why would that render > the output arch-specific? We where talking about *real* content... :-) What I could imaging to happen is something like this (just an example, but it should make my point clear): i386: somescript.sh #!/usr/libexec/blabla/i386-bla-bla/bla/binary ... x86_64: somescript.sh #!/usr/libexec/blabla/x86_64-bla-bla/bla/binary ... alpha: somescript.sh #!/usr/libexec/blabla/alpha-bla-bla/bla/binary ... It's hard to check something like this, isn't it? Well, as long as it's only the interpreter line at the beginning it might be simple, but .... But don't worry, since we do have a few arches already, such bugs should be found quite fast and maybe we can add additional checks later :-) -of From jkeating at redhat.com Wed Jan 7 14:34:56 2009 From: jkeating at redhat.com (Jesse Keating) Date: Wed, 07 Jan 2009 09:34:56 -0500 Subject: Koji feature proposals In-Reply-To: <496462E1.4090500@linux-kernel.at> References: <1231273316.15116.14.camel@burren.bos.redhat.com> <4963C44C.6090805@linux-kernel.at> <1231275869.15116.19.camel@burren.bos.redhat.com> <496462E1.4090500@linux-kernel.at> Message-ID: <1231338896.4036.37.camel@localhost.localdomain> On Wed, 2009-01-07 at 09:08 +0100, Oliver Falk wrote: > Yet another post install section processing script (YAPISPS) :-) > > And yes, if it's not really noarch, it should fail. But shouldn't rpm > itself check that? I mean, if someone writes a script to check that it > should possibly go directly into rpm upstream sources... rpm itself only ever builds for one target at a time, so a script to check to see if what was produced from one target build is different from another target build doesn't necessarily make sense in rpm sources, at least not something ran automated. -- Jesse Keating Fedora -- Freedom? is a feature! identi.ca: http://identi.ca/jkeating -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 197 bytes Desc: This is a digitally signed message part URL: From oliver at linux-kernel.at Wed Jan 7 16:23:19 2009 From: oliver at linux-kernel.at (Oliver Falk) Date: Wed, 07 Jan 2009 17:23:19 +0100 Subject: Koji feature proposals In-Reply-To: <1231338896.4036.37.camel@localhost.localdomain> References: <1231273316.15116.14.camel@burren.bos.redhat.com> <4963C44C.6090805@linux-kernel.at> <1231275869.15116.19.camel@burren.bos.redhat.com> <496462E1.4090500@linux-kernel.at> <1231338896.4036.37.camel@localhost.localdomain> Message-ID: <4964D6F7.9060800@linux-kernel.at> Jesse Keating wrote: > On Wed, 2009-01-07 at 09:08 +0100, Oliver Falk wrote: >> Yet another post install section processing script (YAPISPS) :-) >> >> And yes, if it's not really noarch, it should fail. But shouldn't rpm >> itself check that? I mean, if someone writes a script to check that it >> should possibly go directly into rpm upstream sources... > > rpm itself only ever builds for one target at a time, so a script to > check to see if what was produced from one target build is different > from another target build doesn't necessarily make sense in rpm sources, > at least not something ran automated. OKOK. I thought rpmbuild will automagically produce arch-specific and noarch packages in 'one step'... -of From jkeating at redhat.com Wed Jan 7 16:31:22 2009 From: jkeating at redhat.com (Jesse Keating) Date: Wed, 07 Jan 2009 11:31:22 -0500 Subject: Koji feature proposals In-Reply-To: <4964D6F7.9060800@linux-kernel.at> References: <1231273316.15116.14.camel@burren.bos.redhat.com> <4963C44C.6090805@linux-kernel.at> <1231275869.15116.19.camel@burren.bos.redhat.com> <496462E1.4090500@linux-kernel.at> <1231338896.4036.37.camel@localhost.localdomain> <4964D6F7.9060800@linux-kernel.at> Message-ID: <1231345882.4036.43.camel@localhost.localdomain> On Wed, 2009-01-07 at 17:23 +0100, Oliver Falk wrote: > > OKOK. I thought rpmbuild will automagically produce arch-specific and > noarch packages in 'one step'... It will, for one arch. You do a rpmbuild --rebuild foo (on x86_64) and it'll spit out x86_64 binary packages and potentially a noarch package. What we care about is doing the same command on i386 and generating i386 packages and a noarch package, having the i386 produced noarch package match the x86_64 produced noarch package. To the best of my knowledge there is no way to convince rpmbuild to produce both i386 /and/ x86_64 packages in the same run. -- Jesse Keating Fedora -- Freedom? is a feature! identi.ca: http://identi.ca/jkeating -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 197 bytes Desc: This is a digitally signed message part URL: From mikeb at redhat.com Wed Jan 7 23:23:52 2009 From: mikeb at redhat.com (Mike Bonnet) Date: Wed, 07 Jan 2009 18:23:52 -0500 Subject: [PATCH] mock: set the HOME environment variable Message-ID: <1231370632.13556.16.camel@localhost.localdomain> Signed-off-by: Mike Bonnet --- Set the HOME environment variable to match the chroothome config parameter. Allows commands running under --chroot --unpriv to create dotfiles, etc. py/mock.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/py/mock.py b/py/mock.py index d04c859..677691b 100755 --- a/py/mock.py +++ b/py/mock.py @@ -410,7 +410,6 @@ def main(ret): uidManager = mock.uid.uidManager(unprivUid, unprivGid) uidManager._becomeUser(unprivUid, unprivGid) - del(os.environ["HOME"]) # defaults config_opts = {} @@ -498,6 +497,7 @@ def main(ret): ret["chroot"] = chroot ret["config_opts"] = config_opts os.umask(002) + os.environ["HOME"] = chroot.homedir # New namespace starting from here try: -- 1.6.0.6 From tmz at pobox.com Wed Jan 7 23:59:21 2009 From: tmz at pobox.com (Todd Zullinger) Date: Wed, 7 Jan 2009 18:59:21 -0500 Subject: [PATCH/mock] Fix git log command used for ChangeLog and AUTHORS Message-ID: <20090107235921.GI21007@inocybe.teonanacatl.org> The git-* commands are not in the PATH by default with git >= 1.6.0. Signed-off-by: Todd Zullinger --- Makefile.am | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Makefile.am b/Makefile.am index aefcad8..4fe1e7d 100644 --- a/Makefile.am +++ b/Makefile.am @@ -58,10 +58,10 @@ CLEANFILES += py/*.pyc py/mock/*.pyc py/mock/plugins/*.pyc dist: ChangeLog AUTHORS ChangeLog: - (GIT_DIR=.git git-log > .changelog.tmp && mv .changelog.tmp ChangeLog; rm -f .changelog.tmp) || (touch ChangeLog; echo 'git directory not found: installing possibly empty changelog.' >&2) + (GIT_DIR=.git git log > .changelog.tmp && mv .changelog.tmp ChangeLog; rm -f .changelog.tmp) || (touch ChangeLog; echo 'git directory not found: installing possibly empty changelog.' >&2) AUTHORS: - (GIT_DIR=.git git-log | grep ^Author | sort |uniq > .authors.tmp && mv .authors.tmp AUTHORS; rm -f .authors.tmp) || (touch AUTHORS; echo 'git directory not found: installing possibly empty AUTHORS.' >&2) + (GIT_DIR=.git git log | grep ^Author | sort |uniq > .authors.tmp && mv .authors.tmp AUTHORS; rm -f .authors.tmp) || (touch AUTHORS; echo 'git directory not found: installing possibly empty AUTHORS.' >&2) REPLACE_VARS_ON_INSTALL=$(sbindir)/mock.py install-exec-hook: -- 1.6.1 -- Todd OpenPGP -> KeyID: 0xBEAF0CE3 | URL: www.pobox.com/~tmz/pgp ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Intaxication (n.) Euphoria at getting a tax refund, which lasts until you realize it was your money to start with. From oliver at linux-kernel.at Thu Jan 8 07:42:17 2009 From: oliver at linux-kernel.at (Oliver Falk) Date: Thu, 08 Jan 2009 08:42:17 +0100 Subject: Koji feature proposals In-Reply-To: <1231345882.4036.43.camel@localhost.localdomain> References: <1231273316.15116.14.camel@burren.bos.redhat.com> <4963C44C.6090805@linux-kernel.at> <1231275869.15116.19.camel@burren.bos.redhat.com> <496462E1.4090500@linux-kernel.at> <1231338896.4036.37.camel@localhost.localdomain> <4964D6F7.9060800@linux-kernel.at> <1231345882.4036.43.camel@localhost.localdomain> Message-ID: <4965AE59.6080104@linux-kernel.at> Jesse Keating wrote: > On Wed, 2009-01-07 at 17:23 +0100, Oliver Falk wrote: >> OKOK. I thought rpmbuild will automagically produce arch-specific and >> noarch packages in 'one step'... > > It will, for one arch. You do a rpmbuild --rebuild foo (on x86_64) and > it'll spit out x86_64 binary packages and potentially a noarch package. > What we care about is doing the same command on i386 and generating i386 > packages and a noarch package, having the i386 produced noarch package > match the x86_64 produced noarch package. To the best of my knowledge > there is no way to convince rpmbuild to produce both i386 /and/ x86_64 > packages in the same run. Alright. I got it now - I think :-) So a build in koji will produce a noarch package for every arch. And then you need to decide which noarch package to take and how to find out if there are (arch specific) differences in those "noarch's", right? -of From bryce at zeniv.linux.org.uk Thu Jan 8 12:13:40 2009 From: bryce at zeniv.linux.org.uk (Bryce) Date: Thu, 08 Jan 2009 12:13:40 +0000 Subject: Koji feature proposals In-Reply-To: <4965AE59.6080104@linux-kernel.at> References: <1231273316.15116.14.camel@burren.bos.redhat.com> <4963C44C.6090805@linux-kernel.at> <1231275869.15116.19.camel@burren.bos.redhat.com> <496462E1.4090500@linux-kernel.at> <1231338896.4036.37.camel@localhost.localdomain> <4964D6F7.9060800@linux-kernel.at> <1231345882.4036.43.camel@localhost.localdomain> <4965AE59.6080104@linux-kernel.at> Message-ID: <4965EDF4.7010009@zeniv.linux.org.uk> Oliver Falk wrote: > Jesse Keating wrote: >> On Wed, 2009-01-07 at 17:23 +0100, Oliver Falk wrote: >>> OKOK. I thought rpmbuild will automagically produce arch-specific >>> and noarch packages in 'one step'... >> >> It will, for one arch. You do a rpmbuild --rebuild foo (on x86_64) and >> it'll spit out x86_64 binary packages and potentially a noarch package. >> What we care about is doing the same command on i386 and generating i386 >> packages and a noarch package, having the i386 produced noarch package >> match the x86_64 produced noarch package. To the best of my knowledge >> there is no way to convince rpmbuild to produce both i386 /and/ x86_64 >> packages in the same run. > > Alright. I got it now - I think :-) > > So a build in koji will produce a noarch package for every arch. And > then you need to decide which noarch package to take and how to find > out if there are (arch specific) differences in those "noarch's", right? > > -of > huh? the entire reason a package is noarch is that it doesn't have any difference on what architecture it was built or installed. Usually shell scripts or data sets make up the bulk of noarch rpms. If a noarch package differs when built on different architectures then you can't call it a noarch rpm! Now that said, I have generated noarch packages that are actually x86_64 binaries so that they will install in an x86 environment without a need to pass --ignorearch to rpm, but, they forced me do it *sob* I didn't want to do it that way *cry* it was awful, my soul is dead *breakdown bawl* Phil =--= From jkeating at redhat.com Thu Jan 8 12:22:59 2009 From: jkeating at redhat.com (Jesse Keating) Date: Thu, 08 Jan 2009 07:22:59 -0500 Subject: Koji feature proposals In-Reply-To: <4965AE59.6080104@linux-kernel.at> References: <1231273316.15116.14.camel@burren.bos.redhat.com> <4963C44C.6090805@linux-kernel.at> <1231275869.15116.19.camel@burren.bos.redhat.com> <496462E1.4090500@linux-kernel.at> <1231338896.4036.37.camel@localhost.localdomain> <4964D6F7.9060800@linux-kernel.at> <1231345882.4036.43.camel@localhost.localdomain> <4965AE59.6080104@linux-kernel.at> Message-ID: <1231417379.4755.0.camel@localhost.localdomain> On Thu, 2009-01-08 at 08:42 +0100, Oliver Falk wrote: > Alright. I got it now - I think :-) > > So a build in koji will produce a noarch package for every arch. And > then you need to decide which noarch package to take and how to find out > if there are (arch specific) differences in those "noarch's", right? Yep! You got it now. -- Jesse Keating Fedora -- Freedom? is a feature! identi.ca: http://identi.ca/jkeating -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 197 bytes Desc: This is a digitally signed message part URL: From paul.schroeder at bluecoat.com Fri Jan 9 19:52:20 2009 From: paul.schroeder at bluecoat.com (Paul B Schroeder) Date: Fri, 09 Jan 2009 13:52:20 -0600 Subject: [PATCH] kojira: never delete repos Message-ID: <4967AAF4.1040309@bluecoat.com> Setting "deleted_repo_lifetime" to 0 will tell kojira to keep repos indefinitely... diff --git a/util/kojira b/util/kojira index 19d08e7..93852d3 100755 --- a/util/kojira +++ b/util/kojira @@ -134,7 +134,7 @@ class ManagedRepo(object): self.logger.error("Can't stat repo directory: %s" % path) return True age = time.time() - max(self.event_ts, mtime) - if age < options.deleted_repo_lifetime: + if options.deleted_repo_lifetime == 0 or age < options.deleted_repo_lifetime: #XXX should really be called expired_repo_lifetime return False self.logger.debug("Attempting to delete repo %s.." % self.repo_id) @@ -252,7 +252,7 @@ class RepoManager(object): continue if rinfo['state'] in (koji.REPO_DELETED, koji.REPO_PROBLEM): age = time.time() - max(rinfo['create_ts'], dir_ts) - if age > options.deleted_repo_lifetime: + if options.deleted_repo_lifetime != 0 and age > options.deleted_repo_lifetime: #XXX should really be called expired_repo_lifetime count += 1 logger.info("Removing stray repo (state=%s): %s" % (koji.REPO_STATES[rinfo['state']], repodir)) -- --- Paul B Schroeder Blue Coat Systems, Inc. From paul.schroeder at bluecoat.com Fri Jan 9 19:54:25 2009 From: paul.schroeder at bluecoat.com (Paul B Schroeder) Date: Fri, 09 Jan 2009 13:54:25 -0600 Subject: [PATCH] koji Makfile test-rpm target broken Message-ID: <4967AB71.4040708@bluecoat.com> Shouldn't the 'test-rpm' target depend upon 'test-tarball', not 'tarball'? diff --git a/Makefile b/Makefile index 7432f31..d9b1af9 100644 --- a/Makefile +++ b/Makefile @@ -84,7 +84,7 @@ srpm: tarball rpm: tarball $(RPM_WITH_DIRS) $(DIST_DEFINES) -bb $(SPECFILE) -test-rpm: tarball +test-rpm: test-tarball $(RPM_WITH_DIRS) $(DIST_DEFINES) --define "testbuild 1" -bb $(SPECFILE) tag:: -- --- Paul B Schroeder Blue Coat Systems, Inc. From mikem at fedoraproject.org Fri Jan 9 20:14:51 2009 From: mikem at fedoraproject.org (Mike McLean) Date: Fri, 9 Jan 2009 15:14:51 -0500 Subject: [PATCH] kojira: never delete repos In-Reply-To: <4967AAF4.1040309@bluecoat.com> References: <4967AAF4.1040309@bluecoat.com> Message-ID: <4f50e0680901091214y166915ap8deb0ce76e720e3c@mail.gmail.com> On Fri, Jan 9, 2009 at 2:52 PM, Paul B Schroeder wrote: > Setting "deleted_repo_lifetime" to 0 will tell kojira to keep repos > indefinitely... Easy enough, but why would you want them indefinitely? From paul.schroeder at bluecoat.com Fri Jan 9 20:23:03 2009 From: paul.schroeder at bluecoat.com (Paul B Schroeder) Date: Fri, 09 Jan 2009 14:23:03 -0600 Subject: [PATCH] kojira: never delete repos In-Reply-To: <4f50e0680901091214y166915ap8deb0ce76e720e3c@mail.gmail.com> References: <4967AAF4.1040309@bluecoat.com> <4f50e0680901091214y166915ap8deb0ce76e720e3c@mail.gmail.com> Message-ID: <4967B227.9080002@bluecoat.com> We're building intermittent dev ISOs from our koji repos. We'd like to be able to rebuild them going back to any old repo at any point in time basically. The repo data is small enough to not be of real concern to us either. Mike McLean wrote: > On Fri, Jan 9, 2009 at 2:52 PM, Paul B Schroeder > wrote: >> Setting "deleted_repo_lifetime" to 0 will tell kojira to keep repos >> indefinitely... > > Easy enough, but why would you want them indefinitely? -- --- Paul B Schroeder Blue Coat Systems, Inc. From jkeating at redhat.com Tue Jan 20 00:18:49 2009 From: jkeating at redhat.com (Jesse Keating) Date: Mon, 19 Jan 2009 16:18:49 -0800 Subject: Koji support for larger gpg rpm signatures Message-ID: <1232410729.3539.93.camel@localhost.localdomain> As part of our sha-256 efforts, we're trying to sign rpms with a sha-2 digest. I'm attempting to sign packages with a RSA key that is size 4096, the biggest possible. However I'm running into problems importing this into koji, due to my local signing software haven stolen come code from koji to determine what the sigkey is. The koji code makes an assumption about where the key ID exists in the signature header, and it seems this assumption is wrong when larger keys are used. Mitr who has been helping me says that for a quick hack, when getting the key chunk out of the hex, we can assume that sigkey[13:17] works if sigkey[0] is 0x88, but if 0 is 0x89, we have to go to 14:18. This comes up a few times in koji code, so I thought some discussion was in order before setting off to make a patch. Is there anything better we can do instead of snaking raw data out of headers? -- Jesse Keating Fedora -- Freedom? is a feature! identi.ca: http://identi.ca/jkeating -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 197 bytes Desc: This is a digitally signed message part URL: From jeff at ocjtech.us Tue Jan 20 00:38:17 2009 From: jeff at ocjtech.us (Jeffrey Ollie) Date: Mon, 19 Jan 2009 18:38:17 -0600 Subject: Koji support for larger gpg rpm signatures In-Reply-To: <1232410729.3539.93.camel@localhost.localdomain> References: <1232410729.3539.93.camel@localhost.localdomain> Message-ID: <935ead450901191638y393cb8b5l45a1e474eacd7c28@mail.gmail.com> On Mon, Jan 19, 2009 at 6:18 PM, Jesse Keating wrote: > > Is there anything better we can do instead of snaking raw data out of > headers? Use pygpgme? import gpgme ctx = gpgme.Context() key = ctx.get_key('6D97044E') print [subkey.length for subkey in key.subkeys] The docs for gpgme/pygpgme aren't great but I have a little experience with it so we can probably figure out what needs to be done. Can you point out where exactly in the code the problem is? -- Jeff Ollie "You know, I used to think it was awful that life was so unfair. Then I thought, wouldn't it be much worse if life were fair, and all the terrible things that happen to us come because we actually deserve them? So, now I take great comfort in the general hostility and unfairness of the universe." -- Marcus to Franklin in Babylon 5: "A Late Delivery from Avalon" From skvidal at fedoraproject.org Tue Jan 20 00:42:49 2009 From: skvidal at fedoraproject.org (seth vidal) Date: Mon, 19 Jan 2009 19:42:49 -0500 Subject: Koji support for larger gpg rpm signatures In-Reply-To: <935ead450901191638y393cb8b5l45a1e474eacd7c28@mail.gmail.com> References: <1232410729.3539.93.camel@localhost.localdomain> <935ead450901191638y393cb8b5l45a1e474eacd7c28@mail.gmail.com> Message-ID: <1232412169.4044.12.camel@rosebud> On Mon, 2009-01-19 at 18:38 -0600, Jeffrey Ollie wrote: > On Mon, Jan 19, 2009 at 6:18 PM, Jesse Keating wrote: > > > > Is there anything better we can do instead of snaking raw data out of > > headers? > > Use pygpgme? > > import gpgme > > ctx = gpgme.Context() > > key = ctx.get_key('6D97044E') > > print [subkey.length for subkey in key.subkeys] > > The docs for gpgme/pygpgme aren't great but I have a little experience > with it so we can probably figure out what needs to be done. Can you > point out where exactly in the code the problem is? That won't let you pull the signature out of an rpm header. though. -sv From jeff at ocjtech.us Tue Jan 20 00:58:20 2009 From: jeff at ocjtech.us (Jeffrey Ollie) Date: Mon, 19 Jan 2009 18:58:20 -0600 Subject: Koji support for larger gpg rpm signatures In-Reply-To: <1232412169.4044.12.camel@rosebud> References: <1232410729.3539.93.camel@localhost.localdomain> <935ead450901191638y393cb8b5l45a1e474eacd7c28@mail.gmail.com> <1232412169.4044.12.camel@rosebud> Message-ID: <935ead450901191658n5b2cc576mca986e4a378c348d@mail.gmail.com> On Mon, Jan 19, 2009 at 6:42 PM, seth vidal wrote: > > That won't let you pull the signature out of an rpm header. though. Ah, I wasn't sure where we were looking. -- Jeff Ollie "You know, I used to think it was awful that life was so unfair. Then I thought, wouldn't it be much worse if life were fair, and all the terrible things that happen to us come because we actually deserve them? So, now I take great comfort in the general hostility and unfairness of the universe." -- Marcus to Franklin in Babylon 5: "A Late Delivery from Avalon" From mikem at redhat.com Tue Jan 20 15:07:42 2009 From: mikem at redhat.com (Mike McLean) Date: Tue, 20 Jan 2009 10:07:42 -0500 Subject: Koji support for larger gpg rpm signatures In-Reply-To: <1232412169.4044.12.camel@rosebud> References: <1232410729.3539.93.camel@localhost.localdomain> <935ead450901191638y393cb8b5l45a1e474eacd7c28@mail.gmail.com> <1232412169.4044.12.camel@rosebud> Message-ID: <4975E8BE.3010501@redhat.com> seth vidal wrote: > On Mon, 2009-01-19 at 18:38 -0600, Jeffrey Ollie wrote: >> On Mon, Jan 19, 2009 at 6:18 PM, Jesse Keating wrote: >>> Is there anything better we can do instead of snaking raw data out of >>> headers? >> Use pygpgme? >> >> import gpgme >> >> ctx = gpgme.Context() >> >> key = ctx.get_key('6D97044E') >> >> print [subkey.length for subkey in key.subkeys] >> >> The docs for gpgme/pygpgme aren't great but I have a little experience >> with it so we can probably figure out what needs to be done. Can you >> point out where exactly in the code the problem is? > > That won't let you pull the signature out of an rpm header. though. Well, the code is probably pulling the signature correctly, it's just not reading the key from said signature correctly. Perhaps pygpgme can do that for us? From jiteshs at marvell.com Thu Jan 22 05:31:51 2009 From: jiteshs at marvell.com (Jitesh Shah) Date: Wed, 21 Jan 2009 21:31:51 -0800 Subject: Koji: ServerOffline Message-ID: <1232602311.3077.20.camel@localhost.localdomain> Hi, I've been running a local koji instance at my workplace. It worked absolutely fine for a fortnight. But, since Monday, I am getting this error : "ServerOffline: database outage" for any CLI command that I try to execute. I initially suspected that it might be because koji cannot connect to the database. But, I checked that I can login to the database through command-line. >> psql Also, I wrote a small script in python which mimics koji's method of connecting to the database (with reference to /usr/lib/python2.5/site-packages/koji/db.py) and it works fine! Just to make sure, I restarted postgresql service in vain. Re-initialised the database (Remove /var/lib/pgsql and run "service postgresql initdb" and all the other steps) to no avail. I started seeing this problem after a run of the koji-shadow script. It exited abnormally giving this very error. Since then, I cannot even access koji-cli.. neither koji-web. I have been tracing through the koji scripts and I see that the SSL login succeeds and then the very first rpc call: "getAPIVersion" fails. Wireshark shows that the koji-hub replies with "Connection: close" to the getAPIVersion call. Any idea as to what is going wrong? Any pointers? Jitesh -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliver at linux-kernel.at Thu Jan 22 07:16:13 2009 From: oliver at linux-kernel.at (Oliver Falk) Date: Thu, 22 Jan 2009 08:16:13 +0100 Subject: Koji: ServerOffline In-Reply-To: <1232602311.3077.20.camel@localhost.localdomain> References: <1232602311.3077.20.camel@localhost.localdomain> Message-ID: <49781D3D.1090206@linux-kernel.at> Hi Jitesh! Jitesh Shah wrote: > Hi, > I've been running a local koji instance at my workplace. It worked > absolutely fine for a fortnight. But, since Monday, I am getting this > error : "ServerOffline: database outage" for any CLI command that I try > to execute. > > I initially suspected that it might be because koji cannot connect to > the database. But, I checked that I can login to the database through > command-line. > >> psql > > Also, I wrote a small script in python which mimics koji's method of > connecting to the database (with reference > to /usr/lib/python2.5/site-packages/koji/db.py) and it works fine! > > Just to make sure, I restarted postgresql service in vain. > Re-initialised the database (Remove /var/lib/pgsql and run "service > postgresql initdb" and all the other steps) to no avail. > > I started seeing this problem after a run of the koji-shadow script. It > exited abnormally giving this very error. Since then, I cannot even > access koji-cli.. neither koji-web. > > I have been tracing through the koji scripts and I see that the SSL > login succeeds and then the very first rpc call: "getAPIVersion" fails. > Wireshark shows that the koji-hub replies with "Connection: close" to > the getAPIVersion call. > > Any idea as to what is going wrong? > Any pointers? You have not written if you have tried to restart Apache. koji(hub) has no auto-reconnect if it looses connection to the database. So if for whatever reason it lost the connection to PostgreSQL, it will never reconnect. You have to manually restart Apache! -of From jiteshs at marvell.com Thu Jan 22 07:26:57 2009 From: jiteshs at marvell.com (Jitesh Shah) Date: Wed, 21 Jan 2009 23:26:57 -0800 Subject: Koji: ServerOffline In-Reply-To: <49781D3D.1090206@linux-kernel.at> References: <1232602311.3077.20.camel@localhost.localdomain> <49781D3D.1090206@linux-kernel.at> Message-ID: <1232609217.3077.31.camel@localhost.localdomain> Hi, Thank you for the quick reply Oliver. So, this morning I moved to a newer version of koji. Reconfigured everything (restarted apache after re-configuring koji-hub) and quite surprisingly, I still get the same error! So, if reinitialising both, postgresql and koji, doesn't do the trick, I wonder what might have gone wrong. (Just as a matter of curiosity, I even checked with ipcs that there are no dangling shared memory segments. There aren't any.) Jitesh On Thu, 2009-01-22 at 08:16 +0100, Oliver Falk wrote: > Hi Jitesh! > > Jitesh Shah wrote: > > Hi, > > I've been running a local koji instance at my workplace. It worked > > absolutely fine for a fortnight. But, since Monday, I am getting this > > error : "ServerOffline: database outage" for any CLI command that I try > > to execute. > > > > I initially suspected that it might be because koji cannot connect to > > the database. But, I checked that I can login to the database through > > command-line. > > >> psql > > > > Also, I wrote a small script in python which mimics koji's method of > > connecting to the database (with reference > > to /usr/lib/python2.5/site-packages/koji/db.py) and it works fine! > > > > Just to make sure, I restarted postgresql service in vain. > > Re-initialised the database (Remove /var/lib/pgsql and run "service > > postgresql initdb" and all the other steps) to no avail. > > > > I started seeing this problem after a run of the koji-shadow script. It > > exited abnormally giving this very error. Since then, I cannot even > > access koji-cli.. neither koji-web. > > > > I have been tracing through the koji scripts and I see that the SSL > > login succeeds and then the very first rpc call: "getAPIVersion" fails. > > Wireshark shows that the koji-hub replies with "Connection: close" to > > the getAPIVersion call. > > > > Any idea as to what is going wrong? > > Any pointers? > > You have not written if you have tried to restart Apache. koji(hub) has > no auto-reconnect if it looses connection to the database. So if for > whatever reason it lost the connection to PostgreSQL, it will never > reconnect. You have to manually restart Apache! > > -of > > -- > Fedora-buildsys-list mailing list > Fedora-buildsys-list at redhat.com > https://www.redhat.com/mailman/listinfo/fedora-buildsys-list -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliver at linux-kernel.at Thu Jan 22 07:45:47 2009 From: oliver at linux-kernel.at (Oliver Falk) Date: Thu, 22 Jan 2009 08:45:47 +0100 Subject: Koji: ServerOffline In-Reply-To: <1232609217.3077.31.camel@localhost.localdomain> References: <1232602311.3077.20.camel@localhost.localdomain> <49781D3D.1090206@linux-kernel.at> <1232609217.3077.31.camel@localhost.localdomain> Message-ID: <4978242B.5050906@linux-kernel.at> Jitesh Shah wrote: > So, this morning I moved to a newer version of koji. Reconfigured > everything (restarted apache after re-configuring koji-hub) and quite > surprisingly, I still get the same error! > > So, if reinitialising both, postgresql and koji, doesn't do the trick, I > wonder what might have gone wrong. > (Just as a matter of curiosity, I even checked with ipcs that there are > no dangling shared memory segments. There aren't any.) The settings in /etc/httpd/conf.d/kojihub.conf are correct I guess? Esp. DBName, DBUser, DBHost? And your /var/lib/pgsql/data/pg_hba.conf is also OK? And you can connect via command line? psql -U -h ? The error is quite clear. kojihub is not able to connect to the database, so something is wrong with the config... :-( -of From a.badger at gmail.com Thu Jan 22 08:23:42 2009 From: a.badger at gmail.com (Toshio Kuratomi) Date: Thu, 22 Jan 2009 00:23:42 -0800 Subject: Koji: ServerOffline In-Reply-To: <4978242B.5050906@linux-kernel.at> References: <1232602311.3077.20.camel@localhost.localdomain> <49781D3D.1090206@linux-kernel.at> <1232609217.3077.31.camel@localhost.localdomain> <4978242B.5050906@linux-kernel.at> Message-ID: <49782D0E.1040504@gmail.com> Oliver Falk wrote: > The settings in /etc/httpd/conf.d/kojihub.conf are correct I guess? > Esp. DBName, DBUser, DBHost? And your /var/lib/pgsql/data/pg_hba.conf is > also OK? > > And you can connect via command line? psql -U -h > ? > ... and the psql command is run on the same machine as koji? -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 197 bytes Desc: OpenPGP digital signature URL: From oliver at linux-kernel.at Thu Jan 22 08:30:54 2009 From: oliver at linux-kernel.at (Oliver Falk) Date: Thu, 22 Jan 2009 09:30:54 +0100 Subject: Koji: ServerOffline In-Reply-To: <49782D0E.1040504@gmail.com> References: <1232602311.3077.20.camel@localhost.localdomain> <49781D3D.1090206@linux-kernel.at> <1232609217.3077.31.camel@localhost.localdomain> <4978242B.5050906@linux-kernel.at> <49782D0E.1040504@gmail.com> Message-ID: <49782EBE.1020604@linux-kernel.at> Toshio Kuratomi wrote: > Oliver Falk wrote: >> The settings in /etc/httpd/conf.d/kojihub.conf are correct I guess? >> Esp. DBName, DBUser, DBHost? And your /var/lib/pgsql/data/pg_hba.conf is >> also OK? >> >> And you can connect via command line? psql -U -h >> ? >> > ... and the psql command is run on the same machine as koji? Well I should have mentioned this, thx Toshio for pointing this out! -of From jiteshs at marvell.com Thu Jan 22 09:28:08 2009 From: jiteshs at marvell.com (Jitesh Shah) Date: Thu, 22 Jan 2009 01:28:08 -0800 Subject: Koji: ServerOffline In-Reply-To: <4978242B.5050906@linux-kernel.at> References: <1232602311.3077.20.camel@localhost.localdomain> <49781D3D.1090206@linux-kernel.at> <1232609217.3077.31.camel@localhost.localdomain> <4978242B.5050906@linux-kernel.at> Message-ID: <1232616488.3077.51.camel@localhost.localdomain> Hi, So, I had koji running and building packages for me for nearly a fortnight. And it worked like a charm! I double-checked all the configuration and it is intact. There is the and entry in pg_hba.conf that trusts koji. I run psql command from the "koji" user on the same machine where koji-hub is running. So, psql uses the default values for DBName and DBUser. (which are "koji" and "koji" respectively) (And I don't use the host option). Also, I've checked that DBName and DBUser configuration are OK in the kojihub.conf On a different note, how does one debug a script which uses rpc calls and the error is on the server side? i.e. the debugger will run on the caller side, so, how do we debug the script that is executed on the server? (Except inserting write-to-file commands in the server script) Also, does koji maintain a logfile?... I wasn't able find it anywhere. Jitesh On Thu, 2009-01-22 at 08:45 +0100, Oliver Falk wrote: > Jitesh Shah wrote: > > So, this morning I moved to a newer version of koji. Reconfigured > > everything (restarted apache after re-configuring koji-hub) and quite > > surprisingly, I still get the same error! > > > > So, if reinitialising both, postgresql and koji, doesn't do the trick, I > > wonder what might have gone wrong. > > (Just as a matter of curiosity, I even checked with ipcs that there are > > no dangling shared memory segments. There aren't any.) > > The settings in /etc/httpd/conf.d/kojihub.conf are correct I guess? > Esp. DBName, DBUser, DBHost? And your /var/lib/pgsql/data/pg_hba.conf is > also OK? > > And you can connect via command line? psql -U -h ? > > The error is quite clear. kojihub is not able to connect to the > database, so something is wrong with the config... :-( > > -of > > -- > Fedora-buildsys-list mailing list > Fedora-buildsys-list at redhat.com > https://www.redhat.com/mailman/listinfo/fedora-buildsys-list -------------- next part -------------- An HTML attachment was scrubbed... URL: From a.badger at gmail.com Thu Jan 22 10:18:32 2009 From: a.badger at gmail.com (Toshio Kuratomi) Date: Thu, 22 Jan 2009 02:18:32 -0800 Subject: Koji: ServerOffline In-Reply-To: <1232616488.3077.51.camel@localhost.localdomain> References: <1232602311.3077.20.camel@localhost.localdomain> <49781D3D.1090206@linux-kernel.at> <1232609217.3077.31.camel@localhost.localdomain> <4978242B.5050906@linux-kernel.at> <1232616488.3077.51.camel@localhost.localdomain> Message-ID: <497847F8.8050905@gmail.com> Jitesh Shah wrote: > Hi, > So, I had koji running and building packages for me for nearly a > fortnight. And it worked like a charm! > > I double-checked all the configuration and it is intact. There is the > and entry in pg_hba.conf that trusts koji. > > I run psql command from the "koji" user on the same machine where > koji-hub is running. So, psql uses the default values for DBName and > DBUser. (which are "koji" and "koji" respectively) (And I don't use the > host option). Also, I've checked that DBName and DBUser configuration > are OK in the kojihub.conf > It sounds like you're running the koji db on the same machine as the kojihub? Just to be sure you're testing the same thing with psql as koji is going to run, try psql with the -h HOST option. The reason is that psql without the -h will try to use unix sockets to connect on the local machine. koji, though, is going to use a tcp connection, not a socket. -Toshio -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 197 bytes Desc: OpenPGP digital signature URL: From jiteshs at marvell.com Thu Jan 22 11:47:55 2009 From: jiteshs at marvell.com (Jitesh Shah) Date: Thu, 22 Jan 2009 03:47:55 -0800 Subject: Koji: ServerOffline In-Reply-To: <497847F8.8050905@gmail.com> References: <1232602311.3077.20.camel@localhost.localdomain> <49781D3D.1090206@linux-kernel.at> <1232609217.3077.31.camel@localhost.localdomain> <4978242B.5050906@linux-kernel.at> <1232616488.3077.51.camel@localhost.localdomain> <497847F8.8050905@gmail.com> Message-ID: <1232624875.3077.59.camel@localhost.localdomain> Hi, Yes, I can successfully connect to the psql database using tcp sockets too. Here is the snippet of the interaction in the terminal. [koji at linux-dev ~]$ psql -U koji -h localhost -d koji Welcome to psql 8.2.11, the PostgreSQL interactive terminal. Type: \copyright for distribution terms \h for help with SQL commands \? for help with psql commands \g or terminate with semicolon to execute query \q to quit koji=> I can view all the tables with "\d". So, it is connected all-right. Btw, I also checked in the "users" "permissions" and "user_perms" tables to verify that "koji" user is indeed the "admin" and has all the required privileges. Jitesh On Thu, 2009-01-22 at 02:18 -0800, Toshio Kuratomi wrote: > Jitesh Shah wrote: > > Hi, > > So, I had koji running and building packages for me for nearly a > > fortnight. And it worked like a charm! > > > > I double-checked all the configuration and it is intact. There is the > > and entry in pg_hba.conf that trusts koji. > > > > I run psql command from the "koji" user on the same machine where > > koji-hub is running. So, psql uses the default values for DBName and > > DBUser. (which are "koji" and "koji" respectively) (And I don't use the > > host option). Also, I've checked that DBName and DBUser configuration > > are OK in the kojihub.conf > > > It sounds like you're running the koji db on the same machine as the > kojihub? Just to be sure you're testing the same thing with psql as > koji is going to run, try psql with the -h HOST option. The reason is > that psql without the -h will try to use unix sockets to connect on the > local machine. koji, though, is going to use a tcp connection, not a > socket. > > -Toshio > > -- > Fedora-buildsys-list mailing list > Fedora-buildsys-list at redhat.com > https://www.redhat.com/mailman/listinfo/fedora-buildsys-list -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliver at linux-kernel.at Thu Jan 22 12:28:43 2009 From: oliver at linux-kernel.at (Oliver Falk) Date: Thu, 22 Jan 2009 13:28:43 +0100 Subject: Koji: ServerOffline In-Reply-To: <1232624875.3077.59.camel@localhost.localdomain> References: <1232602311.3077.20.camel@localhost.localdomain> <49781D3D.1090206@linux-kernel.at> <1232609217.3077.31.camel@localhost.localdomain> <4978242B.5050906@linux-kernel.at> <1232616488.3077.51.camel@localhost.localdomain> <497847F8.8050905@gmail.com> <1232624875.3077.59.camel@localhost.localdomain> Message-ID: <4978667B.4060904@linux-kernel.at> You could do the following on your kojihub - I guess it's only running koji!? If not that's bad :-) # service httpd stop # strace -f httpd -X You may want to write it to some strace.log (add 2>&1 > strace.log). Try to analyse this. You should see where it tries to connect to. -of Jitesh Shah wrote: > Hi, > Yes, I can successfully connect to the psql database using tcp sockets > too. Here is the snippet of the interaction in the terminal. > > [koji at linux-dev ~]$ psql -U koji -h localhost -d koji > Welcome to psql 8.2.11, the PostgreSQL interactive terminal. > > Type: \copyright for distribution terms > \h for help with SQL commands > \? for help with psql commands > \g or terminate with semicolon to execute query > \q to quit > > koji=> > > > I can view all the tables with "\d". So, it is connected all-right. Btw, > I also checked in the "users" "permissions" and "user_perms" tables to > verify that "koji" user is indeed the "admin" and has all the required > privileges. > > Jitesh > > On Thu, 2009-01-22 at 02:18 -0800, Toshio Kuratomi wrote: > > Jitesh Shah wrote: > > > Hi, > > > So, I had koji running and building packages for me for nearly a > > > fortnight. And it worked like a charm! > > > > > > I double-checked all the configuration and it is intact. There is the > > > and entry in pg_hba.conf that trusts koji. > > > > > > I run psql command from the "koji" user on the same machine where > > > koji-hub is running. So, psql uses the default values for DBName and > > > DBUser. (which are "koji" and "koji" respectively) (And I don't use the > > > host option). Also, I've checked that DBName and DBUser configuration > > > are OK in the kojihub.conf > > > > > It sounds like you're running the koji db on the same machine as the > > kojihub? Just to be sure you're testing the same thing with psql as > > koji is going to run, try psql with the -h HOST option. The reason is > > that psql without the -h will try to use unix sockets to connect on the > > local machine. koji, though, is going to use a tcp connection, not a > > socket. > > > > -Toshio > > > > -- > > Fedora-buildsys-list mailing list > > Fedora-buildsys-list at redhat.com > > https://www.redhat.com/mailman/listinfo/fedora-buildsys-list > > > ------------------------------------------------------------------------ > > -- > Fedora-buildsys-list mailing list > Fedora-buildsys-list at redhat.com > https://www.redhat.com/mailman/listinfo/fedora-buildsys-list From jiteshs at marvell.com Thu Jan 22 14:30:57 2009 From: jiteshs at marvell.com (Jitesh Shah) Date: Thu, 22 Jan 2009 06:30:57 -0800 Subject: Koji: ServerOffline In-Reply-To: <4978667B.4060904@linux-kernel.at> References: <1232602311.3077.20.camel@localhost.localdomain> <49781D3D.1090206@linux-kernel.at> <1232609217.3077.31.camel@localhost.localdomain> <4978242B.5050906@linux-kernel.at> <1232616488.3077.51.camel@localhost.localdomain> <497847F8.8050905@gmail.com> <1232624875.3077.59.camel@localhost.localdomain> <4978667B.4060904@linux-kernel.at> Message-ID: <1232634657.3077.65.camel@localhost.localdomain> Hi! It finally worked!!! Rejoice! So, while stracing httpd, as suggested by Oliver, I realised that koji was _NOT_ using localhost to connect to psql because the DBHost parameter in the koji configuration had been tampered with and changed to something else. And obviously the request was then denied. Anyway, Thanks to all of you and specially Oliver for pointing me in the right direction! Jitesh On Thu, 2009-01-22 at 13:28 +0100, Oliver Falk wrote: > You could do the following on your kojihub - I guess it's only running > koji!? If not that's bad :-) > > # service httpd stop > # strace -f httpd -X > > You may want to write it to some strace.log (add 2>&1 > strace.log). > > Try to analyse this. You should see where it tries to connect to. > > -of > > Jitesh Shah wrote: > > Hi, > > Yes, I can successfully connect to the psql database using tcp sockets > > too. Here is the snippet of the interaction in the terminal. > > > > [koji at linux-dev ~]$ psql -U koji -h localhost -d koji > > Welcome to psql 8.2.11, the PostgreSQL interactive terminal. > > > > Type: \copyright for distribution terms > > \h for help with SQL commands > > \? for help with psql commands > > \g or terminate with semicolon to execute query > > \q to quit > > > > koji=> > > > > > > I can view all the tables with "\d". So, it is connected all-right. Btw, > > I also checked in the "users" "permissions" and "user_perms" tables to > > verify that "koji" user is indeed the "admin" and has all the required > > privileges. > > > > Jitesh > > > > On Thu, 2009-01-22 at 02:18 -0800, Toshio Kuratomi wrote: > > > Jitesh Shah wrote: > > > > Hi, > > > > So, I had koji running and building packages for me for nearly a > > > > fortnight. And it worked like a charm! > > > > > > > > I double-checked all the configuration and it is intact. There is the > > > > and entry in pg_hba.conf that trusts koji. > > > > > > > > I run psql command from the "koji" user on the same machine where > > > > koji-hub is running. So, psql uses the default values for DBName and > > > > DBUser. (which are "koji" and "koji" respectively) (And I don't use the > > > > host option). Also, I've checked that DBName and DBUser configuration > > > > are OK in the kojihub.conf > > > > > > > It sounds like you're running the koji db on the same machine as the > > > kojihub? Just to be sure you're testing the same thing with psql as > > > koji is going to run, try psql with the -h HOST option. The reason is > > > that psql without the -h will try to use unix sockets to connect on the > > > local machine. koji, though, is going to use a tcp connection, not a > > > socket. > > > > > > -Toshio > > > > > > -- > > > Fedora-buildsys-list mailing list > > > Fedora-buildsys-list at redhat.com > > > https://www.redhat.com/mailman/listinfo/fedora-buildsys-list > > > > > > ------------------------------------------------------------------------ > > > > -- > > Fedora-buildsys-list mailing list > > Fedora-buildsys-list at redhat.com > > https://www.redhat.com/mailman/listinfo/fedora-buildsys-list > > -- > Fedora-buildsys-list mailing list > Fedora-buildsys-list at redhat.com > https://www.redhat.com/mailman/listinfo/fedora-buildsys-list -------------- next part -------------- An HTML attachment was scrubbed... URL: From mikem at redhat.com Thu Jan 22 15:44:31 2009 From: mikem at redhat.com (Mike McLean) Date: Thu, 22 Jan 2009 10:44:31 -0500 Subject: Koji: ServerOffline In-Reply-To: <49781D3D.1090206@linux-kernel.at> References: <1232602311.3077.20.camel@localhost.localdomain> <49781D3D.1090206@linux-kernel.at> Message-ID: <4978945F.3000405@redhat.com> Oliver Falk wrote: > You have not written if you have tried to restart Apache. koji(hub) has > no auto-reconnect if it looses connection to the database. So if for > whatever reason it lost the connection to PostgreSQL, it will never > reconnect. You have to manually restart Apache! Actually, it does reconnect now. From mikem at redhat.com Thu Jan 22 15:53:40 2009 From: mikem at redhat.com (Mike McLean) Date: Thu, 22 Jan 2009 10:53:40 -0500 Subject: Koji: ServerOffline In-Reply-To: <1232616488.3077.51.camel@localhost.localdomain> References: <1232602311.3077.20.camel@localhost.localdomain> <49781D3D.1090206@linux-kernel.at> <1232609217.3077.31.camel@localhost.localdomain> <4978242B.5050906@linux-kernel.at> <1232616488.3077.51.camel@localhost.localdomain> Message-ID: <49789684.8080103@redhat.com> Jitesh Shah wrote: > On a different note, how does one debug a script which uses rpc calls > and the error is on the server side? i.e. the debugger will run on the > caller side, so, how do we debug the script that is executed on the > server? (Except inserting write-to-file commands in the server script) > Also, does koji maintain a logfile?... I wasn't able find it anywhere. kojihub logs to stderr, which under apache goes to /var/log/httpd/error_log (or ssl_error_log). If you don't see anything helpful, try setting KojiDebug to "on" in the config and restarting httpd. Which version of koji are you running? From oliver at linux-kernel.at Thu Jan 22 16:14:22 2009 From: oliver at linux-kernel.at (Oliver Falk) Date: Thu, 22 Jan 2009 17:14:22 +0100 Subject: Koji: ServerOffline In-Reply-To: <4978945F.3000405@redhat.com> References: <1232602311.3077.20.camel@localhost.localdomain> <49781D3D.1090206@linux-kernel.at> <4978945F.3000405@redhat.com> Message-ID: <49789B5E.50608@linux-kernel.at> Mike McLean wrote: > Oliver Falk wrote: >> You have not written if you have tried to restart Apache. koji(hub) >> has no auto-reconnect if it looses connection to the database. So if >> for whatever reason it lost the connection to PostgreSQL, it will >> never reconnect. You have to manually restart Apache! > > Actually, it does reconnect now. Since which version? -of From oliver at linux-kernel.at Thu Jan 22 16:17:41 2009 From: oliver at linux-kernel.at (Oliver Falk) Date: Thu, 22 Jan 2009 17:17:41 +0100 Subject: Koji: ServerOffline In-Reply-To: <49789684.8080103@redhat.com> References: <1232602311.3077.20.camel@localhost.localdomain> <49781D3D.1090206@linux-kernel.at> <1232609217.3077.31.camel@localhost.localdomain> <4978242B.5050906@linux-kernel.at> <1232616488.3077.51.camel@localhost.localdomain> <49789684.8080103@redhat.com> Message-ID: <49789C25.1080401@linux-kernel.at> Mike McLean wrote: > Jitesh Shah wrote: >> On a different note, how does one debug a script which uses rpc calls >> and the error is on the server side? i.e. the debugger will run on the >> caller side, so, how do we debug the script that is executed on the >> server? (Except inserting write-to-file commands in the server script) >> Also, does koji maintain a logfile?... I wasn't able find it anywhere. > > kojihub logs to stderr, which under apache goes to > /var/log/httpd/error_log (or ssl_error_log). If you don't see anything > helpful, try setting KojiDebug to "on" in the config and restarting httpd. I wonder if it would log this error in a way you know what's going on :-) Jitesh, can you check the logs as suggested by Mike, although it's fixed now, I'd like to know if you see the error there... [ ... ] -of From oliver at linux-kernel.at Thu Jan 22 16:20:23 2009 From: oliver at linux-kernel.at (Oliver Falk) Date: Thu, 22 Jan 2009 17:20:23 +0100 Subject: Koji: ServerOffline In-Reply-To: <1232634657.3077.65.camel@localhost.localdomain> References: <1232602311.3077.20.camel@localhost.localdomain> <49781D3D.1090206@linux-kernel.at> <1232609217.3077.31.camel@localhost.localdomain> <4978242B.5050906@linux-kernel.at> <1232616488.3077.51.camel@localhost.localdomain> <497847F8.8050905@gmail.com> <1232624875.3077.59.camel@localhost.localdomain> <4978667B.4060904@linux-kernel.at> <1232634657.3077.65.camel@localhost.localdomain> Message-ID: <49789CC7.6090205@linux-kernel.at> Hi! Jitesh Shah wrote: > It finally worked!!! Rejoice! Great. > So, while stracing httpd, as suggested by Oliver, I realised that koji > was _NOT_ using localhost to connect to psql because the DBHost > parameter in the koji configuration had been tampered with and changed > to something else. And obviously the request was then denied. As I always say. strace is your friend. :-P > Anyway, Thanks to all of you and specially Oliver for pointing me in the > right direction! Crazy me. I'm not even one of the koji developers... :-) Yay, yay. It's all about community, isn't it? 8-] -of From mikem at redhat.com Thu Jan 22 19:39:37 2009 From: mikem at redhat.com (Mike McLean) Date: Thu, 22 Jan 2009 14:39:37 -0500 Subject: Koji: ServerOffline In-Reply-To: <49789B5E.50608@linux-kernel.at> References: <1232602311.3077.20.camel@localhost.localdomain> <49781D3D.1090206@linux-kernel.at> <4978945F.3000405@redhat.com> <49789B5E.50608@linux-kernel.at> Message-ID: <4978CB79.8070407@redhat.com> Oliver Falk wrote: > Mike McLean wrote: >> Actually, it does reconnect now. > > Since which version? The change is in commit 58b7313d. Should be in 1.2.6 https://fedorahosted.org/koji/changeset/58b7313d0af0e0ac1fe804cd9f247486b3130cab From oliver at linux-kernel.at Thu Jan 22 19:47:05 2009 From: oliver at linux-kernel.at (Oliver Falk) Date: Thu, 22 Jan 2009 20:47:05 +0100 Subject: Koji: ServerOffline In-Reply-To: <4978CB79.8070407@redhat.com> References: <1232602311.3077.20.camel@localhost.localdomain> <49781D3D.1090206@linux-kernel.at> <4978945F.3000405@redhat.com> <49789B5E.50608@linux-kernel.at> <4978CB79.8070407@redhat.com> Message-ID: <4978CD39.6040306@linux-kernel.at> Mike McLean schrieb: > Oliver Falk wrote: >> Mike McLean wrote: >>> Actually, it does reconnect now. >> >> Since which version? > > The change is in commit 58b7313d. Should be in 1.2.6 > https://fedorahosted.org/koji/changeset/58b7313d0af0e0ac1fe804cd9f247486b3130cab Well, thx. Good to know this is 'fixed'. -of From bkosick at mxlogic.com Thu Jan 22 21:05:30 2009 From: bkosick at mxlogic.com (Brian Kosick) Date: Thu, 22 Jan 2009 14:05:30 -0700 Subject: Patch for mercurial support... Message-ID: <1232658330.3366.224.camel@localhost.localdomain> Hi All, First off, I'd like to say that I'm still in the n00b period of learning python and have heard that bad coding with it takes a little bit of extra effort. I'm happy to say that I think that the koji maintainers are too lazy to put that extra bit of effort in. I found the code in kojid to be easily readable. The variables, classes and methods were well named and the flow of the code made sense. What are you guys thinking?!?! These are tough times and you should be thinking about naming schemes that only the original author could love. Cryptic variable names and convoluted work-flows are core synergies to job security! Anyways, in between trying to deepen the dent on my desk via the manual "forehead" method, constant interruptions, and managements infuriating refusal to allow me to throttle those same interrupters, I came up with this patch to koji 1.2.6 it allows Mercurial to be used as a SCM. Thanks for the un-convoluted code! Brian -------------- next part -------------- A non-text attachment was scrubbed... Name: hgkoji.patch Type: text/x-patch Size: 1897 bytes Desc: not available URL: From bkosick at mxlogic.com Thu Jan 22 21:17:50 2009 From: bkosick at mxlogic.com (Brian Kosick) Date: Thu, 22 Jan 2009 14:17:50 -0700 Subject: Patch for mercurial support... In-Reply-To: <1232658330.3366.224.camel@localhost.localdomain> References: <1232658330.3366.224.camel@localhost.localdomain> Message-ID: <1232659070.3366.228.camel@localhost.localdomain> Hello, Sorry about that I already noticed an error in it. Here's a fixed one. I've tested the http/https method, but have not used the HG+SSH one yet.... Brian On Thu, 2009-01-22 at 14:05 -0700, Brian Kosick wrote: > Hi All, > > First off, I'd like to say that I'm still in the n00b period of learning > python and have heard that bad coding with it takes a little bit of > extra effort. I'm happy to say that I think that the koji maintainers > are too lazy to put that extra bit of effort in. I found the code in > kojid to be easily readable. The variables, classes and methods were > well named and the flow of the code made sense. > > What are you guys thinking?!?! > > These are tough times and you should be thinking about naming schemes > that only the original author could love. Cryptic variable names and > convoluted work-flows are core synergies to job security! > > Anyways, in between trying to deepen the dent on my desk via the manual > "forehead" method, constant interruptions, and managements infuriating > refusal to allow me to throttle those same interrupters, I came up with > this patch to koji 1.2.6 it allows Mercurial to be used as a SCM. > > Thanks for the un-convoluted code! > > Brian -------------- next part -------------- A non-text attachment was scrubbed... Name: hgkoji-2.patch Type: text/x-patch Size: 1891 bytes Desc: not available URL: From jiteshs at marvell.com Fri Jan 23 04:39:42 2009 From: jiteshs at marvell.com (Jitesh Shah) Date: Thu, 22 Jan 2009 20:39:42 -0800 Subject: Koji: ServerOffline In-Reply-To: <49789684.8080103@redhat.com> References: <1232602311.3077.20.camel@localhost.localdomain> <49781D3D.1090206@linux-kernel.at> <1232609217.3077.31.camel@localhost.localdomain> <4978242B.5050906@linux-kernel.at> <1232616488.3077.51.camel@localhost.localdomain> <49789684.8080103@redhat.com> Message-ID: <1232685582.3037.0.camel@localhost.localdomain> Hi, On Thu, 2009-01-22 at 10:53 -0500, Mike McLean wrote: > Jitesh Shah wrote: > > Which version of koji are you running? I am using 1.2.6-1 > > -- > Fedora-buildsys-list mailing list > Fedora-buildsys-list at redhat.com > https://www.redhat.com/mailman/listinfo/fedora-buildsys-list -------------- next part -------------- An HTML attachment was scrubbed... URL: From jiteshs at marvell.com Fri Jan 23 05:01:26 2009 From: jiteshs at marvell.com (Jitesh Shah) Date: Thu, 22 Jan 2009 21:01:26 -0800 Subject: Koji: ServerOffline In-Reply-To: <49789C25.1080401@linux-kernel.at> References: <1232602311.3077.20.camel@localhost.localdomain> <49781D3D.1090206@linux-kernel.at> <1232609217.3077.31.camel@localhost.localdomain> <4978242B.5050906@linux-kernel.at> <1232616488.3077.51.camel@localhost.localdomain> <49789684.8080103@redhat.com> <49789C25.1080401@linux-kernel.at> Message-ID: <1232686886.3037.10.camel@localhost.localdomain> On Thu, 2009-01-22 at 17:17 +0100, Oliver Falk wrote: > Mike McLean wrote: > I wonder if it would log this error in a way you know what's going on :-) > > Jitesh, can you check the logs as suggested by Mike, although it's fixed > now, I'd like to know if you see the error there... I checked the logs. There is just a python traceback and the actual error "ServerOffline: Database Outage' (This is with KojiDebug off) With KojiDebug turned on, it additionally shows the queries executed and the time taken for each. Nothing that would point to an incorrect configuration or a failure in database connection. Jitesh > > [ ... ] > > -of > > -- > Fedora-buildsys-list mailing list > Fedora-buildsys-list at redhat.com > https://www.redhat.com/mailman/listinfo/fedora-buildsys-list -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliver at linux-kernel.at Fri Jan 23 09:12:37 2009 From: oliver at linux-kernel.at (Oliver Falk) Date: Fri, 23 Jan 2009 10:12:37 +0100 Subject: Koji: ServerOffline In-Reply-To: <1232686886.3037.10.camel@localhost.localdomain> References: <1232602311.3077.20.camel@localhost.localdomain> <49781D3D.1090206@linux-kernel.at> <1232609217.3077.31.camel@localhost.localdomain> <4978242B.5050906@linux-kernel.at> <1232616488.3077.51.camel@localhost.localdomain> <49789684.8080103@redhat.com> <49789C25.1080401@linux-kernel.at> <1232686886.3037.10.camel@localhost.localdomain> Message-ID: <49798A05.6090003@linux-kernel.at> Jitesh Shah wrote: > > On Thu, 2009-01-22 at 17:17 +0100, Oliver Falk wrote: > > Mike McLean wrote: > > > I wonder if it would log this error in a way you know what's going on :-) > > > > Jitesh, can you check the logs as suggested by Mike, although it's fixed > > now, I'd like to know if you see the error there... > > > I checked the logs. There is just a python traceback and the actual > error "ServerOffline: Database Outage' (This is with KojiDebug off) > > With KojiDebug turned on, it additionally shows the queries executed and > the time taken for each. > > Nothing that would point to an incorrect configuration or a failure in > database connection. Fine. Then it wasn't wrong to give you the strace hint :-) -of From mikem at redhat.com Fri Jan 23 20:17:40 2009 From: mikem at redhat.com (Mike McLean) Date: Fri, 23 Jan 2009 15:17:40 -0500 Subject: Koji: ServerOffline In-Reply-To: <1232686886.3037.10.camel@localhost.localdomain> References: <1232602311.3077.20.camel@localhost.localdomain> <49781D3D.1090206@linux-kernel.at> <1232609217.3077.31.camel@localhost.localdomain> <4978242B.5050906@linux-kernel.at> <1232616488.3077.51.camel@localhost.localdomain> <49789684.8080103@redhat.com> <49789C25.1080401@linux-kernel.at> <1232686886.3037.10.camel@localhost.localdomain> Message-ID: <497A25E4.6070006@redhat.com> Jitesh Shah wrote: > I checked the logs. There is just a python traceback and the actual > error "ServerOffline: Database Outage' (This is with KojiDebug off) 'just' a python traceback? Often these are the key to the mystery. From kanarip at kanarip.com Fri Jan 30 01:39:57 2009 From: kanarip at kanarip.com (Jeroen van Meeuwen) Date: Fri, 30 Jan 2009 02:39:57 +0100 Subject: [PATCH 1,2/2] pungi patches Message-ID: <49825A6D.8030203@kanarip.com> Two patches; 0001) Do not include boot.iso on any disc, per https://www.redhat.com/archives/fedora-devel-list/2009-January/msg02070.html 0002) the more cosmetic patch, I just so happened to see this; extraargs had two append() calls while "-o" and isofile were meant to be together. The patch makes this one extend call so that no accidents happen when someone inserts a line in between. Kind regards, Jeroen van Meeuwen -kanarip -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-Do-not-include-boot.iso-on-any-disc.patch Type: text/x-patch Size: 969 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0002-Prevent-errors-from-happening-by-accidently-insertin.patch Type: text/x-patch Size: 1013 bytes Desc: not available URL: From jkeating at redhat.com Fri Jan 30 01:56:30 2009 From: jkeating at redhat.com (Jesse Keating) Date: Thu, 29 Jan 2009 17:56:30 -0800 Subject: [PATCH 1,2/2] pungi patches In-Reply-To: <49825A6D.8030203@kanarip.com> References: <49825A6D.8030203@kanarip.com> Message-ID: <1233280590.3650.5.camel@localhost.localdomain> On Fri, 2009-01-30 at 02:39 +0100, Jeroen van Meeuwen wrote: > 0001) Do not include boot.iso on any disc, per > https://www.redhat.com/archives/fedora-devel-list/2009-January/msg02070.html 1 was already done upstream, https://fedorahosted.org/pungi/changeset/972240b9729dc023dff6a5ea519f2a55944b8c08 Boot.iso needs to not be in the splitdirs that splittree will make or else the size estimation that splittree does will be off, and the first CD won't use up the extra space. > 0002) the more cosmetic patch, I just so happened to see this; extraargs > had two append() calls while "-o" and isofile were meant to be together. > The patch makes this one extend call so that no accidents happen when > someone inserts a line in between. Thanks, I'll apply that. -- Jesse Keating Fedora -- Freedom? is a feature! identi.ca: http://identi.ca/jkeating -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 197 bytes Desc: This is a digitally signed message part URL: From jkeating at redhat.com Fri Jan 30 02:08:34 2009 From: jkeating at redhat.com (Jesse Keating) Date: Thu, 29 Jan 2009 18:08:34 -0800 Subject: [PATCH 1,2/2] pungi patches In-Reply-To: <1233280590.3650.5.camel@localhost.localdomain> References: <49825A6D.8030203@kanarip.com> <1233280590.3650.5.camel@localhost.localdomain> Message-ID: <1233281314.3650.6.camel@localhost.localdomain> On Thu, 2009-01-29 at 17:56 -0800, Jesse Keating wrote: > On Fri, 2009-01-30 at 02:39 +0100, Jeroen van Meeuwen wrote: > > 0001) Do not include boot.iso on any disc, per > > https://www.redhat.com/archives/fedora-devel-list/2009-January/msg02070.html > > 1 was already done upstream, > https://fedorahosted.org/pungi/changeset/972240b9729dc023dff6a5ea519f2a55944b8c08 > > Boot.iso needs to not be in the splitdirs that splittree will make or > else the size estimation that splittree does will be off, and the first > CD won't use up the extra space. I think I see why your patch may be needed, boot.iso may still wind up on the DVD here. Looking closer. -- Jesse Keating Fedora -- Freedom? is a feature! identi.ca: http://identi.ca/jkeating -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 197 bytes Desc: This is a digitally signed message part URL: From kanarip at kanarip.com Fri Jan 30 02:17:13 2009 From: kanarip at kanarip.com (Jeroen van Meeuwen) Date: Fri, 30 Jan 2009 03:17:13 +0100 Subject: [PATCH 1,2/2] pungi patches In-Reply-To: <1233280590.3650.5.camel@localhost.localdomain> References: <49825A6D.8030203@kanarip.com> <1233280590.3650.5.camel@localhost.localdomain> Message-ID: <49826329.8000709@kanarip.com> Jesse Keating wrote: > On Fri, 2009-01-30 at 02:39 +0100, Jeroen van Meeuwen wrote: >> 0001) Do not include boot.iso on any disc, per >> https://www.redhat.com/archives/fedora-devel-list/2009-January/msg02070.html > > 1 was already done upstream, > https://fedorahosted.org/pungi/changeset/972240b9729dc023dff6a5ea519f2a55944b8c08 > That's only in splittree, which I presume is only executed for split media, right? Kind regards, Jeroen van Meeuwen -kanarip From jkeating at redhat.com Fri Jan 30 02:31:37 2009 From: jkeating at redhat.com (Jesse Keating) Date: Thu, 29 Jan 2009 18:31:37 -0800 Subject: [PATCH 1,2/2] pungi patches In-Reply-To: <49826329.8000709@kanarip.com> References: <49825A6D.8030203@kanarip.com> <1233280590.3650.5.camel@localhost.localdomain> <49826329.8000709@kanarip.com> Message-ID: <1233282697.3650.7.camel@localhost.localdomain> On Fri, 2009-01-30 at 03:17 +0100, Jeroen van Meeuwen wrote: > That's only in splittree, which I presume is only executed for split > media, right? Yeah, I see now that it has to be done in both places. I forgot how the DVD mkisofs call was done, and I was only tested the split media stuff last night. Good catch. -- Jesse Keating Fedora -- Freedom? is a feature! identi.ca: http://identi.ca/jkeating -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 197 bytes Desc: This is a digitally signed message part URL: