[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: What Fedora makes sucking for me - or why I am NOT Fedora

Josh Boyer wrote:

Do we have metrics on 'number of brand new packages going out as updates'
versus 'existing packages being bumped to new versions'?

If not, how hard would it be to get those?  They would be rather important
to reviewing this idea at a FESCo level.

I actually modified repodiff yesterday to look at the version string to work out what the change was (see attached).

F9 -> F9+updates:

Added Packages: 831
Removed Packages: 0 (0 obsoleted)
Modified Packages: 1608
        Major changes: 390
        Minor changes: 764
        Release changes: 445
        Release tag changes: 9

F9+updates -> F10+updates:

Added Packages: 258
Removed Packages: 105 (0 obsoleted)
Modified Packages: 4039
        Major changes: 457
        Minor changes: 673
        Release changes: 1334
        Release tag changes: 1575

F10 -> F10+updates:

Added Packages: 134
Removed Packages: 0 (0 obsoleted)
Modified Packages: 396
        Major changes: 61
        Minor changes: 202
        Release changes: 132
        Release tag changes: 1

This is using the Everything repo as the baseline, and I ran this yesterday. 'minor' is an update where only the last part of the version string (after the last .) changed, major is everything else. Its not a perfect heuristic - looking manually at the list, the major updates are being over reported a bit.

Changing this to look at comps to only consider the default package set is left as an exercise for the reader. Ditto for counting security updates separately, or for counting packages that were updated and then had another update come within a week...

I realise that some people want the latest and greatest at all times, but its not like releases aren't infrequent. Yes, new versions of packages fix bugs, but they also introduce risk. Yes, being the first distro to push a new package to stable means that fedora users can find and report bugs quickly, but anyone who wants to find bugs can run rawhide. And constant updates means that an update for a security issue doesn't leapfrog a bunch of versions, but I'm not sure that thats why the updates are happening.

When an update announcement comes in with the only description for the change is "new version" (with or without copying upstream's changelog), or no comment at all, then something is wrong with the process. I'm not suggesting that maintainers should have to cherrypick security/critical bugfixes only, like RHEL does. And I'm sure that new versions do fix bugs that people may be hitting but not reporting in bugzilla.

The yum-presto stuff will reduce the download size (An x86 machine I just updated from F8 to F10 downloaded over 1.2G via preupgrade and then about 400M of updates from the first couple of weeks of the F10 release - thats just wrong...), but while thats important its not the key issue.

Can someone who wants the new versions immediately explain why they don't want to wait an average of 3 months for the next fedora release?


#!/usr/bin/python -tt
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# GNU Library General Public License for more details.
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
# (c) 2007 Red Hat. Written by skvidal fedoraproject org

import yum
import rpmUtils
import sys
import time
import os
import locale
from yum.misc import to_unicode

from optparse import OptionParser

class DiffYum(yum.YumBase):
    def __init__(self):
        self.dy_repos = {'old':[], 'new':[]}
        self.dy_basecachedir = yum.misc.getCacheDir()
        self.dy_archlist = ['src']
    def dy_shutdown_all_other_repos(self):
        # disable all the other repos

    def dy_setup_repo(self, repotype, baseurl):
        repoid = repotype + str (len(self.dy_repos[repotype]) + 1)
        # make our new repo obj
        newrepo = yum.yumRepo.YumRepository(repoid)
        newrepo.name = repoid
        newrepo.baseurl = [baseurl]
        newrepo.basecachedir = self.dy_basecachedir
        newrepo.metadata_expire = 0
        newrepo.timestamp_check = False
        # add our new repo
        # enable that repo
        # setup the repo dirs/etc
        self._getSacks(archlist=self.dy_archlist, thisrepo=repoid)

    def dy_diff(self):
        add = []
        remove = []        
        modified = []
        obsoleted = {} # obsoleted = by
        newsack = yum.packageSack.ListPackageSack()
        for repoid in self.dy_repos['new']:

        oldsack = yum.packageSack.ListPackageSack()
        for repoid in self.dy_repos['old']:

        for pkg in newsack.returnNewestByName():
            tot = self.pkgSack.searchNevra(name=pkg.name)
            if len(tot) == 1: # it's only in new
            if len(tot) > 1:
                if oldsack.contains(name=pkg.name):
                    newest_old = oldsack.returnNewestByName(name=pkg.name)[0]
                    if newest_old.EVR != pkg.EVR:
                        modified.append((pkg, newest_old))

        for pkg in oldsack.returnNewestByName():
            if len(newsack.searchNevra(name=pkg.name)) == 0:

        for po in remove:
            for newpo in add:
                foundit = 0
                for obs in newpo.obsoletes:
                    if po.inPrcoRange('provides', obs):
                        foundit = 1
                        obsoleted[po] = newpo
                if foundit:
        ygh = yum.misc.GenericHolder()
        ygh.add = add
        ygh.remove = remove
        ygh.modified = modified
        ygh.obsoleted = obsoleted
        return ygh

def parseArgs(args):
       Parse the command line args. return a list of 'new' and 'old' repos
    usage = """
    repodiff: take 2 or more repositories and return a list of added, removed and changed
    repodiff --old=old_repo_baseurl --new=new_repo_baseurl """
    parser = OptionParser(version = "repodiff 0.2", usage=usage)
    # query options
    parser.add_option("-n", "--new", default=[], action="append",
                      help="new baseurl[s] for repos")
    parser.add_option("-o", "--old", default=[], action="append",
                      help="old baseurl[s] for repos")
    parser.add_option("-q", "--quiet", default=False, action='store_true')
    parser.add_option("-a", "--archlist", default=[], action="append",
                      help="In addition to src.rpms, any arch you want to include")
    parser.add_option("-s", "--size", default=False, action='store_true',
                      help="Output size changes for any new->old packages")
    (opts, argsleft) = parser.parse_args()

    if not opts.new or not opts.old:

    # sort out the comma-separated crap we somehow inherited.    
    archlist = ['src']
    for a in opts.archlist:
        for arch in a.split(','):

    opts.archlist = archlist             
    return opts

def main(args):
    opts = parseArgs(args)

    my = DiffYum()
    if opts.quiet:
        my.doLoggingSetup(my.conf.debuglevel, my.conf.errorlevel)
    my.dy_archlist = opts.archlist
    if not opts.quiet: print 'setting up repos'
    for r in opts.old:
        if not opts.quiet: print "setting up old repo %s" % r
            my.dy_setup_repo('old', r)
        except yum.Errors.RepoError, e:
            print "Could not setup repo at url  %s: %s" % (r, e)
    for r in opts.new:
        if not opts.quiet: print "setting up new repo %s" % r
            my.dy_setup_repo('new', r)
        except yum.Errors.RepoError, e:
            print "Could not setup repo at url %s: %s" % (r, e)
    if not opts.quiet: print 'performing the diff'
    ygh = my.dy_diff()

    total_sizechange = 0
    add_sizechange = 0
    remove_sizechange = 0
    num_obsolete = 0
    num_reltag = 0
    num_rel = 0
    num_major = 0
    num_minor = 0
    if ygh.add:
        for pkg in sorted(ygh.add):
            print 'New package %s' % pkg.name
            add_sizechange += int(pkg.size)
    if ygh.remove:
        for pkg in sorted(ygh.remove):
            print 'Removed package %s' % pkg.name
            if ygh.obsoleted.has_key(pkg):
                print 'Obsoleted by %s' % ygh.obsoleted[pkg]
                num_obsolete += 1
            remove_sizechange += (int(pkg.size))
    if ygh.modified:
        print 'Updated Packages:\n'
        for (pkg, oldpkg) in sorted(ygh.modified):
            if oldpkg.ver == pkg.ver:
                oldpos = oldpkg.rel.rfind(".fc")
                newpos = pkg.rel.rfind(".fc")
                if oldpos > 0 and newpos > 0 and oldpkg.rel[0:oldpos] == pkg.rel[0:newpos]:
                    msg = "%s: reltag" % (pkg.name)
                    num_reltag += 1
                    msg = "%s: release" % (pkg.name)
                    num_rel += 1
            elif pkg.ver.startswith(oldpkg.ver + '.') or (oldpkg.ver.rsplit('.',1)[0] == pkg.ver.rsplit('.',1)[0]):
                msg = "%s: minor" % (pkg.name)
                num_minor += 1
                msg = "%s: major" % (pkg.name)
                num_major += 1
            msg += "\n%s (%s): %s-%s => %s-%s" % (pkg.name, pkg.size, oldpkg.ver, oldpkg.rel, pkg.ver, pkg.rel)
            if opts.size:
                sizechange = int(pkg.size) - int(oldpkg.size)
                total_sizechange += sizechange

            print msg

    print 'Summary:'
    print 'Added Packages: %s' % len(ygh.add)
    print 'Removed Packages: %s (%s obsoleted)' % (len(ygh.remove), num_obsolete)
    print 'Modified Packages: %s' % len(ygh.modified)
    print "\tMajor changes: %s" % num_major
    print "\tMinor changes: %s" % num_minor
    print "\tRelease changes: %s" % num_rel
    print "\tRelease tag changes: %s" % num_reltag
    if opts.size:
        print 'Size of added packages: %s' % add_sizechange
        print 'Size change of modified packages: %s' % total_sizechange
        print 'Size of removed packages: %s' % remove_sizechange
if __name__ == "__main__":
    import locale
    # This test needs to be before locale.getpreferredencoding() as that
    # does setlocale(LC_CTYPE, "")
        locale.setlocale(locale.LC_ALL, '')
    except locale.Error, e:
        # default to C locale if we get a failure.
        print >> sys.stderr, 'Failed to set locale, defaulting to C'
        os.environ['LC_ALL'] = 'C'
        locale.setlocale(locale.LC_ALL, 'C')
    if True: # not sys.stdout.isatty():
        import codecs
        sys.stdout = codecs.getwriter(locale.getpreferredencoding())(sys.stdout)
        sys.stdout.errors = 'replace'


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]