Heads-up: brand new RPM version about to hit rawhide

[Doug Ledford]

On Tue, 2008-07-15 at 15:26 +0000, Kevin Kofler wrote:
> Andreas Ericsson <ae <at> op5.se> writes:
> > > * make releases without tags: For example, the weekly trunk snapshots of
> > > KDE don't get tags, nor do the extragear tarball releases.
> > 
> > I'm not sure if you saw my email regarding the requirements on the SCM for it
> > to be useful in the scenario Doug Ledford proposes, but right at the top of
> > the list comes the ability to uniquely name one particular commit. If you
> > have that, you don't need tags.
> 
> The problem with commit IDs is that they're a lot less readable and intuitive 
> than tags.
> 
> > It would be extremely poor project policy to move a tag after it's made
> > public
> 
> Are you trying to imply that KDE has "extremely poor project policy"? I think 
> it only makes sense to do respins that way. A release isn't always perfect on 
> the first try.

Well, no offense intended even though I'm sure you're not going to like
my answer, but yeah, large portions of the world think moving tags is an
explicit no-no in publicly available repos.

If you want to avoid respins, then you should do rc candidates out the
wazoo and when it is all baked and perfect, make a final release with
the tag you want.  If, at some later time, it's found that you
absolutely must respin, then it deserves a point update on the version.

To do things any other way is a gross violation of the principle of
least surprise.  You run a very real risk of essentially tricking users
into running the wrong code without knowing it because you change things
silently.  I have, on several occasions, refused to even submit packages
to the review process if I knew that the upstream people changed their
tarballs without changing the version on the tarball.  I let them know
that it was a hard requirement that any updated tarball have an updated
version, and once they complied I would then look at packaging their
software up and submitting it for review.  They listened and agreed to
why it was necessary, and I submitted their package for review after
that ;-)

> > For centralized scms, moving tags doesn't matter in the slightest, since
> > they can't name a commit uniquely anyway.
> 
> That's not true, a SVN commit is uniquely named by the revision number. And in 
> fact, being centralized allows SVN to actually use totally-ordered numbers, not 
> random IDs coming from the checksum of something (which are bound to break the 
> day you run into a collision, by the way).

Nah, they did the math on the sha1 identifiers when they designed the
scms.  The argument of a collision is bogus.  It's almost impossible to
*intentionally* create a collision, let alone by accident.  You're about
as likely to win the lottery as have a collision, and that's saying a
lot for someone like me who doesn't buy lottery tickets.

> The best way to reproducibly name a tag in SVN would be to use the tag name 
> (which would be part of the URL) _and_ the revision number. Tags in SVN are 
> just directories, so you can use the directory at the given revision and you'll 
> get the exact thing you originally checked out.

Well, as has already been discussed, it would be almost impossible to
directly use svn repos.  More like we would just need to import them
into another scm that we can control locally (be that tarballs into a
look aside cache, or exploded source into an scm, either way it's still
local to us).  This is due to the difficulty of being able to actually
have our own tags and modifications relative to upstream's ownership of
the svn repo.

> > Me, being mostly a user who also happens to be a programmer, would love
> > to have an easy way to be able to get a clone of <insert-package-here>,
> > find the sources corresponding exactly to my version of the package and
> > then fix whatever issues I have with it. Even if it was just me willing
> > to do that (which I highly doubt), you'd have a net gain of one extra
> > spare-time developer. You can't possibly argue that making it easier for
> > casual developers to get involved is a bad thing.
> 
> With the current system, you just have to check out the package and run "make" 
> in one of the branches to get all source files (usually just a tarball) 
> downloaded from the lookaside cache. Extract the tarball and you have your 
> sources. The patches aren't applied there, but that's by design. Patches are 
> supposed to be independent, so (with some exceptions, e.g. the kernel which 
> often includes series of interdependent patches automatically generated from 
> some git repository) we develop each patch against the _original_ sources, then 
> only when actually building the package we apply them all.

There's no need to specifically patch against the original sources.
Either the other patches in the rpm will touch the same area of code, in
which case your patch won't apply over the top of them and you need a
fully prepped tree to generate a patch that will apply, or your patch
won't touch the same code as any other patches and regardless of whether
you patch against a prepped tree or original sources the patch will
still apply to the original source with at most some line offsets.  So,
just make prep and go.

> My workflow when I develop a patch for KDE is:
> 1. I extract the tarball (e.g. kdebase-workspace-4.0.98.tar.bz2).
> 2. This creates a directory (e.g. kdebase-workspace-4.0.98).

make prep here

> 3. I copy this directory, appending the name of the patch I intend to write 
> (e.g. kdebase-workspace-4.0.98-consolekit-kdm).

sometimes I copy the directory if I think there will be lots of files
that change, otherwise I just copy the files independently to some
unique backup suffix as I'm preparing to edit them and use gendiff
instead of diff

> 4. I make my changes in that directory (e.g. 
> kdebase-workspace-4.0.98-consolekit-kdm).
> 5. I diff the original vs. the patched directory (e.g. diff -Nur 
> kdebase-workspace-4.0.98 kdebase-workspace-4.0.98-consolekit-kdm 
> >kdebase-workspace-4.0.98-consolekit-kdm.patch).
> 6. I copy the patch to the package branch.
> 7. I cvs add it to the repository.
> 8. I apply it in the specfile (2 lines, e.g. Patch1: 
> kdebase-workspace-4.0.98-consolekit-kdm.patch in the header 
> and %patch1 -p1 -b .consolekit-kdm in the %prep section).
> 9. I commit.
> 10. I run make tag.
> 11. I run make build BUILD_FLAGS=--nowait.
> 12. Koji sends me the results.
> 13. If the build failed, I:
> * repeat steps 4, 5, 6 and 9
> * run make force-tag
> * resubmit the failed build through the Koji web interface
> as often as needed until it builds.
> 
> I know this sounds overly complicated,

It *is* overly complicated.  Not that you did anything to make it that
way, you are doing things just like I do.  I'm just saying the process
is more difficult than it needs to be.

>  but keep in mind that most of these 
> steps are just a couple of mouse clicks or one line in a terminal, I 
> intentionally detailed them so a beginner can understand what's going on. I'm 
> not sure a SCM with fully-exploded sources would really make that easier.

Oh yeah it does.  I'll use the example of git since that's what I work
with most.  But, as I mentioned above, sometimes I copy the entire
directory and run diff, sometimes I just create backup files and run
gendiff.  I have had circumstances where I forgot to make a backup file
and ran gendiff and then edited the spec and ran a build process (which
of course wipes out the make prep directory I was working in) and only
later found out I missed that file, so that file's work is now gone.  Of
course, this is an artifact of my workflow, but it's a pain and has
happened on more than one occasion.  Now, had I been working in git,
then if I had ran git gui, I would have gotten a list of all the files I
modified while working on the source and could select them individually
or all at once for committing to the repo.  There's no missing files
when you do that.  And since there's no make prep necessary, there's no
chance of wiping anything out.  And since I can do my build testing and
basic run testing in the directory before I make the git commit, I can
be assured that once I commit the change, the code will be exactly as I
intended it to be and be functional.

I mentioned in one of my previous mails that this whole thing of an
exploded repo is probably more important to Red Hat than to Fedora, and
that's really true if you are talking about the pure development
advantage (other features of exploded source apply to Fedora as much as
Red Hat, such as having a single repo able to be the canonical source
for all your packages).

The steps you outline above are not that bad when dealing with packages
where your patches are minimal in both total size and quantity.  Where
it becomes important is in places like the kernel where we freeze the
kernel internally but still work on it for years.  We end up with well
over 1,000 individual patches to our kernels long before we quit working
on them.  And some of those patches are *huge*.  Last update (rhel5.2) I
submitted a kernel patch that was 96,000 lines of diff.  And that was
just one of the 1000+ patches in our kernel.  It's when dealing with
this sort of thing that the whole exploded source method becomes *far*
more important in terms of developer productivity.  Right now, both the
rhel4 and rhel5 kernel maintainers actually use exploded source because
CVS was untenable.  But, because the build system doesn't support
exploded source, they have some convoluted scripts that spit out a
tarball and patches from their git repo and check those into CVS solely
for the purpose of building.

Aside from the development advantage that isn't nearly as strong for
Fedora as it is for Red Hat, there are also other advantages to exploded
source scm management.  Which is why I'm trying to drive this here in
Fedora versus just trying to make it happen internally inside Red Hat.

-- 
Doug Ledford <dledford at redhat.com>
              GPG KeyID: CFBFF194
              http://people.redhat.com/dledford

Infiniband specific RPMs available at
              http://people.redhat.com/dledford/Infiniband

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 197 bytes
Desc: This is a digitally signed message part
URL: <http://listman.redhat.com/archives/fedora-devel-list/attachments/20080715/78bc08a6/attachment.sig>