Package submission process

[Tom 'spot' Callaway]

On Fri, 2005-10-14 at 14:43 -0700, Ian MacGregor wrote:

> I just feel that there are items in the process that are unnecessary or 
> that some items can be changed that would eliminate other items.

Ian,

As the individual who wrote the document which describes the steps
necessary to become a contributor to Fedora Extras, I wanted to take the
opportunity to explain why we have done things in the manner that we
have.

We went into Fedora Extras with the following goals:

- Standardization
- Ease of management
- Ease of builds
- Ease of use
- Security
- Visibility

Red Hat has known for some time that simply providing an "upload" style
method for third party packages does not work well. At one point in the
long long gone era of Red Hat Linux (we're talking 5.*), Red Hat
provided a directory on their ftp server called contrib/ where people
could upload addon packages for Red Hat Linux. Unfortunately, without
some way of managing, revisioning, or auditing these packages, it
quickly became a maintenance nightmare, often with several different
copies of the same package being uploaded. Many users had no idea how to
install the packages, and if they did get installed, there was no
guarantee that these packages would be updated, or continue to work as
Red Hat Linux changed and patched itself in the never ending move
forward.

The contrib/ directory fell into disrepair and was eventually cooked
over a BBQ grill behind Red Hat HQ with some hot dogs and burgers.

So, when we looked at Fedora Extras, we knew that we needed to solve the
problems that contrib/ had presented to us. It became obvious to us that
for Fedora Extras to be successful, the people making these packages
would have to take responsibility for those packages, and not simply be
a "dumping ground" for packages that may or may not work in the many
complicated environments that can be created with Fedora Core. We also
wanted to make the environment seamless for our users, so that as they
installed packages from Fedora Core and Fedora Extras, they didn't have
to worry about tracking down the maintainer to get new releases or file
bugs.

There was a lot of discussion (some of it productive) about the best
tools to do this, and at the end of the day we went with the following:

- CVS: To provide a place where spec files and patch files can be kept,
revisioned, and monitored. This enables Fedora Extras to provide an
unparalleled level of visibility into the packages that are provided, as
well as enabling a clean mechanism of branching packages for specific
Fedora Core releases. We also built a lookaside cache, so that source
files (think tarballs) could be kept in a fast location, and only needed
to be uploaded once, regardless of how many branches used it.

- An account management system: We need to control who has access to
CVS, obviously, it isn't a good idea to let the whole world have write
access without some sort of check. This grew into a sponsor system,
where more experienced packagers who were comfortable and familiar with
the Fedora Extras process could guide and help the newer packagers.

- Bugzilla: To provide a single location where users can file bugs and
feature requests for Fedora Core and Fedora Extras packages. No packages
are ever perfect. What works for one may not work for others, and no
matter how good your packages may be, bugs lurk! It also provides a good
way for Fedora maintainers to communicate with upstream maintainers in a
public (and auditable) fashion.

- A Buildsystem: We looked at all sorts of things, and we ended up
taking some good open source tools (mach) and making them better and
more scalable, ending up with something that hooks directly into cvs and
enables package maintainers to be able to request package builds without
lengthy uploads or complicated routines. We also put systems of several
architectures into this buildsystem, so that packages would be available
not just for x86, but for x86_64 and ppc as well, even if the maintainer
doesn't possess that hardware for package generation or testing. This
comes with web pages and utilities for monitoring your builds, as well
as the entire build queue.

- A wiki server: For documentation, guidelines, best practices, and web
information for our users and maintainers.

- A simple infrastructure that ties this all together. For Fedora
Extras, this is as simple as a Makefile, which enables most package
operations to be done with a "make foo" style command, and a
cvs-import.sh script, which allows for a maintainer to upload a SRPM
into the CVS server with a single command line.

Beyond the tools, we knew that we wanted a way to maintain that Fedora
Extras packages met a standard of quality. Thus, all Fedora Extras
packages go through a fairly thorough review process, tracked in
Bugzilla, before being accepted into CVS. This is only done once per
package, not per revision of that package, so it really is only a
checkpoint process. This is very much in the spirit of the "many eyes"
philosophy of open source.

Also, we wanted to make sure that we were doing as much as we could to
ensure package security. By building all packages in a centralized
buildsystem, we can ensure that the binary packages provided by Fedora
Extras came from the specs and source code kept in our CVS server, and
not something which is malicious or otherwise compromised.

Of course, the lawyers also wanted their say in all of this, so we've
had to require some initial paperwork. Life is full of paperwork, we
managed to minimize ours to a ssh key, a digital signature, and some
personal information.

I think that the steps required for someone to become involved in Fedora
Extras are pretty well streamlined. Those of us who were involved in the
early incarnations of Extras remember how painful it was in the
beginning.

One of the things I've done is generate two documents:

- A document for first time packagers
(http://fedoraproject.org/wiki/Extras/Contributors)
- A document for sponsored packagers
(http://fedoraproject.org/wiki/Extras/NewPackageProcess)

The NewPackageProcess is a 14 step process, but it's really only that
long to make it clear. I'll always opt to make a document too clear,
rather than too short.

Once you've gotten through the fun stuff (cvs access, account creation,
paperwork pushing), which is all a one time thing, importing new
packages really can become a 6 step process:

#1. Make a package that meets Fedora Extras standards and guidelines
#2. Submit the SRPM and SPEC for Review.
#3. When the package passes Review, import it into cvs with
cvs-import.sh.
#4. Add the package to the owners.list.
#5. Tell us what branches you want it built for (FC-3? FC-4?)
#6. Check it out of cvs, tag the branches when you're happy, and request
builds.

I hope that you'll decide to participate in Fedora Extras. We've built
something unique and powerful, and we're always looking at ways to make
it better (automating the branch request process, improving the
buildsystem, streamlining the review process, providing more
best-practice documents).

For those reading, I don't mean to imply that Red Hat did all of this by
itself. We built a LOT of this infrastructure on the good ideas and good
works done by the Fedora Project (pre Fedora Core 1), and would not be
here without them.

Thanks,

~spot
-- 
Tom "spot" Callaway: Red Hat Senior Sales Engineer || GPG ID: 93054260
Fedora Extras Steering Committee Member (RPM Standards and Practices)
Aurora Linux Project Leader: http://auroralinux.org
Lemurs, llamas, and sparcs, oh my!