[Pulp-list] New Source Layout

Jay Dobies jason.dobies at redhat.com
Wed Jun 13 13:03:13 UTC 2012

I think we're in a good place to get everyone back on master. The 
migration process for your dev environment isn't the simplest, but 
hopefully this is the last time we have to change the codebase to this 
extent. I'll update the wiki for new developers later; this is meant for 
those of us dealing with the transition.

Yes, this e-mail is long, even if you've come to expect that from me. 
But please make sure you read this one.  :)

= Migrate =

You're going to lose everything in this process, but ultimately it's a 
dev environment, so you probably don't have much in there to begin with.

The first steps are going to be to remove the old Pulp dev install. From 
what I hear, there's issues in pulp-dev's uninstall capabilities when 
dealing with symlinks, so for simplicity we'll do a few manual steps:

1. $ sudo systemctl stop httpd.service
2. $ sudo rm -rf /etc/pulp/*
3. $ sudo rm -rf /var/lib/pulp/*
4. $ sudo rm /usr/bin/pulp-*
5. Drop your database entirely:
    $ mongo
    $ use pulp_database
    $ db.dropDatabase()

Now for the new stuff. If you don't have any outstanding code, it might 
be quicker to just move your code out to a temp location and re-clone 
the repo. It will save you from having old stuff lying around.

1. $ git pull

The next few steps are to clear out things that git doesn't know about, 
such as .pyc files. The following should be run from the git checkout 
root. To be clear: do not do this if you have outstanding changes to 
merge in. That should only be prad/john with plugin changes but everyone 
think before you run these (all of this code has moved and you'll need 
to merge your changes in elsewhere, more on that later).

2. $ rm -rf src  # yes, you're deleting src in root; it's moved
3. $ rm -rf plugins
4. $ rm -rf extensions

Install the dev environment. You'll need to hit both setup.py files in 
platform and rpm_support to get all of the site-packages references created.

5. $ sudo python ./pulp-dev.py -I
6. $ cd platform/src && sudo python ./setup.py develop
7. $ cd rpm-support/src && sudo python ./setup.py develop

At this point, things should largely be back to normal:

8. $ sudo pulp-migrate  # note the version is back to 1 but I'll address 
that soon

I found it easier to clean up my logs so I knew of any new issues coming up:

9a. (optional) $ rm -f /var/log/httpd/*
9b. (optional) $ rm -f /var/log/pulp/*

10. $ sudo systemctl start httpd.service

= Validation =

Keep an eye on /var/log/httpd/error_log to make sure the server starts 
correctly. It won't if the plugins/type definitions aren't happy. If it 
doesn't, ping me, but also check:


to make sure there aren't any symlinks in there that don't point to 
actual files.

= Running the Client =

Apache's mod_ssl seems to have changed a bit in Fedora 17. It used to 
create a default cert with the CN "localhost.localdomain". Now, at least 
on my box, the CN is the hostname of the box itself.

That's going to cause a problem when trying to run the client. The host 
in admin.conf needs to match the CN. We ship with localhost.localdomain 
by default but that's not sufficient anymore.

That means you'll need to change the host in /etc/pulp/admin/admin.conf 
in order to do anything. The problem is that that file is symlinked into 
the codebase, so if you edit it directly you may accidentally commit it.

If you create ~/.pulp/admin.conf it will use that as overrides 
(~/.pulp/consumer.conf is the corresponding consumer override, but I may 
remove that since it doesn't really make sense to scope anything 
consumer related to a specific user). The contents should be:

host = sunflower
# always makes me smile that sayli named her desktop "sunflower" :)

= Where is Everything? =

I made a slight change to what was proposed last week. The bulk of our 
codebase is organized into three subprojects:

platform - All of the plumbing, including server, client, and agent. If 
you ever find yourself mentioning rpm/errata/kickstart/puppet/other_type 
in here, it's wrong.

rpm-support - RPM pluigns/extensions/handlers. If you ever find yourself 
mentioning rpm/errata/kickstart _outside_ of this directory, it's wrong.

builtins - The generic extensions/plugins/handlers that ship as part of 
the Pulp platform. These cover things unrelated to types, such as the 
create repo that doesn't imply an importer or simple login/logout 

In the future, we'll be adding subprojects for ISO support and puppet 
support. Maybe even Windows support if Todd holds my children ransom and 
forces me to.

Within each of the three subprojects there are a few consistent pieces:

src - Source code.
test/unit - Source code for unit tests
test/unit/data - Data needed for unit tests.
etc - Mimics the structure where any files are deployed.

For the plugin-related subprojects, you'll also find:


= Tests =

What the above means is that we have test cases in three different 
places now (one per subproject). This is actually a benefit in many 
ways. During development in a particular subproject it's dirt simple and 
quick as hell to run just the tests for that subproject. We're in a much 
better position for test-driven development practices; we won't grow old 
while running the full suite as a sanity check.

I'm serious about quicker. The platform tests run in 60-70 seconds. RPM 
support runs in around 90. And after stripping out all of the untested 
v1 code, we're floating around 71% total project code coverage. That's 
actually pretty damn awesome. It's not gonna stop me from bitching that 
we need to raise that number, but I do acknowledge that it's a very 
healthy start.

Back to the three locations. There's a script in root called 
run-tests.sh that runs the tests in all three locations and generates a 
coverage report. The coverage report is under cover in the git root 
(pull up index.html in there for the slick HTML view with the colored 
markers of uncovered code).

That said, don't run that script yet. There's currently an issue in the 
RPM support tests that cause them to barf a ton of directories into your 
working directory and not clean them up, so running that script results 
in a bunch of garbage thrown into your git checkout. We'll get those 
fixed ASAP.

Something else to keep in mind. That script will run all three suites in 
the same nosetests process. Jenkins, however, will intentionally run 
them in three separate shells. My fear is that the single process run 
will allow dependencies to bleed across subprojects and not be detected. 
But at the same time, I dig the full project coverage report. So jenkins 
will be our safety net for making sure the process divisions are in 
place between tests in case you ever hit an issue where jenkins 
indicates a failure but the run-tests.sh script does not.

More on testing in a future e-mail; I have a bunch of comments about 
things we've done wrong in the past and how the new base classes work.

= Random Notes =

* pulp-consumer is broken right now. There are a few bugs in play. It's 
always been using the admin certificate instead of its consumer 
certificate, and when I fixed that it's now failing authentication. A 
quick look at the code makes me think the auth code isn't looking at the 
v2 consumers collection, but I really didn't look closely. I'll be 
addressing this later today.

* I stripped out anything not currently being used. We still have it in 
git history if we need to look again, but the goal here was to be as 
clean as possible. That means things related to CDS, auditing, event 
sending, the exporter, and the recent package migration script are all 
deleted. We can always look up the v1 implementations when we go to 
address them in v2 but I'd rather not have such a large sweeping clean 
up again so took care of them now.

* Jeff's still working on the .spec files. There will be two: one for 
Pulp platform and one for RPM support, each producing a number of RPMs. 
I've seen the work done so far and they look great; much better 
organization by created RPM rather than by spec section. Not sure of his 
current status but I suspect we'll be building by the end of the week.

I think that's all I have for now. Ping me if you run into any problems 
with the migration. I'm thinking of setting up an elluminate for the 
whole day that I'll just sit in and people can jump in/screen share when 
they run into specific problems, but we'll see how it goes.

Jay Dobies
Freenode: jdob @ #pulp
http://pulpproject.org | http://blog.pulpproject.org

More information about the Pulp-list mailing list