[Pulp-list] New Source Layout

[Jay Dobies]

One more clean up step before pulp-dev:

$ sudo rm /etc/rc.d/init.d/pulp-server


On 06/13/2012 09:03 AM, Jay Dobies wrote:
> I think we're in a good place to get everyone back on master. The
> migration process for your dev environment isn't the simplest, but
> hopefully this is the last time we have to change the codebase to this
> extent. I'll update the wiki for new developers later; this is meant for
> those of us dealing with the transition.
>
> Yes, this e-mail is long, even if you've come to expect that from me.
> But please make sure you read this one. :)
>
>
> = Migrate =
>
> You're going to lose everything in this process, but ultimately it's a
> dev environment, so you probably don't have much in there to begin with.
>
> The first steps are going to be to remove the old Pulp dev install. From
> what I hear, there's issues in pulp-dev's uninstall capabilities when
> dealing with symlinks, so for simplicity we'll do a few manual steps:
>
> 1. $ sudo systemctl stop httpd.service
> 2. $ sudo rm -rf /etc/pulp/*
> 3. $ sudo rm -rf /var/lib/pulp/*
> 4. $ sudo rm /usr/bin/pulp-*
> 5. Drop your database entirely:
> $ mongo
> $ use pulp_database
> $ db.dropDatabase()
>
> Now for the new stuff. If you don't have any outstanding code, it might
> be quicker to just move your code out to a temp location and re-clone
> the repo. It will save you from having old stuff lying around.
>
> 1. $ git pull
>
> The next few steps are to clear out things that git doesn't know about,
> such as .pyc files. The following should be run from the git checkout
> root. To be clear: do not do this if you have outstanding changes to
> merge in. That should only be prad/john with plugin changes but everyone
> think before you run these (all of this code has moved and you'll need
> to merge your changes in elsewhere, more on that later).
>
> 2. $ rm -rf src # yes, you're deleting src in root; it's moved
> 3. $ rm -rf plugins
> 4. $ rm -rf extensions
>
> Install the dev environment. You'll need to hit both setup.py files in
> platform and rpm_support to get all of the site-packages references
> created.
>
> 5. $ sudo python ./pulp-dev.py -I
> 6. $ cd platform/src && sudo python ./setup.py develop
> 7. $ cd rpm-support/src && sudo python ./setup.py develop
>
> At this point, things should largely be back to normal:
>
> 8. $ sudo pulp-migrate # note the version is back to 1 but I'll address
> that soon
>
> I found it easier to clean up my logs so I knew of any new issues coming
> up:
>
> 9a. (optional) $ rm -f /var/log/httpd/*
> 9b. (optional) $ rm -f /var/log/pulp/*
>
> 10. $ sudo systemctl start httpd.service
>
>
> = Validation =
>
> Keep an eye on /var/log/httpd/error_log to make sure the server starts
> correctly. It won't if the plugins/type definitions aren't happy. If it
> doesn't, ping me, but also check:
>
> /var/lib/pulp/plugins/{distributors,importers,types}
>
> to make sure there aren't any symlinks in there that don't point to
> actual files.
>
>
> = Running the Client =
>
> Apache's mod_ssl seems to have changed a bit in Fedora 17. It used to
> create a default cert with the CN "localhost.localdomain". Now, at least
> on my box, the CN is the hostname of the box itself.
>
> That's going to cause a problem when trying to run the client. The host
> in admin.conf needs to match the CN. We ship with localhost.localdomain
> by default but that's not sufficient anymore.
>
> That means you'll need to change the host in /etc/pulp/admin/admin.conf
> in order to do anything. The problem is that that file is symlinked into
> the codebase, so if you edit it directly you may accidentally commit it.
>
> If you create ~/.pulp/admin.conf it will use that as overrides
> (~/.pulp/consumer.conf is the corresponding consumer override, but I may
> remove that since it doesn't really make sense to scope anything
> consumer related to a specific user). The contents should be:
>
> [server]
> host = sunflower
> # always makes me smile that sayli named her desktop "sunflower" :)
>
>
> = Where is Everything? =
>
> I made a slight change to what was proposed last week. The bulk of our
> codebase is organized into three subprojects:
>
> platform - All of the plumbing, including server, client, and agent. If
> you ever find yourself mentioning rpm/errata/kickstart/puppet/other_type
> in here, it's wrong.
>
> rpm-support - RPM pluigns/extensions/handlers. If you ever find yourself
> mentioning rpm/errata/kickstart _outside_ of this directory, it's wrong.
>
> builtins - The generic extensions/plugins/handlers that ship as part of
> the Pulp platform. These cover things unrelated to types, such as the
> create repo that doesn't imply an importer or simple login/logout
> functionality.
>
> In the future, we'll be adding subprojects for ISO support and puppet
> support. Maybe even Windows support if Todd holds my children ransom and
> forces me to.
>
> Within each of the three subprojects there are a few consistent pieces:
>
> src - Source code.
> test/unit - Source code for unit tests
> test/unit/data - Data needed for unit tests.
> etc - Mimics the structure where any files are deployed.
>
> For the plugin-related subprojects, you'll also find:
>
> extensions
> handlers
> plugins/{distributors,importers,profilers,types}
>
>
> = Tests =
>
> What the above means is that we have test cases in three different
> places now (one per subproject). This is actually a benefit in many
> ways. During development in a particular subproject it's dirt simple and
> quick as hell to run just the tests for that subproject. We're in a much
> better position for test-driven development practices; we won't grow old
> while running the full suite as a sanity check.
>
> I'm serious about quicker. The platform tests run in 60-70 seconds. RPM
> support runs in around 90. And after stripping out all of the untested
> v1 code, we're floating around 71% total project code coverage. That's
> actually pretty damn awesome. It's not gonna stop me from bitching that
> we need to raise that number, but I do acknowledge that it's a very
> healthy start.
>
> Back to the three locations. There's a script in root called
> run-tests.sh that runs the tests in all three locations and generates a
> coverage report. The coverage report is under cover in the git root
> (pull up index.html in there for the slick HTML view with the colored
> markers of uncovered code).
>
> That said, don't run that script yet. There's currently an issue in the
> RPM support tests that cause them to barf a ton of directories into your
> working directory and not clean them up, so running that script results
> in a bunch of garbage thrown into your git checkout. We'll get those
> fixed ASAP.
>
> Something else to keep in mind. That script will run all three suites in
> the same nosetests process. Jenkins, however, will intentionally run
> them in three separate shells. My fear is that the single process run
> will allow dependencies to bleed across subprojects and not be detected.
> But at the same time, I dig the full project coverage report. So jenkins
> will be our safety net for making sure the process divisions are in
> place between tests in case you ever hit an issue where jenkins
> indicates a failure but the run-tests.sh script does not.
>
> More on testing in a future e-mail; I have a bunch of comments about
> things we've done wrong in the past and how the new base classes work.
>
>
> = Random Notes =
>
> * pulp-consumer is broken right now. There are a few bugs in play. It's
> always been using the admin certificate instead of its consumer
> certificate, and when I fixed that it's now failing authentication. A
> quick look at the code makes me think the auth code isn't looking at the
> v2 consumers collection, but I really didn't look closely. I'll be
> addressing this later today.
>
> * I stripped out anything not currently being used. We still have it in
> git history if we need to look again, but the goal here was to be as
> clean as possible. That means things related to CDS, auditing, event
> sending, the exporter, and the recent package migration script are all
> deleted. We can always look up the v1 implementations when we go to
> address them in v2 but I'd rather not have such a large sweeping clean
> up again so took care of them now.
>
> * Jeff's still working on the .spec files. There will be two: one for
> Pulp platform and one for RPM support, each producing a number of RPMs.
> I've seen the work done so far and they look great; much better
> organization by created RPM rather than by spec section. Not sure of his
> current status but I suspect we'll be building by the end of the week.
>
>
> I think that's all I have for now. Ping me if you run into any problems
> with the migration. I'm thinking of setting up an elluminate for the
> whole day that I'll just sit in and people can jump in/screen share when
> they run into specific problems, but we'll see how it goes.
>


-- 
Jay Dobies
Freenode: jdob @ #pulp
http://pulpproject.org | http://blog.pulpproject.org