[Pulp-list] New Source Layout

Wed Jun 13 18:12:12 UTC 2012

pulp-dev.py creates /usr/lib/pulp/agent/handlers,
but then tries to install symlinks into /usr/lib/pulp/agent/handler (lack of
's').

I'm not sure which is right.  I assume it's working for everyone b/c the
handler dir is still around from previous setups, but I'm actually trying in a
pristine environment, so it's failing for me.

On Wed, Jun 13, 2012 at 09:03:13AM -0400, Jay Dobies wrote:
> I think we're in a good place to get everyone back on master. The
> migration process for your dev environment isn't the simplest, but
> hopefully this is the last time we have to change the codebase to
> this extent. I'll update the wiki for new developers later; this is
> meant for those of us dealing with the transition.
> 
> Yes, this e-mail is long, even if you've come to expect that from
> me. But please make sure you read this one.  :)
> 
> 
> = Migrate =
> 
> You're going to lose everything in this process, but ultimately it's
> a dev environment, so you probably don't have much in there to begin
> with.
> 
> The first steps are going to be to remove the old Pulp dev install.
> From what I hear, there's issues in pulp-dev's uninstall
> capabilities when dealing with symlinks, so for simplicity we'll do
> a few manual steps:
> 
> 1. $ sudo systemctl stop httpd.service
> 2. $ sudo rm -rf /etc/pulp/*
> 3. $ sudo rm -rf /var/lib/pulp/*
> 4. $ sudo rm /usr/bin/pulp-*
> 5. Drop your database entirely:
>    $ mongo
>    $ use pulp_database
>    $ db.dropDatabase()
> 
> Now for the new stuff. If you don't have any outstanding code, it
> might be quicker to just move your code out to a temp location and
> re-clone the repo. It will save you from having old stuff lying
> around.
> 
> 1. $ git pull
> 
> The next few steps are to clear out things that git doesn't know
> about, such as .pyc files. The following should be run from the git
> checkout root. To be clear: do not do this if you have outstanding
> changes to merge in. That should only be prad/john with plugin
> changes but everyone think before you run these (all of this code
> has moved and you'll need to merge your changes in elsewhere, more
> on that later).
> 
> 2. $ rm -rf src  # yes, you're deleting src in root; it's moved
> 3. $ rm -rf plugins
> 4. $ rm -rf extensions
> 
> Install the dev environment. You'll need to hit both setup.py files
> in platform and rpm_support to get all of the site-packages
> references created.
> 
> 5. $ sudo python ./pulp-dev.py -I
> 6. $ cd platform/src && sudo python ./setup.py develop
> 7. $ cd rpm-support/src && sudo python ./setup.py develop
> 
> At this point, things should largely be back to normal:
> 
> 8. $ sudo pulp-migrate  # note the version is back to 1 but I'll
> address that soon
> 
> I found it easier to clean up my logs so I knew of any new issues coming up:
> 
> 9a. (optional) $ rm -f /var/log/httpd/*
> 9b. (optional) $ rm -f /var/log/pulp/*
> 
> 10. $ sudo systemctl start httpd.service
> 
> 
> = Validation =
> 
> Keep an eye on /var/log/httpd/error_log to make sure the server
> starts correctly. It won't if the plugins/type definitions aren't
> happy. If it doesn't, ping me, but also check:
> 
> /var/lib/pulp/plugins/{distributors,importers,types}
> 
> to make sure there aren't any symlinks in there that don't point to
> actual files.
> 
> 
> = Running the Client =
> 
> Apache's mod_ssl seems to have changed a bit in Fedora 17. It used
> to create a default cert with the CN "localhost.localdomain". Now,
> at least on my box, the CN is the hostname of the box itself.
> 
> That's going to cause a problem when trying to run the client. The
> host in admin.conf needs to match the CN. We ship with
> localhost.localdomain by default but that's not sufficient anymore.
> 
> That means you'll need to change the host in
> /etc/pulp/admin/admin.conf in order to do anything. The problem is
> that that file is symlinked into the codebase, so if you edit it
> directly you may accidentally commit it.
> 
> If you create ~/.pulp/admin.conf it will use that as overrides
> (~/.pulp/consumer.conf is the corresponding consumer override, but I
> may remove that since it doesn't really make sense to scope anything
> consumer related to a specific user). The contents should be:
> 
> [server]
> host = sunflower
> # always makes me smile that sayli named her desktop "sunflower" :)
> 
> 
> = Where is Everything? =
> 
> I made a slight change to what was proposed last week. The bulk of
> our codebase is organized into three subprojects:
> 
> platform - All of the plumbing, including server, client, and agent.
> If you ever find yourself mentioning
> rpm/errata/kickstart/puppet/other_type in here, it's wrong.
> 
> rpm-support - RPM pluigns/extensions/handlers. If you ever find
> yourself mentioning rpm/errata/kickstart _outside_ of this
> directory, it's wrong.
> 
> builtins - The generic extensions/plugins/handlers that ship as part
> of the Pulp platform. These cover things unrelated to types, such as
> the create repo that doesn't imply an importer or simple
> login/logout functionality.
> 
> In the future, we'll be adding subprojects for ISO support and
> puppet support. Maybe even Windows support if Todd holds my children
> ransom and forces me to.
> 
> Within each of the three subprojects there are a few consistent pieces:
> 
> src - Source code.
> test/unit - Source code for unit tests
> test/unit/data - Data needed for unit tests.
> etc - Mimics the structure where any files are deployed.
> 
> For the plugin-related subprojects, you'll also find:
> 
> extensions
> handlers
> plugins/{distributors,importers,profilers,types}
> 
> 
> = Tests =
> 
> What the above means is that we have test cases in three different
> places now (one per subproject). This is actually a benefit in many
> ways. During development in a particular subproject it's dirt simple
> and quick as hell to run just the tests for that subproject. We're
> in a much better position for test-driven development practices; we
> won't grow old while running the full suite as a sanity check.
> 
> I'm serious about quicker. The platform tests run in 60-70 seconds.
> RPM support runs in around 90. And after stripping out all of the
> untested v1 code, we're floating around 71% total project code
> coverage. That's actually pretty damn awesome. It's not gonna stop
> me from bitching that we need to raise that number, but I do
> acknowledge that it's a very healthy start.
> 
> Back to the three locations. There's a script in root called
> run-tests.sh that runs the tests in all three locations and
> generates a coverage report. The coverage report is under cover in
> the git root (pull up index.html in there for the slick HTML view
> with the colored markers of uncovered code).
> 
> That said, don't run that script yet. There's currently an issue in
> the RPM support tests that cause them to barf a ton of directories
> into your working directory and not clean them up, so running that
> script results in a bunch of garbage thrown into your git checkout.
> We'll get those fixed ASAP.
> 
> Something else to keep in mind. That script will run all three
> suites in the same nosetests process. Jenkins, however, will
> intentionally run them in three separate shells. My fear is that the
> single process run will allow dependencies to bleed across
> subprojects and not be detected. But at the same time, I dig the
> full project coverage report. So jenkins will be our safety net for
> making sure the process divisions are in place between tests in case
> you ever hit an issue where jenkins indicates a failure but the
> run-tests.sh script does not.
> 
> More on testing in a future e-mail; I have a bunch of comments about
> things we've done wrong in the past and how the new base classes
> work.
> 
> 
> = Random Notes =
> 
> * pulp-consumer is broken right now. There are a few bugs in play.
> It's always been using the admin certificate instead of its consumer
> certificate, and when I fixed that it's now failing authentication.
> A quick look at the code makes me think the auth code isn't looking
> at the v2 consumers collection, but I really didn't look closely.
> I'll be addressing this later today.
> 
> * I stripped out anything not currently being used. We still have it
> in git history if we need to look again, but the goal here was to be
> as clean as possible. That means things related to CDS, auditing,
> event sending, the exporter, and the recent package migration script
> are all deleted. We can always look up the v1 implementations when
> we go to address them in v2 but I'd rather not have such a large
> sweeping clean up again so took care of them now.
> 
> * Jeff's still working on the .spec files. There will be two: one
> for Pulp platform and one for RPM support, each producing a number
> of RPMs. I've seen the work done so far and they look great; much
> better organization by created RPM rather than by spec section. Not
> sure of his current status but I suspect we'll be building by the
> end of the week.
> 
> 
> I think that's all I have for now. Ping me if you run into any
> problems with the migration. I'm thinking of setting up an
> elluminate for the whole day that I'll just sit in and people can
> jump in/screen share when they run into specific problems, but we'll
> see how it goes.
> 
> -- 
> Jay Dobies
> Freenode: jdob @ #pulp
> http://pulpproject.org | http://blog.pulpproject.org
> 
> _______________________________________________
> Pulp-list mailing list
> Pulp-list at redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-list
--
-- James Slagle
--