[Pulp-list] pulp-manage-db freezes on fresh installation?

Wed Jun 24 14:36:56 UTC 2015

Barnaby,

There was a previous installation, and I hadn't properly cleared out that 
directory. You picked it correctly.

Ben.

On 24 June 2015 11:33:08 pm Barnaby Court <bcourt at redhat.com> wrote:

> Ben,
>
> Is this a machine that had previously had Pulp installed? I'm wondering if 
> there were old /var/lib/pulp/published/yum/master/* directories that it 
> still had to scan? Migration 20 is scanning that directory and modifying 
> the contents. That should not be particularly intensive unless there is 
> something going on with the filesystem. Regards,
>
> -Barnaby
>
> ----- Original Message -----
> From: "ben stanley" <ben.stanley at exemail.com.au>
> To: "Ben Stanley" <Ben.Stanley at exemail.com.au>
> Cc: "Barnaby Court" <bcourt at redhat.com>, pulp-list at redhat.com
> Sent: Wednesday, June 24, 2015 1:37:31 AM
> Subject: Re: [Pulp-list] pulp-manage-db freezes on fresh installation?
>
> The machine with the problem was shut down yesterday.
>
> I tried pulp-manage-db again today and it worked.
>
> -----------------------------------------------------------------------------
> [root at stig scripts]# sudo -u apache pulp-manage-db
> Mongo database for connection is version 2.6.9
> Loading content types.
> Loading type descriptors [iso_support.json, rpm_support.json, puppet.json]
> Parsing type descriptors
> Validating type descriptor syntactic integrity
> Validating type descriptor semantic integrity
> Updating the database with types [iso, distribution, drpm, erratum,
> package_group, package_category, package_environment, rpm, srpm,
> yum_repo_metadata_file, puppet_module]
> Content types loaded.
> Ensuring the admin role and user are in place.
> Admin role and user are in place.
> Beginning database migrations.
> Migration package pulp.server.db.migrations is up to date at version 12
> Migration package pulp_puppet.plugins.migrations is up to date at version 2
> Applying pulp_rpm.plugins.migrations version 20
> Migration to pulp_rpm.plugins.migrations version 20 complete.
> Applying pulp_rpm.plugins.migrations version 21
> Migration to pulp_rpm.plugins.migrations version 21 complete.
> Database migrations complete.
> [root at stig scripts]#
> -----------------------------------------------------------------------------
>
> It took a while to do version 20. I walked away for an hour. When I got
> back it was done. I only waited 10min the first time. Perhaps it wasn't
> enough?
>
> I am wondering why it takes a long time to migrate a database version when
> there are no repos on this machine. It is a clean install.
>
> Ben.
>
> > Barnaby,
> >
> > Thanks for your reply.
> >
> > This is the secondary pulp system that I am trying to set up. There are NO
> > repositories on it yet (as I un-installed pulp and deleted /var/lib/pulp
> > and /var/lib/mongodb and then re-installed). It was while I was
> > re-installing that the update-db problem occurred.
> >
> > I am having a lot of trouble establishing a disconnected sync method, but
> > that is documented on plan.io . I look forward to you finding that...
> >
> > Ben.
> >
> >
> > On 22 June 2015 11:06:30 pm Barnaby Court <bcourt at redhat.com> wrote:
> >
> >> Hi Ben,
> >>
> >> How many published repositories do you have and how fast is their
> >> filesystem?  pulp_rpm migration 20 is fixing an issue with the directory
> >> structure created for published delta rpm files. If you have many
> >> published
> >> repositories on a slow filesystem this could take a while as it could be
> >> removing a lot of bad symlinks.
> >>
> >> -Barnaby
> >>
> >> ----- Original Message -----
> >> From: "ben stanley" <ben.stanley at exemail.com.au>
> >> To: pulp-list at redhat.com
> >> Sent: Monday, June 22, 2015 1:59:33 AM
> >> Subject: [Pulp-list] pulp-manage-db freezes on fresh installation?
> >>
> >> I had some trouble running pulp-manage-db on a fresh installation.
> >>
> >> The update process locks up at update 20.
> >>
> >> I have provided a script of what I did (attached pulp_installation.txt).
> >> I can reproduce it by removing all the packages, cleaning
> >> out /var/lib/mongodb, and running the pulp installation again from the
> >> top.
> >>
> >> I looked in the log, but there wasn't anything illuminating.
> >>
> >> What could be causing this?
> >>
> >> Ben.
> >>
> >> ------------------------ /var/log/messages -----------------------------
> >> pulp: pulp.server.db.manage:INFO: Migration to
> >> pulp_rpm.plugins.migrations version 16 complete.
> >> pulp: pulp.server.db.manage:INFO: Applying pulp_rpm.plugins.migrations
> >> version 17
> >> pulp: pulp.server.db.manage:INFO: Migration to
> >> pulp_rpm.plugins.migrations version 17 complete.
> >> pulp: pulp.server.db.manage:INFO: Applying pulp_rpm.plugins.migrations
> >> version 18
> >> pulp: pulp.server.db.manage:INFO: Migration to
> >> pulp_rpm.plugins.migrations version 18 complete.
> >> pulp: pulp.server.db.manage:INFO: Applying pulp_rpm.plugins.migrations
> >> version 19
> >> pulp: pulp.server.db.manage:INFO: Migration to
> >> pulp_rpm.plugins.migrations version 19 complete.
> >> pulp: pulp.server.db.manage:INFO: Applying pulp_rpm.plugins.migrations
> >> version 20
> >> ------------------------------------------------------------------------
> >>
> >> ---------------- Output of just the pulp-manage-db ---------------------
> >> [root at stig lib]# sudo -u apache pulp-manage-db
> >> Mongo database for connection is version 2.6.9
> >> Loading content types.
> >> Loading type descriptors [iso_support.json, rpm_support.json,
> >> puppet.json]
> >> Parsing type descriptors
> >> Validating type descriptor syntactic integrity
> >> Validating type descriptor semantic integrity
> >> Updating the database with types [iso, distribution, drpm, erratum,
> >> package_group, package_category, package_environment, rpm, srpm,
> >> yum_repo_metadata_file, puppet_module]
> >> Content types loaded.
> >> Ensuring the admin role and user are in place.
> >> Admin role and user are in place.
> >> Beginning database migrations.
> >> Applying pulp.server.db.migrations version 1
> >> Migration to pulp.server.db.migrations version 1 complete.
> >> Applying pulp.server.db.migrations version 2
> >> Migration to pulp.server.db.migrations version 2 complete.
> >> Applying pulp.server.db.migrations version 3
> >> Migration to pulp.server.db.migrations version 3 complete.
> >> Applying pulp.server.db.migrations version 4
> >> regenerating content unit counts for 0 repositories
> >> Migration to pulp.server.db.migrations version 4 complete.
> >> Applying pulp.server.db.migrations version 5
> >> Migration to pulp.server.db.migrations version 5 complete.
> >> Applying pulp.server.db.migrations version 6
> >> Migration to pulp.server.db.migrations version 6 complete.
> >> Applying pulp.server.db.migrations version 7
> >> Migration to pulp.server.db.migrations version 7 complete.
> >> Applying pulp.server.db.migrations version 8
> >> Migration to pulp.server.db.migrations version 8 complete.
> >> Applying pulp.server.db.migrations version 9
> >> Migration to pulp.server.db.migrations version 9 complete.
> >> Applying pulp.server.db.migrations version 10
> >> Migration to pulp.server.db.migrations version 10 complete.
> >> Applying pulp.server.db.migrations version 11
> >> Migration to pulp.server.db.migrations version 11 complete.
> >> Applying pulp.server.db.migrations version 12
> >> Migration to pulp.server.db.migrations version 12 complete.
> >> Applying pulp_puppet.plugins.migrations version 1
> >> Migrated puppet modules to include checksum
> >> Migration to pulp_puppet.plugins.migrations version 1 complete.
> >> Applying pulp_puppet.plugins.migrations version 2
> >> Migration to pulp_puppet.plugins.migrations version 2 complete.
> >> Applying pulp_rpm.plugins.migrations version 1
> >> Export distributor migration for rpm repositories started
> >> Export distributor migration for rpm repositories finished
> >> Migration to pulp_rpm.plugins.migrations version 1 complete.
> >> Applying pulp_rpm.plugins.migrations version 2
> >> Migrated rpms to include rpm metadata
> >> Migration to pulp_rpm.plugins.migrations version 2 complete.
> >> Applying pulp_rpm.plugins.migrations version 3
> >> Migration to pulp_rpm.plugins.migrations version 3 complete.
> >> Applying pulp_rpm.plugins.migrations version 4
> >> Migration to pulp_rpm.plugins.migrations version 4 complete.
> >> Applying pulp_rpm.plugins.migrations version 5
> >> Migrated rpms to include rpm changelog and filelist metadata
> >> Migration to pulp_rpm.plugins.migrations version 5 complete.
> >> Applying pulp_rpm.plugins.migrations version 6
> >> Migration to pulp_rpm.plugins.migrations version 6 complete.
> >> Applying pulp_rpm.plugins.migrations version 7
> >> No yum repositories found to inventory custom metadata on
> >> Migration to pulp_rpm.plugins.migrations version 7 complete.
> >> Applying pulp_rpm.plugins.migrations version 8
> >> Migration to pulp_rpm.plugins.migrations version 8 complete.
> >> Applying pulp_rpm.plugins.migrations version 9
> >> Migration to pulp_rpm.plugins.migrations version 9 complete.
> >> Applying pulp_rpm.plugins.migrations version 10
> >> Migration to pulp_rpm.plugins.migrations version 10 complete.
> >> Applying pulp_rpm.plugins.migrations version 11
> >> Migration to pulp_rpm.plugins.migrations version 11 complete.
> >> Applying pulp_rpm.plugins.migrations version 12
> >> Migration to pulp_rpm.plugins.migrations version 12 complete.
> >> Applying pulp_rpm.plugins.migrations version 13
> >> Migration to pulp_rpm.plugins.migrations version 13 complete.
> >> Applying pulp_rpm.plugins.migrations version 14
> >> Migration to pulp_rpm.plugins.migrations version 14 complete.
> >> Applying pulp_rpm.plugins.migrations version 15
> >> stripping repomd.xml checksum
> >> from /var/lib/pulp/content/distribution/ks-Red Hat Enterprise Linux
> >> Client-5.7-i386/treeinfo
> >> stripping repomd.xml checksum
> >> from /var/lib/pulp/content/distribution/ks-Red Hat Enterprise Linux
> >> Client-5.7-x86_64/treeinfo
> >> stripping repomd.xml checksum
> >> from /var/lib/pulp/content/distribution/ks-Red Hat Enterprise Linux
> >> Client-5.8-i386/treeinfo
> >> stripping repomd.xml checksum
> >> from /var/lib/pulp/content/distribution/ks-Red Hat Enterprise Linux
> >> Client-5.8-x86_64/treeinfo
> >> stripping repomd.xml checksum
> >> from /var/lib/pulp/content/distribution/ks-Red Hat Enterprise
> >> Linux-Workstation-6.1-i386/treeinfo
> >> stripping repomd.xml checksum
> >> from /var/lib/pulp/content/distribution/ks-Red Hat Enterprise
> >> Linux-Workstation-6.1-x86_64/treeinfo
> >> stripping repomd.xml checksum
> >> from /var/lib/pulp/content/distribution/ks-Red Hat Enterprise
> >> Linux-Workstation-6.2-i386/treeinfo
> >> stripping repomd.xml checksum
> >> from /var/lib/pulp/content/distribution/ks-Red Hat Enterprise
> >> Linux-Workstation-6.2-x86_64/treeinfo
> >> stripping repomd.xml checksum
> >> from /var/lib/pulp/content/distribution/ks-Red Hat Enterprise
> >> Linux-Workstation-6.3-i386/treeinfo
> >> stripping repomd.xml checksum
> >> from /var/lib/pulp/content/distribution/ks-Red Hat Enterprise
> >> Linux-Workstation-6.3-x86_64/treeinfo
> >> stripping repomd.xml checksum
> >> from /var/lib/pulp/content/distribution/ks-Red Hat Enterprise
> >> Linux-Workstation-6.4-i386/treeinfo
> >> stripping repomd.xml checksum
> >> from /var/lib/pulp/content/distribution/ks-Red Hat Enterprise
> >> Linux-Workstation-6.4-x86_64/treeinfo
> >> examining distribution units
> >> Migration to pulp_rpm.plugins.migrations version 15 complete.
> >> Applying pulp_rpm.plugins.migrations version 16
> >> Loaded plugin yum_distributor for types:
> >> 
> rpm,srpm,drpm,erratum,package_group,package_category,distribution,yum_repo_metadata_file
> >> Loaded plugin export_distributor for types:
> >> rpm,srpm,drpm,erratum,distribution,package_category,package_group
> >> Loaded plugin iso_distributor for types: iso
> >> Loaded plugin puppet_distributor for types: puppet_module
> >> Loaded plugin puppet_file_distributor for types: puppet_module
> >> Loaded plugin puppet_install_distributor for types: puppet_module
> >> Loaded plugin group_export_distributor for types:
> >> rpm,srpm,drpm,erratum,distribution,package_category,package_group
> >> Loaded plugin yum_importer for types:
> >> 
> distribution,drpm,erratum,package_group,package_category,rpm,srpm,yum_repo_metadata_file,package_environment
> >> Loaded plugin iso_importer for types: iso
> >> Loaded plugin puppet_importer for types: puppet_module
> >> Loaded plugin yum_profiler for types: rpm,erratum
> >> Loaded plugin puppet_whole_repo_profiler for types: puppet_module
> >> Loaded plugin yum for types: rpm
> >> Loaded plugin rhui for types: rpm
> >> Migration to pulp_rpm.plugins.migrations version 16 complete.
> >> Applying pulp_rpm.plugins.migrations version 17
> >> Migration to pulp_rpm.plugins.migrations version 17 complete.
> >> Applying pulp_rpm.plugins.migrations version 18
> >> Migration to pulp_rpm.plugins.migrations version 18 complete.
> >> Applying pulp_rpm.plugins.migrations version 19
> >> Migration to pulp_rpm.plugins.migrations version 19 complete.
> >> Applying pulp_rpm.plugins.migrations version 20
> >>
> >> <waited 10min>
> >> ------------------------------------------------------------------------
> >>
> >>
> >> _______________________________________________
> >> Pulp-list mailing list
> >> Pulp-list at redhat.com
> >> https://www.redhat.com/mailman/listinfo/pulp-list
> >
> >
> > _______________________________________________
> > Pulp-list mailing list
> > Pulp-list at redhat.com
> > https://www.redhat.com/mailman/listinfo/pulp-list
> >
>
>