[Pulp-list] Upgrade from v1 to v2 failed

[Andreas Piesk]

Hello list,

i tried to upgrade from v1 (1.1.15) to v2 (2.1.3), unfortunately it 
failed upgrading the database. these are the issue i encountered:

1. missing package attributes

some packages miss some attributes which pulp depends on.

# pulp-v1-upgrade
= Upgrading Database =
..
Upgrading: Packages, Errata, and Distributions
Traceback (most recent call last):
    File "/usr/bin/pulp-v1-upgrade", line 50, in <module>
      upgrader.upgrade()
    File "/usr/lib/python2.6/site-packages/pulp/server/upgrade/main.py", 
line 187, in upgrade
      self._upgrade_database()
    File "/usr/lib/python2.6/site-packages/pulp/server/upgrade/main.py", 
line 228, in _upgrade_database
      report = db_call(v1_database, tmp_database)
    File 
"/usr/lib/python2.6/site-packages/pulp/server/upgrade/db/units.py", line 
121, in upgrade
      rpms_success = _rpms(v1_database, v2_database, report)
    File 
"/usr/lib/python2.6/site-packages/pulp/server/upgrade/db/units.py", line 
217, in _rpms
      return _packages(v1_database, v2_database, rpm_coll, all_rpms, 
'rpm', report)
   File 
"/usr/lib/python2.6/site-packages/pulp/server/upgrade/db/units.py", line 
247, in _packages
      'description' : v1_rpm['description'],
KeyError: 'description'

this is one of the the culprits:

SON([(u'repoids', [u'oracle-jdk-x86_64']), (u'checksum',
SON([(u'sha256', 
u'338a454a36b034bc1373dbdda00d99a3eeba184fb2b690d3a81adb3825812727')])),
(u'filename', u'jdk-1.6.0_33-fcs.x86_64.rpm'), (u'epoch', u'2000'),
(u'version', u'1.6.0_33'), (u'_ns', u'packages'), (u'release', u'fcs'),
(u'_id', u'f516b651-f958-4d22-bd1d-e7d8ee66bcc4'), (u'arch', u'x86_64'),
(u'id', u'f516b651-f958-4d22-bd1d-e7d8ee66bcc4'), (u'name', u'jdk')])

i fixed it by assigning defaults:

# diff -Purp
/usr/lib/python2.6/site-packages/pulp/server/upgrade/db/units.py.org
/usr/lib/python2.6/site-packages/pulp/server/upgrade/db/units.py
--- /usr/lib/python2.6/site-packages/pulp/server/upgrade/db/units.py.org
2013-08-18 21:42:25.723204539 +0200
+++ /usr/lib/python2.6/site-packages/pulp/server/upgrade/db/units.py
2013-08-18 22:02:27.581580546 +0200
@@ -244,13 +244,13 @@ def _packages(v1_database, v2_database,
               'version' : v1_rpm['version'],
               'release' : v1_rpm['release'],
               'arch' : v1_rpm['arch'],
-            'description' : v1_rpm['description'],
-            'vendor' : v1_rpm['vendor'],
+            'description' : v1_rpm.get('description',None),
+            'vendor' : v1_rpm.get('vendor', None),
               'filename' : v1_rpm['filename'],
-            'requires' : v1_rpm['requires'],
-            'provides' : v1_rpm['provides'],
-            'buildhost' : v1_rpm['buildhost'],
-            'license' : v1_rpm['license'],
+            'requires' : v1_rpm.get('requires', None),
+            'provides' : v1_rpm.get('provides', None),
+            'buildhost' : v1_rpm.get('buildhost', None),
+            'license' : v1_rpm.get('license', None),

               '_id' : new_rpm_id,
               '_content_type_id' : unit_type_id


2. package names with '.'

# pulp-v1-upgrade
..
Upgrading: Packages, Errata, and Distributions
Traceback (most recent call last):
    File "/usr/bin/pulp-v1-upgrade", line 50, in <module>
      upgrader.upgrade()
    File "/usr/lib/python2.6/site-packages/pulp/server/upgrade/main.py", 
line 187, in upgrade
      self._upgrade_database()
    File "/usr/lib/python2.6/site-packages/pulp/server/upgrade/main.py", 
line 228, in _upgrade_database
      report = db_call(v1_database, tmp_database)
    File 
"/usr/lib/python2.6/site-packages/pulp/server/upgrade/db/units.py", line 
125, in upgrade
      groups_success = _package_groups(v1_database, v2_database, report)
    File 
"/usr/lib/python2.6/site-packages/pulp/server/upgrade/db/units.py", line 
622, in _package_groups
      v2_coll.insert(new_group, safe=True)
    File "/usr/lib64/python2.6/site-packages/pymongo/collection.py", 
line 312, in insert
      continue_on_error, self.__uuid_subtype), safe)
bson.errors.InvalidDocument: key 'openoffice.org-langpack-pt_PT' must 
not contain '.'

I believe, this was introduced in pymongo >2.0.1 and pulp v2 requires 2.1.

RHEL5 contains these openoffice packages and i believe (but i might be 
wrong), rpm allows dots.

Funny thing, this issue was mentioned in 
http://www.pulpproject.org/ug/UGFAQ.html. workaround for this is to 
downgrade.

The issue has been discovered some time ago, see 
http://osdir.com/ml/pulp-list/2013-06/msg00014.html.

To sucessfully upgrade i have to set check_keys=False (w=1 is the same 
as safe=True on recent versions of pymongo):

--- 
/usr/lib/python2.6/site-packages/pulp/server/upgrade/db/units.py.org 
2013-08-18 21:42:25.723204539 +0200
+++ /usr/lib/python2.6/site-packages/pulp/server/upgrade/db/units.py 
2013-08-20 21:57:16.201263427 +0200
@@ -619,7 +619,7 @@ def _package_groups(v1_database, v2_data
                  'translated_name' : v1_group['translated_name'],
                  'user_visible' : v1_group['user_visible'],
              }
-            v2_coll.insert(new_group, safe=True)
+            v2_coll.insert(new_group, check_keys=False, w=1)

              new_association = {
                  '_id' : ObjectId(),


Unfortunately "check_keys=False" must be set on every insert, save, 
update, upsert.


3. upgrade documentation

the upgrade documentation 
https://pulp-user-guide.readthedocs.org/en/pulp-2.1/v1_upgrade.html is 
missing a important step: running pulp-manage-db after db upgrade and 
updating pulp to the last stable version.



Do you want pull requests for issue 1 and/or issue 2?

Regards,
-ap