[Pulp-list] Upgrade from v1 to v2 failed, twice

[Andreas Piesk]

Hello list,

today i tried to upgrade from v1 (1.1.15) to v2 (2.1.3), unfortunately 
it failed upgrading the database:

# pulp-v1-upgrade
= Upgrading Database =
Upgrading: CDS

Upgrading: Consumers

Upgrading: Event

Upgrading: Users, Permissions, and Roles

Upgrading: Tasks

Upgrading: Repositories, Content

Upgrading: File Repositories, Content

Upgrading: Repository Groups, Sync Schedules

Upgrading: Packages, Errata, and Distributions
Traceback (most recent call last):
   File "/usr/bin/pulp-v1-upgrade", line 50, in <module>
     upgrader.upgrade()
   File "/usr/lib/python2.6/site-packages/pulp/server/upgrade/main.py", 
line 187, in upgrade
     self._upgrade_database()
   File "/usr/lib/python2.6/site-packages/pulp/server/upgrade/main.py", 
line 228, in _upgrade_database
     report = db_call(v1_database, tmp_database)
   File 
"/usr/lib/python2.6/site-packages/pulp/server/upgrade/db/units.py", line 
121, in upgrade
     rpms_success = _rpms(v1_database, v2_database, report)
   File 
"/usr/lib/python2.6/site-packages/pulp/server/upgrade/db/units.py", line 
217, in _rpms
     return _packages(v1_database, v2_database, rpm_coll, all_rpms, 
'rpm', report)
   File 
"/usr/lib/python2.6/site-packages/pulp/server/upgrade/db/units.py", line 
247, in _packages
     'description' : v1_rpm['description'],
KeyError: 'description'

this is the culprit:

SON([(u'repoids', [u'oracle-jdk-x86_64']), (u'checksum', 
SON([(u'sha256', 
u'338a454a36b034bc1373dbdda00d99a3eeba184fb2b690d3a81adb3825812727')])), 
(u'filename', u'jdk-1.6.0_33-fcs.x86_64.rpm'), (u'epoch', u'2000'), 
(u'version', u'1.6.0_33'), (u'_ns', u'packages'), (u'release', u'fcs'), 
(u'_id', u'f516b651-f958-4d22-bd1d-e7d8ee66bcc4'), (u'arch', u'x86_64'), 
(u'id', u'f516b651-f958-4d22-bd1d-e7d8ee66bcc4'), (u'name', u'jdk')])

i "fixed" this by assigning defaults:

# diff -Purp 
/usr/lib/python2.6/site-packages/pulp/server/upgrade/db/units.py.org 
/usr/lib/python2.6/site-packages/pulp/server/upgrade/db/units.py
--- 
/usr/lib/python2.6/site-packages/pulp/server/upgrade/db/units.py.org 
2013-08-18 21:42:25.723204539 +0200
+++ /usr/lib/python2.6/site-packages/pulp/server/upgrade/db/units.py 
2013-08-18 22:02:27.581580546 +0200
@@ -244,13 +244,13 @@ def _packages(v1_database, v2_database,
              'version' : v1_rpm['version'],
              'release' : v1_rpm['release'],
              'arch' : v1_rpm['arch'],
-            'description' : v1_rpm['description'],
-            'vendor' : v1_rpm['vendor'],
+            'description' : v1_rpm.get('description',None),
+            'vendor' : v1_rpm.get('vendor', None),
              'filename' : v1_rpm['filename'],
-            'requires' : v1_rpm['requires'],
-            'provides' : v1_rpm['provides'],
-            'buildhost' : v1_rpm['buildhost'],
-            'license' : v1_rpm['license'],
+            'requires' : v1_rpm.get('requires', None),
+            'provides' : v1_rpm.get('provides', None),
+            'buildhost' : v1_rpm.get('buildhost', None),
+            'license' : v1_rpm.get('license', None),

              '_id' : new_rpm_id,
              '_content_type_id' : unit_type_id


but then, it failed again:

# pulp-v1-upgrade
= Upgrading Database =
Upgrading: CDS

Upgrading: Consumers

Upgrading: Event

Upgrading: Users, Permissions, and Roles

Upgrading: Tasks
[-]

Upgrading: Repositories, Content

Upgrading: File Repositories, Content

Upgrading: Repository Groups, Sync Schedules

Upgrading: Packages, Errata, and Distributions
Traceback (most recent call last):
   File "/usr/bin/pulp-v1-upgrade", line 50, in <module>
     upgrader.upgrade()
   File "/usr/lib/python2.6/site-packages/pulp/server/upgrade/main.py", 
line 187, in upgrade
     self._upgrade_database()
   File "/usr/lib/python2.6/site-packages/pulp/server/upgrade/main.py", 
line 228, in _upgrade_database
     report = db_call(v1_database, tmp_database)
   File 
"/usr/lib/python2.6/site-packages/pulp/server/upgrade/db/units.py", line 
125, in upgrade
     groups_success = _package_groups(v1_database, v2_database, report)
   File 
"/usr/lib/python2.6/site-packages/pulp/server/upgrade/db/units.py", line 
622, in _package_groups
     v2_coll.insert(new_group, safe=True)
   File "/usr/lib64/python2.6/site-packages/pymongo/collection.py", line 
312, in insert
     continue_on_error, self.__uuid_subtype), safe)
bson.errors.InvalidDocument: key 'openoffice.org-langpack-pt_PT' must 
not contain '.'


i believe, this was introduced in pymongo >2.0.1 and pulp v2 requires 2.1.

RHEL5 contains these openoffice packages and i believe (but i might be 
wrong), rpm allows dots.

Funny thing, this issue was mentioned in 
http://www.pulpproject.org/ug/UGFAQ.html. workaround for this is to 
downgrade.

The issue has been discovered some time ago, see 
http://osdir.com/ml/pulp-list/2013-06/msg00014.html.

Is there a solution? How is Pulp V2 supposed to work with these rpms and 
repos containing these rpms like RHEL5?


regards,
-ap