[Pulp-list] MongoDB document size errors

[Mike McCune]

I spent some time today tracking down the exception that a few people 
were seeing:

   InvalidDocument: document too large - BSON documents are limited to 4 MB

I wrote a unit test to create a repo and add 5,000 packages to the 
repository and easily hit the above error:

File "/home/mmccune/devel/pulp/test/unit/test_api.py", line 615, in 
test_sync_large_repo
     self.rapi.update(repo)
   File "/home/mmccune/devel/pulp/test/unit/../../src/pulp/auditing.py", 
line 166, in _audit
     _record_event()
   File "/home/mmccune/devel/pulp/test/unit/../../src/pulp/auditing.py", 
line 141, in _record_event
     _objdb.insert(event, safe=False, check_keys=False)
   File "build/bdist.linux-x86_64/egg/pymongo/collection.py", line 245, 
in insert
     message.insert(self.__full_name, docs, check_keys, safe), safe)
InvalidDocument: document too large - BSON documents are limited to 4 MB

Note the trace is contained within the auditing insert() call.

Turns out the Event class has both a 'params' field and a field called 
'action' but the action field also contained a copy of the params so for 
a large Repo document you get an Event document that is 2-3X the size of 
the actual parameter being audited.

Jason is working to trim this down but the short story is, we aren't 
hitting the 4MB limit because of the structure of our actual Repo 
document, we are hitting it because of the way we were auditing the 
parameters of a method call.

Jason also in his latest update took out the repo from the param logging 
in RepoApi.update()

     @audit(params=[])
     def update(self, repo_data):

Also, to make people feel more confortable, with the fix to the audit() 
call I was able to create a repo with 20,000 packages without error 
whereas before it would bail with a repo with 5,000.


-- 
Mike McCune
mmccune AT redhat.com
Red Hat Engineering       | Portland, OR
Systems Management        | 650.254.4248