[Pulp-list] MongoDB document size errors

[Pradeep Kilambi]

On 08/02/2010 11:00 PM, Mike McCune wrote:
> I spent some time today tracking down the exception that a few people 
> were seeing:
>
>   InvalidDocument: document too large - BSON documents are limited to 
> 4 MB
>
> I wrote a unit test to create a repo and add 5,000 packages to the 
> repository and easily hit the above error:
>
> File "/home/mmccune/devel/pulp/test/unit/test_api.py", line 615, in 
> test_sync_large_repo
>     self.rapi.update(repo)
>   File 
> "/home/mmccune/devel/pulp/test/unit/../../src/pulp/auditing.py", line 
> 166, in _audit
>     _record_event()
>   File 
> "/home/mmccune/devel/pulp/test/unit/../../src/pulp/auditing.py", line 
> 141, in _record_event
>     _objdb.insert(event, safe=False, check_keys=False)
>   File "build/bdist.linux-x86_64/egg/pymongo/collection.py", line 245, 
> in insert
>     message.insert(self.__full_name, docs, check_keys, safe), safe)
> InvalidDocument: document too large - BSON documents are limited to 4 MB
>
> Note the trace is contained within the auditing insert() call.
>
> Turns out the Event class has both a 'params' field and a field called 
> 'action' but the action field also contained a copy of the params so 
> for a large Repo document you get an Event document that is 2-3X the 
> size of the actual parameter being audited.
>
> Jason is working to trim this down but the short story is, we aren't 
> hitting the 4MB limit because of the structure of our actual Repo 
> document, we are hitting it because of the way we were auditing the 
> parameters of a method call.
>
> Jason also in his latest update took out the repo from the param 
> logging in RepoApi.update()
>
>     @audit(params=[])
>     def update(self, repo_data):
>
> Also, to make people feel more confortable, with the fix to the 
> audit() call I was able to create a repo with 20,000 packages without 
> error whereas before it would bail with a repo with 5,000.
>
>

Ah nice find Mike. Thanks for tracking this down.

~ Prad