[Pulp-list] pagination design

[Bryan Kearney]

On 03/11/2011 08:32 AM, Jay Dobies wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On 03/11/2011 08:14 AM, Todd B Sanders wrote:
>> On 03/11/2011 03:58 AM, Jason L Connor wrote:
>>> Hi All,
>>>
>>> A quick write up on pagination support for pulp. Comments welcome.
>>> https://fedorahosted.org/pulp/wiki/Pagination
>>>
>>>
>>> _______________________________________________
>>> Pulp-list mailing list
>>> Pulp-list at redhat.com
>>> https://www.redhat.com/mailman/listinfo/pulp-list
>> Thanks Jason.  Question.  Suppose I have 200 packages in a Repo, and I
>> want to present 25 packages per page to the end user.  Obviously this
>> would give me 8 pages.  What would happen if packages are added/removed
>> from the Repo while I am paging through the data?  Since we are hitting
>> mongo for each page set, won't this skew the paging?
>
> Interesting question. Thinking out loud, I think this is an inevitability.
>
> - - We can't preload them into memory, that would get us into trouble very
> quickly.
> - - We can't lock the database from changes while something is being
> paginated for a bunch of reasons.
> - - I worry using some sort of view-like concept in the database would get
> us into trouble as well. By view-like I mean we copy the data they are
> paginating over to some temporary area and they iterate over that. Then
> we'd have to get into expiration of that data, invalidation if they
> re-execute the same query, and reapers to clean it all up since they may
> not tell us they are done with the data set. It'd be kinda brutal
>
> That's just first thoughts on your question though, so don't take that
> as my final answer. More just thinking out loud.
>
>

If you look at many web sites (facebook, digg, reddit) they all suffer 
from this. I would suggest it is ok. As is not knowing the complete size 
of data set (see google)

-- bk