[Freeipa-devel] Paging in Web UI

[Endi Sukma Dewata]

On 8/28/2012 8:30 AM, Petr Vobornik wrote:
> I would like to point out a problem in Web UI related to paging and
> suggest a possible solution:
>
> Current implementation of paging in Web UI is not serving it purpose
> which should be to do operations faster and be able to find all users.
>
> How current implementation should work:
>   1) get all primary keys (pkeys) of certain object type (users, ...)
>   2) load first 20 of them
>   3) from 1) we know record count so we can choose other page (different
> subset of 20)
>
> How it actually works:
>   * get up to 2000 keys, with more items a LDAP limit is reached and
> result is truncated
>   * we don't know how many are there, paging doesn't work as it should

We have a 'search limit' which can be specified by the user as a search 
parameter. It has a default value of 100 which can configured in IPA 
Server -> Configuration. There's also something like 'max result size' 
(I don't remember the actual name) of 2000 entries which is a global 
limit imposed by the LDAP backend. I'm not sure if the 'max result size' 
is configurable, but if I remember correctly if we want to get the 
entire entries in LDAP we'd need to use Simple Paged Results or VLV.

> Problem:
> Let's say that paging is working (no LDAP limit) and consider this: Our
> page size is 20 records. With 400 users it is 20 pages. With 2000 users
> (current search limit) it is 200 pages and so on. Personally I wouldn't
> like to go through 20 pages. 200 or more pages sounds like a punishment.

In the current UI we could jump to a particular page directly, so it's 
not necessary to go through all pages.

> There is also a question if we even need to do it. I don't see a benefit
> of paging here. For smaller setup an increase of page size may help
> admin to see more users on a page more quickly (rolling mouse wheel is
> faster than click on next page button).

We probably can make it look like a single continuous page, but we would 
still use paging to retrieve the entries. So initially it will show the 
entries of the first page, when you scroll down it will get the next 
page, but it will show the entries one-by-one by scrolling up the page. 
If none of the entries in the previous page are visible anymore the page 
can be removed from cache, but we might want to keep several pages in 
the cache.

If we implement this, the interface will look the same regardless of the 
page size or whether we enable/disable paging.

> For large setups, size of the pkey list must also be taken on mind. For
> each record there is a pkey and dn (dn is not required but it would need
> some hacking in python code to remove it). At the moment the list size
> raises drastically because JSON response includes indenting by spaces
> and is not compressed (easy to fix, but still there). With 10 000 or
> more users this pkey and dn list is pretty big (slow load on slower
> networks -vpns). This list is also loaded on each page change
> (record-wise, can be improved - cached).

Yes, ideally the pkey list should be cached and reloaded on new search 
or cache expiration. Or the server can do the query as John described, 
so the UI/CLI will just need to specify the page number.

> IMO with hundreds or thousands of users the most common use case is
> search by login or name or something. Paging is not required this case.
> It may be required if the result count is greater than size limit. In
> such case an option to temporary increase size limit or enable paging
> would be beneficial.

For large setup we might need to use simple paged results or VLV, but 
they have limitations and require server support. Here's a page I wrote 
sometime ago: http://freeipa.org/page/IPAv3_Directory_Browsing

> Apart from this case, paging should be off. It will
> speed up page load because it executes only one request.
>
> Possible solution:
> 1) I suggest to introduce configuration options for paging. They can be
> global (default for all search pages) or individual for pages or Web
> UI's users. Per-user configuration can be stored in browser (to not
> pollute LDAP with application stuff). Global configuration on server in
> config plugin (minor polluting).
>
> Those options should be (per page and global):
>   * enable paging
>   * page size
>
> Note: when paging is disabled page size is basically --sizelimit option.

Agreed. The actual number of entries would be different for each 
deployment and for each entity too, so it makes sense to use different 
settings to optimize the performance.

If we implement the single continuous page above this setting will not 
affect the visual, but it will affect the performance.

> 2) On search pages we should have controls to enable/disable paging and
> to change sizelimit for this particular moment.

If we keep the current UI users might want to save the setting (either 
on the server or on the client as you described) so they don't have to 
reconfigure it every time they log in. But if we implement the single 
continuous page above this might not be necessary anymore because users 
won't see the visual difference.

> 3) Compress JSON responses (on httpd level)

Yes.

> This way admin can configure default's for his deployment and user's can
> adjust for certain situations.
 >
> Btw always including member_xx attributes in find or show commands is
> not good for search pages either but that's another topic.
>
> Comments are welcome.
>
> Regards

I'm also writing a response for the other email.

-- 
Endi S. Dewata