[Freeipa-devel] Paging in Web UI

[Petr Vobornik]

On 08/28/2012 08:31 PM, Endi Sukma Dewata wrote:
> On 8/28/2012 11:23 AM, Petr Vobornik wrote:
>>> Your possible solution does not address how many results are fetched
>>> (unless I misunderstood).
>  >
>> If paging is enabled it doesn't, but it expects, that admin will disable
>> it for larger setups.For smaller setups it isn't of much an issue. If
>> paging is disabled, the limit is server 'search size limit' or
>> --sizelimit option supplied by Web UI.
>>
>>> I'm not sure how per-user preferences are handled in browsers, but don't
>>> forget we now have session support in the server. Server session data is
>>> available for use.
>>
>> I was thinking about using browser local storage (basically key-value
>> DB). It has a benefit over session, that it survives a browser restart
>> but it should contain only non-sensitive data (other users may see it).
>
> You mean browser cookie?

No. https://developer.mozilla.org/en-US/docs/DOM/Storage

Petr Spacek also suggested to have an attribute in user object to store 
IPA specific data as JSON. Therefore changing browser wouldn't be an issue.
>
>>> If you don't want to fetch every record to implement paging smartly in
>>> conjunction with it's performance issues why not do the query on the
>>> server, cache the results in the session, and have the RPC return the
>>> total number of results plus a subset of the result. Another RPC could
>>> retrieve the next/previous subset of results from the cached result in
>>> the session.
>>
>> I think most software do paging like this. I don't know the reasons for
>> not doing it that way first time. My solution was counting with that we
>> still don't want to do it. Endi do you know the reasons? Earlier
>> sessions didn't exists, but it is doable without them too.
>
> Yes, at the time the sessions didn't exist and we needed a
> quick/temporary solution. I agree ideally this should be done on the
> server side, but how would the server maintain the paging information?
>
> The server can keep the search result (either just the pkey list or the
> entire entries) in memcached, but the result might be large and there
> might be multiple users accessing multiple search pages, so the total
> memory requirement could be large.

I can imagine that if used only by admins it may be OK but if such 
functionality would be used by other applications it may become problem.

>
> If we don't want the server to keep the search result, the server could
> rerun the search and return only the requested page and discard most of
> the results each time the user change page. This may affect server
> performance.
>
> We can also use Simple Paged Results, but if I understood correctly it
> requires the httpd to maintain an open connection to the LDAP server for
> each user and for each page. I'm not sure memcached can be used to move
> the connection object among forked httpd processes. Also Simple Paged
> Results can only go forward, so no Prev button unless somebody keeps the
> results.

Seems unusable.

>
> Another possibility is to use VLV, but it seems to require open
> connection too and only works with a predefined filter.

Apart from the possible connection problem I think it is very usable for 
default views of search pages. User can freely page. I don't understand 
from the wiki page if it can reasonably obtain total record count.

>
> Here's the page I wrote about this:
> http://freeipa.org/page/IPAv3_Directory_Browsing
>
> Currently we're using Option #2 but I'm not sure if we use SPR between
> httpd and the LDAP server. But even with SPR the result is still bound
> by some limits. It looks you have to connect as Directory Manager to get
> the complete list of pkey list/entries. See the table in this page:

This is one of the main reasons why I say it is broken.

>
> http://directory.fedoraproject.org/wiki/Simple_Paged_Results_Design
>
> The current implementation (keeping the pkey list in the UI) is
> conceptually similar to the front-end approach described in the above page.
>

IMO a modification of a hybrid solution may be best:

When user opens page a VLV result would be shown. User can page and see 
all the data. Currently we don't support custom sorting so disability of 
custom sorting isn't an issue. With predefined sorts we can even 
improved these possibilities.

If there is many records and user don't want to go through pages he will 
use search. If the search string is good he should fit to search limit 
so I wouldn't use paging at this moment. If not the case and he really 
don't know how to improve the filter he should have option to enable 
paging (current implementation) and go through the records by hand. Such 
record set should fit to limit of 2000 records. If not the filter is 
really bad.

If we don't want to implement VLV or until that time I would set 
not-paging as default setting.

>>> I don't think there any need in JSON formatted data for pretty printing
>>> with indentation. Is it an accident or oversight we're pretty printing
>>> the JSON data in an RPC. For large data sets compression would be a win,
>>> but most of our RPC communication is not large. Compression has an
>>> overhead. With small data you'll use more cycles compressing and
>>> uncompressing than you would sending verbose but small data blobs.
>>> Compression should be an RPC option specified by either side of the
>>> connection and the receiver should be prepared to conditionally
>>> uncompress based on a flag value in the RPC. If you use server session
>>> data to support paging you may not need to introduce compression since
>>> you would only be passing one page of data at a time.
>>>
>>> User specified page size (without limitation) is an absolute necessity.
>>> I am frequently annoyed by web sites which either do not allow me to
>>> specify page size or constrain it to ridiculous hard coded limits such
>>> as 10, 20 or 30.
>>
>> +1
>
> Agreed, but if we implement the single continuous page I described in
> the earlier email the page size won't be much of an issue anymore.
>
It may matter. For example when testing Web UI permissions. It often 
requires to navigate to the last page. I can enter the number directly 
and press enter or extend it for times (hypothetically) but I would 
rather see those 90 permissions right away because I can be at the list 
end in 0.5s.

We can also improve pager to offer some subset of pages. For example:

First Prev 22 23 *24* 25 26 Next Last Page [   ]
where 24 is current page.

Regarding the continuous paging: I would extend the page by clicking at 
a bar at the end of the list and/or by mouse-scrolling at the end of the 
list?
-- 
Petr Vobornik