[katello-devel] search over rest api - interface design

[Brad Buckingham]

On 07/18/2011 02:52 AM, Amos Benari wrote:
>
> ----- Original Message -----
>> From: "Justin Sherrill"<jsherril at redhat.com>
>> To: katello-devel at redhat.com
>> Sent: Monday, July 18, 2011 4:28:17 AM
>> Subject: Re: [katello-devel] search over rest api - interface design
>> On 07/17/2011 11:32 AM, Amos Benari wrote:
>>> ----- Original Message -----
>>>> From: "Lukas Zapletal"<lzap at redhat.com>
>>>> To: katello-devel at redhat.com
>>>> Sent: Thursday, July 14, 2011 4:07:24 PM
>>>> Subject: Re: [katello-devel] search over rest api - interface
>>>> design
>>>> On 07/14/2011 12:01 PM, Amos Benari wrote:
>>>    ...
>>>> Well I am surprised that scoped_search is not a fulltext. As a
>>>> long-time Apache Lucene user I have to recommend this project.
>>> Lucene vs. scoped search
>>> -------------------------
>>> I am not really going to compere them because it's a bit like
>>> comparing oranges to apples.
>>> I am just going to write a short description of scoped search and
>>> try to point what use case will each fit.
>>>
>>> scoped_search is not an indexer, it's also not a data store. It's a
>>> light weight query parser, auto-completer and SQL query builder.
>>> It enables feature rich search box in the GUI and a powerful API, on
>>> top of any RDBMS, no schema change is needed.
>>> scoped search on top of RDBMS can easily handle structured data, it
>>> returns updated data as soon as it's stored in the database.
>>> No scoring, no ranking, just a simple "order by" clause in the SQL.
>>> It will scale as much as the underline database does.
>>>
>>> Lucene will index your data, store it in a document format optimized
>>> for searching.
>>> It will scale-out well as long as you can live with eventual
>>> consistency.
>>>
>>> So does Lucene it fit our needs?
>>> In Kalpana, Foreman and Candlepin the data set is not large enough
>>> to make a scale issue.
>> Is this true? From earlier discussions I seem to remember that Katello
>> should be able to support ~100,000 systems eventually. I would think
>> this could indicate scale may be an issue?
> Well, I guess that if the rest of the application is going to behave nice with
>   ~100K systems then a search over the database should be fine.
> I think that the main area of concern here is if we are going to have a very high
> rate of updates to the systems table, it might slow read operation significantly.
> Otherwise, indexes and cache of the database engine should be able handle that scale.
> Amos.
>
Amos,

Have any performance or benchmark tests been performed using the 
scoped_search and our target DBs to ensure that scalability will not be 
an issue?

If not, I'd suggest that we add a task and plan to run some tests.  
Assuming that we plan to utilize the autocomplete feature, it needs to 
be noted that a given search scenario could generate anywhere from 1-15 
(or more) queries on the database depending on how many characters a 
user types and the time it takes them to type them.  Correct?   At a 
minimum, such tests might indicate that we need to tune the autocomplete 
to minimize impact to the server while optimizing the behavior for the user.

cheers,
Brad

>>> In Pulp we already have a no-sql data store (mongoDB) with it's own
>>> search interface.
>>>
>>> Adding an external index store for searching can be an interesting
>>> idea, but it comes in the cost of:
>>> modeling the searchable data into document format.
>>> Updating the index on write, otherwise when a user update an item,
>>> he might not see the changes he made reflected in the GUI.
>>> RBAC model needs to be re-implemented in the index data store to
>>> prevent reading unauthorized data.
>>>
>>> Seems to me a bit of a complex solution.
>>> Amos.
>>>
>>> _______________________________________________
>>> katello-devel mailing list
>>> katello-devel at redhat.com
>>> https://www.redhat.com/mailman/listinfo/katello-devel
>> _______________________________________________
>> katello-devel mailing list
>> katello-devel at redhat.com
>> https://www.redhat.com/mailman/listinfo/katello-devel
> _______________________________________________
> katello-devel mailing list
> katello-devel at redhat.com
> https://www.redhat.com/mailman/listinfo/katello-devel