[Pulp-list] Asynchronous Task Dispatching

[Jason L Connor]

On Thu, 2011-04-14 at 11:50 -0400, Jay Dobies wrote:
> Why do we need to query the database for tasks? Can we keep the task
> stuff in memory and snap out its state when it changes? Or are you
> trying to solve the cross-process task question at the same time? 

There's a few of reasons:
     1. I want to get the task persistence stuff working, we can look at
        what optimizations we need once it does
     2. I'm currently trying to keep the multi-process deployment option
        open and volatile memory storage is not conducive to that
     3. I'm trying not to introduce any task state consistency bugs, at
        least, not initially
     4. To be honest, dequeueing tasks, running tasks, timing out tasks,
        and canceling tasks (i.e. what the dispatcher does), all
        represent state changes and most would have to hit the db anyway

I'm think that once the persistent stuff actually works I can revisit it
looking for optimizations and features needed to support multi-process
access (if we decided to go that route).


In the meantime, I was thinking about a 30 second delay between task
queue checks. With an on demand dispatcher wake up whenever a new task
is enqueued. This should keep our async sub-system fairly responsive in
terms of repo syncs and the like while keep db io down to something
reasonable.

It fits in with the time granularity of 1 minute that I've been
advertising as well.


Any other thoughts?

-- 
Jason L Connor
linear on freenode #pulp
http://pulpproject.org/
RHCE: 805010912355231
GPG Fingerprint: 2048R/CC4ED7C1
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 490 bytes
Desc: This is a digitally signed message part
URL: <http://listman.redhat.com/archives/pulp-list/attachments/20110414/b4fe0b76/attachment.sig>