[Ovirt-devel] Re: Thoughts about taskomatic redesign

[Daniel P. Berrange]

On Wed, Jun 25, 2008 at 06:32:09PM -0700, Ian Main wrote:
> - Task ordering system: At some point I think a single 
> process needs to separate out the tasks and order them.
> I think it would be useful to actually move them into 
> separate independent queues that can be executed in 
> parallel.  This would be done such that each action in
> a given queue would need to be done in order, but each
> queue will not have dependencies on anything in another
> queue and so they can be worked on in parallel.  While
> this could be a bottleneck I think that the logic required
> to order the tasks will not be onerous and should keep up
> with high loads.

Again this is just a architecturally limited task dependancy
system where you only have linear non-forking task orders.
It is pretty easy to imagine scenarios where you have more
complex dependancy requirements for oVirt. We just need to go
for the dependancy system implemented as a full DAG, as David
outlines.

> Queues could continue to be implemented in postgresql.  It 
> would be nice however to have something that was event driven
> and did not require polling.

I think this is probably overkill. If there aren't many tasks being
added then the datbase won't have much load and so polling won't be
a huge issue. If there are lots of tasks then you'll get pulling new
tasks off the queue pretty often and you'll need a scalable database
already. David's idea of having the process which queue's a tasks
issue some form of notification could be added as an optimization
later, but I'd just go with polling to start off with.

BTW, there is a built-in notification mechanims in postgresql if you
don't mind using PG specific syntax

The taskomatic would do

     LISTEN newtasks

And the WUI would do

     NOTIFY newtasks

whenever it queued a new task. If you have polling as the built-in
default mode of operation, then you could just have this PG specific
bit as an optional optimization.

http://www.postgresql.org/docs/8.3/interactive/sql-notify.html

> I think a single ruby process could be used to order the tasks and
> place them in per-thread/process queues.  If using a DB I think we
> could either migrate the entries to a new 'in-progress' table, or 
> update the row with the ID of the process/thread and possibly the 
> sequence number to be used in implementing the queue.  

This is overkill for the problem - a simple status field in the
tasks table can take care of te tracking what's in progress. Completed
tasks can be purged periodically to stop it growing without bound, or
sent to an archive_tasks table if we really need to keep the data
around long term.

We want to have  'n' x (number of logical CPU cores)   worker processes
for a value of 'n' yet to be determined - if they're mostly waiting
on I/O, then 'n' can be pretty large. We'll have to just try it out and
see what a good number is.

Regards,
Daniel
-- 
|: Red Hat, Engineering, London   -o-   http://people.redhat.com/berrange/ :|
|: http://libvirt.org  -o-  http://virt-manager.org  -o-  http://ovirt.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505  -o-  F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|