[Ovirt-devel] Thoughts about taskomatic redesign

[David Lutterkort]

On Mon, 2008-06-23 at 22:03 +0100, Daniel P. Berrange wrote:
> On Mon, Jun 23, 2008 at 11:06:08AM +0200, Chris Lalancette wrote:
> > Ian, et.al,
> >      I've been doing some thinking about both what taskomatic needs to do in its
> > next incarnation, along with ways of how to do it.
> > 
> > WHAT:
> > 1)  Taskomatic needs to be able to run on multiple machines at the same time,
> > accessing a central database
> 
> This is an over-specialization - basically taskomatic needs to be parallelized
> whether it runs on one or many machines.

Yeah, that was my first thought when I read that, too. Taskomatic should
be structured so that an arbitrary number of workers can work
concurrently on the queue - whether they are all running on the same
machine, how many there are etc. is a config option, but shouldn't
affect the way the queue is managed and processed.

> > 2)  Taskomatic needs to be able to fire off tasks relating to different VMs (or
> > storage pools) concurrently (whether it's just run on one machine or many).
> 
> It strikes me that this is avoiding the more general problem - that there are
> explicit dependancies between tasks. Serializing tasks per-VM is not expressing
> this concept of dependancies directly. 

You definitely need to have a clear definition of which tasks have
dependencies on one another, for each type of task that can be put in
the queue. Since tasks generally talk about other objects (VM's, Storage
pools etc.), whenever a task for a specific object is queued, you need
to be prepared for each object it depends on to be in one of three
states: (1) the object does not exist (whine to user) (2) the object is
being created (task for it in the queue) and (3) the object exists.
Here, 'the object exists' might actually be tests for slightly different
conditions; in Dan's example of 'first shut down VM x; then start VM y',
when inserting the task to start VM y, you'd need to take into account
whether VM x is running or not, and whether there is a task to stop VM x
in the queue or not.

In addition to tasks, you'd therefore also need information between task
dependencies in the DB (where each task can depend on n other tasks);
the workers can then continually pick tasks that do not depend on
anything from the queue, mark them as 'in progress' and process them.

> We only need row locking if you're working on the model where you keep the
> transaction open for the duration of taskomatic's processing for that 
> particular job and commit/rollback on completion. It may be that you simply
> immediately mark a task as 'in progress' and commit that change right at
> the start. Then later have a second transaction where you fill in the result
> of the task whether succes or failure.

Agreed; you definitely also want a clear definition of what states a
task can be in and how to transition between states, where each state
transition needs to be atomic. Whether changing a task's state requires
locking in the DB or relies just on DB transactions is really an
implementation detail.

David