[Ovirt-devel] Re: Thoughts about taskomatic redesign

[Ian Main]

On Wed, 25 Jun 2008 18:32:09 -0700
Ian Main <imain at redhat.com> wrote:

OK, here is an update based on feedback and some more thinking.

Requirements again.. added one new one:

- NEW: Ability to accept tasks from WUI or scripting API. 
- Execute many tasks in a timely manner (presumably via distributed/multi threaded setup)
- Execute tasks in the correct order.
- Implement transaction support (if we can/want it)
- Have good error reporting.
- Include authentication and encryption.
- It would be nice to be able to see the state of the queues in the WUI.
- Implement at least the following tasks: 
   - start/stop/create/destroy VMs
   - clear host functionality.  Destroy all VMs/pools.
   - Migrate vms.
   - Storage pool creation/destruction/management

If you start to think about the scripting API requirement you then have to consider how high level you want this API, and also about where to do task ordering and breakdown of tasks into actions if desired.  Any comments here?

It's pretty clear that we don't want to put the logic into the WUI as then it would need to be duplicated in a scripting API.

So I think:

- Task queuing API.  Provides logic for queuing tasks to the database and inserting dependency information.  We still have race conditions to contend with in terms of inserting new rows with dependency information.. would we need to lock the table to prevent this?  I might be tempted to make the UI in qpid modeling and have a single process deal with this; this also allows all supported qpid modeling languages to be used.  At this point I'm thinking the API should be high level just like the WUI uses now.

- Queuing can be done in postgresql DB as a single table.  We can look at using the NOTIFY optimization at some point if we like.  Each task should be placed in the database with a dependency ID of the previous task if required along with status fields etc.

- Multiple taskomatic threads/processes can run on the WUI or any other machine and query the DB for tasks.  It will select tasks which have no uncompleted dependencies, lock the row, set in-progress status, work on task, set completed/error conditions, unlock the row.  There seems to be no debate on using qpid/amqp modeling for communicating with the nodes so I guess we'll run with that.  I'm thinking thread count can be expanded whenever there are available tasks that aren't being dealt with.  Note that ruby doesn't actually implement real threads.

	Ian