[Ovirt-devel] Re: [PATCH] make taskomatic.rb robust to database outages

[steve linabery]

On Thu, May 8, 2008 at 11:01 PM, steve linabery <slinabery at gmail.com> wrote:
> Hi ovirt,
>
> En route to transaction support, and being of the "if it ain't broke,
> break it!" school of code spelunking, I noticed that taskomatic.rb
> dies if one stops postgresql on the appliance.
>
> So I offer this first attempt at making taskomatic.rb handle this
> situation somewhat more gracefully.
>
> Key points:
> 0) move 'require dutils' above the call to daemonize, so that the
> script will Fail outright if a connection cannot be established before
> daemonization (not sure about this one really, but I thought the
> sysadmin doing 'service ovirt-taskomatic start' would like to know
> immediately that it was failing).
> 1) use ruby exception handling and extremely rudimentary logging for
> exceptions in Task.find section.
> 2) wrap the entire 'case task.action' block (and the following Task
> state updates) in a begin/rescue. This is meant as a first step toward
> isolating/handling the many exceptions that can be raised by these
> statements.
>
> I am VERY open to feedback/suggestions on this; in particular, the
> logging (does anyone know off-hand how to log line numbers in this
> type of situation?), and making the script fail to start with the init
> script if the DB appears unavailable (maybe it should just go daemon
> and keep polling for a connection while logging its error?).
>
> Oh, and I use the 'database_connect' method from active_record_env.rb
> for the case (not sure if it can be reached) where ActiveRecord shows
> that it's not connected (note: this seems to mean something different
> from "there's no currently-functioning connection to the db").
>
> I've tested this with database halts before and during execution, and
> also restarting database during execution. Works for me.
>
> Thanks,
> Steve
>

Hmm, I see from the libvirt list that perhaps I should have called
this "RFC" rather than "Patch", since I don't really mean it's ready
for commit, but rather I'm looking for some feedback on it.

So, feedback at will.

Thanks,
Steve