[Ovirt-devel] [PATCH server] Make taskomatic retry qpidd connection on failure.

[Ian Main]

This patch makes taskomatic keep trying to connect to qpidd in the
event of any kind of failure to do so or if the connection is lost
during runtime.

If others test this and it works out ok I'll do the same for
dbomatic.

Signed-off-by: Ian Main <imain at redhat.com>
---
 src/task-omatic/taskomatic.rb |   44 ++++++++++++++++++++++++++++++++++++----
 1 files changed, 39 insertions(+), 5 deletions(-)

diff --git a/src/task-omatic/taskomatic.rb b/src/task-omatic/taskomatic.rb
index e65db08..a5cf6ed 100755
--- a/src/task-omatic/taskomatic.rb
+++ b/src/task-omatic/taskomatic.rb
@@ -47,11 +47,7 @@ class TaskOmatic
     @nth_host = 0
 
     @session = Qpid::Qmf::Session.new()
-
-    server, port = get_srv('qpidd', 'tcp')
-    raise "Unable to determine qpid server from DNS SRV record" if not server
-
-    @broker = @session.add_broker("amqp://#{server}:#{port}", :mechanism => 'GSSAPI')
+    @broker = nil
 
     do_daemon = true
 
@@ -87,6 +83,42 @@ class TaskOmatic
       $stdout = lf
       $stderr = lf
     end
+
+    # this has to be after daemonizing now because it could take a LONG time to
+    # actually connect if qpidd isn't running yet etc.
+    qpid_ensure_connected
+
+  end
+
+  def qpid_ensure_connected()
+
+    return if @broker and @broker.connected?
+
+    sleepy = 2
+
+    while true do
+      begin
+        server, port = get_srv('qpidd', 'tcp')
+        raise "Unable to determine qpid server from DNS SRV record" if not server
+
+        @broker = @session.add_broker("amqp://#{server}:#{port}", :mechanism => 'GSSAPI')
+
+        # Connection succeeded, go about our business.
+        return
+      rescue Exception => msg
+        puts "Error connecting to qpidd: #{msg}"
+      end
+      sleep(sleepy)
+      sleepy *= 2
+      sleepy = 120 if sleepy > 120
+
+      begin
+        # Could also be a credentials problem?  Try to get them again..
+        get_credentials('qpidd')
+      rescue Exception => msg
+        puts "Error getting qpidd credentials: #{msg}"
+      end
+    end
   end
 
   def find_capable_host(db_vm)
@@ -755,6 +787,8 @@ class TaskOmatic
         end
       end
 
+      qpid_ensure_connected
+
       tasks.each do |task|
 
         task.time_started = Time.now
-- 
1.6.0.6