<html><body><div style="font-family: times new roman, new york, times, serif; font-size: 12pt; color: #000000"><div><br></div><div><br></div><hr id="zwchr"><blockquote style="border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;" data-mce-style="border-left: 2px solid #1010FF; margin-left: 5px; padding-left: 5px; color: #000; font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica,Arial,sans-serif; font-size: 12pt;"><b>From: </b>"Andrew Lau" <andrew@andrewklau.com><br><b>To: </b>"Omer Frenkel" <ofrenkel@redhat.com><br><b>Cc: </b>"Dan Kenigsberg" <danken@redhat.com>, libvir-list@redhat.com, "users" <users@ovirt.org><br><b>Sent: </b>Monday, September 16, 2013 1:38:53 AM<br><b>Subject: </b>Re: [Users] Live Migration failed oVirt 3.3 Nightly<br><div><br></div><div dir="ltr"><div class="gmail_default" style="font-family:tahoma,sans-serif" data-mce-style="font-family: tahoma,sans-serif;"><span style="font-family:arial" data-mce-style="font-family: arial;">On Sun, Sep 15, 2013 at 11:51 PM, Omer Frenkel </span><span dir="ltr" style="font-family:arial" data-mce-style="font-family: arial;"><<a href="mailto:ofrenkel@redhat.com" target="_blank" data-mce-href="mailto:ofrenkel@redhat.com">ofrenkel@redhat.com</a>></span><span style="font-family:arial" data-mce-style="font-family: arial;"> wrote:</span><br></div><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex" data-mce-style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: #cccccc; border-left-style: solid; padding-left: 1ex;"><div class=""><div class="h5"><br> <br> ----- Original Message -----<br> > From: "Dan Kenigsberg" <<a href="mailto:danken@redhat.com" target="_blank" data-mce-href="mailto:danken@redhat.com">danken@redhat.com</a>><br> > To: "Andrew Lau" <<a href="mailto:andrew@andrewklau.com" target="_blank" data-mce-href="mailto:andrew@andrewklau.com">andrew@andrewklau.com</a>><br> > Cc: <a href="mailto:libvir-list@redhat.com" target="_blank" data-mce-href="mailto:libvir-list@redhat.com">libvir-list@redhat.com</a>, "users" <<a href="mailto:users@ovirt.org" target="_blank" data-mce-href="mailto:users@ovirt.org">users@ovirt.org</a>><br> > Sent: Sunday, September 15, 2013 3:47:03 PM<br> > Subject: Re: [Users] Live Migration failed oVirt 3.3 Nightly<br> ><br> > On Sun, Sep 15, 2013 at 09:57:47PM +1000, Andrew Lau wrote:<br> > > On Sun, Sep 15, 2013 at 9:34 PM, Dan Kenigsberg <<a href="mailto:danken@redhat.com" target="_blank" data-mce-href="mailto:danken@redhat.com">danken@redhat.com</a>> wrote:<br> > ><br> > > > On Sun, Sep 15, 2013 at 08:44:18PM +1000, Andrew Lau wrote:<br> > > > > On Sun, Sep 15, 2013 at 8:00 PM, Dan Kenigsberg <<a href="mailto:danken@redhat.com" target="_blank" data-mce-href="mailto:danken@redhat.com">danken@redhat.com</a>><br> > > > wrote:<br> > > > ><br> > > > > > On Sun, Sep 15, 2013 at 06:48:41PM +1000, Andrew Lau wrote:<br> > > > > > > Hi Dan,<br> > > > > > ><br> > > > > > > Certainly, I've uploaded them to fedora's paste bin and tried to<br> > > > > > > snip<br> > > > > > just<br> > > > > > > the relevant details.<br> > > > > > ><br> > > > > > > Sender (<a href="http://hv01.melb.domain.net" target="_blank" data-mce-href="http://hv01.melb.domain.net">hv01.melb.domain.net</a>):<br> > > > > > > <a href="http://paste.fedoraproject.org/39660/92339651/" target="_blank" data-mce-href="http://paste.fedoraproject.org/39660/92339651/">http://paste.fedoraproject.org/39660/92339651/</a><br> > > > > ><br> > > > > > This one has<br> > > > > ><br> > > > > > libvirtError: operation failed: Failed to connect to remote<br> > > > > > libvirt<br> > > > > > URI qemu+tls://<a href="http://hv02.melb.domain.net/system" target="_blank" data-mce-href="http://hv02.melb.domain.net/system">hv02.melb.domain.net/system</a><br> > > > > ><br> > > > > > which is most often related to firewall issues, and some time to key<br> > > > > > mismatch.<br> > > > > ><br> > > > > > Does<br> > > > > > virsh -c qemu+tls://<a href="http://hv02.melb.domain.net/system" target="_blank" data-mce-href="http://hv02.melb.domain.net/system">hv02.melb.domain.net/system</a> capabilities<br> > > > > > work when run from the command line of hv01?<br> > > > > ><br> > > > > > Dan.<br> > > > > > > Receiver (<a href="http://hv02.melb.domain.net" target="_blank" data-mce-href="http://hv02.melb.domain.net">hv02.melb.domain.net</a>): `<br> > > > > > > <a href="http://paste.fedoraproject.org/39661/23406913/" target="_blank" data-mce-href="http://paste.fedoraproject.org/39661/23406913/">http://paste.fedoraproject.org/39661/23406913/</a><br> > > > > > ><br> > > > > > > VM being transfered is ovirt_guest_vm<br> > > > > > ><br> > > > > > > Thanks,<br> > > > > > > Andrew<br> > > > > ><br> > > > ><br> > > > > virsh -c qemu+tls://<a href="http://hv02.melb.domain.net/system" target="_blank" data-mce-href="http://hv02.melb.domain.net/system">hv02.melb.domain.net/system</a><br> > > > > 2013-09-15 10:41:10.620+0000: 23994: info : libvirt version: 0.10.2,<br> > > > > package: 18.el6_4.9 (CentOS BuildSystem <<a href="http://bugs.centos.org" target="_blank" data-mce-href="http://bugs.centos.org">http://bugs.centos.org</a>>,<br> > > > > 2013-07-02-11:19:29, <a href="http://c6b8.bsys.dev.centos.org" target="_blank" data-mce-href="http://c6b8.bsys.dev.centos.org">c6b8.bsys.dev.centos.org</a>)<br> > > > > 2013-09-15 10:41:10.620+0000: 23994: warning :<br> > > > > virNetTLSContextCheckCertificate:1102 : Certificate check failed<br> > > > > Certificate failed validation: The certificate hasn't got a known<br> > > > > issuer.<br> > > ><br> > > > Would you share your<br> > > ><br> > > ><br> > > > openssl x509 -in<br> > > > /etc/pki/vdsm/certs/cacert.pem -text<br> > > ><br> > > > openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem -text<br> > > ><br> > > > on both hosts? This content may be sensitive, and may not<br> > > > provide an answer why libvirt on src cannot contact libvirtd on the<br> > > > other host. So before you do that, would you test if<br> > > ><br> > > ><br> > > > vdsClient -s <a href="http://hv02.melb.domain.net" target="_blank" data-mce-href="http://hv02.melb.domain.net">hv02.melb.domain.net</a> getVdsCapabilities<br> > > ><br> > > > works when run on hv01? It may be that the certificates are fine, but<br> > > > libvirt is not configured to use the correct ones.<br> > > ><br> > > > Dan.<br> > > ><br> > > ><br> > > vdsClient -s <a href="http://hv02.melb.domain.net" target="_blank" data-mce-href="http://hv02.melb.domain.net">hv02.melb.domain.net</a> getVdsCapabilities runs fine<br> > ><br> > > I did a quick comparison between the files on both hosts, they seem to have<br> > > the right details (host names, authority etc.)<br> > > cacert.pem matches<br> > ><br> > > /etc/libvirt/libvirtd.conf<br> > ><br> > > ca_file="/etc/pki/vdsm/certs/cacert.pem"<br> > > cert_file="/etc/pki/vdsm/certs/vdsmcert.pem"<br> > > key_file="/etc/pki/vdsm/keys/vdsmkey.pem"<br> ><br> <br></div></div>this sounds a little like<br> <a href="https://bugzilla.redhat.com/show_bug.cgi?id=996146" target="_blank" data-mce-href="https://bugzilla.redhat.com/show_bug.cgi?id=996146">https://bugzilla.redhat.com/show_bug.cgi?id=996146</a><br> <br> can you try to restart libvirt (on both hosts just to be sure) and try again?<br><div class="im"><br> > Maybe someone on libvir-list could guess why this could be happening?<br></div>> _______________________________________________<br> > Users mailing list<br> > <a href="mailto:Users@ovirt.org" target="_blank" data-mce-href="mailto:Users@ovirt.org">Users@ovirt.org</a><br> > <a href="http://lists.ovirt.org/mailman/listinfo/users" target="_blank" data-mce-href="http://lists.ovirt.org/mailman/listinfo/users">http://lists.ovirt.org/mailman/listinfo/users</a><br> ><br></blockquote></div><br></div><div class="gmail_extra"><div class="gmail_default" style="font-family:tahoma,sans-serif" data-mce-style="font-family: tahoma,sans-serif;">I did try that already</div><div class="gmail_default" style="font-family:tahoma,sans-serif" data-mce-style="font-family: tahoma,sans-serif;"><br></div><div class="gmail_default" style="font-family:tahoma,sans-serif" data-mce-style="font-family: tahoma,sans-serif;">service vdsmd restart</div><div class="gmail_default" style="font-family:tahoma,sans-serif" data-mce-style="font-family: tahoma,sans-serif;"><br></div><div class="gmail_default"><div class="gmail_default"><span style="font-family: tahoma,sans-serif;" data-mce-style="font-family: tahoma,sans-serif;" face="tahoma, sans-serif">[root@hv02 ~]# service vdsmd restart</span></div><div class="gmail_default"><span style="font-family: tahoma,sans-serif;" data-mce-style="font-family: tahoma,sans-serif;" face="tahoma, sans-serif">Shutting down vdsm daemon: </span></div><div class="gmail_default"><span style="font-family: tahoma,sans-serif;" data-mce-style="font-family: tahoma,sans-serif;" face="tahoma, sans-serif">vdsm watchdog stop [ OK ]</span></div><div class="gmail_default"><span style="font-family: tahoma,sans-serif;" data-mce-style="font-family: tahoma,sans-serif;" face="tahoma, sans-serif">vdsm stop [ OK ]</span></div><div class="gmail_default"><span style="font-family: tahoma,sans-serif;" data-mce-style="font-family: tahoma,sans-serif;" face="tahoma, sans-serif">Starting configure libvirt to VDSM ...</span></div><div class="gmail_default"><span style="font-family: tahoma,sans-serif;" data-mce-style="font-family: tahoma,sans-serif;" face="tahoma, sans-serif">libvirt is already configured for vdsm</span></div><div class="gmail_default"><span style="font-family: tahoma,sans-serif;" data-mce-style="font-family: tahoma,sans-serif;" face="tahoma, sans-serif">=Done configuring libvirt=</span></div><div class="gmail_default"><span style="font-family: tahoma,sans-serif;" data-mce-style="font-family: tahoma,sans-serif;" face="tahoma, sans-serif">libvir: Network Filter Driver error : Requested operation is not valid: nwfilter is in use</span></div><div class="gmail_default"><span style="font-family: tahoma,sans-serif;" data-mce-style="font-family: tahoma,sans-serif;" face="tahoma, sans-serif">Checking conflicts ...</span></div><div class="gmail_default"><span style="font-family: tahoma,sans-serif;" data-mce-style="font-family: tahoma,sans-serif;" face="tahoma, sans-serif">SUCCESS: ssl configured to true. No conflicts</span></div><div class="gmail_default"><span style="font-family: tahoma,sans-serif;" data-mce-style="font-family: tahoma,sans-serif;" face="tahoma, sans-serif">Starting up vdsm daemon: </span></div><div class="gmail_default"><span style="font-family: tahoma,sans-serif;" data-mce-style="font-family: tahoma,sans-serif;" face="tahoma, sans-serif">vdsm start [ OK ]</span></div><div style="font-family:tahoma,sans-serif" data-mce-style="font-family: tahoma,sans-serif;"><br></div><div style="font-family:tahoma,sans-serif" data-mce-style="font-family: tahoma,sans-serif;"><br></div><div style="font-family:tahoma,sans-serif" data-mce-style="font-family: tahoma,sans-serif;">Migration still failed. Keep in mind, when I had oVirt 3.3 on these nodes migration was working fine. Only when I upgraded to the nightly and it picked up the new vdsm packages it started to fail.</div><div style="font-family:tahoma,sans-serif" data-mce-style="font-family: tahoma,sans-serif;"><br></div></div><br></div></div></blockquote><div>can you try to restart the libvirtd service itself? not vdsm<br></div></div></body></html>