[libvirt] [Users] Live Migration failed oVirt 3.3 Nightly

[Andrew Lau]

On Mon, Sep 16, 2013 at 5:41 PM, Omer Frenkel <ofrenkel at redhat.com> wrote:

>
>
> ------------------------------
>
> *From: *"Andrew Lau" <andrew at andrewklau.com>
> *To: *"Omer Frenkel" <ofrenkel at redhat.com>
> *Cc: *"Dan Kenigsberg" <danken at redhat.com>, libvir-list at redhat.com,
> "users" <users at ovirt.org>
> *Sent: *Monday, September 16, 2013 1:38:53 AM
>
> *Subject: *Re: [Users] Live Migration failed oVirt 3.3 Nightly
>
> On Sun, Sep 15, 2013 at 11:51 PM, Omer Frenkel <ofrenkel at redhat.com>wrote:
>
>>
>>
>> ----- Original Message -----
>> > From: "Dan Kenigsberg" <danken at redhat.com>
>> > To: "Andrew Lau" <andrew at andrewklau.com>
>> > Cc: libvir-list at redhat.com, "users" <users at ovirt.org>
>> > Sent: Sunday, September 15, 2013 3:47:03 PM
>> > Subject: Re: [Users] Live Migration failed oVirt 3.3 Nightly
>> >
>> > On Sun, Sep 15, 2013 at 09:57:47PM +1000, Andrew Lau wrote:
>> > > On Sun, Sep 15, 2013 at 9:34 PM, Dan Kenigsberg <danken at redhat.com>
>> wrote:
>> > >
>> > > > On Sun, Sep 15, 2013 at 08:44:18PM +1000, Andrew Lau wrote:
>> > > > > On Sun, Sep 15, 2013 at 8:00 PM, Dan Kenigsberg <
>> danken at redhat.com>
>> > > > wrote:
>> > > > >
>> > > > > > On Sun, Sep 15, 2013 at 06:48:41PM +1000, Andrew Lau wrote:
>> > > > > > > Hi Dan,
>> > > > > > >
>> > > > > > > Certainly, I've uploaded them to fedora's paste bin and tried
>> to
>> > > > > > > snip
>> > > > > > just
>> > > > > > > the relevant details.
>> > > > > > >
>> > > > > > > Sender (hv01.melb.domain.net):
>> > > > > > > http://paste.fedoraproject.org/39660/92339651/
>> > > > > >
>> > > > > > This one has
>> > > > > >
>> > > > > >     libvirtError: operation failed: Failed to connect to remote
>> > > > > >     libvirt
>> > > > > > URI qemu+tls://hv02.melb.domain.net/system
>> > > > > >
>> > > > > > which is most often related to firewall issues, and some time
>> to key
>> > > > > > mismatch.
>> > > > > >
>> > > > > > Does
>> > > > > >     virsh -c qemu+tls://hv02.melb.domain.net/systemcapabilities
>> > > > > > work when run from the command line of hv01?
>> > > > > >
>> > > > > > Dan.
>> > > > > > > Receiver (hv02.melb.domain.net): `
>> > > > > > > http://paste.fedoraproject.org/39661/23406913/
>> > > > > > >
>> > > > > > > VM being transfered is ovirt_guest_vm
>> > > > > > >
>> > > > > > > Thanks,
>> > > > > > > Andrew
>> > > > > >
>> > > > >
>> > > > > virsh -c qemu+tls://hv02.melb.domain.net/system
>> > > > > 2013-09-15 10:41:10.620+0000: 23994: info : libvirt version:
>> 0.10.2,
>> > > > > package: 18.el6_4.9 (CentOS BuildSystem <http://bugs.centos.org>,
>> > > > > 2013-07-02-11:19:29, c6b8.bsys.dev.centos.org)
>> > > > > 2013-09-15 10:41:10.620+0000: 23994: warning :
>> > > > > virNetTLSContextCheckCertificate:1102 : Certificate check failed
>> > > > > Certificate failed validation: The certificate hasn't got a known
>> > > > > issuer.
>> > > >
>> > > > Would you share your
>> > > >
>> > > >
>> > > > openssl x509 -in
>> > > > /etc/pki/vdsm/certs/cacert.pem -text
>> > > >
>> > > > openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem -text
>> > > >
>> > > > on both hosts? This content may be sensitive, and may not
>> > > > provide an answer why libvirt on src cannot contact libvirtd on the
>> > > > other host. So before you do that, would you test if
>> > > >
>> > > >
>> > > >   vdsClient -s hv02.melb.domain.net getVdsCapabilities
>> > > >
>> > > > works when run on hv01? It may be that the certificates are fine,
>> but
>> > > > libvirt is not configured to use the correct ones.
>> > > >
>> > > > Dan.
>> > > >
>> > > >
>> > > vdsClient -s hv02.melb.domain.net getVdsCapabilities runs fine
>> > >
>> > > I did a quick comparison between the files on both hosts, they seem
>> to have
>> > > the right details (host names, authority etc.)
>> > > cacert.pem matches
>> > >
>> > > /etc/libvirt/libvirtd.conf
>> > >
>> > > ca_file="/etc/pki/vdsm/certs/cacert.pem"
>> > > cert_file="/etc/pki/vdsm/certs/vdsmcert.pem"
>> > > key_file="/etc/pki/vdsm/keys/vdsmkey.pem"
>> >
>>
>> this sounds a little like
>> https://bugzilla.redhat.com/show_bug.cgi?id=996146
>>
>> can you try to restart libvirt (on both hosts just to be sure) and try
>> again?
>>
>> > Maybe someone on libvir-list could guess why this could be happening?
>> > _______________________________________________
>> > Users mailing list
>> > Users at ovirt.org
>> > http://lists.ovirt.org/mailman/listinfo/users
>> >
>>
>
> I did try that already
>
> service vdsmd restart
>
> [root at hv02 ~]# service vdsmd restart
> Shutting down vdsm daemon:
> vdsm watchdog stop                                         [  OK  ]
> vdsm stop                                                  [  OK  ]
> Starting configure libvirt to VDSM ...
> libvirt is already configured for vdsm
> =Done configuring libvirt=
> libvir: Network Filter Driver error : Requested operation is not valid:
> nwfilter is in use
> Checking conflicts ...
> SUCCESS: ssl configured to true. No conflicts
> Starting up vdsm daemon:
> vdsm start                                                 [  OK  ]
>
>
> Migration still failed. Keep in mind, when I had oVirt 3.3 on these nodes
> migration was working fine. Only when I upgraded to the nightly and it
> picked up the new vdsm packages it started to fail.
>
>
> can you try to restart the libvirtd service itself? not vdsm
>


[root at hv01 ~]# service libvirtd restart
Stopping libvirtd daemon: libvirtd: libvirtd is managed by upstart and
started, use initctl instead
[root at hv01 ~]# initctl restart libvirtd
libvirtd start/running, process 4538


Migration was successful, thanks!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/libvir-list/attachments/20130916/5c2982b6/attachment-0001.htm>