[libvirt-users] LXC + USB passthrough = Operation not permitted

Filip Maj fil at saucelabs.com
Thu Apr 17 18:26:57 UTC 2014


Ah, nevermind, think my last post may not have been entirely correct. We've
spent some more time correlating the log output from our failed LXC startup
via libvirt [1] with libvirt code from the 1.2.2 tag. Interestingly, the
error we get back is different, and digging through the multitudes of logs,
nothing comes up about 'operating not permitted' regarding the USB bus that
I saw earlier. If we turn off the verbose logging, we get the original
error back. The more-verbose error is a bit more cryptic:

libvirt LXC Driver error: internal error: guest failed to start: 2014-

Anyways, here's a rough overview of the logs:

1. it starts to check/detect for cgroup stuff [2]
2. starts to 'prepare host devices' and logs out that the appropriate usb
address is initialized (002/003) [3]
3. sets the appropriate usb bus to 'active host devs' [4]
4. seems to then switch to doing network-related things, and stop talking
about any hostdev/usb stuff [5]
5. eventually invokes `libvirt_lxc` [6]
6. does the cgroup song-n-dance again for some reason? [7]. not sure if
this is critical or not.
7. eventually some kind of socket read wire error comes up [8]
8. and this leads to a 'guest failed to start error' [9], which it probably
truncates to return the error i mentioned above.

What is also confusing is, looking through the libvirt lxc code, shouldn't
we be seeing a series of "Allowing any <blah>" log debug outputs in our
logs? This is part of the virLXCCgroupSetupDeviceACL part of the
lxc_cgroup.c file [10]. I imagine that we'd eventually want the
virCgroupAllowDeviceMajor
method [11] invoked, no?

I realize that you are the author of most of this code, Daniel, so
appreciate your patience with me and hope you are willing to continue
trying to get to the bottom of this with us! Thanks a lot in advance!

Cheers,
Fil

[1] https://gist.github.com/filmaj/207ab3c09f5b881d83b1
[2]
https://gist.github.com/filmaj/207ab3c09f5b881d83b1#file-libvirtd-log-L226-L253
[3]
https://gist.github.com/filmaj/207ab3c09f5b881d83b1#file-libvirtd-log-L257-L266
[4]
https://gist.github.com/filmaj/207ab3c09f5b881d83b1#file-libvirtd-log-L296
[5]
https://gist.github.com/filmaj/207ab3c09f5b881d83b1#file-libvirtd-log-L302
[6]
https://gist.github.com/filmaj/207ab3c09f5b881d83b1#file-libvirtd-log-L685
[7]
https://gist.github.com/filmaj/207ab3c09f5b881d83b1#file-libvirtd-log-L752-L799
[8]
https://gist.github.com/filmaj/207ab3c09f5b881d83b1#file-libvirtd-log-L1075
[9]
https://gist.github.com/filmaj/207ab3c09f5b881d83b1#file-libvirtd-log-L1090
[10]
http://libvirt.org/git/?p=libvirt.git;a=blob;f=src/lxc/lxc_cgroup.c;h=39d955cd8cc2e34b6157e126fbb455d8f71e0647;hb=e8684eb541f01df9b45e87e0a8ce446c7bc90a17#l342
[11]
http://libvirt.org/git/?p=libvirt.git;a=blob;f=src/lxc/lxc_cgroup.c;h=39d955cd8cc2e34b6157e126fbb455d8f71e0647;hb=e8684eb541f01df9b45e87e0a8ce446c7bc90a17#l446


On Wed, Apr 16, 2014 at 5:11 PM, Filip Maj <fil at saucelabs.com> wrote:

> Further followups!
>
> We are correlating DEBUG-level output from libvirt with the libvirt 1.2.2
> code to try to figure out what libvirt is doing under the hood.
>
> Even though we have the log level set to 1 (info) in our libvirtd.conf, we
> are not seeing the VIR_DEBUG() [1] statements being printed out. There are
> tons of other presumably-debug lines of output showing up in our log. We
> are sort of expecting to see the output from [1] in our logs somewhere,
> given the type of usb mounting we are trying to do?
>
> [1]
> http://libvirt.org/git/?p=libvirt.git;a=blob;f=src/lxc/lxc_controller.c;h=c05dfec6428cad927cd5751004a4f3afc67899de;hb=HEAD#l1396
>
>
> On Wed, Apr 16, 2014 at 3:58 PM, Filip Maj <fil at saucelabs.com> wrote:
>
>> To follow up on this a little bit, tail'ing kern.log while trying to get
>> our little container up doesn't yield anything with apparmor complaining,
>> so, unless I'm looking in the wrong spots for apparmor logs (which I don't
>> think so, as I see other apparmor-related log entries in kern.log), I am
>> not entirely sure this is an apparmor issue at this point.
>>
>>
>> On Wed, Apr 16, 2014 at 3:25 PM, Filip Maj <fil at saucelabs.com> wrote:
>>
>>> Yeah, AppArmor is enabled, but I put everything (that I could find) into
>>> complain mode:
>>>
>>> $ sudo apparmor_status
>>> apparmor module is loaded.
>>> 12 profiles are loaded.
>>> 3 profiles are in enforce mode.
>>>    lxc-container-default
>>>    lxc-container-default-with-mounting
>>>    lxc-container-default-with-nesting
>>> 9 profiles are in complain mode.
>>>    /sbin/dhclient
>>>    /usr/bin/lxc-start
>>>    /usr/lib/NetworkManager/nm-dhcp-client.action
>>>    /usr/lib/connman/scripts/dhclient-script
>>>    /usr/lib/libvirt/virt-aa-helper
>>>    /usr/sbin/libvirtd
>>>    /usr/sbin/ntpd
>>>    /usr/sbin/rsyslogd
>>>    /usr/sbin/tcpdump
>>> 3 processes have profiles defined.
>>> 0 processes are in enforce mode.
>>> 2 processes are in complain mode.
>>>    /usr/sbin/libvirtd (30419)
>>>    /usr/sbin/ntpd (3418)
>>> 1 processes are unconfined but have a profile defined.
>>>    /usr/sbin/rsyslogd (626)
>>>
>>> And still get issues. From libvirtd.log:
>>>
>>> 2014-04-16 22:19:10.855+0000: 30419: info : libvirt version: 1.2.2
>>> 2014-04-16 22:19:10.855+0000: 30419: error : virNetSocketReadWire:1446 :
>>> Cannot recv data: Connection reset by peer
>>> 2014-04-16 22:19:10.940+0000: 30420: error : virLXCProcessStart:1299 :
>>> internal error: guest failed to start: Unable to create device
>>> //var/run/libvirt/lxc/oshi32134.dev/bus/usb//002//003: Operation not
>>> permitted
>>>
>>> 2014-04-16 22:19:10.964+0000: 30420: warning :
>>> virLXCDomainReAttachHostUsbDevices:388 : Unable to find device 000.000 in
>>> list of active USB devices
>>>
>>> Thanks in advance for any help, Daniel!
>>>
>>> Cheers,
>>> Fil
>>>
>>>
>>> On Tue, Apr 15, 2014 at 1:33 AM, Daniel P. Berrange <berrange at redhat.com
>>> > wrote:
>>>
>>>> On Fri, Apr 11, 2014 at 05:32:28PM -0700, Filip Maj wrote:
>>>> > Hi!
>>>> >
>>>> > First post, kind of a noobie. I've been working with LXC and libvirt
>>>> for a
>>>> > few months now. Trying to do some interesting things with containers
>>>> and
>>>> > Android devices :D
>>>> > Here's my entire domain definition:
>>>> >
>>>> > <domain type='lxc'>
>>>> >   <name>oshi32134</name>
>>>> >   <uuid>xxxxx</uuid>
>>>> >   <memory unit='KiB'>3145728</memory>
>>>> >   <currentMemory unit='KiB'>3145728</currentMemory>
>>>> >   <vcpu placement='static'>1</vcpu>
>>>> >   <resource>
>>>> >     <partition>/machine</partition>
>>>> >   </resource>
>>>> >   <os>
>>>> >     <type arch='i686'>exe</type>
>>>> >     <init>/sbin/init</init>
>>>> >   </os>
>>>> >   <clock offset='utc'/>
>>>> >   <on_poweroff>destroy</on_poweroff>
>>>> >   <on_reboot>restart</on_reboot>
>>>> >   <on_crash>destroy</on_crash>
>>>> >   <devices>
>>>> >     <emulator>/usr/lib/libvirt/libvirt_lxc</emulator>
>>>> >     <filesystem type='mount' accessmode='passthrough'>
>>>> >       <source dir='/some/valid/filesystem/location'/>
>>>> >       <target dir='/'/>
>>>> >     </filesystem>
>>>> >     <filesystem type='mount' accessmode='passthrough'>
>>>> >       <source dir='/another/valid/filesystem/location'/>
>>>> >       <target dir='/mnt/android'/>
>>>> >     </filesystem>
>>>> >     <interface type='bridge'>
>>>> >       <mac address='xx:xx:xx:xx:xx:xx'/>
>>>> >       <source bridge='br1'/>
>>>> >     </interface>
>>>> >     <console type='pty'>
>>>> >       <target type='lxc' port='0'/>
>>>> >     </console>
>>>> >     <hostdev mode='capabilities' type='misc'>
>>>> >       <source>
>>>> >         <char>/dev/kvm</char>
>>>> >       </source>
>>>> >     </hostdev>
>>>> >     <hostdev mode='subsystem' type='usb' managed='yes'>
>>>> >       <source>
>>>> >         <vendor id='0x04e8'/>
>>>> >         <product id='0x6860'/>
>>>> >   </source>
>>>> >     </hostdev>
>>>> >   </devices>
>>>> > </domain>
>>>>
>>>> Your config looks fine here.
>>>>
>>>> >
>>>> > Everything worked fine until I added the USB <hostdev> element. I'm
>>>> > essentially trying to get access to a physical Android device
>>>> connected to
>>>> > the host from inside a container. When I go to start the container, I
>>>> get
>>>> > an error about Operation not permitted. Here's the relevant bits from
>>>> > /var/log/libvirt/lxc/machine.log:
>>>> >
>>>> > 2014-04-11 22:46:40.491+0000: starting up
>>>> > PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin
>>>> > LIBVIRT_DEBUG=3 LIBVIRT_LOG_OUTPUTS=3:stderr
>>>> /usr/lib/libvirt/libvirt_lxc
>>>> > --name oshi32134 --console 24 --security=none --handshake 27
>>>> --background
>>>> > --veth vnet1
>>>> > 2014-04-11 22:46:40.597+0000: 685: info : libvirt version: 1.2.2
>>>> > 2014-04-11 22:46:40.597+0000: 685: error :
>>>> > virLXCControllerSetupHostdevSubsysUSB:1390 : Unable to create device
>>>> > //var/run/libvirt/lxc/oshi32134.dev/bus/usb//002//003: Operation not
>>>> > permitted
>>>> > Unable to create device
>>>> > //var/run/libvirt/lxc/oshi32134.dev/bus/usb//002//003: Operation not
>>>> > permitted
>>>>
>>>> Do you have AppArmour enabled on the machine. That seems like the
>>>> most likely thing that would result in libvirt getting that permission
>>>> error.
>>>>
>>>> Regards,
>>>> Daniel
>>>> --
>>>> |: http://berrange.com      -o-
>>>> http://www.flickr.com/photos/dberrange/ :|
>>>> |: http://libvirt.org              -o-
>>>> http://virt-manager.org :|
>>>> |: http://autobuild.org       -o-
>>>> http://search.cpan.org/~danberr/ :|
>>>> |: http://entangle-photo.org       -o-
>>>> http://live.gnome.org/gtk-vnc :|
>>>>
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/libvirt-users/attachments/20140417/031f4040/attachment.htm>


More information about the libvirt-users mailing list