[libvirt-users] LXC + USB passthrough = Operation not permitted

Filip Maj fil at saucelabs.com
Fri Apr 18 17:13:10 UTC 2014 

Indeed, we've applied that code change, compiled and installed the patched
source, and things work great!

That was a fun code dive :D


On Thu, Apr 17, 2014 at 2:44 PM, Filip Maj <fil at saucelabs.com> wrote:

> Me again!
>
> Think we've found it.
>
> By diving into the LXC logs for the specific container, we found this:
>
> 2014-04-17 21:07:06.066+0000: 2861: debug : virCgroupSetValueStr:678 : Set
> value '/sys/fs/cgroup/devices/machine/oshi32134.libvirt-lxc/devices.allow'
> to 'c 189:130 rw'
>
> Looks like libvirt the permission to 'rw', not 'rmw' [1], so no surprise
> that when it goes to call mknod [2], it can't do so.
>
> We are going to patch this so it gives 'rwm' access in the cgroup
> whitelist, try to compile and replace libvirt to see if that works out for
> us. Will report back :)
>
> [1]
> http://libvirt.org/git/?p=libvirt.git;a=blob;f=src/lxc/lxc_cgroup.c;h=39d955cd8cc2e34b6157e126fbb455d8f71e0647;hb=e8684eb541f01df9b45e87e0a8ce446c7bc90a17#l334
> [2]
> http://libvirt.org/git/?p=libvirt.git;a=blob;f=src/lxc/lxc_controller.c;h=5ca960f13e53501315b28f9086eaa389475b8feb;hb=e8684eb541f01df9b45e87e0a8ce446c7bc90a17#l1387
>
>
> On Thu, Apr 17, 2014 at 11:26 AM, Filip Maj <fil at saucelabs.com> wrote:
>
>> Ah, nevermind, think my last post may not have been entirely correct.
>> We've spent some more time correlating the log output from our failed LXC
>> startup via libvirt [1] with libvirt code from the 1.2.2 tag.
>> Interestingly, the error we get back is different, and digging through the
>> multitudes of logs, nothing comes up about 'operating not permitted'
>> regarding the USB bus that I saw earlier. If we turn off the verbose
>> logging, we get the original error back. The more-verbose error is a bit
>> more cryptic:
>>
>> libvirt LXC Driver error: internal error: guest failed to start: 2014-
>>
>> Anyways, here's a rough overview of the logs:
>>
>> 1. it starts to check/detect for cgroup stuff [2]
>> 2. starts to 'prepare host devices' and logs out that the appropriate usb
>> address is initialized (002/003) [3]
>> 3. sets the appropriate usb bus to 'active host devs' [4]
>> 4. seems to then switch to doing network-related things, and stop talking
>> about any hostdev/usb stuff [5]
>> 5. eventually invokes `libvirt_lxc` [6]
>> 6. does the cgroup song-n-dance again for some reason? [7]. not sure if
>> this is critical or not.
>> 7. eventually some kind of socket read wire error comes up [8]
>> 8. and this leads to a 'guest failed to start error' [9], which it
>> probably truncates to return the error i mentioned above.
>>
>> What is also confusing is, looking through the libvirt lxc code,
>> shouldn't we be seeing a series of "Allowing any <blah>" log debug outputs
>> in our logs? This is part of the virLXCCgroupSetupDeviceACL part of the
>> lxc_cgroup.c file [10]. I imagine that we'd eventually want the virCgroupAllowDeviceMajor
>> method [11] invoked, no?
>>
>> I realize that you are the author of most of this code, Daniel, so
>> appreciate your patience with me and hope you are willing to continue
>> trying to get to the bottom of this with us! Thanks a lot in advance!
>>
>> Cheers,
>> Fil
>>
>> [1] https://gist.github.com/filmaj/207ab3c09f5b881d83b1
>> [2]
>> https://gist.github.com/filmaj/207ab3c09f5b881d83b1#file-libvirtd-log-L226-L253
>> [3]
>> https://gist.github.com/filmaj/207ab3c09f5b881d83b1#file-libvirtd-log-L257-L266
>> [4]
>> https://gist.github.com/filmaj/207ab3c09f5b881d83b1#file-libvirtd-log-L296
>> [5]
>> https://gist.github.com/filmaj/207ab3c09f5b881d83b1#file-libvirtd-log-L302
>> [6]
>> https://gist.github.com/filmaj/207ab3c09f5b881d83b1#file-libvirtd-log-L685
>> [7]
>> https://gist.github.com/filmaj/207ab3c09f5b881d83b1#file-libvirtd-log-L752-L799
>> [8]
>> https://gist.github.com/filmaj/207ab3c09f5b881d83b1#file-libvirtd-log-L1075
>> [9]
>> https://gist.github.com/filmaj/207ab3c09f5b881d83b1#file-libvirtd-log-L1090
>> [10]
>> http://libvirt.org/git/?p=libvirt.git;a=blob;f=src/lxc/lxc_cgroup.c;h=39d955cd8cc2e34b6157e126fbb455d8f71e0647;hb=e8684eb541f01df9b45e87e0a8ce446c7bc90a17#l342
>> [11]
>> http://libvirt.org/git/?p=libvirt.git;a=blob;f=src/lxc/lxc_cgroup.c;h=39d955cd8cc2e34b6157e126fbb455d8f71e0647;hb=e8684eb541f01df9b45e87e0a8ce446c7bc90a17#l446
>>
>>
>> On Wed, Apr 16, 2014 at 5:11 PM, Filip Maj <fil at saucelabs.com> wrote:
>>
>>> Further followups!
>>>
>>> We are correlating DEBUG-level output from libvirt with the libvirt
>>> 1.2.2 code to try to figure out what libvirt is doing under the hood.
>>>
>>> Even though we have the log level set to 1 (info) in our libvirtd.conf,
>>> we are not seeing the VIR_DEBUG() [1] statements being printed out. There
>>> are tons of other presumably-debug lines of output showing up in our log.
>>> We are sort of expecting to see the output from [1] in our logs somewhere,
>>> given the type of usb mounting we are trying to do?
>>>
>>> [1]
>>> http://libvirt.org/git/?p=libvirt.git;a=blob;f=src/lxc/lxc_controller.c;h=c05dfec6428cad927cd5751004a4f3afc67899de;hb=HEAD#l1396
>>>
>>>
>>> On Wed, Apr 16, 2014 at 3:58 PM, Filip Maj <fil at saucelabs.com> wrote:
>>>
>>>> To follow up on this a little bit, tail'ing kern.log while trying to
>>>> get our little container up doesn't yield anything with apparmor
>>>> complaining, so, unless I'm looking in the wrong spots for apparmor logs
>>>> (which I don't think so, as I see other apparmor-related log entries in
>>>> kern.log), I am not entirely sure this is an apparmor issue at this point.
>>>>
>>>>
>>>> On Wed, Apr 16, 2014 at 3:25 PM, Filip Maj <fil at saucelabs.com> wrote:
>>>>
>>>>> Yeah, AppArmor is enabled, but I put everything (that I could find)
>>>>> into complain mode:
>>>>>
>>>>> $ sudo apparmor_status
>>>>> apparmor module is loaded.
>>>>> 12 profiles are loaded.
>>>>> 3 profiles are in enforce mode.
>>>>>    lxc-container-default
>>>>>    lxc-container-default-with-mounting
>>>>>    lxc-container-default-with-nesting
>>>>> 9 profiles are in complain mode.
>>>>>    /sbin/dhclient
>>>>>    /usr/bin/lxc-start
>>>>>    /usr/lib/NetworkManager/nm-dhcp-client.action
>>>>>    /usr/lib/connman/scripts/dhclient-script
>>>>>    /usr/lib/libvirt/virt-aa-helper
>>>>>    /usr/sbin/libvirtd
>>>>>    /usr/sbin/ntpd
>>>>>    /usr/sbin/rsyslogd
>>>>>    /usr/sbin/tcpdump
>>>>> 3 processes have profiles defined.
>>>>> 0 processes are in enforce mode.
>>>>> 2 processes are in complain mode.
>>>>>    /usr/sbin/libvirtd (30419)
>>>>>    /usr/sbin/ntpd (3418)
>>>>> 1 processes are unconfined but have a profile defined.
>>>>>    /usr/sbin/rsyslogd (626)
>>>>>
>>>>> And still get issues. From libvirtd.log:
>>>>>
>>>>> 2014-04-16 22:19:10.855+0000: 30419: info : libvirt version: 1.2.2
>>>>> 2014-04-16 22:19:10.855+0000: 30419: error : virNetSocketReadWire:1446
>>>>> : Cannot recv data: Connection reset by peer
>>>>> 2014-04-16 22:19:10.940+0000: 30420: error : virLXCProcessStart:1299 :
>>>>> internal error: guest failed to start: Unable to create device
>>>>> //var/run/libvirt/lxc/oshi32134.dev/bus/usb//002//003: Operation not
>>>>> permitted
>>>>>
>>>>> 2014-04-16 22:19:10.964+0000: 30420: warning :
>>>>> virLXCDomainReAttachHostUsbDevices:388 : Unable to find device 000.000 in
>>>>> list of active USB devices
>>>>>
>>>>> Thanks in advance for any help, Daniel!
>>>>>
>>>>> Cheers,
>>>>> Fil
>>>>>
>>>>>
>>>>> On Tue, Apr 15, 2014 at 1:33 AM, Daniel P. Berrange <
>>>>> berrange at redhat.com> wrote:
>>>>>
>>>>>> On Fri, Apr 11, 2014 at 05:32:28PM -0700, Filip Maj wrote:
>>>>>> > Hi!
>>>>>> >
>>>>>> > First post, kind of a noobie. I've been working with LXC and
>>>>>> libvirt for a
>>>>>> > few months now. Trying to do some interesting things with
>>>>>> containers and
>>>>>> > Android devices :D
>>>>>> > Here's my entire domain definition:
>>>>>> >
>>>>>> > <domain type='lxc'>
>>>>>> >   <name>oshi32134</name>
>>>>>> >   <uuid>xxxxx</uuid>
>>>>>> >   <memory unit='KiB'>3145728</memory>
>>>>>> >   <currentMemory unit='KiB'>3145728</currentMemory>
>>>>>> >   <vcpu placement='static'>1</vcpu>
>>>>>> >   <resource>
>>>>>> >     <partition>/machine</partition>
>>>>>> >   </resource>
>>>>>> >   <os>
>>>>>> >     <type arch='i686'>exe</type>
>>>>>> >     <init>/sbin/init</init>
>>>>>> >   </os>
>>>>>> >   <clock offset='utc'/>
>>>>>> >   <on_poweroff>destroy</on_poweroff>
>>>>>> >   <on_reboot>restart</on_reboot>
>>>>>> >   <on_crash>destroy</on_crash>
>>>>>> >   <devices>
>>>>>> >     <emulator>/usr/lib/libvirt/libvirt_lxc</emulator>
>>>>>> >     <filesystem type='mount' accessmode='passthrough'>
>>>>>> >       <source dir='/some/valid/filesystem/location'/>
>>>>>> >       <target dir='/'/>
>>>>>> >     </filesystem>
>>>>>> >     <filesystem type='mount' accessmode='passthrough'>
>>>>>> >       <source dir='/another/valid/filesystem/location'/>
>>>>>> >       <target dir='/mnt/android'/>
>>>>>> >     </filesystem>
>>>>>> >     <interface type='bridge'>
>>>>>> >       <mac address='xx:xx:xx:xx:xx:xx'/>
>>>>>> >       <source bridge='br1'/>
>>>>>> >     </interface>
>>>>>> >     <console type='pty'>
>>>>>> >       <target type='lxc' port='0'/>
>>>>>> >     </console>
>>>>>> >     <hostdev mode='capabilities' type='misc'>
>>>>>> >       <source>
>>>>>> >         <char>/dev/kvm</char>
>>>>>> >       </source>
>>>>>> >     </hostdev>
>>>>>> >     <hostdev mode='subsystem' type='usb' managed='yes'>
>>>>>> >       <source>
>>>>>> >         <vendor id='0x04e8'/>
>>>>>> >         <product id='0x6860'/>
>>>>>> >   </source>
>>>>>> >     </hostdev>
>>>>>> >   </devices>
>>>>>> > </domain>
>>>>>>
>>>>>> Your config looks fine here.
>>>>>>
>>>>>> >
>>>>>> > Everything worked fine until I added the USB <hostdev> element. I'm
>>>>>> > essentially trying to get access to a physical Android device
>>>>>> connected to
>>>>>> > the host from inside a container. When I go to start the container,
>>>>>> I get
>>>>>> > an error about Operation not permitted. Here's the relevant bits
>>>>>> from
>>>>>> > /var/log/libvirt/lxc/machine.log:
>>>>>> >
>>>>>> > 2014-04-11 22:46:40.491+0000: starting up
>>>>>> > PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin
>>>>>> > LIBVIRT_DEBUG=3 LIBVIRT_LOG_OUTPUTS=3:stderr
>>>>>> /usr/lib/libvirt/libvirt_lxc
>>>>>> > --name oshi32134 --console 24 --security=none --handshake 27
>>>>>> --background
>>>>>> > --veth vnet1
>>>>>> > 2014-04-11 22:46:40.597+0000: 685: info : libvirt version: 1.2.2
>>>>>> > 2014-04-11 22:46:40.597+0000: 685: error :
>>>>>> > virLXCControllerSetupHostdevSubsysUSB:1390 : Unable to create device
>>>>>> > //var/run/libvirt/lxc/oshi32134.dev/bus/usb//002//003: Operation not
>>>>>> > permitted
>>>>>> > Unable to create device
>>>>>> > //var/run/libvirt/lxc/oshi32134.dev/bus/usb//002//003: Operation not
>>>>>> > permitted
>>>>>>
>>>>>> Do you have AppArmour enabled on the machine. That seems like the
>>>>>> most likely thing that would result in libvirt getting that permission
>>>>>> error.
>>>>>>
>>>>>> Regards,
>>>>>> Daniel
>>>>>> --
>>>>>> |: http://berrange.com      -o-
>>>>>> http://www.flickr.com/photos/dberrange/ :|
>>>>>> |: http://libvirt.org              -o-
>>>>>> http://virt-manager.org :|
>>>>>> |: http://autobuild.org       -o-
>>>>>> http://search.cpan.org/~danberr/ :|
>>>>>> |: http://entangle-photo.org       -o-
>>>>>> http://live.gnome.org/gtk-vnc :|
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/libvirt-users/attachments/20140418/13d39a2a/attachment.htm>

More information about the libvirt-users mailing list