<div dir="ltr">Indeed, we've applied that code change, compiled and installed the patched source, and things work great!<div><br></div><div>That was a fun code dive :D</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">
On Thu, Apr 17, 2014 at 2:44 PM, Filip Maj <span dir="ltr"><<a href="mailto:fil@saucelabs.com" target="_blank">fil@saucelabs.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Me again!<div><br></div><div>Think we've found it.</div><div><br></div><div>By diving into the LXC logs for the specific container, we found this:</div><div><br></div><div>2014-04-17 21:07:06.066+0000: 2861: debug : virCgroupSetValueStr:678 : Set value '/sys/fs/cgroup/devices/machine/oshi32134.libvirt-lxc/devices.allow' to 'c 189:130 rw'<br>
</div><div><br></div><div>Looks like libvirt the permission to 'rw', not 'rmw' [1], so no surprise that when it goes to call mknod [2], it can't do so.</div><div><br></div><div>We are going to patch this so it gives 'rwm' access in the cgroup whitelist, try to compile and replace libvirt to see if that works out for us. Will report back :)</div>
<div><br></div><div>[1] <a href="http://libvirt.org/git/?p=libvirt.git;a=blob;f=src/lxc/lxc_cgroup.c;h=39d955cd8cc2e34b6157e126fbb455d8f71e0647;hb=e8684eb541f01df9b45e87e0a8ce446c7bc90a17#l334" target="_blank">http://libvirt.org/git/?p=libvirt.git;a=blob;f=src/lxc/lxc_cgroup.c;h=39d955cd8cc2e34b6157e126fbb455d8f71e0647;hb=e8684eb541f01df9b45e87e0a8ce446c7bc90a17#l334</a></div>
<div>[2] <a href="http://libvirt.org/git/?p=libvirt.git;a=blob;f=src/lxc/lxc_controller.c;h=5ca960f13e53501315b28f9086eaa389475b8feb;hb=e8684eb541f01df9b45e87e0a8ce446c7bc90a17#l1387" target="_blank">http://libvirt.org/git/?p=libvirt.git;a=blob;f=src/lxc/lxc_controller.c;h=5ca960f13e53501315b28f9086eaa389475b8feb;hb=e8684eb541f01df9b45e87e0a8ce446c7bc90a17#l1387</a></div>
</div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Apr 17, 2014 at 11:26 AM, Filip Maj <span dir="ltr"><<a href="mailto:fil@saucelabs.com" target="_blank">fil@saucelabs.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Ah, nevermind, think my last post may not have been entirely correct. We've spent some more time correlating the log output from our failed LXC startup via libvirt [1] with libvirt code from the 1.2.2 tag. Interestingly, the error we get back is different, and digging through the multitudes of logs, nothing comes up about 'operating not permitted' regarding the USB bus that I saw earlier. If we turn off the verbose logging, we get the original error back. The more-verbose error is a bit more cryptic:<div>
<br></div><div>libvirt LXC Driver error: internal error: guest failed to start: 2014-<br><div>
<br></div><div>Anyways, here's a rough overview of the logs:<div><br></div><div>1. it starts to check/detect for cgroup stuff [2]</div><div>2. starts to 'prepare host devices' and logs out that the appropriate usb address is initialized (002/003) [3]</div>
<div>3. sets the appropriate usb bus to 'active host devs' [4]</div><div>4. seems to then switch to doing network-related things, and stop talking about any hostdev/usb stuff [5]</div><div>5. eventually invokes `libvirt_lxc` [6]</div>
<div>6. does the cgroup song-n-dance again for some reason? [7]. not sure if this is critical or not.</div><div>7. eventually some kind of socket read wire error comes up [8]</div><div>8. and this leads to a 'guest failed to start error' [9], which it probably truncates to return the error i mentioned above.</div>
<div><br></div><div>What is also confusing is, looking through the libvirt lxc code, shouldn't we be seeing a series of "Allowing any <blah>" log debug outputs in our logs? This is part of the <span style="font-size:12px;white-space:pre-wrap;font-family:monospace">virLXCCgroupSetupDeviceACL </span>part of the lxc_cgroup.c file [10]. I imagine that we'd eventually want the <span style="font-size:12px;white-space:pre-wrap;font-family:monospace">virCgroupAllowDeviceMajor </span>method [11] invoked, no?</div>
<div><br></div><div>I realize that you are the author of most of this code, Daniel, so appreciate your patience with me and hope you are willing to continue trying to get to the bottom of this with us! Thanks a lot in advance!</div>
<div><br></div><div>Cheers,</div><div>Fil</div><div><div>
<br></div><div>[1] <a href="https://gist.github.com/filmaj/207ab3c09f5b881d83b1" target="_blank">https://gist.github.com/filmaj/207ab3c09f5b881d83b1</a></div></div><div>[2] <a href="https://gist.github.com/filmaj/207ab3c09f5b881d83b1#file-libvirtd-log-L226-L253" target="_blank">https://gist.github.com/filmaj/207ab3c09f5b881d83b1#file-libvirtd-log-L226-L253</a></div>
<div>[3] <a href="https://gist.github.com/filmaj/207ab3c09f5b881d83b1#file-libvirtd-log-L257-L266" target="_blank">https://gist.github.com/filmaj/207ab3c09f5b881d83b1#file-libvirtd-log-L257-L266</a></div><div>[4] <a href="https://gist.github.com/filmaj/207ab3c09f5b881d83b1#file-libvirtd-log-L296" target="_blank">https://gist.github.com/filmaj/207ab3c09f5b881d83b1#file-libvirtd-log-L296</a></div>
<div>[5] <a href="https://gist.github.com/filmaj/207ab3c09f5b881d83b1#file-libvirtd-log-L302" target="_blank">https://gist.github.com/filmaj/207ab3c09f5b881d83b1#file-libvirtd-log-L302</a></div><div>[6] <a href="https://gist.github.com/filmaj/207ab3c09f5b881d83b1#file-libvirtd-log-L685" target="_blank">https://gist.github.com/filmaj/207ab3c09f5b881d83b1#file-libvirtd-log-L685</a></div>
<div>[7] <a href="https://gist.github.com/filmaj/207ab3c09f5b881d83b1#file-libvirtd-log-L752-L799" target="_blank">https://gist.github.com/filmaj/207ab3c09f5b881d83b1#file-libvirtd-log-L752-L799</a></div></div><div>[8] <a href="https://gist.github.com/filmaj/207ab3c09f5b881d83b1#file-libvirtd-log-L1075" target="_blank">https://gist.github.com/filmaj/207ab3c09f5b881d83b1#file-libvirtd-log-L1075</a></div>
<div>[9] <a href="https://gist.github.com/filmaj/207ab3c09f5b881d83b1#file-libvirtd-log-L1090" target="_blank">https://gist.github.com/filmaj/207ab3c09f5b881d83b1#file-libvirtd-log-L1090</a></div></div><div>[10] <a href="http://libvirt.org/git/?p=libvirt.git;a=blob;f=src/lxc/lxc_cgroup.c;h=39d955cd8cc2e34b6157e126fbb455d8f71e0647;hb=e8684eb541f01df9b45e87e0a8ce446c7bc90a17#l342" target="_blank">http://libvirt.org/git/?p=libvirt.git;a=blob;f=src/lxc/lxc_cgroup.c;h=39d955cd8cc2e34b6157e126fbb455d8f71e0647;hb=e8684eb541f01df9b45e87e0a8ce446c7bc90a17#l342</a></div>
<div>[11] <a href="http://libvirt.org/git/?p=libvirt.git;a=blob;f=src/lxc/lxc_cgroup.c;h=39d955cd8cc2e34b6157e126fbb455d8f71e0647;hb=e8684eb541f01df9b45e87e0a8ce446c7bc90a17#l446" target="_blank">http://libvirt.org/git/?p=libvirt.git;a=blob;f=src/lxc/lxc_cgroup.c;h=39d955cd8cc2e34b6157e126fbb455d8f71e0647;hb=e8684eb541f01df9b45e87e0a8ce446c7bc90a17#l446</a></div>
</div><div><div><div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, Apr 16, 2014 at 5:11 PM, Filip Maj <span dir="ltr"><<a href="mailto:fil@saucelabs.com" target="_blank">fil@saucelabs.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Further followups!<div><br></div><div>We are correlating DEBUG-level output from libvirt with the libvirt 1.2.2 code to try to figure out what libvirt is doing under the hood.</div>
<div><br></div><div>Even though we have the log level set to 1 (info) in our libvirtd.conf, we are not seeing the VIR_DEBUG() [1] statements being printed out. There are tons of other presumably-debug lines of output showing up in our log. We are sort of expecting to see the output from [1] in our logs somewhere, given the type of usb mounting we are trying to do?</div>
<div><br></div><div>[1] <a href="http://libvirt.org/git/?p=libvirt.git;a=blob;f=src/lxc/lxc_controller.c;h=c05dfec6428cad927cd5751004a4f3afc67899de;hb=HEAD#l1396" target="_blank">http://libvirt.org/git/?p=libvirt.git;a=blob;f=src/lxc/lxc_controller.c;h=c05dfec6428cad927cd5751004a4f3afc67899de;hb=HEAD#l1396</a></div>
</div><div><div><div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, Apr 16, 2014 at 3:58 PM, Filip Maj <span dir="ltr"><<a href="mailto:fil@saucelabs.com" target="_blank">fil@saucelabs.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">To follow up on this a little bit, tail'ing kern.log while trying to get our little container up doesn't yield anything with apparmor complaining, so, unless I'm looking in the wrong spots for apparmor logs (which I don't think so, as I see other apparmor-related log entries in kern.log), I am not entirely sure this is an apparmor issue at this point.</div>
<div><div>
<div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, Apr 16, 2014 at 3:25 PM, Filip Maj <span dir="ltr"><<a href="mailto:fil@saucelabs.com" target="_blank">fil@saucelabs.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Yeah, AppArmor is enabled, but I put everything (that I could find) into complain mode:<div><br></div><div><div>$ sudo apparmor_status</div><div>apparmor module is loaded.</div><div>12 profiles are loaded.</div>
<div>3 profiles are in enforce mode.</div><div> lxc-container-default</div><div> lxc-container-default-with-mounting</div><div> lxc-container-default-with-nesting</div><div>9 profiles are in complain mode.</div><div>
/sbin/dhclient</div><div> /usr/bin/lxc-start</div><div> /usr/lib/NetworkManager/nm-dhcp-client.action</div><div> /usr/lib/connman/scripts/dhclient-script</div><div> /usr/lib/libvirt/virt-aa-helper</div><div> /usr/sbin/libvirtd</div>
<div> /usr/sbin/ntpd</div><div> /usr/sbin/rsyslogd</div><div> /usr/sbin/tcpdump</div><div>3 processes have profiles defined.</div><div>0 processes are in enforce mode.</div><div>2 processes are in complain mode.</div>
<div> /usr/sbin/libvirtd (30419)</div><div> /usr/sbin/ntpd (3418)</div><div>1 processes are unconfined but have a profile defined.</div><div> /usr/sbin/rsyslogd (626)</div></div><div><br></div><div>And still get issues. From libvirtd.log:</div>
<div><br></div><div><div>2014-04-16 22:19:10.855+0000: 30419: info : libvirt version: 1.2.2</div><div>2014-04-16 22:19:10.855+0000: 30419: error : virNetSocketReadWire:1446 : Cannot recv data: Connection reset by peer</div>
<div>2014-04-16 22:19:10.940+0000: 30420: error : virLXCProcessStart:1299 : internal error: guest failed to start: Unable to create device //var/run/libvirt/lxc/oshi32134.dev/bus/usb//002//003: Operation not permitted</div>
<div><br></div><div>2014-04-16 22:19:10.964+0000: 30420: warning : virLXCDomainReAttachHostUsbDevices:388 : Unable to find device 000.000 in list of active USB devices</div></div><div><br></div><div>Thanks in advance for any help, Daniel!</div>
<div><br></div><div>Cheers,</div><div>Fil</div></div><div><div><div class="gmail_extra"><br><br><div class="gmail_quote">On Tue, Apr 15, 2014 at 1:33 AM, Daniel P. Berrange <span dir="ltr"><<a href="mailto:berrange@redhat.com" target="_blank">berrange@redhat.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div>On Fri, Apr 11, 2014 at 05:32:28PM -0700, Filip Maj wrote:<br>
> Hi!<br>
><br>
> First post, kind of a noobie. I've been working with LXC and libvirt for a<br>
> few months now. Trying to do some interesting things with containers and<br>
> Android devices :D<br>
</div><div><div>> Here's my entire domain definition:<br>
><br>
> <domain type='lxc'><br>
> <name>oshi32134</name><br>
> <uuid>xxxxx</uuid><br>
> <memory unit='KiB'>3145728</memory><br>
> <currentMemory unit='KiB'>3145728</currentMemory><br>
> <vcpu placement='static'>1</vcpu><br>
> <resource><br>
> <partition>/machine</partition><br>
> </resource><br>
> <os><br>
> <type arch='i686'>exe</type><br>
> <init>/sbin/init</init><br>
> </os><br>
> <clock offset='utc'/><br>
> <on_poweroff>destroy</on_poweroff><br>
> <on_reboot>restart</on_reboot><br>
> <on_crash>destroy</on_crash><br>
> <devices><br>
> <emulator>/usr/lib/libvirt/libvirt_lxc</emulator><br>
> <filesystem type='mount' accessmode='passthrough'><br>
> <source dir='/some/valid/filesystem/location'/><br>
> <target dir='/'/><br>
> </filesystem><br>
> <filesystem type='mount' accessmode='passthrough'><br>
> <source dir='/another/valid/filesystem/location'/><br>
> <target dir='/mnt/android'/><br>
> </filesystem><br>
> <interface type='bridge'><br>
> <mac address='xx:xx:xx:xx:xx:xx'/><br>
> <source bridge='br1'/><br>
> </interface><br>
> <console type='pty'><br>
> <target type='lxc' port='0'/><br>
> </console><br>
> <hostdev mode='capabilities' type='misc'><br>
> <source><br>
> <char>/dev/kvm</char><br>
> </source><br>
> </hostdev><br>
> <hostdev mode='subsystem' type='usb' managed='yes'><br>
> <source><br>
> <vendor id='0x04e8'/><br>
> <product id='0x6860'/><br>
> </source><br>
> </hostdev><br>
> </devices><br>
> </domain><br>
<br>
</div></div>Your config looks fine here.<br>
<div><br>
><br>
> Everything worked fine until I added the USB <hostdev> element. I'm<br>
> essentially trying to get access to a physical Android device connected to<br>
> the host from inside a container. When I go to start the container, I get<br>
> an error about Operation not permitted. Here's the relevant bits from<br>
> /var/log/libvirt/lxc/machine.log:<br>
><br>
> 2014-04-11 22:46:40.491+0000: starting up<br>
> PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin<br>
> LIBVIRT_DEBUG=3 LIBVIRT_LOG_OUTPUTS=3:stderr /usr/lib/libvirt/libvirt_lxc<br>
> --name oshi32134 --console 24 --security=none --handshake 27 --background<br>
> --veth vnet1<br>
> 2014-04-11 22:46:40.597+0000: 685: info : libvirt version: 1.2.2<br>
> 2014-04-11 22:46:40.597+0000: 685: error :<br>
> virLXCControllerSetupHostdevSubsysUSB:1390 : Unable to create device<br>
> //var/run/libvirt/lxc/oshi32134.dev/bus/usb//002//003: Operation not<br>
> permitted<br>
> Unable to create device<br>
> //var/run/libvirt/lxc/oshi32134.dev/bus/usb//002//003: Operation not<br>
> permitted<br>
<br>
</div>Do you have AppArmour enabled on the machine. That seems like the<br>
most likely thing that would result in libvirt getting that permission<br>
error.<br>
<div><div><br>
Regards,<br>
Daniel<br>
--<br>
|: <a href="http://berrange.com" target="_blank">http://berrange.com</a> -o- <a href="http://www.flickr.com/photos/dberrange/" target="_blank">http://www.flickr.com/photos/dberrange/</a> :|<br>
|: <a href="http://libvirt.org" target="_blank">http://libvirt.org</a> -o- <a href="http://virt-manager.org" target="_blank">http://virt-manager.org</a> :|<br>
|: <a href="http://autobuild.org" target="_blank">http://autobuild.org</a> -o- <a href="http://search.cpan.org/~danberr/" target="_blank">http://search.cpan.org/~danberr/</a> :|<br>
|: <a href="http://entangle-photo.org" target="_blank">http://entangle-photo.org</a> -o- <a href="http://live.gnome.org/gtk-vnc" target="_blank">http://live.gnome.org/gtk-vnc</a> :|<br>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>