<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"> <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> <meta name="Generator" content="Microsoft Word 14 (filtered medium)"> <style></style> </head> <body lang="ZH-CN" link="blue" vlink="purple"> <div class="WordSection1"> <p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D">Hi Filip,<o:p></o:p></span></p> <p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D">I just checked about my machine.<o:p></o:p></span></p> <p class="MsoListParagraph" style="margin-left:18.0pt;text-indent:-18.0pt;mso-list:l1 level1 lfo1"> <![if !supportLists]><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D"><span style="mso-list:Ignore">1)<span style="font:7.0pt "Times New Roman""> </span></span></span><![endif]><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D">Just like you said, in my container, the read/write permission of the exposed PCIe device is also different from the host machine. Only the root can read/write the PCIe card. Other users/groups are not allowed to access the card.<o:p></o:p></span></p> <p class="MsoListParagraph" style="margin-left:18.0pt;text-indent:-18.0pt;mso-list:l1 level1 lfo1"> <![if !supportLists]><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D"><span style="mso-list:Ignore">2)<span style="font:7.0pt "Times New Roman""> </span></span></span><![endif]><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D">Both my host machine and lxc are fedora16. As I login in the lxc, I became root automatically, so I can access the card without problem. As I know, if Ubuntu is used, the root might be treated differently from Fedora. I am not sure if this is the reason.<o:p></o:p></span></p> <p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D">For next step<o:p></o:p></span></p> <p class="MsoListParagraph" style="margin-left:18.0pt;text-indent:-18.0pt;mso-list:l0 level1 lfo2"> <![if !supportLists]><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D"><span style="mso-list:Ignore">1)<span style="font:7.0pt "Times New Roman""> </span></span></span><![endif]><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D">You can try fedora<o:p></o:p></span></p> <p class="MsoListParagraph" style="margin-left:18.0pt;text-indent:-18.0pt;mso-list:l0 level1 lfo2"> <![if !supportLists]><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D"><span style="mso-list:Ignore">2)<span style="font:7.0pt "Times New Roman""> </span></span></span><![endif]><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D">Open a new post to see if lxc experts can help you<o:p></o:p></span></p> <p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D">Good luck!<o:p></o:p></span></p> <p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D">Cheng<o:p></o:p></span></p> <p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p> <p class="MsoNormal"><b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Filip Maj [mailto:fil@saucelabs.com] <br> <b>Sent:</b> 2014</span><span style="font-size:10.0pt">年</span><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">4</span><span style="font-size:10.0pt">月</span><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">15</span><span style="font-size:10.0pt">日</span><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> 1:57<br> <b>To:</b> WANG Cheng D<br> <b>Cc:</b> libvirt-users@redhat.com<br> <b>Subject:</b> Re: [libvirt-users] LXC + USB passthrough = Operation not permitted<o:p></o:p></span></p> <p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p> <div> <p class="MsoNormal"><span lang="EN-US">I'd also like to point out that, when setting up a linux container with _just_ the lxc tools available in ubuntu, all that was needed to be done was whitelist the appropriate cgroups for the Android devices. The lxc tools seem to create the appropriate character devices when you detach/attach the Android device via USB, and the Android `adb` tool recognized the devices.<o:p></o:p></span></p> <div> <p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p> </div> <div> <p class="MsoNormal"><span lang="EN-US">So, a tangential question would be, what libvirt domain XML ends up "converting" into a cgroup whitelist line in the LXC configuration file? (see <a href="https://wiki.archlinux.org/index.php/Linux_Containers#Host_device_access_settings">https://wiki.archlinux.org/index.php/Linux_Containers#Host_device_access_settings</a> for examples of cgroups configuration in a vanilla LXC config file).<o:p></o:p></span></p> </div> </div> <div> <p class="MsoNormal" style="margin-bottom:12.0pt"><span lang="EN-US"><o:p> </o:p></span></p> <div> <p class="MsoNormal"><span lang="EN-US">On Mon, Apr 14, 2014 at 10:44 AM, Filip Maj <<a href="mailto:fil@saucelabs.com" target="_blank">fil@saucelabs.com</a>> wrote:<o:p></o:p></span></p> <div> <p class="MsoNormal"><span lang="EN-US">So, if I remove the hostdev element related to the USB device, running `lsusb` and `usb-devices` inside the container lists the proper device, but I do not see the appropriate character device bus for the Android device under /dev/bus/usb. We are not entirely sure that `lsusb` and `usb-devices` queries the appropriate location - it somehow gets information from the host, not the guest container, as far as we can tell.<o:p></o:p></span></p> <div> <p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p> </div> <div> <p class="MsoNormal"><span lang="EN-US">If we specify the appropriate usb bus character device path (like you suggest, Cheng), then it does appear inside the container, but the permissions and ownership on the device seem to be incorrect. Here are the details of the appropriate device busses in the host (002 and 003 are the specific Android devices):<o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p> </div> <div> <div> <p class="MsoNormal"><span lang="EN-US">$ ll /dev/bus/usb/002<o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span lang="EN-US">total 0<o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span lang="EN-US">drwxr-xr-x 2 root root 100 Apr 9 18:14 ./<o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span lang="EN-US">drwxr-xr-x 10 root root 200 Apr 9 18:14 ../<o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span lang="EN-US">crw-rw-r-- 1 root root 189, 128 Apr 9 18:14 001<o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span lang="EN-US">crw-rw-r-- 1 root plugdev 189, 129 Apr 9 18:14 002<o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span lang="EN-US">crw-rw-r-- 1 root plugdev 189, 130 Apr 10 15:25 003<o:p></o:p></span></p> </div> </div> <div> <p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p> </div> <div> <p class="MsoNormal"><span lang="EN-US">Then, I removed the old mode="subsystem" hostdev element and added the following hostdev elements:<o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p> </div> <div> <div> <div> <p class="MsoNormal"><span lang="EN-US"> <hostdev mode='capabilities' type='misc'><o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span lang="EN-US"> <source><o:p></o:p></span></p> </div> </div> <div> <p class="MsoNormal"><span lang="EN-US"> <char>/dev/bus/usb/002/002</char><o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span lang="EN-US"> </source><o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span lang="EN-US"> </hostdev><o:p></o:p></span></p> </div> <div> <div> <p class="MsoNormal"><span lang="EN-US"> <hostdev mode='capabilities' type='misc'><o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span lang="EN-US"> <source><o:p></o:p></span></p> </div> </div> <div> <p class="MsoNormal"><span lang="EN-US"> <char>/dev/bus/usb/002/003</char><o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span lang="EN-US"> </source><o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span lang="EN-US"> </hostdev><o:p></o:p></span></p> </div> </div> <div> <p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p> </div> <div> <p class="MsoNormal"><span lang="EN-US">Then I get inside the guest container and list out the usb busses we just mounted:<o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p> </div> <div> <div> <p class="MsoNormal"><span lang="EN-US">$ ll /dev/bus/usb/002/<o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span lang="EN-US">total 0<o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span lang="EN-US">drwxr-xr-x 2 root root 80 Apr 14 17:43 ./<o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span lang="EN-US">drwxr-xr-x 3 root root 60 Apr 14 17:43 ../<o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span lang="EN-US">crwx------ 1 root root 189, 129 Apr 14 17:43 002<o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span lang="EN-US">crwx------ 1 root root 189, 130 Apr 14 17:43 003<o:p></o:p></span></p> </div> </div> <div> <p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p> </div> <div> <p class="MsoNormal"><span lang="EN-US">You can see the group that owns the character devices is different, and the read/write permissions are different, too. Not sure if this is the root of the issue, but, I noticed that difference. In any case, the Android devices are still not recognized by the container.<o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p> </div> <div> <p class="MsoNormal"><span lang="EN-US">Any help is appreciated!<o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p> </div> <div> <p class="MsoNormal"><span lang="EN-US">Cheers,<o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span lang="EN-US">Fil<o:p></o:p></span></p> </div> </div> <div> <div> <div> <p class="MsoNormal" style="margin-bottom:12.0pt"><span lang="EN-US"><o:p> </o:p></span></p> <div> <p class="MsoNormal"><span lang="EN-US">On Sun, Apr 13, 2014 at 8:50 PM, WANG Cheng D <<a href="mailto:Cheng.d.Wang@alcatel-sbell.com.cn" target="_blank">Cheng.d.Wang@alcatel-sbell.com.cn</a>> wrote:<o:p></o:p></span></p> <div> <div> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D">Can you see the USB device in the lxc?</span><span lang="EN-US"><o:p></o:p></span></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D">My answering assumes you cannot see the USB device in the container. If you can see it, please ignore my answer.</span><span lang="EN-US"><o:p></o:p></span></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D">According to my understanding, “hostdev mode='subsystem' “ is used with KVM only. We use this if we want to passthrough a device to a KVM virtual machine. After doing that, the host machine will lose control of the device. If container is used, the container is not a real virtual machine, it’s just a group of processes and it shares the OS kernel with the host. If a device cannot be seen by the host machine, I guess it also cannot been seen by the container.</span><span lang="EN-US"><o:p></o:p></span></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D">I used the following XML scripts to expose a third-party PCIe card to the container successfully, and I can access the card in the lxc.</span><span lang="EN-US"><o:p></o:p></span></p> <div> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D"><hostdev mode=’capabilities’ type=’misc’></span><span lang="EN-US"><o:p></o:p></span></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D"> <source></span><span lang="EN-US"><o:p></o:p></span></p> </div> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D"> <char/dev/mydevice</char></span><span lang="EN-US"><o:p></o:p></span></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D"> </source></span><span lang="EN-US"><o:p></o:p></span></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D"></hostdev></span><span lang="EN-US"><o:p></o:p></span></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D">Although I am not sure if my card is a char device, but it really works.</span><span lang="EN-US"><o:p></o:p></span></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D">If I use “ hostdev mode=’subsystem’”, it will fail to expose the PCIe card.</span><span lang="EN-US"><o:p></o:p></span></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><span lang="EN-US"><o:p></o:p></span></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Filip Maj [mailto:<a href="mailto:fil@saucelabs.com" target="_blank">fil@saucelabs.com</a>] <br> <b>Sent:</b> 2014</span><span style="font-size:10.0pt">年</span><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">4</span><span style="font-size:10.0pt">月</span><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">14</span><span style="font-size:10.0pt">日</span><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> 11:26<br> <b>To:</b> WANG Cheng D<br> <b>Cc:</b> <a href="mailto:libvirt-users@redhat.com" target="_blank">libvirt-users@redhat.com</a><br> <b>Subject:</b> RE: [libvirt-users] LXC + USB passthrough = Operation not permitted</span><span lang="EN-US"><o:p></o:p></span></p> <div> <div> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> <o:p></o:p></span></p> <p><span lang="EN-US">Hi Chang Weng,<o:p></o:p></span></p> <p><span lang="EN-US">Thanks for answering!<o:p></o:p></span></p> <p><span lang="EN-US">Could you explain why subsystem=USB does not make sense? I was simply following the documentation for USB devices (right above the section you linked to).<o:p></o:p></span></p> <p><span lang="EN-US">For what it's worth, I've also tried mounting the specific character device related to the appropriate android device into the container in the way you suggest, with no luck; `adb` still does not recognize the device.<o:p></o:p></span></p> <div> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US">On Apr 13, 2014 7:10 PM, "WANG Cheng D" <<a href="mailto:Cheng.d.Wang@alcatel-sbell.com.cn" target="_blank">Cheng.d.Wang@alcatel-sbell.com.cn</a>> wrote:<o:p></o:p></span></p> <div> <div> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US">Dear Fil,<o:p></o:p></span></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US">I am not sure if my answer can help you.<o:p></o:p></span></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US">I had ever asked a similar question to Daniel and I was using a thirty-party card. As a container uses a shared kernel with the host, so hostdev mode='subsystem' doesn’t make sense. Maybe you can try to use hostdev mode='capabilities’. Please see <a href="http://libvirt.org/formatdomain.html#elementsHostDevCaps" target="_blank"> http://libvirt.org/formatdomain.html#elementsHostDevCaps</a><o:p></o:p></span></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US">Hope this helps<o:p></o:p></span></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US">Cheng Wang<o:p></o:p></span></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> <o:p></o:p></span></p> </div> </div> </div> </div> </div> </div> </div> </div> <p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p> </div> </div> </div> </div> <p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p> </div> </div> </body> </html>