[vfio-users] Trying to pass through USB controller.

Tue Feb 23 15:57:37 UTC 2016

Hello,

On 02/23/2016 06:46 AM, Jonathan Scruggs wrote:
> How can I pass through the USB ports to the guest so that the items
> plugged into the ports appear to the guest?

In short, "sudo setpci -s0:14.0 0xd0.W=0x0000" will pass through all of
your ports.

I was planning to write a blog post about it this month, but I haven't
gotten to it yet, so I guess I'll write it here. I have been using this
setup since December, and everything works without any issues. I had
done like you (playing with the firmware options) and gotten nowhere; I
finally found a Google+ post[0] (the user appears to have been deleted)
with a link to the 9 series PCH datasheet[1], which has been immensely
useful.

I have only tried this with an AsRock Z97E-ITX/ac. It might not work on
your board if your firmware disables switching ports (see section
16.1.37). Your chipset datasheet is here[2]. The xHCI registers are in
chapter 17 in your datasheet, so you will have to adapt the section
references.

Step 1: Map your USB ports
==========================
This is easiest to do if you turn USB3 (xHCI) off completely. First, you
need to map your EHCI controllers to USB buses as seen by the kernel.
These numbers can change when you reboot your machine, so do this after
you have disabled USB3. Run `lspci` and look at the numbers after the
EHCI controllers (#1/#2). Then run the command below. You should get
similar output.

$ readlink -f /sys/bus/usb/devices/usb?
/sys/devices/pci0000:00/0000:00:14.0/usb1
/sys/devices/pci0000:00/0000:00:1a.0/usb2
/sys/devices/pci0000:00/0000:00:1d.0/usb3

Run `watch lsusb -t` in a terminal. Take a flash drive/mouse/etc. and
plug it into each port on your machine. There are 14 of them. If you
cannot find some, they might be in a mini-PCIe or M.2 slot, or just not
connected.

Use the bus numbers you found earlier, and the output of lsusb, to map
physical ports to logical ports on controllers. Eight of them will be on
the EHCI controller marked #1, and the other six will be on #2. Now sort
them, starting with EHCI #1 port 1, and ending with EHCI #2 port 6.

Next you should turn USB3 back on in your firmware settings.

Step 2: Pass through your controller
====================================
This is the easy part. Just add whichever controller you want to use in
the VM to your libvirt XML or qemu command line. Personally, I only ever
need to use USB2 devices in Windows, so I pass through both EHCI
controllers. However, I believe the EHCI controllers on the PCH do not
support MSI, so that may affect performance with devices like flash
drives. With a keyboard or mouse, it shouldn't make any difference.

Step 3: Route your USB ports
============================
Looking at section 16.1.36, we can connect individual ports to either
the EHCI or xHCI controller. On your sorted list of ports, mark which
ones you want to have available in the VM, and which ones you want to
use on the host. This is switchable at runtime, so you can create
several lists--I have some ports that are routed to EHCI all of the
time, and others I toggle back and forth.

Turn each list of port routes into a binary mask: 0 for each EHCI port,
and 1 for each xHCI port. The least significant bit is EHCI #1 port 1.
Extend it to 16 bits by setting the two most significant bits to 0. Then
convert it to hexadecimal. It should be something between 0x0000 and
0x3fff. Finally, run setpci (as root) to change the routing. Moved USB
devices will reset immediately. The command is

# setpci -s<xHCI PCI address> 0xd0.W=0x<the mask in hex>

For example, to move everything to the xHCI controller:

# setpci -s0:14.0 0xd0.W=0x3fff

Again, even though you are running this command to configure the xHCI
controller, the bits are in the order the ports appear to the EHCI
controllers. If, for some reason, you cannot turn off xHCI to find that
order, use table 2-2 in section 2.2 to map xHCI ports (what you see in
lsusb minus 1) to EHCI ports.

If you are connecting a USB3 (blue) port to a EHCI controller, there's
one additional step you (probably) need to take. You will need to either
1) remember to never plug a USB3 device into the port, or 2) turn off
USB3 on that port. See section 16.1.38 for the register to change. I
have not needed to do this, so I am not certain, but I believe the port
numbers here are as they appear to the xHCI controller.

Step 4: Toggle your port mask
=============================
If you have more than one keyboard/mouse or a KVM switch or synergy, you
are done. Personally, I use this method to avoid all of that, so I have
to have a way to switch my keyboard back and forth while running the VM.
The first step is to have a service running as root, that will run
setpci for you when you tell it to switch ports. I use the following
python script. You should only need to change the masks and port number
(at the bottom). Hopefully it doesn't get mangled in the email.

----------------SNIP----------------
#!/usr/bin/env python3

import socketserver
import subprocess
import syslog

def change_usb_state(action):
     global state
     if action == state: return
     if action == "toggle": action = "guest" if state == "host" else "host"
     syslog.syslog("Setting xHCI port routing mask to 
{:#x}".format(masks[action]))
     if subprocess.call(["setpci", "-s0:14.0", 
"0xd0.L=0x{:08x}".format(masks[action])]) == 0: state = action
     else: syslog.syslog("Calling setpci failed. Assuming routing is 
unchanged.")

class PCIHandler(socketserver.StreamRequestHandler):
     def handle(self):
         for line in self.rfile:
             action = line.strip().decode("utf-8")
             syslog.syslog("Got action request: {}".format(action))
             if action in ["guest", "host", "toggle"]: 
change_usb_state(action)

class ThreadedTCPServer(socketserver.ThreadingMixIn, 
socketserver.TCPServer): pass

masks = { "guest": 0x0fc, "host": 0x0ff }
state = "host"
server = ThreadedTCPServer(("127.0.0.1", 10), PCIHandler)
server.serve_forever()
----------------SNIP----------------

On Linux, you can then use netcat as any user to toggle your ports. I
use i3, and have a key mapped like so, which toggles my USB ports and
then locks my screen:

----------------SNIP----------------
bindsym Control+Mod1+Menu exec --no-startup-id busybox nc 127.0.0.1 10 
<<< toggle; exec --no-startup-id slock
----------------SNIP----------------

The second part is to connect qemu to your service. (You could also have
the service accept remote connections, but then anyone on your network
could control your USB ports. I only recommend this if you have an
isolated network to use with qemu, and you only have the service listen
on that network.) The simple and secure way to do this is to connect a
serial port in the VM to your service. I use Q35 and qemu directly; you
will have to adapt this to libvirt yourself.

----------------SNIP----------------
     -device isa-serial,chardev=com1,index=0 \
       -chardev socket,host=127.0.0.1,id=com1,port=10
----------------SNIP----------------

Third, you need some way to connect to the serial port from within the
guest. On a Linux guest, you can use picocom, socat, etc. On Windows,
the simple way is to use a PowerShell script:

----------------SNIP----------------
$port = New-Object System.IO.Ports.SerialPort COM1,115200,None,8,One
$port.Open()
$port.WriteLine("toggle")
$port.Close()
----------------SNIP----------------

Personally, I use AutoHotKey to map the same key in Windows to switching
my ports and locking my screen. I'd be happy to share it, but it is too
long to paste here.

Hopefully that should get you going. I apologize if this is too long,
but I wanted to be thorough. It definitely took a weekend to get set up,
but once it works, it is really rather seamless.

[0] 
https://plus.google.com/wm/quevuelvamatt/app/basic/stream/z13ki3ixwvubfpk4e04chfi45r3lyv4zeiw
[1] 
http://www.intel.com/content/dam/www/public/us/en/documents/datasheets/9-series-chipset-pch-datasheet.pdf
[2] 
http://www.intel.com/content/dam/www/public/us/en/documents/datasheets/x99-chipset-pch-datasheet.pdf

(See also sections 5.19-5.21 of the datasheet for more information.)

--
Regards,
Samuel Holland <samuel at sholland.org>