[vfio-users] Another Nvidia code 43 error

[paigeadele at gmail.com]

Post a block comment of your domxml or script on my gist I sent you last night if you can, congrats!

Sent from my Windows 10 phone

From: Thomas Mashos
Sent: Saturday, November 19, 2016 10:00 AM
To: Will Marler
Cc: vfio-users
Subject: Re: [vfio-users] Another Nvidia code 43 error

Hold on everyone..... I got it to work (and by I, I truly mean it was a group effort).  Here's what I did.

So out of the 10's of different combinations I tried, the one thing I hadn't tried was flashing UEFI onto my GTX760, so I did that, rebooted the host, fired up the VM and for the first time since removing the spice display adapter I saw a resolution higher than 640x480. Opening up Device Manager shows that the GTX 760 is working properly. ( I have another PCI device that is unknown, but this is at least some progress. I think the other PCI device is the sound part of my GTX 760, since the VM says there are no sound devices).

I tried this because of the other thread I saw this morning (Support for Passthrough) and reading through that and the links indicated that OVMF wouldn't work without UEFI on the card. I tried just having the rom loaded at VM boot but that didn't work.

I followed this guide for both finding the UEFI rom to flash as well as the tools to flash it (there are linux tools for flashing it from nvidia).  http://www.tomshardware.com/faq/id-2288384/upgrade-gpu-bios.html 



On Fri, Nov 18, 2016 at 12:30 PM Will Marler <will at wmarler.com> wrote:
Sorry, I'm at a loss : (. Maybe there's a BIOS setting you need to play with ... maybe the motherboard just can't handle it. I only see 2 entries for an "ASUS X79 Deluxe" (there are lots of Asus P9X19 Deluxe's though) in https://docs.google.com/spreadsheets/d/1LnGpTrXalwGVNy0PWJDURhyxa3sgqkGXmvNCIvIMenk/edit?pli=1#gid=0 (both from the same guy with the exact same specs and one lists working "no" and the other with working "yes", so I'm hesitant to find it a useful entry), which doesn't bode well.

I've never tried a headless host with the guest using passed-throughV VGA card ... if you want to try that route I'd suggest booting the VM with no video card connected and using VNC to uninstall the nVidia drivers (or make a new guest) and seeing if Windows will boot & recognize the card and if that works then you can go to the next step of nVidia drivers.

On Fri, Nov 18, 2016 at 12:43 PM, Thomas Mashos <thomas at mashos.com> wrote:
Ok, tried having the 760 in the first slot and the 430 in the second slot. Same issue, when I boot the Win 10 VM the whole host seems to hang (can't access it locally or remotely, doesn't respond to ping). Does this indicate that Linux isn't giving up the card properly or something else?

On Fri, Nov 18, 2016 at 11:26 AM Thomas Mashos <thomas at mashos.com> wrote:
Hi Will,

So booting with just the one card is more work than just pulling the 430. I can boot linux, but it's using the 760 even though it's loading the pci-stub driver. As soon as I boot the win10 VM (after changing the VM to include the 01 PCI device instead of the 02 PCI device) The whole box seems to hang. I'll try shutting it down and having the 430 be the second card in the box to see if that makes any difference.

On Thu, Nov 17, 2016 at 4:46 PM Thomas Mashos <thomas at mashos.com> wrote:
Hi Will,

I haven't tried swapping the two cards yet, nor pulling just the one. I'll try that tomorrow and see what happens. 

The motherboard I have is an ASUS X79-DELUXE.

On Thu, Nov 17, 2016 at 11:17 AM Will Marler <will at wmarler.com> wrote:
Sorry about that, I just took the time to read through the full thread and I found your updated XML. I see you've already tried with all of the hyperv lines removed. 

Early in the thread you mentioned that "if I pull the GT 430 from the box that I can see the boot on the GTX 760." If you pull the GT 430, do you clear up the code 43 error? This could be worth trying, though I've never tried running a guest on a headless machine personally (I've read others have done it).

Have you tried swapping the slots that the 760 & 430 occupy?

Maybe there's something in the BIOS... What kind of motherboard do you have? 

On Thu, Nov 17, 2016 at 11:57 AM, Thomas Mashos <thomas at mashos.com> wrote:
Hi Will,

No, I've not been able to get the NVidia drivers working at all, in any of my tests. 

On Thu, Nov 17, 2016 at 10:55 AM Will Marler <will at wmarler.com> wrote:
Sorry if you've said this already, but are you able to get the nVidia drivers to work if you remove all hyper-v enablements? 

From http://vfio.blogspot.com/2015/05/vfio-gpu-how-to-series-part-4-our-first.html:
"Within the <features> section, remove everything between the <hyperv> tags, including the tags themselves.  In their place add the following tags:

    <kvm>
      <hidden state='on'/>
    </kvm>

Additionally, within the <clock> tag, find the timer named hypervclock, remove the line containing this tag completely."

You have the <kvm><hidden state='on'/></kvm> tags and you removed the hypervclock within the <clock> tags, but you have <features><hyperv><vendor_id state='on' value='whatever'/></hyperv>

That might be confusing the nVidia drivers.

If you get it working as per the blog post, then try moving forward to the next increment, enabling hyperv. For reference, here are the relevant sections of my XML, with hyper-v enabled (I have a GeForce 750ti)

  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
      <vendor_id state='on' value='123456789ab'/>
    </hyperv>
    <kvm>
      <hidden state='on'/>
    </kvm>
    <vmport state='off'/>
  </features>

  <cpu mode='host-passthrough'>
    <topology sockets='1' cores='2' threads='1'/>
  </cpu>

  <clock offset='localtime'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
    <timer name='hypervclock' present='yes'/>
  </clock>

One other thing to note: the "value" string of the vendor_id tag might be required to be a certain number of characters. I recall reading that somewhere, but I can't quote it, so I could be wrong. It's an easy thing to test though (after you have the nVidia driver working without hyperv enabled first!)

On Thu, Nov 17, 2016 at 10:08 AM, Thomas Mashos <thomas at mashos.com> wrote:
Hello Brett,

I just ran GPU-Z and the UEFI box is not checked. So no UEFI it seems 

On Thu, Nov 17, 2016 at 8:33 AM Brett Peckinpaugh <erylflynn at gmail.com> wrote:
Has anyone verified if the video card has uefi firmware? 

On November 17, 2016 7:33:32 AM PST, Thomas Mashos <thomas at mashos.com> wrote:
Sorry, guess I didn't reply all last time.

Just uninstalled the Nvidia driver and let windows search for and install the driver. Let it reboot a few times, no deal. Still code 43 :(

I'm still using the default Ubuntu kernel
Linux smitty 4.4.0-47-generic #68-Ubuntu SMP Wed Oct 26 19:39:52 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

Here's my current XML, which I tried changing the CPU topology to match yours, still getting error code 43.
http://termbin.com/3n44

Not sure what else to try here, maybe I should try a Win 7 install? I think I still have one of those DVD's laying around.


On Wed, Nov 16, 2016 at 10:33 PM Eddie Yen <missile0407 at gmail.com> wrote:
OK....now things change to difficult.

Can you post your XML again after you edit?
Also, did you change kernel or still using Ubuntu official version?
I don't want to suspect but I think it had few connection inside.

And here is my XML code when running K420 /w Windows 7, as a reference.
Also don't forget to reply the mail back to VFIO milling list, maybe other people can help :)

2016-11-17 13:34 GMT+08:00 Thomas Mashos <thomas at mashos.com>:
Just uninstalled the Nvidia driver and let windows search for and install the driver. Let it reboot a few times, no deal. Still code 43 :(

On Wed, Nov 16, 2016 at 7:33 PM Eddie Yen <missile0407 at gmail.com> wrote:
Roger that,

What about try the driver version from Windows 10, not NVIDIA official download?
I remember Windows 10 has built GeForce 700 Series driver inside.

2016-11-17 11:04 GMT+08:00 Thomas Mashos <thomas at mashos.com>:
I've now tried both those options (I tried a few different machine types too) with no luck.

I forgot to mention that I also tried the setting the interrupts to MSI via the registry for the GTX 760, but that didn't seem to switch it to MSI on reboot.

On Wed, Nov 16, 2016 at 6:08 PM Eddie Yen <missile0407 at gmail.com> wrote:
Hi Thomas,

Basically KVM <hidden state=on/> equals kvm=off, so you can keep it.

And there is no need to reinstall NVIDIA driver whenever you change XML code.

If you already tried remove Hyper-V features and still got Code 43, there're few ways you can try.

1. Set CPU as "host-passthrough" mode
2. You can type virsh capabilities to see how many machine type you can use, and try different i440fx version.
Some GPU may got a problem if using higher version of i440fx

I'm running K420 on server, and guest OS is Windows 7. But I'm not using third-party ppa to install newer QEMU and libvirt, all I do is compile and install by myself. Using OVMF and i440fx-2.6 and it works perfectly without Hyper-V (Because Windows 7 can't enable Hyper-V in OVMF mode.)

I'm not sure these tip works on you or not, just give it a try :)

2016-11-17 9:40 GMT+08:00 Thomas Mashos <thomas at mashos.com>:

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/vfio-users/attachments/20161119/4cb46f79/attachment.htm>