[vfio-users] Another Nvidia code 43 error

Sat Nov 19 17:58:41 UTC 2016

Hold on everyone..... I got it to work (and by I, I truly mean it was a
group effort).  Here's what I did.

So out of the 10's of different combinations I tried, the one thing I
hadn't tried was flashing UEFI onto my GTX760, so I did that, rebooted the
host, fired up the VM and for the first time since removing the spice
display adapter I saw a resolution higher than 640x480. Opening up Device
Manager shows that the GTX 760 is working properly. ( I have another PCI
device that is unknown, but this is at least some progress. I think the
other PCI device is the sound part of my GTX 760, since the VM says there
are no sound devices).

I tried this because of the other thread I saw this morning (Support for
Passthrough) and reading through that and the links indicated that OVMF
wouldn't work without UEFI on the card. I tried just having the rom loaded
at VM boot but that didn't work.

I followed this guide for both finding the UEFI rom to flash as well as the
tools to flash it (there are linux tools for flashing it from nvidia).
http://www.tomshardware.com/faq/id-2288384/upgrade-gpu-bios.html

On Fri, Nov 18, 2016 at 12:30 PM Will Marler <will at wmarler.com> wrote:

Sorry, I'm at a loss : (. Maybe there's a BIOS setting you need to play
with ... maybe the motherboard just can't handle it. I only see 2 entries
for an "ASUS X79 Deluxe" (there are lots of Asus P9X19 Deluxe's though) in
https://docs.google.com/spreadsheets/d/1LnGpTrXalwGVNy0PWJDURhyxa3sgqkGXmvNCIvIMenk/edit?pli=1#gid=0
(both from the same guy with the exact same specs and one lists working
"no" and the other with working "yes", so I'm hesitant to find it a useful
entry), which doesn't bode well.

I've never tried a headless host with the guest using passed-throughV VGA
card ... if you want to try that route I'd suggest booting the VM with no
video card connected and using VNC to uninstall the nVidia drivers (or make
a new guest) and seeing if Windows will boot & recognize the card and if *that
*works then you can go to the next step of nVidia drivers.

On Fri, Nov 18, 2016 at 12:43 PM, Thomas Mashos <thomas at mashos.com> wrote:

Ok, tried having the 760 in the first slot and the 430 in the second slot.
Same issue, when I boot the Win 10 VM the whole host seems to hang (can't
access it locally or remotely, doesn't respond to ping). Does this indicate
that Linux isn't giving up the card properly or something else?

On Fri, Nov 18, 2016 at 11:26 AM Thomas Mashos <thomas at mashos.com> wrote:

Hi Will,

So booting with just the one card is more work than just pulling the 430. I
can boot linux, but it's using the 760 even though it's loading the
pci-stub driver. As soon as I boot the win10 VM (after changing the VM to
include the 01 PCI device instead of the 02 PCI device) The whole box seems
to hang. I'll try shutting it down and having the 430 be the second card in
the box to see if that makes any difference.

On Thu, Nov 17, 2016 at 4:46 PM Thomas Mashos <thomas at mashos.com> wrote:

Hi Will,

I haven't tried swapping the two cards yet, nor pulling just the one. I'll
try that tomorrow and see what happens.

The motherboard I have is an ASUS X79-DELUXE.

On Thu, Nov 17, 2016 at 11:17 AM Will Marler <will at wmarler.com> wrote:

Sorry about that, I just took the time to read through the full thread and
I found your updated XML. I see you've already tried with all of the hyperv
lines removed.

Early in the thread you mentioned that "if I pull the GT 430 from the box
that I can see the boot on the GTX 760." If you pull the GT 430, do you
clear up the code 43 error? This could be worth trying, though I've never
tried running a guest on a headless machine personally (I've read others
have done it).

Have you tried swapping the slots that the 760 & 430 occupy?

Maybe there's something in the BIOS... What kind of motherboard do you
have?

On Thu, Nov 17, 2016 at 11:57 AM, Thomas Mashos <thomas at mashos.com> wrote:

Hi Will,

No, I've not been able to get the NVidia drivers working at all, in any of
my tests.

On Thu, Nov 17, 2016 at 10:55 AM Will Marler <will at wmarler.com> wrote:

Sorry if you've said this already, but are you able to get the nVidia
drivers to work if you remove all hyper-v enablements?

From
http://vfio.blogspot.com/2015/05/vfio-gpu-how-to-series-part-4-our-first.html
:
"Within the <features> section, remove everything between the <hyperv>
tags, including the tags themselves.  In their place add the following tags:

    <kvm>
      <hidden state='on'/>
    </kvm>

Additionally, within the <clock> tag, find the timer named hypervclock,
remove the line containing this tag completely."

You have the <kvm><hidden state='on'/></kvm> tags and you removed the
hypervclock within the <clock> tags, but you have
<features><hyperv><vendor_id state='on' value='whatever'/></hyperv>

That might be confusing the nVidia drivers.

If you get it working as per the blog post, then try moving forward to the
next increment, enabling hyperv. For reference, here are the relevant
sections of my XML, with hyper-v enabled (I have a GeForce 750ti)

  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
      <vendor_id state='on' value='123456789ab'/>
    </hyperv>
    <kvm>
      <hidden state='on'/>
    </kvm>
    <vmport state='off'/>
  </features>

  <cpu mode='host-passthrough'>
    <topology sockets='1' cores='2' threads='1'/>
  </cpu>

  <clock offset='localtime'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
    <timer name='hypervclock' present='yes'/>
  </clock>

One other thing to note: the "value" string of the vendor_id tag might be
required to be a certain number of characters. I recall reading that
somewhere, but I can't quote it, so I could be wrong. It's an easy thing to
test though (*after* you have the nVidia driver working without hyperv
enabled first!)

On Thu, Nov 17, 2016 at 10:08 AM, Thomas Mashos <thomas at mashos.com> wrote:

Hello Brett,

I just ran GPU-Z and the UEFI box is not checked. So no UEFI it seems

On Thu, Nov 17, 2016 at 8:33 AM Brett Peckinpaugh <erylflynn at gmail.com>
wrote:

Has anyone verified if the video card has uefi firmware?

On November 17, 2016 7:33:32 AM PST, Thomas Mashos <thomas at mashos.com>
wrote:

Sorry, guess I didn't reply all last time.

Just uninstalled the Nvidia driver and let windows search for and install
the driver. Let it reboot a few times, no deal. Still code 43 :(

I'm still using the default Ubuntu kernel
Linux smitty 4.4.0-47-generic #68-Ubuntu SMP Wed Oct 26 19:39:52 UTC 2016
x86_64 x86_64 x86_64 GNU/Linux

Here's my current XML, which I tried changing the CPU topology to match
yours, still getting error code 43.
http://termbin.com/3n44

Not sure what else to try here, maybe I should try a Win 7 install? I think
I still have one of those DVD's laying around.

On Wed, Nov 16, 2016 at 10:33 PM Eddie Yen <missile0407 at gmail.com> wrote:

OK....now things change to difficult.

Can you post your XML again after you edit?
Also, did you change kernel or still using Ubuntu official version?
I don't want to suspect but I think it had few connection inside.

And here is my XML code when running K420 /w Windows 7, as a reference.
Also don't forget to reply the mail back to VFIO milling list, maybe other
people can help :)

2016-11-17 13:34 GMT+08:00 Thomas Mashos <thomas at mashos.com>:

Just uninstalled the Nvidia driver and let windows search for and install
the driver. Let it reboot a few times, no deal. Still code 43 :(

On Wed, Nov 16, 2016 at 7:33 PM Eddie Yen <missile0407 at gmail.com> wrote:

Roger that,

What about try the driver version from Windows 10, not NVIDIA official
download?
I remember Windows 10 has built GeForce 700 Series driver inside.

2016-11-17 11:04 GMT+08:00 Thomas Mashos <thomas at mashos.com>:

I've now tried both those options (I tried a few different machine types
too) with no luck.

I forgot to mention that I also tried the setting the interrupts to MSI via
the registry for the GTX 760, but that didn't seem to switch it to MSI on
reboot.

On Wed, Nov 16, 2016 at 6:08 PM Eddie Yen <missile0407 at gmail.com> wrote:

Hi Thomas,

Basically KVM <hidden state=on/> equals kvm=off, so you can keep it.

And there is no need to reinstall NVIDIA driver whenever you change XML
code.

If you already tried remove Hyper-V features and still got Code 43,
there're few ways you can try.

1. Set CPU as "host-passthrough" mode
2. You can type virsh capabilities to see how many machine type you can
use, and try different i440fx version.
Some GPU may got a problem if using higher version of i440fx

I'm running K420 on server, and guest OS is Windows 7. But I'm not using
third-party ppa to install newer QEMU and libvirt, all I do is compile and
install by myself. Using OVMF and i440fx-2.6 and it works perfectly without
Hyper-V (Because Windows 7 can't enable Hyper-V in OVMF mode.)

I'm not sure these tip works on you or not, just give it a try :)

2016-11-17 9:40 GMT+08:00 Thomas Mashos <thomas at mashos.com>:
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/vfio-users/attachments/20161119/1318c68b/attachment.htm>