Nvidia GPU passthrough fail with code 43 - gpu

Question

I'm currently trying to pass a nvidia GPU to Windows 10 guest using qemu 2.5 and libvirt 1.3.5.
I see "Error 43" on Nvidia GPU in Device Manager.
I had tried to hide the hypervisor by adding "kvm=off" and "hv_vendor_id=123456780ab", but it does not work for me. I searched in google and people solved the problem in this way.
And I also saw Virtual Machine : Yes in task manager.
Did I use in the wrong way? I can pass a AMD gpu to windows guest(AMD does not check the kvm virtualization).
Can I spoof nvidia in other way?
My system information:
#uname -a
Linux ns.mqcache.net 4.2.0-1.el7.elrepo.x86_64 #1 SMP Sun Aug 30 21:25:29 EDT 2015 x86_64 x86_64 x86_64 GNU/Linux
#/root/qemu25/qemu/x86_64-softmmu/qemu-system-x86_64 --version
QEMU emulator version 2.5.1.1, Copyright (c) 2003-2008 Fabrice Bellard
GPU:
02:00.0 VGA compatible controller: NVIDIA Corporation GF119 [GeForce GT 620 OEM] (rev a1)
02:00.1 Audio device: NVIDIA Corporation GF119 HDMI Audio Controller (rev a1)
libvirt.xml
<domain xmlns:qemu="http://libvirt.org/schemas/domain/qemu/1.0" type="kvm">
<name>win10</name>
<os>
<type machine="q35">hvm</type>
<boot dev="hd"/>
<boot dev="cdrom"/>
</os>
<features>
<acpi/>
<apic/>
<hyperv>
<vendor_id state='on' value='1234567890ab'/>
</hyperv>
<kvm>
<hidden state='on'/>
</kvm>
</features>
<clock offset="localtime">
<timer name="rtc" tickpolicy="catchup"/>
<timer name="pit" tickpolicy="delay"/>
<timer name="hpet" present="no"/>
<timer name='hypervclock' present='yes'/>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<vcpu current="4">4</vcpu>
<cpu mode="host-passthrough">
<topology sockets="1" cores="4" threads="1"/>
</cpu>
<memory>8388608</memory>
<currentMemory>8388608</currentMemory>
<devices>
<emulator>/root/qemu25/qemu/x86_64-softmmu/qemu-system-x86_64</emulator>
<disk device="disk" type="file">
<driver name="qemu" type="qcow2"/>
<source file="/root/vm/win10/image.qcow2"/>
<target bus="virtio" dev="vda"/>
</disk>
<sound model="ac97"/>
<interface type="bridge">
<mac address="fa:16:3e:81:00:03"/>
<source bridge="eucabr"/>
<model type="virtio"/>
<driver name="qemu"/>
<alias name="net0"/>
</interface>
<hostdev mode="subsystem" type="pci" managed="yes">
<source>
<address domain="0x0000" bus="0x02" slot="0x00" function="0x1"/>
</source>
</hostdev>
</devices>
<qemu:commandline>
<qemu:arg value="-machine"/>
<qemu:arg value="smm=off"/>
<qemu:arg value="-device"/>
<qemu:arg value="ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1"/>
<qemu:arg value="-device"/>
<qemu:arg value="vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on"/>
<qemu:arg value="-vga"/>
<qemu:arg value="none"/>
</qemu:commandline>
</domain>
qemu command
/root/qemu25/qemu/x86_64-softmmu/qemu-system-x86_64 \
-name win10 \
-machine q35,accel=kvm,usb=off \
-cpu host,kvm=off,hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time,hv_vendor_id=blah \
-m 2048 \
-realtime mlock=off \
-smp 2,sockets=1,cores=2,threads=1 \
-no-user-config \
-nodefaults \
-rtc base=localtime \
-no-shutdown \
-boot strict=on \
-device i82801b11-bridge,id=pci.1,bus=pcie.0,addr=0x1e \
-device pci-bridge,chassis_nr=2,id=pci.2,bus=pci.1,addr=0x1 \
-drive file=/root/vm/win10/snap.qcow2,if=none,id=drive-virtio-disk0,format=qcow2 \
-device virtio-blk-pci,scsi=off,bus=pci.2,addr=0x2,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 \
-k en-us \
-device virtio-balloon-pci,id=balloon0,bus=pci.2,addr=0x4 \
-machine smm=off \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=02:00.1,bus=root.1,addr=00.1 \
-msg timestamp=on \
-vga none
Look forward to your help!

Answer 1

You need to pass copy of unmodified videocard ROM to VM.
You need a secondary GPU that you can use as the primary for this
process. You cannot dump a clean copy of the BIOS without having the
passthrough GPU as a secondary card
Put the extra card in the primary slot and the intended passthrough card in another pci-e port and bootup.
Find your intended GPU again via lspci -v. In my case it had about the same address.
Now you can dump the ROM to a file:
# echo "0000:05:00.0" > /sys/bus/pci/drivers/vfio-pci/unbind
# cd /sys/bus/pci/devices/0000\:05\:00.0
# echo 1 > rom
# cat rom > /home/username/KVM/evga_gtx970.dump
# echo 0 > rom
# echo "0000:05:00.0" > /sys/bus/pci/drivers/vfio-pci/bind
In this case, 0000:05:00.0 is my PCI card address. You don't really need the bind step at the bottom since you'll be rebooting anyways.
You can check the integrity of the ROM dump with this handy utility at https://github.com/awilliam/rom-parser. My rom looks like:
# ./rom-parser evga_gtx970.dump
Valid ROM signature found #0h, PCIR offset 1a0h
PCIR: type 0 (x86 PC-AT), vendor: 10de, device: 13c2, class: 030000
PCIR: revision 0, vendor revision: 1
Valid ROM signature found #f400h, PCIR offset 1ch
PCIR: type 3 (EFI), vendor: 10de, device: 13c2, class: 030000
PCIR: revision 3, vendor revision: 0
EFI: Signature Valid, Subsystem: Boot, Machine: X64
Last image
You should have both an EFI and a non-EFI x86 ROM in the dump ( I think most cards have both)
Turn off the machine and put your GTX 1070 back in the primary slot.
After booting, edit your VM xml and in the section for your GPU (if you have already assigned the GPU to the VM) there should be a section. Add a file='path/to/dump/here' statement to it. My full section looks like:
<hostdev mode='subsystem' type='pci' managed='yes'>
<source>
<address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
</source>
<rom bar='on' file='/home/username/KVM/evga_gtx970.dump'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</hostdev>
This will have the VM start the card with that BIOS instead of whatever the kernel gives it.
source
Please note that you have to use OVMF (EFI) because SeaBIOS does not use card ROM properly.

Answer 2

If you're on OVMF or some other UEFI, make sure to triple-check that your card is UEFI-ready, especially on stuff that is older than ~2014.
I was under false impression that mine was (GTX 770), while, in fact, it wasn't (looked at the wrong version of ROM online) and wasted almost 2 days ripping my hair out. Look up UEFI support like so and look for ROM updates here.
I flashed my card, but I think you can put an UEFI-enabled ROM as romfile=. It appears that other manufacturers' ROMs could work too, if yours doesn't have an UEFI fix for you.