My working solution was to use the first pcie x16 slot for host and the second for guest. And why are you using the 5th slot - it is pcie x8. The third slot is pcie x16. Use vga=none to find out the real reason for the problem.