I've been thinking of trying out ESXi on my desktop system and running my actual desktop as a VM with gpu passthrough. This weekend I finally did it. It took some experimenting and there isn't a lot of good information out there. Initially, I tried using my standard centos 7 template. It saw the gpu but wouldn't actually output anything to the monitors. Then I tried my windows 2019 template and it had a similar problem. So finally I installed a fresh copy of Fedora and set the bios mode to efi and it actually worked. The usb controller for my keyboard and mouse just worked as well. The only devices I had problems with were the wifi adapter (possibly shared by the in use ethernet adapter) and the audio controller(claimed it was in use). The audio one is easily solved by getting a usb DAC since I know the usb controller works. I also have a quad port nic I can use to test my theory of passing through but wifi and ethernet adapters at the same time.
What I Did
- Create a new "AMD" cluster in vCenter
- Add the new desktop host to the AMD cluster
- add vlan/portgroups as needed
- add a datastore
- set graphics to "shared direct" per virtuallyGhetto
- set the pci devices you want to passthrough to "available" in the passthrough configuration
- Create a new VM and install Fedora KDE
- ensure the VM is set to EFI
- After installation and OS updates we're ready to connect the gpu
- add the gpu to the VM (some people suggest also adding the hdmi audio device that goes along with it, but I didn't need to)
svga.present = FALSEin the advanced VM Options (this will essentially disable the built in vmware graphics console)
- add usb controllers or other devices you need (adding one device at a time makes it easier to troubleshoot)
- Power on the VM and test it out
- Rebooting the host will cause orphaned VMs. For some reason I have trouble with host settings persisting. After a reboot it may be necessary to make the PCI devices available for passthrough again. Also completely unrelated but enabling the ssh server to start at boot didn't persist either.
- VM unable to power on: pci device is likely "in use" and unable to be reset and assigned to the VM
- Gigabyte B450 Aorus Pro WIFI
- 4x16GB Mushkin Redline DDR4 2666
- AMD R5 1600
- AMD RX 570 4GB
- Samsung 960 Evo NVME drive
- Coming soon: Fosi Audio Q5 DAC
For consumer level hardware this works amazingly well. Obviously I lose a little bit of freedom in that I now need to manage things through vcenter but that's not terrible. I still need to load it up and do some power usage tests but if I set the storage server up similarly I could see myself being able to power down the lab servers. If I upgraded to 4x32GB RAM I could almost imagine moving the storage server into the desktop as well and basically have an all in one system. If I moved the VMs to a local ssd datastore I would have less concerns of a nested storage server... It might be worth crunching the power numbers after all.
Now I just need to configure vagrant to deploy things on vCenter, other than that I believe my workflow will be surprisingly unchanged.