How to configure IOMMU device for nested Proxmox hypervisor (PVE) VM - PCIe Passthrough

Overview

Configuring PCIe Passthrough for Nested Virtualization on Proxmox

Summary:

If you are running bare-metal L0 (level 0) Proxmox (PVE) hypervisor with nested PVE hypervisor in L1 VM, and you wish to passthrough a PCIe device, physically attached to the bare-metal machine, to a VM inside the L1 VM (nested PVE), then this may be for you.

This was tested on a machine using Intel Xeon X5670, which has the required Virtualization Technology for Directed I/O (VT-d) support with Proxmox PVE v7.0-2.

Challenges:

I have a PCIe-to-SATA expansion card added to my computer and a SATA DVD drive was connected to it. Ultimately, I wanted the L2 VM to be able to fully control the drive.

Most PCIe passthrough guides found all over the Internet do not cover how one can set up passthrough for L2 VM residing in L1 nested hypervisor on top of L0 bare-metal hypervisor due to nested VM being two levels deep, which is needless for most people due to potential performance issues and no live migration support.

If you are like me, who prefers to experiment with passthrough on VMs directly on L1 PVEs due to clustering, then you have come to the right place.

Unfortunately, even Proxmox's PCIe passthrough guide doesn't cover nested virtualization. The minimal OS configuration needed for PCIe passthrough for supported Intel machine is updating the kernel command-line parameter, intel_iommu=on, which will be covered in later steps.

Requirements:

This guide assumes some familiarity with Proxmox (e.g. creating VMs).

There are two sets of requirements we need to fulfill as one is intended for nested virtualization and the other is for PCIe passthrough.

The below requirements are intended for the L1 PVE VM. Not the L2 VM guest.

Nested Virtualization:
  • CPU on bare-metal machine requires hardware-assisted virtualization extensions (e.g. Intel VT-x, AMD-V) which needs to be enabled in BIOS/UEFI.
  • Nested PVE (L1) using the same PVE version as the bare-metal PVE (L0)
  • Nested PVE VM needs to have CPU set to host type (ex. qm set --cpu host )
  • Nested PVE VM needs to use ovmf BIOS implementation (ex. qm set --bios ovmf )
  • Nested PVE VM requires q35 QEMU machine type (ex. qm set --machine q35 )

NOTE: It's extremely important to use q35 QEMU machine type and ovmf BIOS implementation together before booting up the machine. Otherwise, a kernel panic may occur, even at the bare-metal level and a hard reboot may be required to unseize the machine.

Nested virtualization may already be enabled for you if you are running the latest version of PVE. PVE 7.0 seems to have it enabled by default. To check, run cat /sys/module/kvm_intel/parameters/nested on L0 and you'd get either Y or N.

You can also configure the options through the web GUI instead of qm command. It's recommended to add only EFI disk on a storage location with no extra options but not required.

Screenshot of Create Virtual Machine - System option with OVMF BIOS and q35 machine selected

PCIe Passthrough:
  • Bare-metal requires CPU and motherboard with IOMMU (I/O Memory Management Unit) interrupt remapping support (e.g. Intel VT-d, AMD-Vi). Many modern CPUs and motherboards support this. This needs to be enabled in BIOS/UEFI settings.
  • For nested PVE, CPU also needs to be host type, ovmf BIOS implementation is required, and q35 QEMU machine type is necessary. This requirement is already outlined in the Nested Virtualization section.

Configuring Passthrough:

Enable IOMMU for L0:

Before we can proceed, we'd need to make sure the L0 hypervisor (bare-metal PVE) has IOMMU enabled, which can be done by tuning the kernel command-line parameter.

As mentioned in Proxmox documentation, you need to add intel_iommu=on in the bootloader configuration file depending on which one one you are using. Usually BIOS are found on legacy systems whereas UEFI are on modern systems. GRUB bootloader is typically for BIOS system, and EFI bootloader is for UEFI. Please see Proxmox documentation on editing the kernel command-line.

A reboot would be required after updating the kernel parameter in bootloader configuration file.

To verify, run dmesg | grep -e DMAR -e IOMMU -e AMD-Vi and if you see lines containing IOMMU, Directed I/O, or Interrupt Remapping being enabled, then you are set.

Enable IOMMU for L1 PVE:

This assumes L1 PVE has been created and you are ready to create a VM from L1 PVE virtual machine.

Before we can get to the next step, you also need to modify the kernel parameter for the L1 PVE the same way you did for L0 PVE in the previous section.

Create L2 VM:

You should already have the ability to add PCI device to the L1 VM through the Hardware settings.

At this point, you should be able to passthrough a PCIe device from L0 to L1 PVE. It would be listed in lspci -nnk output in L1 PVE's shell.

Proceed to create the L2 VM using any compatible Linux guest OS of your choice (Windows hasn't been tested) with the following virtual hardware requirements.

  • OVMF BIOS implementation must be used
  • q35 machine type is required

Go ahead attempt to add the passthrough PCI device as well.

What did you get? A warning like the one below?

No IOMMU detected error while adding PCI device to L2 VM

Despite updating the kernel parameter, you are faced with this error until you add a vIOMMU device to the L1 PVE, which was missing all along.

Where is the vIOMMU option?

You may wonder where you may have missed this option while creating the PVE. It turns out, it's not natively included with Proxmox tools at the time of this writing as VT-d emulation would need to be added in QEMU.

How VM is run on PVE is through the /usr/bin/qemu-system-x86_64 command. You would see that /usr/bin/kvm is a symbolic link to the previous command that powers the VMs on PVE.

Run the qm showcmd command for any configured VM of your choice, and you would see a very long kvm command. The long command derives from the configurations that you'd make to the VM.

To view the VM configuration via command, run qm config , which is human-readable as all options are in alphabetical order. Please see more information on qm command here.

According to the QEMU wiki article previously mentioned, the options needed to add the vIOMMU device to L1 PVE can theoretically be done through the qm set --args command.

qm set 100 --args '-device intel-iommu,intremap=on,caching-mode=on -machine accel=kvm,kernel-irqchip=split'

The above change would also reflect in the qm config and qm showcmd commands.

However... Because qm set --args would simply append the missing options to the end of the long kvm command, it would not work. The QEMU article said the intel-iommu device must be specificed before all other device parameters that you'd see in the kvm command.

You would have to shutdown the PVE, modifying the kvm command and add the missing options. The major drawback is that this change would have to be made each time you boot if you need IOMMU support for passthrough.

Adding vIOMMU to L1 PVE:

Fortunately, I've created a Python script to simplify the job. Hopefully, one day the Proxmox team can embed the vIOMMU option in a future release.

Just simply go to the L0 PVE's shell and run the commands below using the L1 PVE's VM ID. The script will not send a reboot signal and it'd passively wait for the VM to shutdown (not reboot as the script would be booting back up the VM).

wget https://raw.githubusercontent.com/bashtheshell/IOMMU-nested-pve/main/add_vIOMMU.py
python3 add_vIOMMU.py 
   

   

Please note that the above script may not be compatible with older version of PVE as this was tested on 7.0. At least Python 3.5 would be required.

Also, the script would have to be run at each boot when IOMMU is needed.

Re-attempt to add passthrough PCI device to L2 VM:

Previously, you may have received an error message on L2 VM that IOMMU isn't detected. Now that should no longer be the case.

Upon booting up the L2 VM, you should be able to see it in lspci -nnk output inside the guest OS. If yes, YOU ARE ALL SET!

If you are still facing challenges, then chances are you may be required to do further tweaking. Please review Proxmox's PCIe passthrough documentations.

Special Thanks:

Thanks to thenickdude for his contribution to my Reddit post, which steered me to the solution.

And of course, the Proxmox and QEMU/KVM teams for making all things possible for us homelabbers and tinkers.

Owner
Travis Johnson
Linux system administration enthusiast. Loves to tinker with programming and networking.
Travis Johnson
Home-Assistant MQTT bridge for Panasonic Comfort Cloud

Panasonic Comfort Cloud MQTT Bridge Home-Assistant MQTT bridge for Panasonic Comfort Cloud. Note: Currently this brige is a one evening prototype proj

Santtu Järvi 2 Jan 04, 2023
Programmable Rainbow Redstone Computer

Programmable Rainbow Redstone Computer Table of contents What is it? Program flasher How to use it What is it? PRRC is Programmable Rainbow Redstone C

Fern H 2 Jun 07, 2022
A battery pack simulation tool that uses the PyBaMM framework

Overview of liionpack liionpack takes a 1D PyBaMM model and makes it into a pack. You can either specify the configuration e.g. 16 cells in parallel a

PyBaMM Team 40 Jan 05, 2023
A python script for macOS to enable scrolling with the 3M ergonomic mouse EM500GPS in any application.

A python script for macOS to enable scrolling with the 3M ergonomic mouse EM500GPS in any application.

3 Feb 19, 2022
Bucatini: a soft PIPE PHY for FPGA SerDes

Bucatini: a soft PIPE PHY for FPGA SerDes Bucatini is a noodly gateware layer capable of transforming an FPGA SerDes into a PIPE PHY, allowing you to

Great Scott Gadgets 28 Dec 02, 2022
CPU benchmark by calculating Pi, powered by Python3

cpu-benchmark Info: CPU benchmark by calculating Pi, powered by Python 3. Algorithm The program calculates pi with an accuracy of 10,000 decimal place

Alex Dedyura 20 Jan 03, 2023
Rasberry Pie GPIO memory game. Press the corresponding key to the lit LED.

RPie-keyboard-game Rasberry Pie GPIO memory game. Press the corresponding key to the lit LED. Randem LED (general output) is lit up on rasberrypi rand

Shawn Dowling 1 Oct 24, 2021
CO2Ampel - This RaspberryPi project uses weather data to estimate the share of renewable energy in the power grid

CO2Ampel This RaspberryPi project uses weather data to estimate the share of ren

Felix 4 Jan 19, 2022
Baseline model for Augmented Home Assistant

Dataset Preparation Step 1. Rename the Virtual-Home output directory to 'vh.[name]', for example: 'vh.door' Make sure the directory contains 100+ fram

Stanford HCI 1 Aug 24, 2022
A Macropad using the Raspberry Pi Pico, programmed with CircuitPython.

A Macropad using the Raspberry Pi Pico, programmed with CircuitPython.

15 Oct 14, 2022
Lenovo Legion 5 Pro 2021 Linux RGB Keyboard Light Controller

Lenovo Legion 5 Pro 2021 Linux RGB Keyboard Light Controller This util allows to drive RGB keyboard light on Lenovo Legion 5 Pro 2021 Laptop Requireme

36 Dec 16, 2022
Python Client for ESPHome native API. Used by Home Assistant.

aioesphomeapi aioesphomeapi allows you to interact with devices flashed with ESPHome. Installation The module is available from the Python Package Ind

ESPHome 76 Jan 04, 2023
Kwcpu - An unobtrusive CPU meter that fits in the default Windows 11 taskbar. Supports up to 32 cores.

kwcpu An unobtrusive CPU meter that fits in the default Windows 11 taskbar. Supports up to 32 cores. kwcpu is provided as a Rainmeter skin. By default

Jay Oster 2 Nov 07, 2022
A lightweight script for updating custom components for Home Assistant

Updater for Home Assistant This is a lightweight script for updating custom components for Home Assistant. If for some reason you do not want to use H

Alex X 12 Sep 21, 2022
A script for performing OTA update over BLE on ESP32

A script for performing OTA update over BLE on ESP32

Felix Biego 18 Dec 15, 2022
Python library to interact with the GCE Electronics IPX800 device

A python library to control a GCE-Electronics IPX800 V4 device through its API.

Marc-Aurèle Brothier 2 Oct 20, 2021
♟️ QR Code display for P4wnP1 (SSH, VNC, any text / URL)

♟️ Display QR Codes on P4wnP1 (p4wnsolo-qr) 🟢 QR Code display for P4wnP1 w/OLED (SSH, VNC, P4wnP1 WebGUI, any text / URL / exfiltrated data) Note: Th

PawnSolo 4 Dec 19, 2022
Home Assistant custom integration for e-distribución

e-Distribución is an energy distribution company that covers most of South Spain area. If you live in this area, you probably are able to register into their website to get some information about you

VMG 17 Sep 07, 2022
View your medication from Medisafe Cloud in Home Assistant

Medisafe View your medication from Medisafe Cloud in Home Assistant. This integration adds sensors for today's upcoming, taken, skipped, and missed do

Sam Steele 12 Dec 27, 2022
A iot Bike sytem based on RaspberryPi, Ardiuino

Cyclic 's Kernel ---- A iot Bike sytem based on RaspberryPi, Ardiuino, etc 0x1 What is This? Cyclic 's Kernel is an independent System With self-produ

Retr0mous 2 Oct 09, 2022