GPU passthrough

You can configure direct connection (passthrough) of hardware with a graphics processing unit (GPU) to a virtual machine (VM). This will allow you to:

use the GPU performance on the VM for user tasks (for example, machine learning and modeling);
run graphics-intensive applications (game platforms, computer-aided design systems, image processing programs) on VMs;
isolate GPU device resources — each GPU device will only be used by a specific VM.

Feature limitations

GPU passthrough is supported for PCI (PCIe) devices. Only the platform administrator can manage GPU passthrough.

GPU passthrough and the following platform features are not supported to work together:

enable high availability in the cluster if a GPU VM is on the list of VMs to recover;
enabling the balancer in the cluster;
evacuation of VMs with GPU in cluster node maintenance mode;
To switch a node that has a VM with a GPU to maintenance mode, you can:
- not to evacuate these VMs;
- disable GPU passthrough on these VMs.
VM snapshot creation.

Migration of VMs with GPUs is not supported.

Preparation

Checking devices

Make sure that GPU devices are correctly detected by the system before configuring the passthrough:

Connect to the cluster node via SSH. For more information about connecting via SSH, see Workstation setup.
Run the command:
```
lspci | grep -iE "vga|3d" 
```

Examine the command output.

Example output

00:02.0 VGA compatible controller: Intel Corporation UHD Graphics 630 (rev 0c)
01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 470/480]
02:00.0 VGA compatible controller: NVIDIA Corporation TU106 [GeForce RTX 2060 SUPER] (rev a1)

If the command output does not contain device information, make sure that:
1. The device is properly installed in the server.
2. The device is active in the BIOS (UEFI) of the server.

Configuring IOMMU

IOMMU (Input/Output Memory Management Unit) is a memory management technology for I/O devices. IOMMU provides virtualization of physical memory addresses used by PCI devices and isolates resources between guest operating systems.

For correct device passthrough, enable IOMMU in the GRUB bootloader settings:

Connect to the cluster node via SSH. For more information about connecting via SSH, see Workstation setup.
In the /etc/default/grub file, add the IOMMU parameters to the end of the GRUB_CMDLINE_LINUX_DEFAULT variable. The parameter names depend on the type of processor installed in the server:
- Intel — iommu=on, intel_iommu=on;
  Example variable
```
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash iommu=on intel_iommu=on"
```
- AMD — iommu=on, amd_iommu=on;
  Example variable
```
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash iommu=on amd_iommu=on"
```

Update the GRUB configuration:

AlmaLinux

sudo grub2-mkconfig -o /boot/grub2/grub.cfg

Ubuntu

sudo update-grub

Reboot the server:
```
sudo reboot
```

Managing GPU devices

To get information about the GPU devices in a cluster, go to Clusters → select the cluster → Parameters button → PCI devices tab.

Tab interface

To manage GPU devices on a cluster node, go to Nodes → select the node → Parameters button → PCI devices tab. The tab displays information about the devices:

name;
vendor;
number;
usage — using by VN, occupied by node, free;
The status "Occupied by node" means that the device is used by the operating system or belongs to IOMMU Group number 0.
IOMMU Group — a group of PCI devices sharing a common IOMMU memory address space. A device group is created automatically by the system based on the hardware architecture and PCI bus configuration. Passthrough of devices from IOMMU Group 0 is not supported;
GPU passthrough — enable or disable;
note.

Tab interface

To enable or disable GPU passthrough for a device, click the radio button in the GPU passthrough column. You can enable or disable GPU passthrough for multiple devices at the same time. To do this, highlight the desired devices and click the Enable GPU passthrough (Disable GPU passthrough) button at the top of the table.

If the device is used by a VM, then disabling the passthrough is not supported.

If the device is used by a node, GPU passthrough is not available.

To change the note for the device, click the icon → Note.

To update the device information, click the Update device list button.

Configuring GPU on the VM

The number of GPU devices available for passthrough to the VM is set in the Passthrough GPU devices parameter of the VM configuration. Selecting specific GPUs for passthrough is not supported.

You can change this parameter in the VM's fine-tuning settings. If you decrease the setting, some GPUs will be disabled. Selecting specific GPUs to be disabled is not supported. After reducing the parameter, the VM will be rebooted. While the VM is rebooting, changing the setting will not be available.

The number of passthrough devices is displayed in the VM table in the GPU column. By default, this column is disabled. To enable its display, in the table header, click the icon → select the GPU checkbox.

Selling VMs with GPU via BILLmanager

To configure the sale of virtual server services with GPUs:

On the VMmanager side:
1. Create a VM configuration with the Passthrough GPU devices option. For more information, see the article Configurations of virtual machines.
2. Add the created configuration to the distribution filters on nodes with GPUs:
  1. Go to Nodes section → select the node → Parameters button → VM distribution settings tab.
  2. In the VM parameters field select Configurations (select).
  3. In the Parameter value field select the created configuration.
  4. Click the Save button.
On the BILLmanager side, when creating a tariff, select the created configuration in the VPS template parameter.

You can configure the sale of virtual server services with a specific type of GPU as follows. For example, you need to configure rates for selling virtual server services with RTX 6000 and RTX 4090 devices. To do this:

Install RTX 6000 devices in one of the cluster nodes (N1). Create the RTX 6000 configuration and specify it in the distribution filters on node N1.
Create the "Virtual Server with RTX 6000" tariff and specify the VPS template for it — RTX 6000.
Install RTX 4090 devices in another cluster node (N2). Create the RTX 4090 configuration and specify it in the distribution filters on node N2.
Create the "Virtual Server with RTX 4090" tariff and specify the VPS template for it — RTX 4090.

Possible issues

Driver initialization errors

After Nvidia devices passthrough, errors may occur in Linux guest operating systems due to attempts to initialize the nouveau driver. To fix the errors, install Nvidia drivers in the guest OS or add the nouveau kernel module to the blacklist. To add a module to the blacklist:

Connect to the VM via SSH.
Add lines to the configuration file /etc/modprobe.d/blacklist.conf:
```
# VMmanager GPU passthrough fix for Nvidia
blacklist nouveau
```

Useful tips