Using vGPU

vGPU (virtual graphics processing unit) is a software emulation of a physical GPU (graphics processing unit). vGPU allows you to distribute graphics processor and graphics card memory resources among several virtual machines (VMs). VMs take turns using the GPU power and allocate a part of video memory for their needs. When a VM is shut down or suspended, vGPU resources are released. For more information about the technology, read the Nvidia documentation.

vGPU technology in VMmanager allows:

• users to create VMs with GPUs to accelerate graphics computing (for example, for applications in virtual desktops);
• administrators to allocate resources of a single GPU device among multiple VMs.

If there are multiple GPU devices installed on a cluster node, some of them can be used for GPU passthrough and the rest can be used to create vGPU-enabled VMs.

GPU resources are allocated between VMs using special configurations — vGPU profiles. Profiles allow you to allocate a certain amount of resources to each VM.

Feature limitations

vGPU technology is only supported for certain Nvidia GPU devices. For a list of supported devices, see the Nvidia documentation.

vGPU and the following platform features are not supported to work together:

• enable high availability in the cluster if a VM with vGPU is on the list of VMs to recover;
• enabling the balancer in the cluster;
• evacuation of VMs with vGPU in cluster node maintenance mode; 

  To switch a node that has a VM with a vGPU to maintenance mode, you can:

  • not to evacuate these VMs;
  • disable vGPU on these VMs.
• VM snapshot creation;
• VM migration.

Preparation

BIOS configuration

For correct vGPU operation, enable IOMMU (Input/Output Memory Management Unit) and SR-IOV (Single Root I/O Virtualization) support in the BIOS settings on the cluster nodes.

Checking devices

Make sure that GPU devices are correctly detected by the system before configuring the passthrough:

1. Connect to the cluster node via SSH. For more information about connecting via SSH, see Workstation setup.
2. Run the command:

   lspci | grep -iE "vga|3d" 

3. Examine the command output. 
   Example output

   00:02.0 VGA compatible controller: Intel Corporation UHD Graphics 630 (rev 0c)
   01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 470/480]
   02:00.0 VGA compatible controller: NVIDIA Corporation TU106 [GeForce RTX 2060 SUPER] (rev a1)

4. If the command output does not contain device information, make sure that:
   1. The device is properly installed in the server.
   2. The device is active in the BIOS (UEFI) of the server.

Configuring IOMMU

IOMMU (Input/Output Memory Management Unit) is a memory management technology for I/O devices. IOMMU provides virtualization of physical memory addresses used by PCI devices and isolates resources between guest operating systems.

For correct device passthrough, enable IOMMU in the GRUB bootloader settings:

1. Connect to the cluster node via SSH. For more information about connecting via SSH, see Workstation setup.
2. In the /etc/default/grub file, add the IOMMU parameters to the end of the GRUB_CMDLINE_LINUX_DEFAULT variable. The parameter names depend on the type of processor installed in the server:
   • Intel — iommu=on, intel_iommu=on; 
     Example variable

     GRUB_CMDLINE_LINUX_DEFAULT="quiet splash iommu=on intel_iommu=on"

   • AMD — iommu=on, amd_iommu=on; 
     Example variable

     GRUB_CMDLINE_LINUX_DEFAULT="quiet splash iommu=on amd_iommu=on"

3. Update the GRUB configuration:
   AlmaLinux

   sudo grub2-mkconfig -o /boot/grub2/grub.cfg

   Ubuntu

   sudo update-grub

4. Reboot the server: 

   sudo reboot

Installing the Nvidia driver

Install the Nvidia driver for vGPUs on the cluster nodes following the instructions in the official documentation.

Installing mdevctl

Install mdevctl, the vGPU management utility, on the cluster nodes:

sudo dnf -y install mdevctl

sudo apt -y install mdevctl

vGPU management

Device selection

To select the devices for which the vGPU will be used:

1. Go to Nodes → select a node → Parameters button → PCI devices section → PCI devices tab.
2. For each device, in the Using column, click the icon → select vGPU.
3. If any devices on the node have previously been used for GPU passthrough, reboot the cluster node.

To disable vGPU usage for a device:

1. Disable vGPU in the fine-tuning settings of VMs that use this device.
2. Go to Nodes → select a node → Parameters button → PCI devices section → PCI devices tab.
3. In the Using column, click the  icon → select Not used.
   These actions will remove the vGPU profiles from the platform and VM configurations.

Tab interface

Profile information

To view the vGPU profiles, go to Nodes → select a node → Parameters button → PCI devices section → vGPU profiles tab. When vGPU mode is enabled, a list of profiles appears on the tab.

To update the profile information, click the Update profile list button.

The profile table contains the following information:

• PCI device name;
• device number;
• profile name;
• number of displays;
• frame rate;
• frame buffer;
• maximum resolution;
• number of VMs using the profile.

Tab interface

Configuring vGPU on VMs

To enable VMs to use vGPU, when creating the configuration in the Virtualization parameters section:

1. In the Using the graphics card (GPU) parameter, select vGPU.
2. Select a vGPU profile from the list. 
   The list displays all device profiles, including those that are temporarily unavailable.

You can change these settings in the VM's fine-tuning settings. If you change the parameters, the VM will be rebooted. During the VM reboot period, the parameter change will not be available.

Example settings