Error "Timed out during operation: cannot acquire state change lock" when stopping or starting a VM

Description

The issue manifests as an inability to stop or restart a virtual machine (VM). When performing these operations, the following error appears in the vmctl.log:

libvirt.libvirtError: Timed out during operation: cannot acquire state change lock (held by monitor=remoteDispatchDomainBlockStatsFlags)

In most cases, this problem affects only one cluster. Rebooting the node may restore its functionality temporarily but does not eliminate the root cause of the failure.

The cause of the problem is unstable operation of one or more physical network connections between the cluster nodes and the storage system. Packet loss leads to nodes losing access to the shared disk space. As a result, the libvirt service cannot complete VM operations due to timeouts while waiting for access to the VM disks, which causes lock acquisition failures.

Diagnostics

To confirm the root cause:

1. Connect to the server with the platform via SSH. For more information about connecting via SSH, see Workstation setup.
2. Check the vmctl.log:
   

   docker exec -it vm_box cat /var/log/vmctl.log

   Expected result: The error "cannot acquire state change lock" is present.
   
   Example output

     File "/usr/local/lib64/python3.9/site-packages/libvirt.py", line 2789, in setMetadata
       raise libvirtError('virDomainSetMetadata() failed')
   libvirt.libvirtError: Timed out during operation: cannot acquire state change lock (held by monitor=remoteDispatchDomainBlockStatsFlags)

3. Check for issues on the physical storage system connection channels. If problems are present, this confirms the cause.

Solution

To resolve the issue, eliminate the faults in the network hardware. This will restore a stable network connection between the cluster and the storage system.