Platform log files

VMmanager saves information about the operation of services in log files. The data from log files can be used to diagnose the platform operation.

Log files list

Main log files

The main log files of VMmanager services are stored in the vm_box container in the /var/log/ directory. Usually the name of the log file is the same as the service name. For example, the log file of the vmwatch service is called vmwatch.log.

File name	Contents
Common services
vm_1_writer.log	tasks created by the platform; API requests executed with POST and DELETE methods; database operations performed using INSERT and UPDATE commands
vm_1_reader.log	API requests executed with GET method
monitor.log	monitoring service for VMs and cluster nodes
run_script.log	running scripts
vmctl.log	parameters and the result of running scripts: operations with virtual machines (VMs); sending mail; collecting information about the cluster node; downloading and copying of VM images and OS templates; Ansible configuration management system logs
vmwatch.log	libvirt events, VM synchronization status
Operations with VMs
check_storage.log	VM storage check
connect_storages.log	VM storage connection
disk_resize.log	VM disk resizing operations
gosockify.log	proxy service for access to VMs via VNC
host.log	creating and deleting VMs
host_action.log	starting, stopping, restarting VMs
host_change_password.log	changing the root password for a VM
host_firewall.log	firewall configuration
host_image_create.log	creating VM images
host_image_delete.log	deleting VM images
host_migration.log	VM migration
host_redefine.log	changing VM settings, setting VNC password
host_snapshot.log	operations with VM snapshots
host_source_copy.log	downloading and copying of VM images and OS templates;
image_relocate.log	relocating VM images
iso.log	downloading and mounting ISO images
uploader.log	ISO images uploading
websockify.log	access service for VMs via VNC
Operations with LXD containers
lxdconsole.log	cloning LXD containers
lxd_init.log	creating LXD containers
Operations with cluster nodes
hawatch.log	hawatch microservice. Read more in Operating principle of high availability clusters
lxd_node_info.log	collecting information about LXD cluster nodes
network_configure.log node_network.log	network configuration on the cluster node
node_create/{node_id}.log	collecting information about adding of the cluster nodes and adding errors
node_info.log	collecting information about the cluster node (information about the OS, disk, etc.)

Platform installation

The platform installation log is saved in the file /opt/ispsystem/vm/install.log.

Various services

Function	Container	File path
Licensing	license	container stdout
Notification service	msgsender	/var/log/msgsender_1_reader.1.log /var/log/msgsender_1_writer.1.log
Mail notifications about creating, reinstalling the operating system, or changing the VM password	msgsender	/var/log/email.log
Operation of Telegram bot. Read more in Sending notifications to Telegram	msgsender	/var/log/telegram.log
Collecting information about cluster node uptime, QEMU and libvirt versions	nodewatch	/var/log/nodewatch.log
Migration from VMmanager 5	importer	/var/log/importer.log
Address space migration	ipmgr	/var/log/ip_ipmgr5_proxy.log
Synchronization with the LDAP directory	ldap	/var/log/isp_ldap.log
Execution of query sequences	batch	container stdout

Log files on the cluster node

Function	File path
Collecting metrics for statistics	/var/log/gomon.log
ha-agent service. Read more in Operating principle of high availability clusters	/var/log/ha-agent.log

Working with logs

Copy log file from a container

To copy a log file from the container to the current directory, run the command:

docker cp <container_name>:<path_to_log> ./

Comments to the command

Example command

docker cp auth:/var/log/licupdate.log ./

View log files in a container

Enter the required container:
```
docker exec -it <container_name> sh
```
Comments to the command
<container_name> — container name
Example command
```
docker exec -it vm_box sh
```
View the log file using standard Linux utilities. For example, cat, tail, less, etc.
Example command
```
less /var/log/host.log
```
To view the logs of a specific task:
1. In the platform interface go to the Tasks → select the task → copy the request_id value.
2. Run the command:
  grep <request_id> /var/log/*
  Comments to the command
  <request_id> — request_id value
Exit the container:
```
exit
```

Viewing log files via stdout

Separate log files are not created for some services. Logs of these services are written to stdout. To view such logs, run the command:

docker logs <container_name> --tail <lines> --since <period>

Comments to the command

Example command: Display the last 100 log lines of vm_box contatiner created in the last 60 minutes

docker logs vm_box --tail 100 --since 60m

Collecting platform logs

To collect all the log files of the platform in one directory:

Connect to the server with the platform via SSH.

Create the bash script logs_collect.sh with the following content:

#!/bin/bash

rm /home/logs -fr

DOCKER_CONTAINER_NAMES=`docker ps --format '{{.Names}}'`

SERVICES=($DOCKER_CONTAINER_NAMES)
cd /home
mkdir -p logs
cd logs
for service in ${SERVICES[@]}
do
 echo -e "----\033[0;31mCopying logs from $service\033[0m----\n"
 mkdir -p $service
 docker cp $service:/var/log/. $service/.
 docker logs $service > $service/${service}_stdout.log 2>&1
done

cp -r /opt/ispsystem/*/install.log install.log

ARCHIVE_NAME="logs_$(date +'%Y_%m_%d_%H_%M_%S').tar.gz"

tar -cvzf /home/$ARCHIVE_NAME -C /home/logs .

Run the script:
```
bash logs_collect.sh
```

The log files will be saved in the /home/logs/ directory, the log archive will be saved in the /home/ directory.

Logging to journald

By default, logging in the journald system service is disabled in the platform. To enable logging:

Connect to a server with the platform via SSH.

Create the patch file /opt/ispsystem/vm/journald.yaml with the contents:

version: "3.5"
services:
  auth:
    volumes:
    - /var/run/systemd/journal/:/var/run/systemd/journal/
  ldap:
    volumes:
    - /var/run/systemd/journal/:/var/run/systemd/journal/
  vm_box:
    volumes:
    - /opt/ispsystem/vm/backup:/opt/ispsystem/vm/backup
    - /opt/ispsystem/license:/opt/ispsystem/license
    - /opt/ispsystem/vm/socket/:/opt/ispsystem/vm/vmbox/
    - /var/run/systemd/journal/:/var/run/systemd/journal/

Apply the patch:

vm add-patch -p journald -f /opt/ispsystem/vm/journald.yaml

Useful tips