DCImanager 6

Platform log files

DCImanager 6 saves information about the operation of services in log files. The data from log files can be used to diagnose the platform operation. Log files are stored in the /var/log directories of docker containers.

Log files list

Main log files on platform server

ContainerFile pathContents
dci_auth_back_1/var/log/licupdate.log logs of the license validation
dci_back/var/log/dci_1_writer.log
/var/log/dci_1_reader.log
logs of the main DCImanager 6 service
 /var/log/dci_location_check_update.loglogs of the update service for containers at the location. The service starts when the container is restarted and updates the containers at the location, if necessary
/var/log/dci_locationctl.log
/var/log/dci_ansiblectl_location_<location id>.log
/var/log/ansible.log 
logs of creating and configuring a location
/var/log/dci_operationctl.loglog of operations performed at the location
dci_consumer_1/var/log/worker_daemon.loglogs of periodic equipment polls and obtaining statistical data
dci_dns_proxy_1/var/log/dns_proxy_integration.log
/var/log/dns_proxy_service_1_reader.log
/var/log/dns_proxy_service_1_writer.log

logs of the modules:

dci_ipmgr_1/var/log/ip_1_reader.log
/var/log/ip_1_writer.log
logs of address space management service
dci_ipmi_proxy_v2_client_1/var/log/ipmi_proxy_goproxy.loggoproxy service logs. The service transmits data to the VNC client and proxies Websocket requests to the VNC server
/var/log/ipmi_proxy_update_checker.loglogs of the update service for BMC proxy module at the location. The service starts each time the container is restarted and updates the module if necessary
vm_ldap_1/var/log/isp_ldap.loglogs of synchronization with the LDAP directory
migrator/var/log/migrate.log
/var/log/bill_migrate.log
migration of objects from DCImanager 5

The platform installation log is saved in the file /opt/ispsystem/dci/install.log.

The log files of the diagnostics operation are listed in the Diagnostics check article.

The authorization service logs are written to the standard output of the dci_auth_back4_1 container.

Main log files on location server

ContainerFile pathContents
eservice_handler

/var/log/dci_switch_<id>_handler.loglogs of switch handlers
/var/log/dci_pdu_<id>_handler.loglogs of PDU handlers
/var/log/dci_ipmi_<id>_handler.loglogs of BMC handlers
/var/log/dci_intel_amt_handler.log logs of Intel AMT handlers
Comments

Working with logs

Copy log file from a container

To copy a log file from the container to the current directory, run the command:

docker cp <container_name>:<path_to_log> ./
Comments to the command
Example command
docker cp dci_auth_back_1:/var/log/licupdate.log ./

View log files in a container

  1. Enter the required container: 

    docker exec -it <container_name> sh
    Comments to the command
    Example command
    docker exec -it dci_back sh
  2. View the log file using standard Linux utilities. For example, cat, tail, less, grep, etc.

    Example command: Find out when the switch was polled
    grep 'Switch status start' /var/log/dci_switch_1_handler.log
  3. Exit the container:

    exit

Viewing log files via stdout

Separate log files are not created for some services. Logs of these services are written to stdout. To view such logs, run the command: 

docker logs <container_name> --tail <lines> --since <period>
Comments to the command
Example command: Display the last 100 log lines of the authorization service created in the last 60 minutes
docker logs dci_consul_1 --tail 100 --since 60m

Collecting platform logs

To collect all the log files of the platform in one directory:

  1. Connect to the server with the platform via SSH.
  2. Create the bash script getlog.sh with the following content:

    #!/bin/bash
    
    BEGIN_DATE=$(date +'%Y-%m-%d %H:%M:%S' --date '-1 day')
    SINCE=${1:-${BEGIN_DATE}}
    
    /bin/echo "Remove old files"
    /bin/rm -rf /home/logs && /bin/mkdir -p /home/logs/container && /bin/mkdir -p /home/logs/service && /bin/cd /home/logs
    
    /bin/echo "---===Step 1/3===---"
    DOCKER_CONTAINER_NAMES=`journalctl --field CONTAINER_NAME`
    CONTAINERS=($DOCKER_CONTAINER_NAMES)
    
    /bin/echo "Get logs of the containers"
    for container in ${CONTAINERS[@]}
    do
        /bin/echo "Get log of the container: ${container}"
        /bin/journalctl --no-pager CONTAINER_NAME=${container} --since "${SINCE}" > /home/logs/container/${container}.log
    done
    
    /bin/echo "---===Step 2/3===---"
    DOCKER_SERVICES_NAMES=`journalctl --field SYSLOG_IDENTIFIER`
    SERVICES=($DOCKER_SERVICES_NAMES)
    
    /bin/echo "Get logs of the services"
    for service in ${SERVICES[@]}
    do
        /bin/echo "Get log of the service: ${service}"
        /bin/journalctl --no-pager SYSLOG_IDENTIFIER=${service} --since "${SINCE}" > /home/logs/service/${service}.log
    done
    
    /bin/echo "---===Step 3/3===---"
    ARCHIVE_NAME="logs_$(date +'%Y_%m_%d_%H_%M_%S').tar.gz"
    tar -cvzf /home/$ARCHIVE_NAME -C /home/logs .
    
    /bin/echo "Done"
  3. Run the script using one of the commands: 

    Before collecting logs, the script deletes all files from the /home/logs/ directory.
    Collect logs for the last day
    bash getlog.sh
    Collect logs since a certain date
    bash getlog.sh "YYYY-MM-DD"
    Collect logs since a certain date and time
    bash getlog.sh "YYYY-MM-DD HH:MM:SS"
    Comments to the commands

    The script:

    1. Saves container logs in the /home/logs/container/ directory.
    2. Saves service logs in the /home/logs/service/ directory.
    3. Creates an archive logs_<current_date>_.tar.gz in the /home/ directory.

Logging in journald

Logging settings

The platform uses the journald system service to collect logs. journald collects logs for all containers except dci_consul_1. You can change the journald logging level, read more in the Logging level management.

The journald service on the platform server and the location server is configured automatically. The service configuration is saved in the /usr/lib/systemd/journald.conf.d/dci.conf file. If the service has already been configured on the server, the configuration set by the platform will have a higher priority.

Configuring the journald service may fail if the syslog and rsyslog service settings on the server have been changed.

Configuration parameters set by the platform:

  • Storage=persistent — store files on disk;
  • Compress=yes — compress data in logs;
  • SystemMaxFileSize=500M — the maximum size of the log file is 500 Mb. When this value is reached, logs are rotated;
  • SystemMaxUse=20G — the maximum disk size for storing logs is 20 GB. When this value is reached, older files will be deleted.

Operations with logs

journald saves logs in the /var/log/journal/ directory. You can view the logs using the journalctl utility. Examples of how to use the utility:

Get id of containers for which logging is performed
journalctl --field CONTAINER_NAME
View container logs
journalctl -f CONTAINER_NAME=<container_id>
Get id of services for which logging is performed
journalctl --field SYSLOG_IDENTIFIER
View service logs
journalctl -f SYSLOG_IDENTIFIER=<service_id>
Get logs of the authorization service from the certain date
journalctl --no-pager CONTAINER_NAME=dci_auth_back4_1 --since "YYYY-MM-DD"
Comments to the command
Useful tips

Related topics: