VMmanager saves information about the operation of services in log files. The data from log files can be used to diagnose the platform operation.

Log files list


Main log files

The main log files of VMmanager services are stored in the vm_box container in the /var/log/ directory. Usually the name of the log file is the same as the service name. For example, the log file of the vmwatch service is called vmwatch.log.

File name

Contents
Common services
vm_1_writer.log 
  • tasks created by the platform;
  • API requests executed with POST and DELETE methods;
  • database operations performed using INSERT and UPDATE commands
vm_1_reader.logAPI requests executed with GET method
monitor.logmonitoring service for VMs and cluster nodes
run_script.logrunning scripts
vmctl.log 
  • parameters and the result of running scripts:
    • operations with virtual machines (VMs);
    • sending mail;
    • collecting information about the cluster node;
    • downloading and copying of VM images and OS templates;
  • Ansible configuration management system logs
vmwatch.loglibvirt events, VM synchronization status
Operations with VMs
check_storage.log

VM storage check

connect_storages.log

VM storage connection

disk_resize.log

VM disk resizing operations

gosockify.log

proxy service for access to VMs via VNC

host.log

creating and deleting VMs

host_action.log

starting, stopping, restarting VMs

host_change_password.log

changing the root password for a VM

host_firewall.log

firewall configuration

host_image_create.log

creating VM images

host_image_delete.log

deleting VM images

host_migration.log

VM migration

host_redefine.log

changing VM settings, setting VNC password

host_snapshot.log

operations with VM snapshots

host_source_copy.log

downloading and copying of VM images and OS templates;

image_relocate.log

relocating VM images

iso.logdownloading and mounting ISO images
uploader.log
ISO images uploading
websockify.logaccess service for VMs via VNC
Operations with LXD containers
lxdconsole.log

cloning LXD containers

lxd_init.log

creating LXD containers

Operations with cluster nodes

hawatch.log

hawatch microservice. Read more in Operating principle of high availability clusters

lxd_node_info.log

collecting information about LXD cluster nodes

network_configure.log

node_network.log

network configuration on the cluster node

node_create/{node_id}.log

collecting information about adding of the cluster nodes

node_info.log

collecting information about the cluster node (information about the OS, disk, etc.)

Platform installation

The platform installation log is saved in the file /opt/ispsystem/vm/install.log.

Various services

FunctionContainerFile path
Licensingvm_license_1container stdout
Notification servicevm_msgsender_1/var/log/msgsender_1_reader.1.log
/var/log/msgsender_1_writer.1.log
Mail notifications about creating, reinstalling the operating system, or changing the VM passwordvm_msgsender_1/var/log/email.log
Operation of Telegram bot. Read more in Sending notifications to Telegramvm_msgsender_1/var/log/telegram.log
Collecting information about cluster node uptime, QEMU and libvirt versionsnodewatch/var/log/nodewatch.log
Migration from VMmanager 5importer/var/log/importer.log
Address space migrationvm_ipmgr_1/var/log/ip_ipmgr5_proxy.log
Synchronization with the LDAP directoryvm_ldap_1/var/log/isp_ldap.log
Execution of query sequencesvm_batch_1 (vm-batch-1)container stdout

Log files on the cluster node

FunctionFile path
Collecting metrics for statistics/var/log/gomon.log
ha-agent service. Read more in Operating principle of high availability clusters/var/log/ha-agent.log

Working with logs


Copy log file from a container

To copy a log file from the container to the current directory, run the command:

docker cp <container_name>:<path_to_log> ./
BASH

<container_name> — container name

<path_to_log> — log file path and name

Example command

docker cp vm_auth_back_1:/var/log/licupdate.log ./
BASH

View log files in a container

  1. Enter the required container: 

    docker exec -it <container_name> sh
    BASH

    <container_name> — container name

    Example command

    docker exec -it vm_box sh
    BASH
  2. View the log file using standard Linux utilities. For example, cat, tail, less, etc.

    Example command

    less /var/log/host.log
    BASH

    To view the logs of a specific task:

    1. In the platform interface go to the Tasks → select the task → copy the request_id value.
    2. Run the command:

      grep <request_id> /var/log/*
      CODE

      <request_id> — request_id value

  3. Exit the container:

    exit
    BASH

Viewing log files via stdout

Separate log files are not created for some services. Logs of these services are written to stdout. To view such logs, run the command: 

docker logs <container_name> --tail <lines> --since <period>
BASH

<container_name> — container name

<lines> — number of lines

<period> — logging period

Example command: Display the last 100 log lines of vm_box contatiner created in the last 60 minutes

docker logs vm_box --tail 100 --since 60m
CODE

Collecting platform logs


To collect all the log files of the platform in one directory:

  1. Connect to the server with the platform via SSH.
  2. Create the bash script logs_collect.sh with the following content:

    #!/bin/bash
    
    rm /home/logs -fr
    
    DOCKER_CONTAINER_NAMES=`docker ps --format '{{.Names}}'`
    
    SERVICES=($DOCKER_CONTAINER_NAMES)
    cd /home
    mkdir -p logs
    cd logs
    for service in ${SERVICES[@]}
    do
     echo -e "----\033[0;31mCopying logs from $service\033[0m----\n"
     mkdir -p $service
     docker cp $service:/var/log/. $service/.
     docker logs $service > $service/${service}_stdout.log 2>&1
    done
    
    cp -r /opt/ispsystem/*/install.log install.log
    
    ARCHIVE_NAME="logs_$(date +'%Y_%m_%d_%H_%M_%S').tar.gz"
    
    tar -cvzf /home/$ARCHIVE_NAME -C /home/logs .
    BASH
  3. Run the script: 

    bash logs_collect.sh
    BASH

The log files will be saved in the /home/logs/ directory, the log archive will be saved in the /home/ directory.

Logging to journald


By default, logging in the journald system service is disabled in the platform. To enable logging:

  1. Connect to a server with the platform via SSH.
  2. Create the patch file /opt/ispsystem/vm/journald.yaml with the contents:

    version: "3.5"
    services:
      auth_back4:
        volumes:
        - /var/run/systemd/journal/:/var/run/systemd/journal/
      ldap:
        volumes:
        - /var/run/systemd/journal/:/var/run/systemd/journal/
      vm_box:
        volumes:
        - /opt/ispsystem/vm/backup:/opt/ispsystem/vm/backup
        - /opt/ispsystem/license:/opt/ispsystem/license
        - /opt/ispsystem/vm/socket/:/opt/ispsystem/vm/vmbox/
        - /var/run/systemd/journal/:/var/run/systemd/journal/
    CODE
  3. Apply the patch:

    vm add-patch -p journald -f /opt/ispsystem/vm/journald.yaml
    BASH

Useful tips

Related topics: