DCImanager 6 saves information about the operation of services in log files. The data from log files can be used to diagnose the platform operation. Log files are stored in the /var/log directories of docker containers.

Log files list


Main log files on platform server

ContainerFile pathContents
dci_auth_back_1/var/log/licupdate.log logs of the license validation
dci_back/var/log/dci_1_writer.log
/var/log/dci_1_reader.log
logs of the main DCImanager 6 service
 /var/log/dci_location_check_update.loglogs of the update service for containers at the location. The service starts when the container is restarted and updates the containers at the location, if necessary
/var/log/dci_locationctl.log
/var/log/dci_ansiblectl_location_<location id>.log
/var/log/ansible.log 
logs of creating and configuring a location
/var/log/dci_operationctl.loglog of operations performed at the location
dci_consumer_1/var/log/worker_daemon.loglogs of periodic equipment polls and obtaining statistical data
dci_dns_proxy_1/var/log/dns_proxy_integration.log
/var/log/dns_proxy_service_1_reader.log
/var/log/dns_proxy_service_1_writer.log

logs of the modules:

dci_ipmgr_1/var/log/ip_1_reader.log
/var/log/ip_1_writer.log
logs of address space management service
dci_ipmi_proxy_v2_client_1/var/log/ipmi_proxy_goproxy.loggoproxy service logs. The service transmits data to the VNC client and proxies Websocket requests to the VNC server
/var/log/ipmi_proxy_update_checker.loglogs of the update service for BMC proxy module at the location. The service starts each time the container is restarted and updates the module if necessary
vm_ldap_1/var/log/isp_ldap.loglogs of synchronization with the LDAP directory
migrator/var/log/migrate.log
/var/log/bill_migrate.log
migration of objects from DCImanager 5

The platform installation log is saved in the file /opt/ispsystem/dci/install.log.

The log files of the diagnostics operation are listed in the Diagnostics check article.

The authorization service logs are written to the standard output of the dci_auth_back4_1 container.

Main log files on location server

ContainerFile pathContents
eservice_handler

/var/log/dci_switch_<id>_handler.loglogs of switch handlers
/var/log/dci_pdu_<id>_handler.loglogs of PDU handlers
/var/log/dci_ipmi_<id>_handler.loglogs of BMC handlers
/var/log/dci_intel_amt_handler.log logs of Intel AMT handlers

<id> — device ID in DCImanager 6. For example, for the switch with ID 3, the file is called dci_switch_3_handler.log

To view BMC logs, use the BMC id, not the server id. To get the BMC id, perform an API request:

GET https://domain.com/dci/v3/ipmi
CODE

domain.com — domain name or IP address of the server with the platform


Working with logs


Copy log file from a container

To copy a log file from the container to the current directory, run the command:

docker cp <container_name>:<path_to_log> ./
BASH

<container_name> — container name

<path_to_log> — log file path and name

Example command

docker cp dci_auth_back_1:/var/log/licupdate.log ./
BASH

View log files in a container

  1. Enter the required container: 

    docker exec -it <container_name> sh
    BASH

    <container_name> — container name

    Example command

    docker exec -it dci_back sh
    BASH
  2. View the log file using standard Linux utilities. For example, cat, tail, less, grep, etc.

    Example command: Find out when the switch was polled

    grep 'Switch status start' /var/log/dci_switch_1_handler.log
    BASH
  3. Exit the container:

    exit
    BASH

Viewing log files via stdout

Separate log files are not created for some services. Logs of these services are written to stdout. To view such logs, run the command: 

docker logs <container_name> --tail <lines> --since <period>
BASH

<container_name> — container name

<lines> — number of lines

<period> — logging period

Example command: Display the last 100 log lines of the authorization service created in the last 60 minutes

docker logs dci_auth_back4_1 --tail 100 --since 60m
BASH

Collecting platform logs


To collect all the log files of the platform in one directory:

  1. Connect to the server with the platform via SSH.
  2. Create the bash script logs_collect.sh with the following content:

    #!/bin/bash
    
    rm /home/logs -fr
    
    DOCKER_CONTAINER_NAMES=`docker ps --format '{{.Names}}'`
    
    SERVICES=($DOCKER_CONTAINER_NAMES)
    cd /home
    mkdir -p logs
    cd logs
    for service in ${SERVICES[@]}
    do
     echo -e "----\033[0;31mCopying logs from $service\033[0m----\n"
     mkdir -p $service
     docker cp $service:/var/log/. $service/.
     docker logs $service > $service/${service}_stdout.log 2>&1
    done
    
    cp -r /opt/ispsystem/*/install.log install.log
    
    ARCHIVE_NAME="logs_$(date +'%Y_%m_%d_%H_%M_%S').tar.gz"
    
    tar -cvzf /home/$ARCHIVE_NAME -C /home/logs .
    BASH
  3. Run the script: 

    bash logs_collect.sh
    BASH

The log files will be saved in the /home/logs/ directory, the log archive will be saved in the /home/ directory.

Logging in journald


Logging settings

The platform uses the journald system service to collect logs. journald collects logs for all containers except dci_consul_1.

The journald service on the platform server and the location server is configured automatically. The service configuration is saved in the /usr/lib/systemd/journald.conf.d/dci.conf file. If the service has already been configured on the server, the configuration set by the platform will have a higher priority.

Configuring the journald service may fail if the syslog and rsyslog service settings on the server have been changed.

Configuration parameters set by the platform:

  • Storage=persistent — store files on disk;
  • Compress=yes — compress data in logs;
  • SystemMaxFileSize=500M — the maximum size of the log file is 500 Mb. When this value is reached, logs are rotated;
  • SystemMaxUse=20G — the maximum disk size for storing logs is 20 GB. When this value is reached, older files will be deleted.

Operations with logs

journald saves logs in the /var/log/journal/ directory. You can view the logs using the journalctl utility. Examples of how to use the utility:

View dci_back container logs

journalctl -f CONTAINER_NAME=dci_back
BASH

View logs of the equipment operation service

journalctl -f SYSLOG_IDENTIFIER=equip_daemon
BASH

To create an archive with logs, run the command:

dci log --export <output_dir>
BASH

<output_dir> — directory to save the archive

The name of the archive is YYYY-MM-DD_HH:MM:SS_dci_logs.tar.gz. For example, 2023-03-23_10:25:30_dci_logs.tar.gz.