The VM status monitoring subsystem is specially designed to send the information about virtual machines from libvirt to VMmanager KVM. The main goals of this subsystem:

  • Synchronizes data in the control panel about virtual machines and their actual statuses
  • Transmits statuses of virtual machines with minimal time lags
  • Registers changes of VM statuses

Monitoring subsystem:

  • Participates in OS installation on a virtual machine
  • Controls duplicated of virtual machines

Subsystem architecture


Subsystem consists of 2 services: vmwatch-master and vmwatch-node.

A message queue, which is implemented using ZeroMQ library via the PUSH-PULL pattern, is used for interaction between services.

vmwatch-master

This service runs on the node, where the control panel is installed.  It receives messages on the IP address and port specified in the configuration file, processes them, and calls the corresponding API functions of a control panel functions to send data about changes in statuses of virtual machines.


vmwatch-node

The service runs on every cluster node. It sends messages to the IP address and port specified in the configuration file. It receives information about changes of VM statuses from libvirtd. If the connection with libvirt is lost, the service will be stopped.

Service configuration


System V (System D for CentOS 7) controls service run and auto-run. vmwatch-master service and its control script are kept in vmmanager-kvm package for VMmanager KVM. vmwatch-node service and its control script are kept in vmmanager-pkg-vmnode package.

Services run automatically when booting the OS or by the control panel during the configuration process.

After VMmanager installation vmwatch-master service is blocked until the first cluster node is added into the cluster.

Service work parameters are added into the vmwatch-master.conf and vmwatch-node.conf configuration files that are kept in /usr/local/mgr5/etc catalog. The configuration files are created by the control panel automatically. 

The configuration file is recorded in the NAME_PARAMETER=VALUE format, one parameter on one line.

vmwatch-master.conf configuration file

It may contain the following parameters:

  • disabled - If parameter has yes value, it blocks service execution.
  • listen - Parameter value in 'protocol://IP:port' format assigns socket where service receives messages.
  • secret - Undefined symbol line (A-Z, 0-9), the parameter value is used for incoming messages authentication.

vmwatch-node.conf configuration file

It may contain the following parameters:

  • connect - Parameter value in 'protocol://IP:port' format assigns socket where service sends messages
  • secret - Parameter value is used for incoming messages authentication. It has to match with a secret parameter value of vmwatch-master configuration file
  • node_id - cluster node id which is used in messages for message source identification

Service configuration

The control panel configures the service automatically when needed: 

  • During the upgrade to the version which supports the monitoring subsystem, and if there are nodes in the cluster;
  • When a cluster node is added into the cluster;

IP address assignment, on which vmwatch-master receives messages, is executed in the following way:

  • the system checks the VmwatchListenIp parameter value of the configuration file;
  • If IP address is not assigned by the parameter, a cluster node where the control panel is installed is searched; 
  • If there is no local cluster node, IP address scanning is started at all network interfaces of the server where the control panel is installed, it chooses the first IP address from the list which is not a local one.

The port where vmwatch-master receives messages is defined by the VmwatchListenPort parameter of the control panel configuration file. The default value is 1515.

To change the IP address and/or port, add the corresponding parameters into the control panel configuration file, restart the panel, and call API vmwatch.configure function. This function configures all monitoring services according to the indicated parameters.

Service status control


vmwatch-master service status control is run by the panel when periodic API function is called. It is run by the cron every 60 minutes.

If the control function detects that service is stopped, it will register the notification about this problem. When this problem is solved, the system will configure the whole monitoring subsystem once again.

vmwatch-node service status control is run by the panel when periodic API function is called. It is run by the cron every 5 minutes when checking the cluster node status (vmmini).

If the service is stopped, the control function registers the notification about this problem. When the problem is  solved, the system will  configure the service on this cluster node

VM status check


In the Cluster settings→ Policy you can enable the check of VM statuses.

The system will use statistics data sent by VM monitoring services. The status check function analyses packages that were sent over the network and the number of write requests to a VM disk. Every 10 minutes vmwatch-nodecollects statistics from all domains and sends it to vmwatch-master. If any of the values equals 0, the corresponding warning will be displayed in the list of virtual machines.

Service log

vmwatch-master service log is kept in the /usr/local/mgr5/var/vmwatch-master.log file, vmwatch-node service log is kept in the /usr/local/mgr5/var/vmwatch-node.log file.