If maintenance work needs to be performed on a cluster node, you can switch the node to maintenance mode. The following features are not supported on the node in maintenance mode:

  • creating new virtual machines (VMs);

  • migrating VMs from other nodes.

When you switch a node to maintenance mode, you can start evacuation of its objects — automatic migration of VMs, disks, images and backups to another node. After maintenance is complete, the evacuated objects will be returned to the source node. If object evacuation is not performed, all VMs on that node will be stopped when maintenance mode is enabled.

Restrictions

Maintenance mode is unavailable in:

  • HA clusters;
  • clusters with "Routing" configuration type;
  • clusters with LXD virtualization.

VM evacuation is unavailable if:

  • the cluster nodes do not have enough resources for migration;
  • there are no cluster nodes with network parameters suitable for migration;
  • snapshots have been created for VMs of the source node.

Work logic

Platform actions when the node is switched to maintenance mode:

  1. If the Forcibly cancel all tasks on the node option is enabled, all running tasks are terminated. If the option is disabled, the platform waits until the running tasks are completed.
  2. If node objects need to be evacuated:

    1. An evacuation plan is created. The plan contains actions to migrate all objects of the node. The selection of nodes for migration depends on the cluster settings. Read more in the article Selecting a cluster node for the virtual machine.

    2. Space is reserved on node disks, where objects from the maintained node will be migrated to. The reserved space will not be used to create new VMs.
    3. The objects of the maintained node are migrated to other nodes according to the evacuation plan. If the migration of any object fails, the plan execution will continue. If live migration for a particular VM is not supported, that VM will be stopped before migration.
  3. If evacuation of node objects is not required, all VMs on the node are stopped.
  4. The node status changes to In maintenance mode.

Platform actions when a node is out of maintenance mode:

  1. If the Return evacuated objects option is enabled, the migrated objects are returned to the source node.
  2. VMs that were stopped when the mode was enabled are started.

  3. The status of the node In maintenance mode is disabled.

Maintenance mode management

Enabling maintenance mode

If node objects need to be evacuated, delete VM snapshots before enabling the mode. Read more in the article Snapshots of virtual machines.

  1. Go to the Nodes section → select a node → menu → Maintenance mode.
  2. To allow a node to enter maintenance mode only after all tasks on the node are completed, disable the Forcibly cancel all tasks on the node option.
  3. To evacuate objects, activate the Initiate object evacuation from this node switch.
  4. Click the Enable button. When the node is ready for maintenance, its status will change to In maintenance mode.

If the evacuation of any objects fails, migrate them manually. Read more in the article Migration of virtual machines.

Disabling maintenance mode

  1. Go to the Nodes section → select a node → menu → Maintenance mode.
  2. If the migrated objects do not need to be returned to the node, disable the Return evacuated objects option.
  3. Click the Disable button.

Logs

Maintenance mode logs are saved to the vm_box container on the server with the platform:

  • /var/log/vm_1_writer.log — evacuation planning; 

    Example command for viewing

    docker exec -it vm_box grep "node/{node_id}/maintenance" /var/log/vm_1_writer.log
    BASH

  • /var/log/node_{node_id}_maintenance.log — performed actions.

{node_id} — id of the node in maintenance mode

Useful tips

Related topics: