VMmanager 6: Administrator guide

IP fabric

ISPSystem

IP fabric is one the network cluster configuration types. This configuration type allows you to create virtual machines (VMs) with public IP addresses in a private network.

Virtual machines in the cluster are assigned IP addresses with the /32 mask. A separate virtual interface is used as the default gateway for each VM. Cluster nodes act as routers.

Routing in the cluster is performed using the iBGP protocol. Route Reflector (RR) equipment is used to transmit information about iBGP routes. Physical or virtual routers and/or servers can be used in this function. We recommend using at least two sets of RR equipment in one cluster — the main and the backup set.

Note

To operate an IPv4 network in a cluster, use at least one set of RR equipment with an IPv4 address; for an IPv6 network, at least one set with an IPv6 address.

Advantages of IP fabric:

  • reduced service traffic;
  • saving the address space;
  • isolation of VM traffic;
  • IP addresses are not bound to cluster nodes;
  • VMs can be migrated promptly among cluster nodes.

Operation logic


When creating a VM in an IP fabric cluster:

  1. VMmanager platform:
    1. Creates a separate virtual interface <vmname>_net0. For example, for a VM with the name sea_gold, the interface will be called sea_gold_net0. All virtual interfaces on the node will have the same IP and MAC addresses.
    2. Sets the IP address of the created interface as the gateway for the VM. Routing between the VM and the gateway is performed through a point-to-point connection (PtP).
    3. Adds the created route to the FRR service configuration on the cluster node.
  2. The FRR service transmits information about the created route via iBGP protocol to RR equipment.
  3. RR equipment transmits the route information to Core Gateway equipment. After that, the VM becomes available from the external network (Internet).

 

Example of IP fabric cluster operation

iBGP — dynamic routing protocol responsible for route exchange between nodes and network equipment.

Node 1, Node 2, Node 3 — cluster nodes.

VM1, VM2, VM3... — virtual machines.

VNET — virtual network interfaces.

Core Gateway — equipment that provides access to the external network (Internet).

FRR — service on the cluster node that implements the iBGP protocol.

Linux CORE — OS core of cluster node.

Route Reflector — server or network equipment that receives route information from nodes and transmits it to Core Gateway.

Backup Route Reflector — backup set of Route Reflector equipment.

Configuration order


  1. In VMmanager:
    1. Create an IP pool for the VMs. Read more in Pools management.
    2. Create a cluster with the IP fabric network configuration type. Read more in Creating a cluster. When creating the cluster, specify the following:
      1. gateway IP address for the VM. VMmanager will assign this address of the gateway to all virtual interfaces on the cluster node and use it only for routing between the VM within the node. We recommend to specify a private address (e.g. 10.0.0.1). This address should not be in use anywhere else in the local network.
      2. BGP community.
      3. IP addresses and numbers of autonomous systems of RR equipment.
    3. Add the nodes to the cluster. Read more in Managing servers in the cluster.

      Note

      The nodes of the IP-fabric with KVM virtualization must have AlmaLinux 8, with LXD virtualization — Ubuntu 20.

  2. Configure BGP sessions for each cluster node on the RR equipment: specify the IP addresses of nodes and numbers of autonomous BGP systems in the settings.

Example of configuration


In this example, we consider the IP-fabric configuration in a cluster of servers used as hypervisors. The role of Core Gateway is performed by a Juniper MX router. Physical servers with Linux are used as Route Reflector.

Preparation

The following equipment will be required for the setup:

  • a server with the VMmanager platform;
  • one or more cluster nodes with AlmaLinux 8 or Ubuntu 20.04 installed;
  • one or two Route Reflector servers;
  • Juniper MX switch.

Before configuration:

  1. Get information from your hosting provider about BGP settings: autonomous system number and community.
  2. Make sure that the BGP protocol is configured and functioning properly.
  3. Create an IP address pool for virtual machines in VMmanager.

Configuring the Core Gateway router

To configure the "neighborhood" of Core Gateway and Route Reflector:

  1. Add a new filter to the router configuration:

    set policy-options policy-statement VM term isp-ipv4 from protocol bgp
            set policy-options policy-statement VM term isp-ipv4 from route-filter <filter> orlonger
            set policy-options policy-statement VM term isp-ipv4 then accept
            set policy-options policy-statement VM then reject
            set policy-options policy-statement reject-all then reject

    <filter> — list of networks from VMmanager

  2. Add a new group to the router configuration:

     set protocols bgp group VM import VM
            set protocols bgp group VM export reject-all
            set protocols bgp group VM peer-as <as>
            set protocols bgp group VM neighbor <rr_ip>

    <as> — autonomous system number

    <rr_ip> — IP address of Route Reflector

  3. Check that BGP routes have been received:

    commit check
    commit confirmed 5
  4. Проверьте, что маршруты BGP получены:

     show bgp group VM detail
  5. Confirm the configuration changes:

    commit

Configuring Route Reflector servers

To configure "neighborhood" among Route Reflector, Core Gateway and cluster nodes, install and configure FRRouting (FRR) software. You can automate this process with a script.

If you are using a VMmanager virtual machine for Route Reflector, run the Route Reflector script on it:

  1. Go to Virtual machines menu → Run script.
  2. Select the Route Reflector script and specify its parameters:
    1. AS — BGP autonomous system number.
    2. NEIGHBOR — IPv4 address of the "neighbor" — VMmanager cluster node.
    3. PREFIX — prefix of the IPv4 networks that Route Reflector will receive.
    4. VxLAN support — whether it is necessary to transfer information about virtual networks (VxLANs).
  3. Press Run script.

If you are using a physical server for Route Reflector:

  1. Create the route_reflector.sh file with the following content: 

    #!/bin/sh
    # tags: debian10,debian11,ubuntu2004,alma8
    RNAME=route_reflector
    
    set -x
    
    LOG_PIPE=/tmp/log.pipe.$$                                                                                                                                                                                                                    
    mkfifo ${LOG_PIPE}
    LOG_FILE=/root/${RNAME}.log
    touch ${LOG_FILE}
    chmod 600 ${LOG_FILE}
    
    tee < ${LOG_PIPE} ${LOG_FILE} &
    
    exec > ${LOG_PIPE}
    exec 2> ${LOG_PIPE}
    
    killjobs() {
        jops="$(jobs -p)"
        test -n "${jops}" && kill ${jops} || :
    }
    trap killjobs INT TERM EXIT
    
    echo
    echo "=== Recipe ${RNAME} started at $(date) ==="
    echo
    
    if [ -f /etc/redhat-release ]; then
        OSNAME=centos
    else
        OSNAME=debian
    fi
    
    Service() {
        # $1 - name
        # $2 - command
    
        if [ -n "$(which systemctl 2>/dev/null)" ]; then
            systemctl ${2} ${1}.service
        else
            if [ "${2}" = "enable" ]; then
                if [ "${OSNAME}" = "debian" ]; then
                    update-rc.d ${1} enable
                else
                    chkconfig ${1} on
                fi
            else
                service ${1} ${2}
            fi
        fi
    }
    
    if [ "${OSNAME}" = "debian" ]; then
        export DEBIAN_FRONTEND="noninteractive"
    
        # Wait firstrun script
        while ps uxaww | grep  -v grep | grep -Eq 'apt-get|dpkg' ; do echo "waiting..." ; sleep 3 ; done
        apt-get update --allow-releaseinfo-change || :
        apt-get update
        test -f /usr/bin/which || apt-get -y install which
        which lsb_release 2>/dev/null || apt-get -y install lsb-release
        which logger 2>/dev/null || apt-get -y install bsdutils
        OSREL=$(lsb_release -s -c)
        apt install -y curl ca-certificates
        curl -s https://deb.frrouting.org/frr/keys.asc | apt-key add -
        FRRVER="frr-stable"
        echo deb https://deb.frrouting.org/frr $(lsb_release -s -c) $FRRVER | tee -a /etc/apt/sources.list.d/frr.list
        apt update
        apt -y install frr frr-pythontools
    else
        OSREL=$(printf '%.0f' $(rpm -qf --qf '%{version}' /etc/redhat-release))
        FRRVER="frr-stable"
        yum install -y curl
        curl -O https://rpm.frrouting.org/repo/$FRRVER-repo-1-0.el8.noarch.rpm
        yum install -y ./$FRRVER*
        yum install -y frr frr-pythontools
    fi
    
    sed -i '/bgpd=/s/no/yes/' /etc/frr/daemons
    CONFIG="/etc/frr/frr.conf"
    ip=$(ip route get 1 | grep -Po '(?<=src )[^ ]+')
    if grep -q "router bgp" $CONFIG; then
        # Already exists. adding new neighbor or new network
        if grep -q "($NEIGHBOR)" $CONFIG; then
            # This neighbor already exists
            true
        else
            sed -i "/extended-nexthop/a    neighbor ($NEIGHBOR) peer-group fabric" $CONFIG
            sed -i "/extended-nexthop/a    neighbor ($NEIGHBOR) remote-as ($AS)" $CONFIG
        fi
        if [ "($VXLAN)" = "yes" ]; then
            # need to add vxlan?
            if grep -q "address-family l2vpn evpn" $CONFIG; then
                # already enabled
                true
            else
                sed -i '/exit-address-family/cexit-address-family\
        !\
        address-family l2vpn evpn\
            neighbor fabric activate\
            neighbor fabric route-reflector-client\
            advertise-all-vni\
        exit-address-family' $CONFIG
            fi
        fi
        if grep -q "($PREFIX)" $CONFIG; then
            # already added
            true
        else
            if ! [ "($PREFIX)" = "()" ]; then
                cat << EOF >> $CONFIG
    ip prefix-list IPV4_PLIST permit ($PREFIX) ge 32 le 32
    !
    EOF
            fi
        fi
    else
        cat << EOF > $CONFIG
    router bgp ($AS)
        bgp router-id $ip
        bgp log-neighbor-changes
        no bgp default ipv4-unicast
        neighbor fabric peer-group
        neighbor fabric capability extended-nexthop
        neighbor ($NEIGHBOR) remote-as ($AS)
        neighbor ($NEIGHBOR) peer-group fabric
        !
        address-family ipv4 unicast
            neighbor fabric activate
            neighbor fabric route-map IPV4_IMPORT in
            neighbor fabric route-reflector-client
        exit-address-family
        !
    EOF
        if [ "($VXLAN)" = "yes" ]; then
            cat << EOF >> $CONFIG
        address-family l2vpn evpn
            neighbor fabric activate
            neighbor fabric route-reflector-client
            advertise-all-vni
        exit-address-family
    EOF
        fi
        cat << EOF >> $CONFIG
    exit
    !
    route-map IPV4_IMPORT permit 5
        match ip address prefix-list IPV4_PLIST
    exit
    !
    ip nht resolve-via-default
    !
    EOF
    if ! [ "($PREFIX)" = "()" ]; then
        cat << EOF >> $CONFIG
    ip prefix-list IPV4_PLIST permit ($PREFIX) ge 32 le 32
    !
    EOF
        fi
        Service frr enable
    fi
    sed -i '/Nice/d' /usr/lib/systemd/system/frr.service
    sed -i '/Nuce/d' /lib/systemd/system/frr.service
    systemctl daemon-reload
    Service frr restart
    
    
  2. Replace the macros in the file text:
    1. ($AS) — with the BGP autonomous system number.
    2. ($NEIGHBOR) — with the IPv4 address of the "neighbor" — VMmanager cluster node.
    3. ($PREFIX) — with the prefix of the IPv4 networks that Route Reflector will receive.
    4. ($VXLAN) — with:
      • уеs — if the service should transfer information about virtual networks (VxLANs);
      • no — if there is no need to transfer information about virtual networks.
  3. Run the script: 

    sh route_reflector.sh

Note

When new nodes are added to the cluster, you will need to add the information about them to the FRR configuration file. To do this, re-run the script with the necessary parameters or make changes manually in the /etc/frr/frr.conf file.

VMmanager configuration

To configure IP-fabric in VMmanager:

  1. Create a cluster with IP-fabric network type. When creating, specify the settings for connections to the Core Gateway and Route Reflector.
  2. Connect the servers to the created cluster.
  3. Create VMs in the cluster.
Thank you for your feedback!
Was the article useful?
Tell us why not:

it does not match the title

it was hard to understand

I didn't find the answer

recommendations from the article did not help

other

Tell us what you didn't like in the article: