LVS Cluster Configuration HOWTO

Wayne Sherrill (sherrill@redhat.com)

This document explains how to set up and administer a Linux Virtual Server (LVS) cluster that provides highly available Web and FTP services.

Table of Contents
Introduction
Components of an LVS Cluster
Background of the LVS Cluster
Hardware/Network Requirements
LVS Router Prerequisites
Cluster Node Interconnection Prerequisites
Installing the Software
Configuring an LVS Cluster
Example -- Setting Up a Five-node Cluster

Introduction

A Linux Virtual Server (LVS) cluster is a collection of servers that have been specially configured to provide highly available Web and FTP services. The diagram below illustrates how an LVS cluster works. Service requests arriving at an LVS cluster are addressed to a virtual server: a publicly advertised, fully-qualified domain name that is associated with a floating IP address, and which can be migrated to a different host.

Figure 1. LVS Cluster Interactions

  
                             _/\__/\_ 
                            |        | 
                           / Internet \ 
                           \_  _  _  _/ 
                             \/ \/ \/
                                 |
                                 |
                       Virtual Server IP/FQDN
             |------------------------------------------|
             |eth0                                      |eth0
       ------|-----                              -------|-----
       | Primary  |                              |   Backup  |
       |  Node    |                              |    Node   |    LVS
       |          |                              |           |  routers
       ------|-----                              -------|-----
             |eth1                                      |eth1
             |-------|------------|-----------------|---|
                     |            |                 | 
                     |            |                 |
                     |            |                 |
                |----|----|  |----|----|       |----|----|
                | Web/FTP |  | Web/FTP |       | Web/FTP |        real
                |  Node#1 |  |  Node#2 |  ...  |  Node#n |       servers
                |_________|  |_________|       |_________|

An LVS cluster consists of one or two router nodes (top of figure) and a variable number of Web/FTP servers (bottom). We will refer to the LVS router nodes as LVS routers, and to the pool of Web/FTP servers as real servers. The real servers are connected via a private network. The LVS routers are connected to this private network and also to the public network. The adapters connecting the LVS routers to the public and private networks (eth0 and eth1 in the figure) may be any device but must be the same on each router.

Only one LVS router is active at a time. The role of the active router is to redirect service requests from the virtual server address to the real servers. The redirection is based on one of the four supported load-balancing algorithms (described in Table 1). The active router dynamically monitors the health of the real servers, and the workload on each, via one of three supported methods (described in Table 2). If a real server becomes disabled, the active router stops sending jobs to the server until it returns to normal operation.

Periodically, the LVS routers exchange "I'm alive" heartbeat messages over their public connection. Should the backup node fail to receive a heartbeat message within an expected interval, it initiates failover in order to assume the role of the active router. During failover, the backup router takes over the failed router's floating IP addresses (as indicated in the cluster configuration file), and uses the technique known as ARP spoofing: starts announcing itself as the destination for IP packets addressed to the failed node. When the failed node returns to service, it assumes the hot backup role.

Currently, the LVS cluster supports one routing method, Network Address Translation (NAT). (In the future, tunneling and direct routing will be added.) The diagram below illustrates how a NAT virtual server works.

Figure 2. An LVS Cluster Implemented with NAT Routing

                             _/\__/\_ 
                            |        | 
                           / Internet \ 
                           \_  _  _  _/ 
                             \/ \/ \/
                                 |
                                 |
             |------------------------------------------| public
                                 |                        network
                            eth0 | virtual server IP address
                       --------------------         
                       |   active router  |
                       |                  |
                       --------------------
                            eth1 | NAT router IP address
                                 |
             |-------|-------------------------|--------| private
                     |                         |          network
                     |                         |
                     |                         |
                |----|----------|  |-----------|---|
                | real server#1 |  | real server#2 |
                |               |  |               |  ...
                |_______________|  |_______________|

Client requests for service arrive at a virtual server IP address. This is a publicly-advertised address that an administrator at the site will have associated with a fully-qualified domain name (for example, lvs.ajax.com). The illustration shows only one virtual server address, but there may be many. A unique virtual server address is a triplet comprising a protocol (TCP or UDP), IP address, and port number.

The IP components of virtual server addresses are floating addresses. They may be aliased to the device (for example, eth0:1) that connects the LVS routers to the public network, or each could be associated with a separate device. The NAT router IP address, also a floating IP address, is the default route used by each real server on the private network to communicate with the active router. As with virtual server addresses, the NAT router IP address may be aliased to the device (for example, eth1:1) connecting the virtual server to the network of real servers, or it could be associated with a separate device.

Virtual server and NAT addresses are enabled only on the active router. Thus, should the active router fail, the backup router enables the virtual server and NAT addresses during take-over of the floating IP addresses. In the topology shown in Figure 2, virtual server addresses are enabled on device eth0 and the NAT router address on eth1.

The IPVS table in the kernel maps requests from the virtual server address to a real server on the private network. For example, a TCP request addressed to port 80 on virtual server 1.2.3.1 might be routed to port 80 on real server 192.168.1.2. The actual mapping in the IPVS table of jobs to real servers is based on which load-balancing algorithm is in use. Table 1 describes the supported load-balancing methods.

Table 1. Load-balancing Methods

Name Description
Round robin Distribute jobs equally among the real servers.
Least-connections Distribute more jobs to real servers with fewer active connections. (The IPVS table stores active connections.)
Weighted round robin Distribute more jobs to servers with greater capacity. Capacity is indicated by the user-assigned weight, which is adjusted upward or downward by dynamic load information.
Weighted least-connections Distribute more jobs to servers with fewer active connections relative to their capacity. Capacity is indicated by the user-assigned weight, which is adjusted upward or downward by dynamic load information.

As a real server processes a request, it returns packets to the active router, where the address of the real server in the packets is replaced by the virtual server address. In this manner, the private network of real servers is masqueraded from requesting clients.

    Next
    Components of an LVS Cluster
 

Home

2010 3/18/2010 9:43:31 AM