I love Music, Swimming and …

Feb 11, 2016

Seesaw 101 : scalable and robust load balancing #1

Google said that they released and deployed successfully Seesaw V2(http://google-opensource.blogspot.kr/2016/01/seesaw-scalable-and-robust-load.html) in their platforms. Load balancing architectures have been already known and used. After reading my post, you will be able to easily understand Seesaw project.

In this post, I will explain 1) Unicast and Anycast with VIPs 2) NAT and DSR(also known as DR) 3) performing health check against the backend and 4) how to setup this Seesaw in your Linux machine. In the next post, I will review software architecture of Seesaw, the automated deployment for Seesaw, and the performance result with other load balancer in the virtual machine.

1. Unicast and Anycast with VIPs

There are three types of communication in general. 1) Unicast 2) Multicast 3) Broadcast. Anycast is not a widely accepted type of communication in IPv4, but it is present in IPv6. Anycast is used in IPv4 occasionally. Let’s take look at Unicast and Anycast.

IPv4 Unicast

This communication is a one-to-one type. One network device communicates with another network device. Fig. 1 shows Unicast flow, and It use a Layer 3 address for Unicast that is IPv4 Class A, Class B, and Class C addresses and a Layer 2 address is a unicast MAC address. Examples include to browse a website, download files using FTP and connect to another device using SSH(Secure Shell).

IPv6/IPv4 Anycast

This communication uses to identify an interface from a group of interfaces, which connect to the same service. For example, there is DNS service used in USA. It can have two DNS servers, one in USA, and the other in South Korea. If you want to resolve FQDN “www.times.com” to an IP address, the best choice will be DNS server from USA. In this example, Anycast can find DNS server which is near to the client within routing distance. Anycast can help a routing protocol of DNS service. The other example is 6to4 IPv6 transition protocol. 6to4 gateways announce their presence on a specific IP, 192.168.99.1. The clients are connecting to use the 6to4 gateway and send traffic to 192.168.99.1.

VIPs

As you know, VIPs(Virtual IP address) is an IP address that doesn’t correspond to an actual physical network interface.

Open Source Projects of Load Balancer

The most important thing is the features of Load balancer for cloud Environments. It can need to handle a service replicas and a health-check with any protocol to increase reliability.

There are several load balancers like HAProxy and LVS(Linux Virtual Server).

Table 1. The difference of Load Balancers

Most of load balancers don’t have the specific features that are backend server monitoring. Seesaw doesn’t use DNS functions for health check and high availability. The more detailed scenario for that will be described in next chapter.

2. NAT and DSR

What is NAT?

A traditional NAT would allow hosts within a private network to transparently access hosts in the external network, in most cases.

Load balancing

There are several load balancing methods on OSI Layer 2/3/4. Seesaw used layer 4 technique with NAT and DSR(also known as DR).

DR(Direct Routing) mode is a very high performance solution and if you use this mode, it will make a very little change to an existing infrastructure. Please see the reference for another mode.

3. Performing health check against the backend

Most of backend systems use a load balanced system for their scalability and reliability. But the health check against the backend system doesn’t have the perfect solution in some specific conditions. We may have to look into the health check way of Seesaw. I’m going to review some scenario how to check their backend systems and load balancer in general.

Normal Scenario

We can simply design high available system in Fig 5. All or some of backend systems may shutdown by unknown reasons. This is why we have to do health check in a normal way. The easiest way is that health checking server is used, that server check from external server to backend servers using HTTP or TCP protocol on Fig 5. But this scenario can’t solve this failure situation in a specific environment that went down all of backend servers. The load balancers do not connect to the server. We have to find another solutions.

DNS Scenario

We can see DNS system on Fig 6. It has two load balancers. In this scenario, we may have to apply public IP to load balancer. If all of backend systems behind of one of load balancer are shutdown, DNS does not resolve the failed load balancer. It is good to see, but this scenario have a critical thing. That is this scenario use a TTL time of DNS records. Even if it update DNS records, it have to wait the recored TTL time. Mostly the TTL time has 1 day or 1 hour. During the TTL time, our system do not operate at all. So we need another techniques or proper solutions.

Anycast for failover of Seesaw

Seesaw support a cluster-mode. With Fig 7, we can see how to solve it in case of the failover of load balancer. BGP(Border Gateway Protocol) 4 is applied for managing backend servers which have IP address, port and so on. Quagga for implementation of BGP is used. For cluster resource management software, Heartbeat from Linux-HA project is used additionally. Hearbeat automatically can be checked the bringing up and down of network interfaces and software modules.

Seesaw use ldirectord for backend monitoring and failover. It was very useful to bring up and down with Anycast IP address for backend status. You can see ldirectord communicates directly with ifconfig and ip_vs in Fig 8. I will also have to review ip_vs, ldirectord, and Quagga for understanding how to process it in the next post. Actually, ip_vs is a module of Linux Kernel and have some functions like load balancing, tunnelling, half NAT and Direct Routing(DR).

4. How to setup this Seesaw in your Linux machine

Seesaw v2 is a load balancing cluster. Then it needs two virtual machines to operate well. Each node must have two network interfaces. One for the host and the other for cluster VIPs.

Building

I write a Vagrantfile for building and installing in my VirtualBox. It set up using precise64 of ubuntu box.

# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant.configure(“2”) do |config|
config.vm.box = “precise64”
config.vm.provider “virtualbox”
config.vm.hostname = “seesaw-01”
config.vm.network :private_network, ip: “192.168.100.10”
config.vm.network “forwarded_port”, guest: 80, host: 80

config.vm.provider :virtualbox do |vb|
vb.customize [
“modifyvm”, :id,
“ — memory”, “1024”,
“ — hwvirtex”, “off”
]
end

config.vm.synced_folder “.”, “/src”, :nfs => true, :nfs_udp => false

And I prepare for building by running as :

$ sudo apt-get update

$ sudo apt-get install curl git gcc make

$ sudo apt-get install protobuf-compiler

$ sudo apt-get install libnl-3-dev libnl-genl-3-dev

$ sudo sudo apt-get install libcap2-bin

$ curl -fsSL “https://storage.googleapis.com/golang/go1.5.3.linux-amd64.tar.gz

$ tar -xzC /usr/local

$ vi ~/.bashrc

PATH /go/bin:/usr/local/go/bin:$PATH

Now, It has to install Go packages in your GOPATH:

$ go get -u golang.org/x/crypto/ssh

$ go get -u github.com/dlintw/goconf

$ go get -u github.com/golang/glog

$ go get -u github.com/golang/protobuf/{proto,protoc-gen-go}

$ go get -u github.com/miekg/dns

Next, it has to get, compile and install Seesaw code from github.

$ go get -u github.com/google/seesaw

$ cd github.com/google/seesaw

$ make test

$ make install

You can write the scripts for a convenient install.

Create seesaw_install.sh in ${GOPATH}/bin and execute below script.

SEESAW_BIN=”/usr/local/seesaw”
SEESAW_ETC=”/etc/seesaw”
SEESAW_LOG=”/var/log/seesaw”

install -d “${SEESAW_BIN}” “${SEESAW_ETC}” “${SEESAW_LOG}”

install “${GOPATH}/bin/seesaw_cli” /usr/bin/seesaw

for component in {ecu,engine,ha,healthcheck,ncc,watchdog}; do
install “${GOPATH}/bin/seesaw_${component}” “${SEESAW_BIN}”
done

install “etc/init/seesaw_watchdog.conf” “/etc/init”
install “etc/seesaw/watchdog.cfg” “${SEESAW_ETC}”

# Enable CAP_NET_RAW for seesaw binaries that require raw sockets.
/sbin/setcap cap_net_raw+ep “${SEESAW_BIN}/seesaw_ha”
/sbin/setcap cap_net_raw+ep “${SEESAW_BIN}/seesaw_healthcheck”

Configure Seesaw

I changed seesaw.cfg as below :

$ vi /etc/seesaw/seesaw.cfg

[cluster]
anycast_enabled = false
name = au-syd
node_ipv4 = 192.168.100.10 # <- changed this VM ip
node_ipv6 = 2015:cafe::10
peer_ipv4 = 192.168.100.3 # <- changed peer VM ip
peer_ipv6 = 2015:cafe::3
vip_ipv4 = 192.168.100.1 # <- changed this cluster VIP
vip_ipv6 = 2015:cafe::1

[config_server]
primary = https://seesaw-config1.example.com/
secondary = https://seesaw-config2.example.com/
tertiary = https://seesaw-config3.example.com/

[interface]
node = eth0
lb = eth1

In conclusion, I reviewed the some issues and terms of load balancing architecture. I explained and introduced the several open source projects related Seesaw. Seesaw v2 is on-going project and deployed in Google infrastructure. I expect that Seesaw improve dramatically the cloud environment, and has an opportunity to apply into Docker environment. In the next post, I will introduce detailed software architecture of Seesaw.