HAproxy and keepAlived for Multiple Kubernetes Master Nodes

.

Background

Several months ago, in a project, I was trying to deploy a Kubernetes cluster with kubespray. Considering to build an high available cluster with multiple master nodes and separate etcd (not in container but directly on file system), an haproxy can be a good option. I built my own haproxy to enable a floating IP for multiple Kubernetes APIs to which worker nodes connect.

Architecture

The above diagram is drawn in plantuml. Source code attached at the bottom

Description:

• haproxy and keepalived are installed on 10.0.10.14 and 10.0.10.15 VMs respectively
• 10.0.10.200 is the floating virtual IP failover between 10.0.10.14 and 10.0.10.15
• Load Balancing (LB) service, bound to 10.0.10.200 over port 8383. Request hitting 10.0.10.200 over port 8383 will be forwarded to those 3 Kubernetes master nodes (10.0.10.14/15/16) over port 6443

Note: etcd cluster is created by Ansible as well

Reference >
https://kubesphere.io/docs/installing-on-linux/high-availability-configurations/set-up-ha-cluster-using-keepalived-haproxy/
https://kubespray.io/#/docs/ha-mode

Install

• Install haproxy and keepalived on 10.0.10.14 and 10.0.10.15

Configure haproxy

• /etc/haproxy/haproxy.cfg on 10.0.10.14 and 10.0.10.15

They’re identical on 2 HAproxy machines

ubuntu@node1:~$ cat /etc/haproxy/haproxy.cfg
global
	log /dev/log	local0
	log /dev/log	local1 notice
	chroot /var/lib/haproxy
	stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
	stats timeout 30s
	user haproxy
	group haproxy
	daemon
	# Default SSL material locations
	ca-base /etc/ssl/certs
	crt-base /etc/ssl/private
	# See: https://ssl-config.mozilla.org/#server=haproxy&server-version=2.0.3&config=intermediate
        ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
        ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
        ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets
defaults
	log	global
	mode	http
	option	httplog
	option	dontlognull
        timeout connect 5000
        timeout client  50000
        timeout server  50000
	errorfile 400 /etc/haproxy/errors/400.http
	errorfile 403 /etc/haproxy/errors/403.http
	errorfile 408 /etc/haproxy/errors/408.http
	errorfile 500 /etc/haproxy/errors/500.http
	errorfile 502 /etc/haproxy/errors/502.http
	errorfile 503 /etc/haproxy/errors/503.http
	errorfile 504 /etc/haproxy/errors/504.http
frontend kube-apiserver
  bind *:8383
  mode tcp
  option tcplog
  default_backend kube-apiserver
backend kube-apiserver
    mode tcp
    option tcplog
    option tcp-check      
    balance roundrobin
    default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 250 maxqueue 256 weight 100
    server kube-apiserver-1 10.0.10.14:6443 check # Replace the IP address with your own.
    server kube-apiserver-2 10.0.10.15:6443 check # Replace the IP address with your own.
    server kube-apiserver-3 10.0.10.16:6443 check # Replace the IP address with your own.

Configure keepalived

• /etc/keepalived/keepalived.conf on 10.0.10.14

unicast_src_ip = the current machine
unicast_peer = the remote peer(s). There could be multiple peers

ubuntu@node1:~$ cat /etc/keepalived/keepalived.conf
global_defs {
  notification_email {
  }
  router_id LVS_DEVEL
  vrrp_skip_check_adv_addr
  vrrp_garp_interval 0
  vrrp_gna_interval 0
}
vrrp_script chk_haproxy {
  script "killall -0 haproxy"
  interval 2
  weight 2
}
vrrp_instance haproxy-vip {
  state BACKUP
  priority 100
  interface eth0                # Network card
  virtual_router_id 60
  advert_int 1
  authentication {
    auth_type PASS
    auth_pass 1111
  }
  unicast_src_ip 10.0.10.14	# The IP address of this machine
  unicast_peer {
    10.0.10.15			# The IP address of peer machines
  }
  virtual_ipaddress {
    10.0.10.200/24              # The virtualIP address
  }
  track_script {
    chk_haproxy
  }
}

Change unicase_arc_ip and unicast_peer if adding additional peer(s)

• Restart haproxy and keepalived

systemctl restart haproxy keepalived
systemctl enable haproxy keepalived

Test

systemctl stop haproxy

This will force the floatIP floating to the peer of haproxy

Reference > https://github.com/kubernetes-sigs/kubespray/blob/master/docs/ha-mode.md