【博文推薦】基于云端虛擬機的LVS/DR+Keepalived+nginx的高可用集群架構配置
最近，公司要我部署一個集群架構，選來選去還是選擇了大家都很熟悉的基于DR的LVS+Keepalived做負載分發，然后使用輕量級的nginx做中 間代理層，這里呢先暫時實現一個簡單web應用功能，對于代理功能和后續的web層部署、數據層安排將擇機更新！
首先看一下集群框架：
下面開始我們的集群搭建行程：
一、IP規劃：
做一件事情需要想想怎么去做既簡單又有條理，我們做項目也應該如此。所以這里一定先做一個簡單的規劃，對于你后續測試的時候出現不能分發的情況有所解決。
負載均衡層 代理層 WEB層（待定）
LVS1：10.124.20.248 nginx1:10.124.20.250 web1
nginx2:10.124.20.251
LVS2：10.124.20.249 nginx3:10.124.20.252 web2
VIP：10.124.20.20 對應綁定的外網IP：117.78.3.253
備注：在云端虛擬機做架構，比在自己本地虛擬機做實驗，要略有不同。初次接觸云端虛擬機的同學可能有點犯迷糊，不知道這個ip地址是怎么分配的，怎么去管 理。我的老師教給一個最能解決問題的辦法：在任何服務之前，首先要了解整個網絡環境，然后根據環境部署服務，更改配置、規則等。關于云端虛擬機，它在整個 硬件防火墻和整個路由器封閉起來的一個大的局域網集群（這里為了好理解我們姑且認為云端就一個防火墻和一個路由器）。在這個集群中，又有很多網段的局域 網，每個企業在申請虛擬機的時候一般會建立一個虛擬機組，這個機組則對應著一個網段。
這里使用的網段是10.124.20.0/24， 給各臺服務器分配唯一的IP地址。這里要特別說明一下，華為云技術人員建議使用的服務器固定ip在32-254之間，在2-31之間的IP地址作為候補之 用，比如說VIP就選用這里面，還有臨時加虛擬機的話也可以選用。所以對于IP地址的規劃是很重要的。另外說一下，這也是在生產環境中的經驗吧，VIP必 須是和集群IP地址在一個網段，還要必須有別于集群IP地址，另外要提供網站訪問服務或者測試，則需要將內網的VIP與一個公網IP進行綁定，在外部網絡 瀏覽器訪問外網IP就會直接映射到內網VIP上，然后就會實現我們所熟悉的功能，VIP漂移，LVS分發。小技巧：公網IP與單純的一個內網ip做綁定是 不成功的，所以先把需要綁定公網IP的內網IP綁定到一個虛擬機上，然后再進行綁定，綁定完之后再進行更改IP地址。（比如說我要使用10.124.20.20做VIP，首先我把10.124.20.20綁定到一臺虛擬機上，通過后臺將外網IP117.78.3.253與10.124.20.20綁定，也就是做映射，最后再將10.124.20.20與虛擬解綁，這樣就實現了預期目的）
二、源碼安裝&&配置
1.LVS+Keepalived搭建
1）LVS1：10.124.20.248&&LVS2：10.124.20.249
- cd /usr/local/src/
- tar xf ipvsadm-1.24.tar.gz
- tar xf keepalived-1.1.20.tar.gz
ipvsadm編譯安裝
- ln -s /usr/src/kernels/2.6.32-279.el6.x86_64/ /usr/src/linux #創建軟連接
- cd /usr/local/src/ipvsadm-1.24
- make && make install
- #/sbin/ipvsadm -v #檢測ipvsadm是否安裝成功(或者：#watch ipvsadm –ln)
- ipvsadm v1.24 2005/12/10 (compiled with getopt_long and IPVS v1.2.1)
將ipvsadm設置成開機自啟動服務
- chkconfig --add keepalived
- chkconfig keepalived on
Keepalived編譯安裝
- /usr/local/src/keepalived-1.1.20
- ./configure --prefix=/usr/local/keepalived --with-kernel-dir=/usr/src/kernels/2.6.32-279.el6.x86_64
- 返回信息：
- Keepalived configuration
- ------------------------
- Keepalived version : 1.1.20
- Compiler : gcc
- Compiler flags : -g -O2
- Extra Lib : -lpopt -lssl -lcrypto
- Use IPVS Framework : Yes
- IPVS sync daemon support : Yes
- Use VRRP Framework : Yes
- Use Debug flags : No
- make && make install
- 返回信息：
- install -d /usr/local/keepalived/bin
- install -m 755 ../bin/genhash /usr/local/keepalived/bin/
- install -d /usr/local/keepalived/share/man/man1
- install -m 644 ../doc/man/man1/genhash.1 /usr/local/keepalived/share/man/man1
將Keepalived做成服務：
- cd /usr/local/keepalived/
- cp etc/rc.d/init.d/keepalived /etc/rc.d/init.d/
- cp etc/sysconfig/keepalived /etc/sysconfig/
- mkdir /etc/keepalived
- cp etc/keepalived/keepalived.conf /etc/keepalived/
- cp sbin/keepalived /usr/sbin/
2）修改Keepalived.conf配置文件
- #cat /etc/keepalived/keepalived.conf #主LVS1：10.124.20.248
- ! Configuration File for keepalived
- global_defs {
- notification_email {
- qingbo.song@apicloud.com
- }
- notification_email_from qingbo.song@apicloud.com
- smtp_server 127.0.0.1
- # smtp_connect_timeout 30
- router_id LVS_DEVEL
- }
- vrrp_sync_group VGM {
- group {
- VI_1
- }
- notify_master "/opt/shell/arp.sh"
- notify_backup "/opt/shell/arp.sh"
- }
- vrrp_script chk_nginx {
- script "/opt/shell/check_nginx.sh"
- interval 1
- weight -2
- }
- vrrp_script chk_nginx2 {
- script "/opt/shell/check_nginx2.sh"
- interval 1
- weight -2
- }
- vrrp_script chk_nginx3 {
- script "/opt/shell/check_nginx3.sh"
- interval 1
- weight -2
- }
- vrrp_instance VI_1 {
- state MASTER
- interface eth0
- virtual_router_id 51
- priority 100
- advert_int 1
- authentication {
- auth_type PASS
- auth_pass yzkjTest
- }
- track_script {
- chk_nginx
- chk_nginx2
- chk_nginx3
- }
- virtual_ipaddress {
- 10.124.20.20
- }
- }
- virtual_server 10.124.20.20 80 {
- delay_loop 5
- lb_algo sh
- lb_kind DR
- persistence_timeout 60
- protocol TCP
- real_server 10.124.20.250 80 {
- weight 1
- TCP_CHECK {
- connect_timeout 10
- nb_get_retry 3
- delay_before_retry 3
- connect_port 80
- }
- }
- real_server 10.124.20.251 80 {
- weight 1
- TCP_CHECK {
- connect_timeout 10
- nb_get_retry 3
- delay_before_retry 3
- connect_port 80
- }
- }
- real_server 10.124.20.252 80 {
- weight 1
- TCP_CHECK {
- connect_timeout 10
- nb_get_retry 3
- delay_before_retry 3
- connect_port 80
- }
- }
- }
- #cat /etc/keepalived/keepalived.conf #備LVS1：10.124.20.249
- ! Configuration File for keepalived
- global_defs {
- notification_email {
- qingbo.song@apicloud.com
- }
- notification_email_from qingbo.song@apicloud.com
- smtp_server 127.0.0.1
- # smtp_connect_timeout 30
- router_id LVS_DEVEL
- }
- vrrp_sync_group VGM {
- group {
- VI_1
- }
- notify_master "/opt/shell/arp.sh"
- notify_backup "/opt/shell/arp.sh"
- }
- vrrp_script chk_nginx {
- script "/opt/shell/check_nginx.sh"
- interval 1
- weight -2
- }
- vrrp_script chk_nginx2 {
- script "/opt/shell/check_nginx2.sh"
- interval 1
- weight -2
- }
- vrrp_script chk_nginx3 {
- script "/opt/shell/check_nginx3.sh"
- interval 1
- weight -2
- }
- vrrp_instance VI_1 {
- state BACKUP
- interface eth0
- virtual_router_id 51
- priority 99
- advert_int 1
- authentication {
- auth_type PASS
- auth_pass yzkjTest
- }
- track_script {
- chk_nginx
- chk_nginx2
- chk_nginx3
- }
- virtual_ipaddress {
- 10.124.20.20
- }
- }
- virtual_server 10.124.20.20 80 {
- delay_loop 5
- lb_algo sh
- lb_kind DR
- persistence_timeout 60
- protocol TCP
- real_server 10.124.20.250 80 {
- weight 1
- TCP_CHECK {
- connect_timeout 10
- nb_get_retry 3
- delay_before_retry 3
- connect_port 80
- }
- }
- real_server 10.124.20.251 80 {
- weight 1
- TCP_CHECK {
- connect_timeout 10
- nb_get_retry 3
- delay_before_retry 3
- connect_port 80
- }
- }
- real_server 10.124.20.252 80 {
- weight 1
- TCP_CHECK {
- connect_timeout 10
- nb_get_retry 3
- delay_before_retry 3
- connect_port 80
- }
- }
- }
相關檢測腳本：
- #cat /opt/shell/arp.sh
- #!/bin/bash
- VIP=10.124.20.20
- GATEWAY=10.124.20.1
- /sbin/arping -I eth0 -c 5 -s $VIP $GATEWAY &>/dev/null
- #cat /opt/shell/check_nginx.sh
- !/bin/bash
- url="http://10.124.20.250"
- status=$(/usr/bin/curl -s --head "$url" | awk '/HTTP/ {print $2}')
- if [ "$status" != "200" ]; then
- /opt/shell/arp.sh start
- fi
- #cat /opt/shell/check_nginx2.sh
- !/bin/bash
- url="http://10.124.20.251"
- status=$(/usr/bin/curl -s --head "$url" | awk '/HTTP/ {print $2}')
- if [ "$status" != "200" ]; then
- /opt/shell/arp.sh start
- fi
- #cat /opt/shell/check_nginx3.sh
- !/bin/bash
- url="http://10.124.20.252"
- status=$(/usr/bin/curl -s --head "$url" | awk '/HTTP/ {print $2}')
- if [ "$status" != "200" ]; then
- /opt/shell/arp.sh start
- fi
3）Keepalived服務啟動（ifconfig eth0:1 10.124.20.248 netmask 255.255.255.255 up）
- service keepalived start #服務啟動
- #ps -aux|grep keepalived #檢測服務啟動狀態
- （/usr/local/keepalived/sbin/keepalived -D -f /etc/keepalived/keepalived.conf -D 顯示在日志記錄 -f 指定配置文件目錄）
- Warning: bad syntax, perhaps a bogus '-'? See /usr/share/doc/procps-3.2.8/FAQ
- root 21205 0.0 0.0 36776 732 ? Ss 17:06 0:00 keepalived -D
- root 21207 0.0 0.1 38880 1936 ? S 17:06 0:00 keepalived -D
- root 21208 0.0 0.1 38880 1232 ? S 17:06 0:00 keepalived -D
- root 21301 0.0 0.0 103240 840 pts/0 S+ 17:07 0:00 grep keepalived
4）#設置開機啟動服務
- （##echo “/usr/local/keepalived/sbin/keepalived -D -f /etc/keepalived/keepalived.conf” >> /etc/rc.d/rc.local）
- chkconfig --add keepalived
- chkconfig keepalived on
- service iptables stop
5）添加防火墻規則：
- iptables -A INPUT -m state --state NEW -m tcp -p tcp --dport 80 -j ACCEPT #使80端口提供對外服務
- iptables -A INPUT -d 224.0.0.0/8 -j ACCEPT
- iptables -A INPUT -p vrrp -j ACCEPT #基于DR模式，當用戶發出請求后，只有DR響應ARP廣播包，允許vrrp虛擬路由器冗余協議
- /etc/rc.d/init.d/iptables save #保存規則到iptables文件，使重啟后生效）
6）開啟轉發功能；
- #vim /etc/sysctl.conf
- net.ipv4.ip_forward = 1
- #sysctl -p #使轉發能夠生效
三、nginx服務器的搭建
1）安裝pcre-8.36.tar.gz
- cd /usr/local/src
- tar xf pcre-8.36.tar.gz
- cd pcre-8.36
- ./configure --prefix=/usr/local/pcre
- make && make install
2）安裝nginx-1.5.1.tar.gz
- cd /usr/local/src
- tar xf nginx-1.5.1.tar.gz
- cd nginx-1.5.1
- ./configure --prefix=/usr/local/nginx --with-http_ssl_module --with-http_stub_status_module --with-pcre=/usr/local/src/pcre-8.36 --with-openssl=/usr/local/src/openssl-1.0.0c //這里跟的目錄是源碼目錄，而不是源碼安裝后的目錄
- make && make install
3）啟動
編寫啟動腳本:
- vim /etc/init.d/nginx | chmod +x /etc/init.d/nginx
- #!/bin/sh
- #
- # nginx - this script starts and stops the nginx daemin
- #
- # chkconfig: - 85 15
- # description: Nginx is an HTTP(S) server, HTTP(S) reverse \
- # proxy and IMAP/POP3 proxy server
- # processname: nginx
- # config: /usr/local/nginx/conf/nginx.conf
- # pidfile: /usr/local/nginx/logs/nginx.pid
- # Source function library.
- . /etc/rc.d/init.d/functions
- # Source networking configuration.
- . /etc/sysconfig/network
- # Check that networking is up.
- [ "$NETWORKING" = "no" ] && exit 0
- nginx="/usr/local/nginx/sbin/nginx"
- prog=$(basename $nginx)
- NGINX_CONF_FILE="/usr/local/nginx/conf/nginx.conf"
- lockfile=/var/lock/subsys/nginx
- start() {
- [ -x $nginx ] || exit 5
- [ -f $NGINX_CONF_FILE ] || exit 6
- echo -n $"Starting $prog: "
- daemon $nginx -c $NGINX_CONF_FILE
- retval=$?
- echo
- [ $retval -eq 0 ] && touch $lockfile
- return $retval
- }
- stop() {
- echo -n $"Stopping $prog: "
- killproc $prog -QUIT
- retval=$?
- echo
- [ $retval -eq 0 ] && rm -f $lockfile
- return $retval
- }
- restart() {
- configtest || return $?
- stop
- start
- }
- reload() {
- configtest || return $?
- echo -n $"Reloading $prog: "
- killproc $nginx -HUP
- RETVAL=$?
- echo
- }
- force_reload() {
- restart
- }
- configtest() {
- $nginx -t -c $NGINX_CONF_FILE
- }
- rh_status() {
- status $prog
- }
- rh_status_q() {
- rh_status >/dev/null 2>&1
- }
- case "$1" in
- start)
- rh_status_q && exit 0
- $1
- ;;
- stop)
- rh_status_q || exit 0
- $1
- ;;
- restart|configtest)
- $1
- ;;
- reload)
- rh_status_q || exit 7
- $1
- ;;
- force-reload)
- force_reload
- ;;
- status)
- rh_status
- ;;
- condrestart|try-restart)
- rh_status_q || exit 0
- ;;
- *)
- echo $"Usage: $0 {start|stop|status|restart|condrestart|try-restart|reload|force-reload|configtest}"
- exit 2
- esac
service nginx start 啟動服務
4）RS配置腳本
啟動RS配置：
- sh /opt/RS.sh start
四、測試
瀏覽器訪問測試117.78.3.253，能夠返回想要的信息即為成功！
五、總結
使用LVS+Keepalived做負載均衡，首先先將內部各個服務器的iptables與selinux關閉，確保內部網絡通暢，一般而言，將上述防火 墻關閉之后，安裝上述配置文件提供的參數進行配置，是能夠實現預期功能的。我的文檔中可能提到過要設置iptables規則，這個要在測試通過之后，再打 開iptables，然后寫上防火墻規則。通過查閱很多資料，看到很多人都在防火墻關閉的情況實驗成功，看來防火墻與LVS的分發功能是多少有點沖突，對 于這方面的知識學習，有待于我們在今后的工作和學習中進行探索，我也將對后續知識在被人博客上做介紹，希望以后能在這個平臺與大家一起交流、學習、成長！
qingbo.song@apicloud.com
2014-12-02
