運維監控系統之Open-Falcon

一、Open-Falcon介紹

1、監控系統，可以從運營級別（基本配置即可），以及應用級別（二次開發，通過端口進行日誌上報），對服務器、操作系統、中間件、應用進行全面的監控，及報警，對我們的系統正常運行的作用非常重要。

2、基礎監控

CPU、Load、內存、磁盤、IO、網絡相關、內核參數、ss 統計輸出、端口採集、核心服務的進程存活信息採集、關鍵業務進程資源消耗、NTP offset採集、DNS解析採集，這些指標，都是open-falcon的agent組件直接支持的。

Linux運維基礎採集項：http://book.open-falcon.org/zh/faq/linux-metrics.html

對於這些基礎監控選項全部理解透徹的時刻，也就是對Linux運行原理及命令進階的時刻。

3、第三方監控

術業有專攻，運行在OS上的應用甚多，Open-Falcon的開發團隊不可能把所有的第三方應用的監控全部做完，這個就需要開源社區提供更多的插件，當前對於很多常用的第三方應用都有相關插件了。

4、JVM監控

對於Java作爲主要開發語言的大多數公司，對於JVM的監控不可或缺。

每個JVM應用的參數，比如GC、類加載、JVM內存、進程、線程，都可以上報給Falcon，而這些參數的獲得，都可以通過MxBeans實現。

使用 Java 平臺管理 bean：http://www.ibm.com/developerworks/cn/java/j-mxbeans/

5、業務應用監控

對於業務需要監控的接口，比如響應時間等。可以根據業務的需要，上報相關數據到Falcon，並通過Falcon查看結果。

二、Zabbix、Nagios、Open-Falcon這3大開源運維監控工具的比較

三、Open-Falcon編寫的整個腦洞歷程

參考：http://mp.weixin.qq.com/s?__biz=MjM5OTcxMzE0MQ==&mid=400225178&idx=1&sn=c98609a9b66f84549e41cd421b4df74d

小米開源監控系統OpenFalcon應對高併發7種手段：http://h2ex.com/894

官網：

http://open-falcon.org/

https://book.open-falcon.org

https://github.com/open-falcon/falcon-plus/releases

說明：Open-Falcon部署分爲前端（客戶端）和後端（服務端）

四、後端部署（服務端）

參考官方文檔：https://book.open-falcon.org/zh_0_2/quick_install/prepare.html

1、系統環境

[root@open-falcon-server ~]# cat /etc/redhat-release

CentOS Linux release 7.2.1511 (Core)

2、系統優化

#安裝下載軟件

yum install wget -y

#更換aliyun源

mv /etc/yum.repos.d/CentOS-Base.repo /etc/yum.repos.d/CentOS-Base.repo.backup

wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo

#下載epel源

yum install epel-release.noarch -y

rpm -Uvh http://mirrors.aliyun.com/epel/epel-release-latest-7.noarch.rpm

yum clean all

yum makecache

#下載常用軟件

yum install git telnet net-tools tree nmap sysstat lrzsz dos2unix tcpdump ntpdate -y

#配置時間同步

ntpdate cn.pool.ntp.org

#更改主機名

hostnamectl set-hostname open-falcon-server

hostname open-falcon-server

#開啓緩存

sed -i 's#keepcache=0#keepcache=1#g' /etc/yum.conf

grep keepcache /etc/yum.conf

#關閉selinux

sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config

setenforce 0

#關閉防火牆

systemctl stop firewalld.service

systemctl disable firewalld.service

3、軟件環境準備

#安裝 redis

1	`yum` `install` `redis -y`

#redis常用命令

redis-server redis 服務端

redis-cli　　　　redis 命令行客戶端

redis-benchmark redis 性能測試工具

redis-check-aof 　　AOF文件修復工具

redis-check-dump RDB文件修復工具

redis-sentinel 　　 Sentinel 服務端

#啓動redis

#啓動redis

[root@open-falcon-server ~]# redis-server &

[1] 3308

[root@open-falcon-server ~]

# 3308:C 27 Oct 08:37:26.435 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf

3308:M 27 Oct 08:37:26.436 * Increased maximum number of open files to 10032 (it was originally set to 1024).

_._

_.-``__ ''-._

_.-`` `. `_. ''-._ Redis 3.2.10 (00000000/0) 64 bit

.-`` .-```. ```\/ _.,_ ''-._

( ' , .-` | `, ) Running in standalone mode

|`-._`-...-` __...-.``-._|'` _.-'| Port: 6379

| `-._ `._ / _.-' | PID: 3308

`-._ `-._ `-./ _.-' _.-'

|`-._`-._ `-.__.-' _.-'_.-'|

| `-._`-._ _.-'_.-' | http://redis.io

`-._ `-._`-.__.-'_.-' _.-'

|`-._`-._ `-.__.-' _.-'_.-'|

| `-._`-._ _.-'_.-' |

`-._ `-._`-.__.-'_.-' _.-'

`-._ `-.__.-' _.-'

`-._ _.-'

`-.__.-'

3308:M 27 Oct 08:37:26.437

# WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.

3308:M 27 Oct 08:37:26.437 # Server started, Redis version 3.2.10

3308:M 27 Oct 08:37:26.437

# WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.

3308:M 27 Oct 08:37:26.438

# WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.

3308:M 27 Oct 08:37:26.438 * The server is now ready to accept connections on port 6379

#安裝mysql

[root@open-falcon-server ~]#wget http://repo.mysql.com/mysql-community-release-el7-5.noarch.rpm

[root@open-falcon-server ~]# ll

total 12

-rw-------. 1 root root 1262 Oct 27 2017 anaconda-ks.cfg

-rw-r--r--. 1 root root 6140 Nov 12 2015 mysql-community-release-el7-5.noarch.rpm

[root@open-falcon-server ~]# rpm -ivh mysql-community-release-el7-5.noarch.rpm

Preparing... ################################# [100%]

Updating / installing...

1:mysql-community-release-el7-5 ################################# [100%]

[root@open-falcon-server ~]# ll /etc/yum.repos.d/

total 48

-rw-r--r--. 1 root root 2573 Nov 21 2014 CentOS-Base.repo

-rw-r--r--. 1 root root 1664 Dec 9 2015 CentOS-Base.repo.backup

-rw-r--r--. 1 root root 1309 Dec 9 2015 CentOS-CR.repo

-rw-r--r--. 1 root root 649 Dec 9 2015 CentOS-Debuginfo.repo

-rw-r--r--. 1 root root 290 Dec 9 2015 CentOS-fasttrack.repo

-rw-r--r--. 1 root root 630 Dec 9 2015 CentOS-Media.repo

-rw-r--r--. 1 root root 1331 Dec 9 2015 CentOS-Sources.repo

-rw-r--r--. 1 root root 1952 Dec 9 2015 CentOS-Vault.repo

-rw-r--r--. 1 root root 951 Jun 24 23:04 epel.repo

-rw-r--r--. 1 root root 1050 Jun 24 23:04 epel-testing.repo

-rw-r--r--. 1 root root 1209 Jan 29 2014 mysql-community.repo

-rw-r--r--. 1 root root 1060 Jan 29 2014 mysql-community-source.repo

[root@open-falcon-server ~]#yum install -y mysql-server<br>

#啓動mysql並設置開機自啓動

[root@open-falcon-server ~]# systemctl start mysqld.service

[root@open-falcon-server ~]# systemctl enable mysqld.service

[root@open-falcon-server ~]# mysql -uroot -p

Enter password:

Welcome to the MySQL monitor. Commands end with ; or \g.

Your MySQL connection id is 2

Server version: 5.6.38 MySQL Community Server (GPL)

Oracle is a registered trademark of Oracle Corporation and/or its

affiliates. Other names may be trademarks of their respective

owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> exit

Bye

#檢查服務

[root@open-falcon-server ~]# netstat -lntup

Active Internet connections (only servers)

Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name

tcp        0      0 0.0.0.0:6379            0.0.0.0:*               LISTEN      3308

/redis-server *

tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1443

/sshd

tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      1634

/master

tcp6       0      0 :::3306                 :::*                    LISTEN      3283

/mysqld

tcp6       0      0 :::6379                 :::*                    LISTEN      3308

/redis-server *

tcp6       0      0 :::22                   :::*                    LISTEN      1443

/sshd

tcp6       0      0 ::1:25                  :::*                    LISTEN      1634

/master

udp        0      0 0.0.0.0:68              0.0.0.0:*                           2042

/dhclient

udp        0      0 0.0.0.0:4817            0.0.0.0:*                           2042

/dhclient

udp6       0      0 :::20380                :::*                                2042

/dhclient

4、初始化MySQL表結構（備註：導入表結構後，注意檢查）

cd /tmp/ && git clone https://github.com/open-falcon/falcon-plus.git

cd /tmp/falcon-plus/scripts/mysql/db_schema/

mysql -h 127.0.0.1 -u root -p < 1_uic-db-schema.sql

mysql -h 127.0.0.1 -u root -p < 2_portal-db-schema.sql

mysql -h 127.0.0.1 -u root -p < 3_dashboard-db-schema.sql

mysql -h 127.0.0.1 -u root -p < 4_graph-db-schema.sql

mysql -h 127.0.0.1 -u root -p < 5_alarms-db-schema.sql

rm -rf /tmp/falcon-plus/

#設置數據庫密碼 (備註：用戶名：root ,密碼: password )

[root@open-falcon-server open-falcon]# mysqladmin -uroot password "password"

Warning: Using a password on the command line interface can be insecure.

#檢查導入的數據庫表

[root@open-falcon-server open-falcon]# mysql -uroot -p

Enter password:

Welcome to the MySQL monitor. Commands end with ; or \g.

Your MySQL connection id is 35

Server version: 5.6.38 MySQL Community Server (GPL)

Oracle is a registered trademark of Oracle Corporation and/or its

affiliates. Other names may be trademarks of their respective

owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show databases;

+--------------------+

| Database |

+--------------------+

| information_schema |

| alarms |

| dashboard |

| falcon_portal |

| graph |

| mysql |

| performance_schema |

| uic |

+--------------------+

8 rows in set (0.00 sec)

mysql> exit

Bye

5、配置Go語言開發環境

#安裝go語言開發包（備註：安裝需要epel源）

1	`yum` `install` `golang -y`

#檢查版本

[root@open-falcon-server ~]# go version

go version go1.8.3 linux/amd64

#查看go的安裝路徑

[root@open-falcon-server ~]# find / -name go

/etc/alternatives/go

/var/lib/alternatives/go

/usr/bin/go

/usr/lib/golang/src/cmd/go #需要的是這個路徑

/usr/lib/golang/src/go

/usr/lib/golang/bin/go

/usr/lib/golang/pkg/linux_amd64/go

#設置環境變量GOROOT和GOPATH

export GOROOT=/usr/lib/golang

export GOPATH=/home

6、下載編譯好的二進制版本

https://github.com/open-falcon/falcon-plus/releases

直接下載官方編譯好的二進制版本(x86 64位平臺)。

#開始安裝open-falcon

[root@open-falcon-server ~]# export FALCON_HOME=/home/work

[root@open-falcon-server ~]# export WORKSPACE=$FALCON_HOME/open-falcon

[root@open-falcon-server ~]# mkdir -p $WORKSPACE

#上傳軟件

[root@open-falcon-server ~]# ll

-rw-r--r--. 1 root root 40031472 Oct 29 2017 open-falcon-v0.2.1.tar.gz

#解壓

[root@open-falcon-server ~]# tar -xf open-falcon-v0.2.1.tar.gz -C $WORKSPACE

[root@open-falcon-server ~]# cd $WORKSPACE

<br>#查看解壓結果

[root@open-falcon-server open-falcon]# ll

total 3896

drwxrwxr-x. 7 501 501 67 Aug 15 19:16 agent

drwxrwxr-x. 5 501 501 40 Aug 15 19:16 aggregator

drwxrwxr-x. 5 501 501 40 Aug 15 19:16 alarm

drwxrwxr-x. 6 501 501 51 Aug 15 19:16 api

drwxrwxr-x. 5 501 501 40 Aug 15 19:16 gateway

drwxrwxr-x. 6 501 501 51 Aug 15 19:16 graph

drwxrwxr-x. 5 501 501 40 Aug 15 19:16 hbs

drwxrwxr-x. 5 501 501 40 Aug 15 19:16 judge

drwxrwxr-x. 5 501 501 40 Aug 15 19:16 nodata

-rwxrwxr-x. 1 501 501 3987469 Aug 15 19:16 open-falcon

lrwxrwxrwx. 1 501 501 16 Aug 15 19:16 plugins -> ./agent/plugins/

lrwxrwxrwx. 1 501 501 15 Aug 15 19:16 public -> ./agent/public/

drwxrwxr-x. 5 501 501 40 Aug 15 19:16 transfer

7、修改配置文件cfg.json

坑：

部分模塊依賴連接數據庫，因爲如果不修改配置文件，aggregator模塊會出現無法啓動，graph、hbs、nodata、api、alarm模塊會出現開啓不報錯但是狀態爲開啓失敗的情況。

如果需要每個模塊都能正常啓動，需要將上面模塊的cfg.json的數據庫信息進行修改。根據本教程的配置，需要修改配置文件所在的目錄：

模塊	配置文件所在路徑
aggregator	/home/work/aggregator/config/cfg.json
graph	/home/work/graph/config/cfg.json
hbs	/home/work/hbs/config/cfg.json
nodata	/home/work/nodata/config/cfg.json
api	/home/work/api/config/cfg.json
alarm	/home/work/alarm/config/cfg.json

1）修改aggregator的配置文件

vim /home/work/aggregator/config/cfg.json