簡介
MogileFS是一個開源的分佈式文件存儲系統，由LiveJournal旗下的DangaInteractive公司開發。Danga團隊開發了包括Memcached、MogileFS、Perlbal等多個知名的開源項目。
特性
1、應用層：用戶空間文件系統，無須特殊的核心組件
2、無單點：（tracker, mogstore, database(MySQL)）
3、自動文件複製：複製的最小單位不是文件，而class；
4、傳輸中立，無特殊協議：可以通過NFS或HTTP進行通信；
5、簡單的命名空間： 每個文件對應一個key：用於domain定義名稱空間
組成部分
MogileFS：
tracker：追蹤元數據
database：存儲元數據
storage：存儲數據
tracker：
mogilefsd（守護進程）,職責：
replication：節點之間的文件複製
deletion：刪除文件
queryworker：響應客戶請求的文件元數據訪問請求
reaper：在存儲失敗後將文件複製請求重新放置於隊列中
monitor：檢測主機和設備的健康狀態
database：
存儲mogilefs的元數據，一般使用mysql，建議使用冗餘方案保證可用性（MMM,MHA）
mogilefs專門提供了數據結構管理工具mogdbsetup；
storage：
mogstored（進程名），一個準備停當的mogstored節點可通過mogadm命令添加至現在的集羣中，
存儲節點需定義“設備dev”用作存儲空間；每個設備在當前集羣中需通過一個唯一的DevID標識。
client：
客戶端用於與mogilefs簡歷通信，完成數據存取；
安裝包：
MogileFS-Server-2.46-2.el6.noarch.rpm
MogileFS-Server-mogilefsd-2.46-2.el6.noarch.rpm
MogileFS-Server-mogstored-2.46-2.el6.noarch.rpm
MogileFS-Utils-2.19-1.el6.noarch.rpm
perl-MogileFS-Client-1.14-1.el6.noarch.rpm
perl-Net-Netmask-1.9015-8.el6.noarch.rpm
perl-Perlbal-1.78-1.el6.noarch.rpm
架構：
1.安裝mogilefs和數據庫
# ls
MogileFS-Server-2.46-2.el6.noarch.rpm perl-MogileFS-Client-1.14-1.el6.noarch.rpm
MogileFS-Server-mogilefsd-2.46-2.el6.noarch.rpm perl-Net-Netmask-1.9015-8.el6.noarch.rpm
MogileFS-Server-mogstored-2.46-2.el6.noarch.rpm perl-Perlbal-1.78-1.el6.noarch.rpm
MogileFS-Utils-2.19-1.el6.noarch.rpm
# yum localinstall *.rpm -y
# yum install mysql-server mysql -y
2.數據庫授權遠程訪問，並繼承授權權限
mysql> grant all on *.* to root@'192.168.95.%' identified by 'liaobin' with grant option;
mysql> flush privileges;
3.創建mofilefs的數據庫用戶以及授權
mysql> grant all on *.* to mguser@'192.168.95.%' identified by 'liaobin';
mysql> flush privileges;
4.用mogilefs自帶工具mogdbsetup初始化mogilefs數據庫
# mogdbsetup --dbhost=192.168.95.30 --dbname=mogilefs --dbrootuser=root --dbrootpass=liaobin --dbuser=mguser --dbpass=liaobin --yes
查看初始化的數據庫
mysql> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| mogilefs |
| mysql |
| test |
+--------------------+
4 rows in set (0.01 sec)
5.修改tracker（mogilefsd）配置文件
# vim /etc/mogilefs/mogilefsd.conf
# Enable daemon mode to work in background and use syslog
daemonize = 1
# Where to store the pid of the daemon (must be the same in the init script)
pidfile = /var/run/mogilefsd/mogilefsd.pid
# Database connection information
db_dsn = DBI:mysql:mogilefs:host=192.168.95.30 #配置database的連接信息
db_user = mguser #配置mogilefs的數據庫用戶密碼
db_pass = liaobin
# IP:PORT to listen on for mogilefs client requests
listen = 0.0.0.0:7001 #配置監聽的地址和端口
# Optional, if you don't define the port above.
conf_port = 7001
# Number of query workers to start by default.
query_jobs = 10 #生成多少個用於查詢的工作進程
# Number of delete workers to start by default.
delete_jobs = 1 #生成多少個用於刪除的工作進程
# Number of replicate workers to start by default.
replicate_jobs = 5 #生成多少個用於刪除的工作進程
# Number of reaper workers to start by default.
# (you don't usually need to increase this)
reaper_jobs = 1
# Number of fsck workers to start by default.
# (these can cause a lot of load when fsck'ing)
#fsck_jobs = 1
# Minimum amount of space to reserve in megabytes
# default: 100
# Consider setting this to be larger than the largest file you
# would normally be uploading.
#min_free_space = 200
# Number of seconds to wait for a storage node to respond.
# default: 2
# Keep this low, so busy storage nodes are quickly ignored.
#node_timeout = 2
# Number of seconds to wait to connect to a storage node.
# default: 2
# Keep this low so overloaded nodes get skipped.
#conn_timeout = 2
# Allow replication to use the secondary node get port,
# if you have apache or similar configured for GET's
#repl_use_get_port = 1
6.啓動mogilefsd服務
[root@localhost mgfs]# /etc/init.d/mogstored start
7.爲每個節點新增加分區，作爲存儲
# fdisk /dev/sda
.........
# partx -a /dev/sda
# mkfs.ext4 /dev/sda5
# mkdir /mgfs
# vim /etc/fstab
#新增如下條目
/dev/sda5 /mgfs ext4 defaults 0 0
# mount -a
# mount
/dev/sda5 on /mgfs type ext4 (rw)
8.三臺服務器分別創建存儲目錄，並修改權限
95.30：
# mkdir /mgfs/dev1
# chown -R /mgfs/dev1
95.34：
# mkdir /mgfs/dev2
# chown -R /mgfs/dev2
95.35：
# mkdir /mgfs/dev3
# chown -R /mgfs/dev3
9.修改store(mogstored)的配置文件
#vim /etc/mogilefs/mogstored.conf
maxconns = 10000
httplisten = 0.0.0.0:7500
mgmtlisten = 0.0.0.0:7501
#注意此處，寫到dev(1,2,3)的上級目錄就好
docroot = /mgfs
10.啓動mogstored服務
# /etc/init.d/mogstored start
11.用mogadm管理
創建存儲主機：
# mogadm --trackers=192.168.95.30:7001 host add store1 --ip=192.168.95.30
# mogadm --trackers=192.168.95.30:7001 host add store2 --ip=192.168.95.34
# mogadm --trackers=192.168.95.30:7001 host add store3 --ip=192.168.95.35
查看創建的主機：
# mogadm --trackers=192.168.95.30:7001 host list
創建存儲設備：
# mogadm --trackers=192.168.95.30:7001 device add store1 1 --status=alive
# mogadm --trackers=192.168.95.30:7001 device add store2 2 --status=alive
# mogadm --trackers=192.168.95.30:7001 device add store3 3 --status=alive
查看創建的存儲設備：
# mogadm --trackers=192.168.95.30:7001 device list
創建域domain：
mogadm --trackers=192.168.95.30:7001 domain add img
mogadm --trackers=192.168.95.30:7001 domain add conf
查看創建的域：
mogadm --trackers=192.168.95.30:7001 domain list
上傳文件測試：
mogupload --trackers=192.168.95.30:7001 --domain=img --key="1.jpg" --file="/root/1.jpg"
查看上傳的文件：
# mogfileinfo --trackers=192.168.95.30:7001 --domain=img --key="1.jpg"
- file: 1.jpg
class: default
devcount: 2
domain: img
fid: 9
key: 1.jpg
length: 34892
- http://192.168.95.35:7500/dev3/0/000/000/0000000009.fid
- http://192.168.95.30:7500/dev1/0/000/000/0000000009.fid
用fid的url訪問測試：
正常訪問
nginx反向代理mogilefs
1.安裝nginx和mogilefs模塊
安裝好開發環境：
# yum groupinstall "Development tools" "Server Platform Development" -y
# yum install pcre-devel -y
下載解壓nginx包和mogilefs包：
開始編譯安裝：
#cd nginx-1.6.2
# ./configure \
--prefix=/usr/local/nginx \
--sbin-path=/usr/local/nginx/sbin/nginx \
--conf-path=/etc/nginx/nginx.conf \
--error-log-path=/var/log/nginx/error.log \
--http-log-path=/var/log/nginx/access.log \
--pid-path=/var/run/nginx/nginx.pid \
--lock-path=/var/lock/nginx.lock \
--user=nginx \
--group=nginx \
--with-http_ssl_module \
--with-http_flv_module \
--with-http_stub_status_module \
--with-http_gzip_static_module \
--http-client-body-temp-path=/var/tmp/nginx/client/ \
--http-proxy-temp-path=/var/tmp/nginx/proxy/ \
--http-fastcgi-temp-path=/var/tmp/nginx/fcgi/ \
--http-uwsgi-temp-path=/var/tmp/nginx/uwsgi \
--http-scgi-temp-path=/var/tmp/nginx/scgi \
--with-pcre --add-module=../nginx_mogilefs_module-1.0.4
# make && make install
給nginx創建服務腳本
# vim /etc/init.d/nginx
#!/bin/sh
#
# nginx - this script starts and stops the nginx daemon
#
# chkconfig: - 85 15
# description: Nginx is an HTTP(S) server, HTTP(S) reverse \
# proxy and IMAP/POP3 proxy server
# processname: nginx
# config: /etc/nginx/nginx.conf
# config: /etc/sysconfig/nginx
# pidfile: /var/run/nginx.pid
# Source function library.
. /etc/rc.d/init.d/functions
# Source networking configuration.
. /etc/sysconfig/network
# Check that networking is up.
[ "$NETWORKING" = "no" ] && exit 0
nginx="/usr/local/nginx/sbin/nginx"
prog=$(basename $nginx)
NGINX_CONF_FILE="/etc/nginx/nginx.conf"
[ -f /etc/sysconfig/nginx ] && . /etc/sysconfig/nginx
lockfile=/var/lock/subsys/nginx
make_dirs() {
# make required directories
user=`nginx -V 2>&1 | grep "configure arguments:" | sed 's/[^*]*--user=\([^ ]*\).*/\1/g' -`
options=`$nginx -V 2>&1 | grep 'configure arguments:'`
for opt in $options; do
if [ `echo $opt | grep '.*-temp-path'` ]; then
value=`echo $opt | cut -d "=" -f 2`
if [ ! -d "$value" ]; then
# echo "creating" $value
mkdir -p $value && chown -R $user $value
fi
fi
done
}
start() {
[ -x $nginx ] || exit 5
[ -f $NGINX_CONF_FILE ] || exit 6
make_dirs
echo -n $"Starting $prog: "
daemon $nginx -c $NGINX_CONF_FILE
retval=$?
echo
[ $retval -eq 0 ] && touch $lockfile
return $retval
}
stop() {
echo -n $"Stopping $prog: "
killproc $prog -QUIT
retval=$?
echo
[ $retval -eq 0 ] && rm -f $lockfile
return $retval
}
restart() {
configtest || return $?
stop
sleep 1
start
}
reload() {
configtest || return $?
echo -n $"Reloading $prog: "
killproc $nginx -HUP
RETVAL=$?
echo
}
force_reload() {
restart
}
configtest() {
$nginx -t -c $NGINX_CONF_FILE
}
rh_status() {
status $prog
}
rh_status_q() {
rh_status >/dev/null 2>&1
}
case "$1" in
start)
rh_status_q && exit 0
$1
;;
stop)
rh_status_q || exit 0
$1
;;
restart|configtest)
$1
;;
reload)
rh_status_q || exit 7
$1
;;
force-reload)
force_reload
;;
status)
rh_status
;;
condrestart|try-restart)
rh_status_q || exit 0
;;
*)
echo $"Usage: $0 {start|stop|status|restart|condrestart|try-restart|reload|force-reload|configtest}"
exit 2
esac
給予執行權限：
# chmod +x /etc/init.d/nginx
創建nginx用戶和必須目錄：
# useradd -r nginx
# mkdir /var/tmp/nginx/client -pv
修改nginx配置文件讓其支持mogilefs反向代理（紅色字體爲添加項）：
# vim /etc/nginx/nginx.conf
#user nobody;
worker_processes 1;
#error_log logs/error.log;
#error_log logs/error.log notice;
#error_log logs/error.log info;
#pid logs/nginx.pid;
events {
worker_connections 1024;
}
http {
include mime.types;
default_type application/octet-stream;
#log_format main '$remote_addr - $remote_user [$time_local] "$request" '
# '$status $body_bytes_sent "$http_referer" '
# '"$http_user_agent" "$http_x_forwarded_for"';
#access_log logs/access.log main;
sendfile on;
#tcp_nopush on;
#keepalive_timeout 0;
keepalive_timeout 65;
#gzip on;
upstream stores {
server 192.168.95.30:7001;
server 192.168.95.34:7001;
server 192.168.95.35:7001;
}
server {
listen 80;
server_name localhost;
#charset koi8-r;
#access_log logs/host.access.log main;
location / {
root html;
index index.html index.htm;
}
location /img/ {
mogilefs_tracker stores;
mogilefs_domain img;
mogilefs_pass {
proxy_pass $mogilefs_path;
proxy_hide_header Content-Type;
proxy_buffering off;
}
}
location /upload/ {
mogilefs_tracker stores;
mogilefs_domain img;
mogilefs_methods PUT GET DELETE;
mogilefs_pass {
proxy_pass $mogilefs_path;
proxy_hide_header Content-Type;
proxy_buffering off;
}
}
#error_page 404 /404.html;
}
用路徑訪問測試nginx代理：
再上傳一個文件測試：
# mogupload --trackers=192.168.95.30:7001 --domain=img --key="2.jpg" --file="/root/2.jpg"
測試刪除文件：
# curl -X DELETE http://192.168.95.34/upload/1.jpg
