FastDFS是一個開源的輕量級分佈式文件系統，它對文件進行管理，功能包括：文件存儲、文件同步、文件訪問（文件上傳、文件下載）等，解決了大容量存儲和負載均衡的問題，同時也能做到在集羣環境下一臺機子上傳文件，同時該組下的其他節點下也備份了上傳的文件。做分佈式系統開發時，其中要解決的一個問題就是圖片、音視頻、文件共享的問題和數據備份，分佈式文件系統正好可以解決這個需求。FastDFS的服務主要有兩個角色Tracker和Storage，Tracker服務用於負責調度storage節點與client通信，在訪問上起負載均衡的作用，和記錄storage節點的運行狀態，是連接client和storage節點的樞紐，Storage用於保存文件
-
FastDFS集羣部署
-
整體部署模塊圖
![]()
-
環境準備
-
名稱 描述 centos系統版本 6.9 libfatscommon FastDFS分離出的一些公用函數包 FastDFS FastDFS主程序 fastdfs-nginx-module FastDFS和nginx的關聯模塊 nginx nginx1.15.5 安裝編譯環境
yum install git gcc gcc-c++ make automake autoconf libtool pcre pcre-devel zlib zlib-devel openssl-devel wget vim -y-
磁盤安裝路徑說明
說明 位置 FastDFS所以安裝包安裝位置 /usr/local/src tracker數據 /data/fdfs/tracker Storage數據 /data/fdfs/Storage 配置文件路徑 /etc/fdfs
-
-
安裝libfatscommon
-
解壓、安裝
unzip libfastcommon-master.zip cd libfastcommon-master ./make.sh && ./make.sh install #編譯安裝
-
安裝FastDFS
-
解壓、安裝
unzip fastdfs-master.zip cd fastdfs-master ./make.sh && ./make.sh install #編譯安裝 cp /etc/fdfs/tracker.conf.sample /etc/fdfs/tracker.conf cp /etc/fdfs/storage.conf.sample /etc/fdfs/storage.conf cp /etc/fdfs/client.conf.sample /etc/fdfs/client.conf #客戶端文件，測試用 cp /usr/local/src/fastdfs/conf/http.conf /etc/fdfs/ #供nginx訪問使用 cp /usr/local/src/fastdfs/conf/mime.types /etc/fdfs/ #供nginx訪問使用
![]()
-
-
安裝fastdfs-nginx-module
-
解壓、安裝
unzip fastdfs-nginx-module-master.zip cp /usr/local/src/fastdfs-nginx-module-master/src/mod_fastdfs.conf /etc/fdfs #複製配置文件到fdfs目錄
-
安裝nginx
-
解壓、安裝
tar -zxvf nginx-1.15.5.tar.gz cd nginx-1.15.5 #添加fastdfs-nginx-module模塊 ./configure --add-module=/usr/local/src/fastdfs-nginx-module-master/src/ make && make install #編譯安裝
-
FastDFS集羣部署配置
- tracker配置
#服務器ip爲 xxx.xxx.78.12, xxx.xxx.78.13 vim /etc/fdfs/tracker.conf #需要修改的內容如下 port=22122 # tracker服務器端口（默認22122,一般不修改） base_path=/data/fdfs/tracker #存儲日誌和數據的根目錄![]()
Storage配置
```
vim /etc/fdfs/storage.conf
#需要修改的內容如下
port=23000 # storage服務端口（默認23000,一般不修改）
base_path=/data/fdfs/storage # 數據和日誌文件存儲根目錄
store_path0=/data/fdfs/storage # 第一個存儲目錄
tracker_server=xxx.xxx.78.12:22122 # 服務器1
tracker_server=xxx.xxx.78.13:22122 # 服務器2
http.server_port=8888 # http訪問文件的端口(默認8888,看情況修改,和nginx中保持一致)
```
- client配置
```
vim /etc/fdfs/client.conf
#需要修改的內容如下
base_path=/home/moe/dfs
tracker_server=xxx.xxx.78.12:22122 # 服務器1
tracker_server=xxx.xxx.78.13:22122 # 服務器2
```
- 配置nginx訪問
```
vim /etc/fdfs/mod_fastdfs.conf
#需要修改的內容如下
tracker_server=xxx.xxx.78.12:22122 # 服務器1
tracker_server=xxx.xxx.78.13:22122 # 服務器2
url_have_group_name=true
store_path0=/data/fdfs/storage
#配置nginx.config
vim /usr/local/nginx/conf/nginx.conf
#添加如下配置
server {
listen 8888; ## 該端口爲storage.conf中的http.server_port相同
server_name localhost;
location ~/group[0-9]/ {
ngx_fastdfs_module;
}
......
......
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root html;
}
}
```
-
啓動服務、測試
啓動之前我們還需要在防火牆開通端口 vim /etc/sysconfig/iptables -A INPUT -m state --state NEW -m tcp -p tcp --dport 22122 -j ACCEPT -A INPUT -m state --state NEW -m tcp -p tcp --dport 23000 -j ACCEPT -A INPUT -m state --state NEW -m tcp -p tcp --dport 8888 -j ACCEPT service iptables restart #重啓防火牆![]()
-
每個服務的啓動、關閉和重啓操作
#tracker /etc/init.d/fdfs_trackerd start #啓動tracker服務 /etc/init.d/fdfs_trackerd restart #重啓動tracker服務 /etc/init.d/fdfs_trackerd stop #停止tracker服務 chkconfig fdfs_trackerd on #自啓動tracker服務 #storage /etc/init.d/fdfs_storaged start #啓動storage服務 /etc/init.d/fdfs_storaged restart #重動storage服務 /etc/init.d/fdfs_storaged stop #停止動storage服務 chkconfig fdfs_storaged on #自啓動storage服務 #nginx /usr/local/nginx/sbin/nginx #啓動nginx /usr/local/nginx/sbin/nginx -s reload #重啓nginx /usr/local/nginx/sbin/nginx -s stop #停止nginx ```  -
檢測集羣
# 會顯示會有幾臺storage服務器,有2臺就會顯示 Storage 1-Storage 2的詳細信息 /usr/bin/fdfs_monitor /etc/fdfs/storage.conf![]()
![]()
-
圖片上傳測試
#上傳成功返回 文件訪問 ID # fdfs_upload_file 客戶端配置文件 上傳文件路徑 fdfs_upload_file /etc/fdfs/client.conf /data/test.png![]()
- 測試文件訪問
測試nginx默認端口80 訪問剛剛上傳的文件，兩個地址都能訪問通一個文件，達到數據備份目的。http://xxx.xxx.78.12/group1/M00/00/00/rB9ODFvXuSiAWBYBAALSAkm_6RQ360.png http://xxx.xxx.78.13/group1/M00/00/00/rB9ODFvXuSiAWBYBAALSAkm_6RQ360.png
至此，FastDFS服務器部署完成
-
FastDFS客戶端集成到SpringBoot
-
首先根據官方源碼提示，我們先下載源碼使用maven編譯成jar包放到公司maven私服（Nexus），或者你本地的maven私服（也有其他ant等方式，具體請查看github）
FastDFS-java-client-SDK源碼下載地址#編譯jar包（解壓下載的FastDFS-java-client-SDK源碼，使用mvn命令需要先有maven環境） mvn clean install![]()
![]()
-
maven項目pom.xml中添加依賴
<dependency> <groupId>org.csource</groupId> <artifactId>fastdfs-client-java</artifactId> <version>1.27-SNAPSHOT</version> </dependency> -
接下來我們在項目resources目錄下添加fdfs_client.conf文件
connect_timeout = 30 network_timeout = 30 charset = UTF-8 http.tracker_http_port = 80 http.anti_steal_token = no http.secret_key = 123456 #前面配置的集羣tracker服務器地址 tracker_server = xxx.xxx.78.12:22122 tracker_server = xxx.xxx.78.13:22122 -
寫一個上傳文件對象類
/** * @Author: maoqitian * @Date: 2018/10/26 0026 17:57 * @Description: FastDFS 文件類 */ public class FastDFSFileEntity { //文件名稱 private String name; //內容 private byte[] content; //文件類型 private String ext; //md5值 private String md5; //作者 private String author; public FastDFSFileEntity(String name, byte[] content, String ext, String height, String width, String author) { super(); this.name = name; this.content = content; this.ext = ext; this.author = author; } public FastDFSFileEntity(String name, byte[] content, String ext) { super(); this.name = name; this.content = content; this.ext = ext; } public String getName() { return name; } public void setName(String name) { this.name = name; } public byte[] getContent() { return content; } public void setContent(byte[] content) { this.content = content; } public String getExt() { return ext; } public void setExt(String ext) { this.ext = ext; } public String getMd5() { return md5; } public void setMd5(String md5) { this.md5 = md5; } public String getAuthor() { return author; } public void setAuthor(String author) { this.author = author; } } -
編寫FastDFS操作類，主要是加載初始化配置Tracker服務器，文件上傳，下載，刪除等操作工具類
/** * @Author: maoqitian * @Date: 2018/10/29 0029 9:30 * @Description: FastDFS 操作類 */ public class FastDFSClient { private static org.slf4j.Logger logger = LoggerFactory.getLogger(FastDFSClient.class); //雙重守護單例 private static volatile FastDFSClient mInstance; /** * 加載配置信息 **/ static { try { String filePath=new ClassPathResource("fdfs_client.conf").getFile().getAbsolutePath(); ClientGlobal.init(filePath); }catch (Exception e){ logger.error("FastDFS Client Init Fail!",e); } } private FastDFSClient(){ } public static FastDFSClient getInstance(){ if(mInstance == null){ synchronized (FastDFSClient.class){ if(mInstance == null){ mInstance=new FastDFSClient(); } } } return mInstance; } /** * @Author maoqitian * @Description 上傳文件 * @Date 2018/10/29 0029 9:42 * @Param [fastDFSFileEntity] * @return java.lang.String[] **/ public String[] upload(FastDFSFileEntity file){ logger.info("File Name: " + file.getName() + "File Length:" + file.getContent().length); NameValuePair[] metalist=new NameValuePair[1]; metalist[0]=new NameValuePair("author",file.getAuthor()); long startTime = System.currentTimeMillis(); String[] uploadResults= null; StorageClient storageClient=null; try { storageClient=getTrackerClient(); uploadResults = storageClient.upload_file(file.getContent(),file.getExt(),metalist); }catch (IOException e){ logger.error("IO Exception when uploadind the file:"+file.getName(),e); } catch (Exception e){ logger.error("Non IO Exception when uploadind the file:"+file.getName(),e); } logger.info("upload_file time used:" + (System.currentTimeMillis() - startTime) + " ms"); if(uploadResults==null && storageClient!=null){ logger.error("upload file fail, error code:" + storageClient.getErrorCode()); } String groupName = uploadResults[0]; String remoteFileName = uploadResults[1]; logger.info("upload file successfully!!!" + "group_name:" + groupName + ", remoteFileName:" + " " + remoteFileName); return uploadResults; } public FileInfo getFile(String groupName, String remoteFileName) { try { StorageClient storageClient = getTrackerClient(); return storageClient.get_file_info(groupName, remoteFileName); } catch (IOException e) { logger.error("IO Exception: Get File from Fast DFS failed", e); } catch (Exception e) { logger.error("Non IO Exception: Get File from Fast DFS failed", e); } return null; } public InputStream downFile(String groupName, String remoteFileName) { try { StorageClient storageClient = getTrackerClient(); byte[] fileByte = storageClient.download_file(groupName, remoteFileName); InputStream ins = new ByteArrayInputStream(fileByte); return ins; } catch (IOException e) { logger.error("IO Exception: Get File from Fast DFS failed", e); } catch (Exception e) { logger.error("Non IO Exception: Get File from Fast DFS failed", e); } return null; } /** * @Author maoqitian * @Description * @Date 2018/10/31 0031 11:19 * @Param [remoteFileName] * @return int -1 失敗 0成功 **/ public int deleteFile(String remoteFileName) throws Exception { StorageClient storageClient = getTrackerClient(); int i = storageClient.delete_file("group1", remoteFileName); logger.info("delete file successfully!!!" + i); return i; } public StorageServer[] getStoreStorages(String groupName) throws IOException { TrackerClient trackerClient = new TrackerClient(); TrackerServer trackerServer = trackerClient.getConnection(); return trackerClient.getStoreStorages(trackerServer, groupName); } public ServerInfo[] getFetchStorages(String groupName, String remoteFileName) throws IOException { TrackerClient trackerClient = new TrackerClient(); TrackerServer trackerServer = trackerClient.getConnection(); return trackerClient.getFetchStorages(trackerServer, groupName, remoteFileName); } public String getTrackerUrl() throws IOException { return "http://"+getTrackerServer().getInetSocketAddress().getHostString()+":"+ClientGlobal.getG_tracker_http_port()+"/"; } /** * @Author maoqitian * @Description 獲取 StorageClient * @Date 2018/10/29 0029 10:33 * @Param [] * @return org.csource.fastdfs.StorageClient **/ private StorageClient getTrackerClient() throws IOException{ TrackerServer trackerServer=getTrackerServer(); StorageClient storageClient=new StorageClient(trackerServer,null); return storageClient; } /** * @Author maoqitian * @Description 獲取 TrackerServer * @Date 2018/10/29 0029 10:34 * @Param [] * @return org.csource.fastdfs.TrackerServer **/ private TrackerServer getTrackerServer() throws IOException { TrackerClient trackerClient=new TrackerClient(); TrackerServer trackerServer = trackerClient.getConnection(); return trackerServer; } -
Controller編寫，接收請求並上傳文件返回文件訪問路徑（這裏寫一個文件上傳的例子，其他文件下載，刪除等功能可根據自己需求進行編寫）
/** * @Author maoqitian * @Description 上傳文件 * @Date 2018/10/30 0030 15:07 * @Param [file] * @return com.gxxmt.common.utils.ResultApi **/ @RequestMapping("/upload") public ResultApi upload(@RequestParam("file") MultipartFile file) throws Exception { if (file.isEmpty()) { throw new RRException("上傳文件不能爲空"); } String url; //此處域名獲取可以根據自需求編寫 String domainUrl = OSSFactory.build().getDomainPath(); logger.info("配置的域名爲"+domainUrl); if (StringUtils.isNotBlank(domainUrl)){ url = uploadFile(file,domainUrl); return ResultApi.success.put("url",url); }else { return ResultApi.error("域名配置爲空,請先配置對象存儲域名"); } } /** * @Author maoqitian * @Description 上傳文件到 FastDFS * @Date 2018/10/29 0029 11:11 * @Param [file] * @Param [domainName] 域名 * @return path 文件訪問路徑 **/ public String uploadFile(MultipartFile file,String domainName) throws IOException { String[] fileAbsolutePath={}; String fileName=file.getOriginalFilename(); String ext=fileName.substring(fileName.lastIndexOf(".")+1); byte[] file_buff=null; InputStream inputStream = file.getInputStream(); if(inputStream!=null){ int available = inputStream.available(); file_buff=new byte[available]; inputStream.read(file_buff); } inputStream.close(); FastDFSFileEntity fastDFSFileEntity=new FastDFSFileEntity(fileName,file_buff,ext); try { fileAbsolutePath=FastDFSClient.getInstance().upload(fastDFSFileEntity); logger.info(fileAbsolutePath.toString()); }catch (Exception e){ logger.error("upload file Exception!",e); throw new RRException("文件上傳出錯"+e); } if(fileAbsolutePath == null){ logger.error("upload file failed,please upload again!"); throw new RRException("文件上傳失敗，請重新上傳"); } String path=domainName+fileAbsolutePath[0]+ "/"+fileAbsolutePath[1]; return path; }
-
-
測試一下該方法，上傳一個圖片
-
由日誌打印我們可以看出圖片已經上傳成功
![]()
-
測試訪問上傳的圖片
![]()
-
到此，FastDFS服務器集羣部署和集成客戶端到SpringBoot中已經完成，以後我們就可以愉快的使用FastDFS服務保存我們的圖片等並備份。如果文章中有寫得不對的地方，請給我留言指出，大家一起學習進步。如果覺得我的文章給予你幫助，也請給我一個喜歡和關注。