強大的日誌分析工具AWStats

原文地址：http://yahoon.blog.51cto.com/13184/40392

前言,公司要用awstats做web服務器的流量統計,到51cto找了下,發現文章很少,所以寫了這篇文章,記錄下自己配置的過程,也希望能幫到有需要的朋友.強大的日誌分析工具AWStats主頁[url]http://awstats.sourceforge.net[/url]注:此文基本按照官方文檔的操作一步步進行,在安裝過程中參考了相關的文檔作了必要的改動.環境 :redhat 9Apache/2.0.54 源碼安裝,路徑爲/usr/local/apache2主機IP 192.168.0.111域名 s1.domain1.com所用的版本awstats-6.5.tar.gz實現功能:通過awstats統計s1.domain1.com的訪問信息,並提供web頁面查看AWStats的功能很多,具體的可點擊上面的官網,我在此主要用它來分析apache服務器的日誌.使用之前還是說說大致的原理:awstats提供一系列的perl腳本實現:服務配置,日誌讀取,報表生成等功能.而功能實現的具體執行過程是:首先當然是apache將訪問情況記錄到日誌中,AWStats每次執行更新時讀取這些日誌,分析日誌數據,將結果存儲到數據庫中,(這個數據庫是awstats自帶的,並不需要第三方軟件支持.),最後AWStats提供一個cgi程序通過web頁面來顯示數據庫中所統計的數據.首先看當前apache的配置,vi httpd.conf

<VirtualHost *:80>ServerAdmin [email protected]DocumentRoot /var/www/html/s1ServerName s1.domain1.comErrorLog logs/s1_web-error_logTransferLog logs/s1_web-access_log</VirtualHost>

訪問s1.domain1.com的日誌記錄到/usr/local/apache2/logs/s1_web-access_log安裝過程如下將軟件包解壓縮爲/usr/local/awstats第一步是爲網站生成配置文件,其實有很多方式,按照官方文檔的說明推薦使用腳本,而且這樣也是比較簡單的一種方式.轉到/usr/local/awstats/tools,此目錄下有很多不同功能的perl腳本.執行[root@server1 tools]# perl awstats_configure.pl會進入一個交互式的配置程序,過程中會有些問題要你回答,當然這些都是可以在以後的配置文件中修改的.我將所有的內容和註解列出如下

----- AWStats awstats_configure 1.0 (build 1.6) (c) Laurent Destailleur -----This tool will help you to configure AWStats to analyze statistics forone web server. You can try to use it to let it do all that is possiblein AWStats setup, however following the step by step manual setupdocumentation (docs/index.html) is often a better idea. Above all if:- You are not an administrator user,- You want to analyze downloaded log files without web server,- You want to analyze mail or ftp log files instead of web log files,- You need to analyze load balanced servers log files,- You want to ‘understand’ all possible ways to use AWStats…Read the AWStats documentation (docs/index.html).---à Running OS detected: Linux, BSD or Unix

檢測到了我當前的操作系統

-----> Check for web server installFound Web server Apache config file '/usr/local/apache2/conf/httpd.conf'

檢測到了我當前apache的配置文件

-----> Check and complete web server config file '/usr/local/apache2/conf/httpd.conf'Warning: You Apache config file contains directives to write 'common' log filesThis means that some features can't work (os, browsers and keywords detection).Do you want me to setup Apache to write 'combined' log files [y/N] ?y

AWStats推薦使用combined格式的日誌,apache默認使用common格式.所以這個地方提示要不要修改apache的配置文件,將日誌格式改爲combined,當然回答yes.

Add 'Alias /awstatsclasses "/usr/local/awstats/wwwroot/classes/"'Add 'Alias /awstatscss "/usr/local/awstats/wwwroot/css/"'Add 'Alias /awstatsicons "/usr/local/awstats/wwwroot/icon/"'Add 'ScriptAlias /awstats/ "/usr/local/awstats/wwwroot/cgi-bin/"'Add '<Directory>' directiveAWStats directives added to Apache config file.

向apache的配置文件裏面增加了上述內容,裏面是一些目錄和別名的設置.這部分一般加到了文件的最後面.

-----> Update model config file '/usr/local/awstats/wwwroot/cgi-bin/awstats.model.conf'File awstats.model.conf updated.

更新了模板文件

-----> Need to create a new config file ?Do you want me to build a new AWStats config/profilefile (required if first install) [y/N] ? y

是否需要創建一個新配置文件,選y

-----> Define config file name to createWhat is the name of your web site or profile analysis ?Example: [url]www.mysite.com[/url]Example: demoYour web site, virtual server or profile name:s1.domain1.com

這個地方填網站的域名

-----> Define config file pathIn which directory do you plan to store your config file(s) ?Default: /etc/awstatsDirectory path to store config file(s) (Enter for default):> 回車

生成的配置文件存放的路徑,默認是/etc/awstats,我不修改,直接回車.

-----> Create config file '/etc/awstats/awstats.s1.domain1.com.conf'Config file /etc/awstats/awstats.s1.domain1.com.conf created.

可以看到文件已經創建了,路徑爲/etc/awstats/awstats.s1.domain1.com.conf'

-----> Restart Web server with '/sbin/service httpd restart'Stopping httpd: [OK]Starting httpd: httpd: Could not determine the server's fully qualified domain name, using 127.0.0.1 for ServerName[OK]

由於修改了httpd.conf,所以它會自動重啓apache來使配置文件生效,但是這裏有個問題,因爲我的apache並沒有採用系統自帶的,而是源碼安裝的,也並沒有加入service,所以此處啓動的雖然成功了,但並不是我需要的.很簡單,等會把它關閉,然後開啓我自己的.

-----> Add update process inside a schedulerSorry, configure.pl does not support automatic add to cron yet.You can do it manually by adding the following command to your cron:/usr/local/awstats/wwwroot/cgi-bin/awstats.pl -update -config=s1.domain1.comOr if you have several config files and prefer having only one command:/usr/local/awstats/tools/awstats_updateall.pl nowPress ENTER to continue...

由於要得到新信息,就必須更新數據庫,也就是說重新做讀取日誌.分析日誌提取裏面新增的部分進行分析,將更新的數據存入數據庫的過程.最好就讓它定時執行.這段話是提醒你,要實現這個功能就需要手動把上面粗體的兩行中的任一行寫入crontab,讓它定時執行.其中第一句是更新s1.domain1.com,而第二句是在有多個站點情況下,將所有的站點都作更新.我在此不做這個操作.回車.

A SIMPLE config file has been created: /etc/awstats/awstats.s1.domain1.com.confYou should have a look inside to check and change manually main parameters.You can then manually update your statistics for 's1.domain1.com' with command:> perl awstats.pl -update -config=s1.domain1.comYou can also read your statistics for 's1.domain1.com' with URL:[url]http://localhost/awstats/awstats.pl?config=s1.domain1.com[/url]Press ENTER to finish...

按回車來結束安裝.這段話就寫的很清楚,接下來的工作就是檢查配置文件執行perl awstats.pl -update -config=s1.domain1.com來更新數據庫通過[url]http://localhost/awstats/awstats.pl?config=s1.domain1.com[/url]來查看統計的信息這段腳本執行完,我們的httpd.conf也更新了,下面是它主要改動的部分

將CustomLog logs/access_log common改爲CustomLog logs/access_log combined在文件的最後面增加了# Directives to allow use of AWStats as a CGIAlias /awstatsclasses "/usr/local/awstats/wwwroot/classes/"Alias /awstatscss "/usr/local/awstats/wwwroot/css/"Alias /awstatsicons "/usr/local/awstats/wwwroot/icon/"ScriptAlias /awstats/ "/usr/local/awstats/wwwroot/cgi-bin/"# This is to permit URL access to scripts/files in AWStats directory.<Directory "/usr/local/awstats/wwwroot">Options NoneAllowOverride NoneOrder allow,denyAllow from all</Directory>

在修改之前,還是先停掉腳本啓動的apache,將正確的apache啓動[root@server1 tools]# service httpd stop[root@server1 tools]# /usr/local/apache2/bin/apachectl start訪問網站s1.domain1.com正常顯示,這次訪問就應該被記錄到了日誌裏面.訪問[url]http://192.168.0.111t/awstats/awstats.pl?config=s1.domain1.com[/url],提示

Forbidden

You don't have permission to access /awstats/awstats.pl on this server

很明白是權限問題,到/usr/local下執行chmod -R 777 awstats (應該755就行了,因爲我只是測試而已)檢查配置文件的關鍵項目vi /etc/awstats/awstats.s1.domain1.com.conf檢查並做如下修改

# LogFile="/var/log/httpd/mylog.log"LogFile="/usr/local/apache2/logs/s1_web-access_log"指到apche的日誌

LogType=W表示分析的是web日誌

LogFormat=1表示日誌格式爲combined

SiteDomain="s1.domain1.com"域名

HostAliases="s1.domain1.com [url]www.s1.domain1.com[/url] 127.0.0.1 localhost"這個變量的意思是這個域的別名.即多個域名對應同一網站的情況,這句是自動生成的.我這裏沒有用到所以就沒有改.

更新數據庫cd /usr/local/awstats/wwwroot/cgi-binperl awstats.pl -config=s1.domain1.com -update得到如下錯誤

Error: AWStats database directory defined in config file by 'DirData' parameter (/var/lib/awstats) does not exist or is not writable.Setup ('/etc/awstats/awstats.s1.domain1.com.conf' file, web server or permissions) may be wrong.Check config file, permissions and AWStats documentation (in 'docs' directory).

同時訪問[url]http://192.168.0.111t/awstats/awstats.pl?config=s1.domain1.com[/url]提示

Error: AWStats database directory defined in config file by 'DirData' parameter (/var/lib/awstats) does not exist or is not writable.Setup ('/etc/awstats/awstats.s1.domain1.com.conf' file, web server or permissions) may be wrong.Check config file, permissions and AWStats documentation (in 'docs' directory).

很明顯是/var/lib/awstats這個目錄不存在這裏需要說明,這個路徑是由配置文件/etc/awstats/awstats.s1.domain1.com.conf裏面的DirData參數決定的,表示存放數據庫的地方.我沒有修改所以依然是默認值DirData="/var/lib/awstats"既然提示不存在,現在就來新建它cd /var/libmkdir awstats進行數據庫更新[root@server1 cgi-bin]# perl awstats.pl -config=s1.domain1.com -update信息如下

Update for config "/etc/awstats/awstats.s1.domain1.com.conf"With data in log file "/usr/local/apache2/logs/s1_web-access_log"...Phase 1 : First bypass old records, searching new record...Searching new records from beginning of log file...Jumped lines in file: 0Parsed lines in file: 1處理的總行數(apache的日誌是一行代表一個記錄)Found 0 dropped records,Found 1 corrupted records, 發現了一個損壞的記錄Found 0 old records,Found 0 new qualified records.

查看日誌的內容[root@server1 cgi-bin]# less /usr/local/apache2/logs/s1_web-access_log192.168.0.28 - - [30/Aug/2007:10:19:59 +0800] "GET / HTTP/1.1" 200 23訪問[url]http://192.168.0.111/awstats/awstats.pl?config=s1.domain1.com[/url]查看統計信息很是奇怪,爲什麼會這樣呢?打開httpd.conf發現虛擬主機的部分並沒有改動,所以記錄下來的日誌類型並沒有改變,需要單獨設置使用的日誌類型.<VirtualHost *:80>ServerAdmin [email protected]DocumentRoot /var/www/html/s1ServerName s1.domain1.comLogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined1CustomLog logs/s1_web-access_log combined1ErrorLog logs/s1_web-error_log</VirtualHost>可以看到我這裏將這種日誌格式命名爲combine1,只是名稱而已,可以隨便起.修改後重啓apache將日誌清空,然後訪問s1.domain1.com,產生日誌如下[root@server1 cgi-bin]# less /usr/local/apache2/logs/s1_web-access_log

192.168.0.28 - - [30/Aug/2007:13:04:05 +0800] "GET / HTTP/1.1" 304 - "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;SV1; TencentTraveler )"192.168.0.28 - - [30/Aug/2007:13:05:30 +0800] "GET / HTTP/1.1" 304 - "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;SV1; TencentTraveler )"192.168.0.28 - - [30/Aug/2007:13:05:30 +0800] "GET / HTTP/1.1" 304 - "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;SV1; TencentTraveler )"192.168.0.28 - - [30/Aug/2007:13:05:30 +0800] "GET / HTTP/1.1" 304 - "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;SV1; TencentTraveler )"192.168.0.28 - - [30/Aug/2007:13:05:31 +0800] "GET / HTTP/1.1" 304 - "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;SV1; TencentTraveler )"192.168.0.28 - - [30/Aug/2007:13:05:31 +0800] "GET / HTTP/1.1" 304 - "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;SV1; TencentTraveler )"……

再次更新數據庫[root@server1 cgi-bin]# perl awstats.pl -config=s1.domain1.com -update

Update for config "/etc/awstats/awstats.s1.domain1.com.conf"With data in log file "/usr/local/apache2/logs/s1_web-access_log"...Phase 1 : First bypass old records, searching new record...Direct access to last remembered record has fallen on another record.So searching new records from beginning of log file...Phase 2 : Now process new records (Flush history on disk after 20000 hosts)...Jumped lines in file: 0Parsed lines in file: 6Found 0 dropped records,Found 0 corrupted records,Found 0 old records,Found 6 new qualified records.

訪問[url]http://192.168.0.111/awstats/awstats.pl?config=s1.domain1.com[/url]最後需要說明的是,在文中已經多次提到過,要能反應最新信息就必須更新數據庫.也就是說你可能已經對它進行了多次訪問,但是查看統計頁面發現沒變化.這是因爲這些日誌還沒有更新到數據庫.簡單的方法就是定時執行.當然現在寫的也只是這個軟件的基本安裝和使用,它的使用遠不止於這些.尤其其中涉及到了apache日誌的相關知識,例如rotate日誌循環,負載均衡的日誌合併等等.我也剛接觸,希望大家多提寶貴意見.下面是相關的網頁[url]http://www.chinaunix.net/jh/13/641101.html[/url][url]http://www.chedong.com/tech/awstats.html[/url][url]http://www.chedong.com/tech/rotate_merge_log.html[/url]

本文出自 “yahoon的小屋” 博客，請務必保留此出處http://yahoon.blog.51cto.com/13184/40392