| layout | title | subtitle | date | author | tags |
|---|---|---|---|---|---|
post |
使用Google開源庫breakpad實現錯誤報告功能 |
Google大法好 |
2017-08-10 |
cj |
breakpad c++ linux google bugreport |
前陣子寫的微信公衆號後臺服務器自動崩潰重啓了一次，看日誌沒任何頭緒，看來需要core dump。
但是搜索一陣子發現，這玩意真難用，要ulimit -c unlimited後纔會生成dump。又搜索一番，發現Google出品的breakpad，谷歌出品，必屬精品，就它了！
- 按照官方教程，下載源碼
git clone https://chromium.googlesource.com/breakpad/breakpad- 有幾個第三方庫默認情況沒有下載，按需手動下載到breakpad/src/thirdparty中
cd breakpad
git clone https://chromium.googlesource.com/linux-syscall-support src/third_party/lss- 編譯安裝
./configure
make
make check
sudo make install安裝目錄默認爲/usr/local/include/breakpad，庫目錄/usr/local/lib/libbreakpad.a, libbreakpad_client.a
- 應用
方便起見，寫了一個自動生成symbol調試信息的腳本build_symbols.sh:
#!/bin/bash
out=$1
sym="${out}.sym"
dump_syms $out > $sym
line=$(head -n1 ${sym})
arr=($line)
sdir="./symbols/${out}/${arr[3]}"
mkdir -p $sdir
mv $sym $sdir用例./build_symbols.sh test
我把它寫在了makefile中的make release段內。
崩潰後自動調用寫好的腳本生成stack walk併發送郵件給自己的郵箱
#include <client/linux/handler/exception_handler.h>
void crash_send(const std::string& dmp_path)
{
try {
auto path = bfs::canonical("../tools/send_mail_.sh");
if (bfs::exists(path)) {
std::string cmd = path.string() + " server_crash " + dmp_path + " &";
printf("cmd=%s\n", cmd.c_str());
int ret = std::system(cmd.c_str());
printf("ret=%d\n", ret);
}
} catch (bfs::filesystem_error& e) {
printf("%s\n", e.what());
} catch (std::exception& e) {
printf("%s\n", e.what());
}
}
static bool dumpCallback(const google_breakpad::MinidumpDescriptor& descriptor, void* context, bool succeeded)
{
printf("Dump path: %s\n", descriptor.path());
crash_send(descriptor.path());
return succeeded;
}
int main()
{
google_breakpad::MinidumpDescriptor descriptor("/tmp");
google_breakpad::ExceptionHandler eh(descriptor, nullptr, dumpCallback, nullptr, true, -1);
}以下是send_mail_.sh部分代碼：
#!/bin/bash
basedir=/home/ubuntu/wechat_server/tools
#basedir=/home/jack/projects/test
to_user_file="${basedir}/mail_to.txt"
crashed="${basedir}/crashed.sh"
dmp2html="${basedir}/dmp2html.sh"
subject=""
preifx=`echo "<html lang='zh_CN'><head><meta charset='utf8'/></head><body>"`
time=`date +"%Y年%m月%d日 %H:%M:%S"`
param=$1
domain=$2
message=""
if [ "${param}" = "server_stop" ]; then
subject=`echo 微信後臺重啓`
message=`echo "<br/>微信公衆號後臺服務器已停止運行，正在重啓</body></html>"`
elif [ "$param" = "server_crash" ]; then
subject=`echo 微信後臺崩潰`
crash_dump=`${crashed} $2 "${basedir}/symbols"`
crash_dump=`${dmp2html} "${2}.txt"`
message=`echo "<br/>Crash Dump:<br/>"${crash_dump}"<br/></body></html>"`
fi
message=${preifx}${time}${message}
echo 'message=' ${message}
#exit
while IFS= read -r line
do
echo "Sending email to $line"
(
echo "From: admin";
echo "To: $line";
echo "Subject: ${subject}";
echo "Content-Type: text/html";
echo "MIME-Version: 1.0";
echo "";
echo "${message}";
) | /usr/sbin/sendmail -t
echo "Done!"
done <"$to_user_file"如代碼所示，當傳入參數爲server_crash、dump_path時，使用dmp收集腳本crashed.sh生成stackwalk信息並保存爲文件，再使用dmp2html.sh將文件的換行替換爲<br/>，然後就發送郵件。
crashed.sh內容：
#!/bin/bash
dmp_path=$1
sym_path=$2
minidump_stackwalk $dmp_path $sym_path &> ${dmp_path}.txtdmp2html.sh內容：
#!/bin/bash
dmp=$1
while IFS= read -r line
do
echo "${line}<br/>"
done <"$dmp"以下是收到的郵件部分內容：
2017年08月09日 21:57:06
Crash Dump:
2017-08-09 21:57:06: minidump.cc:4811: INFO: Minidump opened minidump /tmp/5f4b59c2-19b1-43e2-5fec0e9c-76809ea0.dmp
……
Operating system: Linux
0.0.0 Linux 4.4.0-87-generic #110-Ubuntu SMP Tue Jul 18 12:55:35 UTC 2017 x86_64
CPU: amd64
family 16 model 6 stepping 2
2 CPUs
GPU: UNKNOWN
Crash reason: SIGABRT
Crash address: 0x43c7
Process uptime: not available
Thread 0 (crashed)
0 libc-2.23.so + 0x35428
rax = 0x0000000000000000 rdx = 0x0000000000000006
rcx = 0x00007eff4892a428 rbx = 0x00007eff4b022000
rsi = 0x00000000000043c7 rdi = 0x00000000000043c7
rbp = 0x00000000005e74de rsp = 0x00007fffe172e378
r8 = 0xfefefefefefefeff r9 = 0x0000000000000001
r10 = 0x0000000000000008 r11 = 0x0000000000000202
r12 = 0x00000000000001a4 r13 = 0x00000 000005e8bc0
r14 = 0x0000000000000001 r15 = 0x0000000000000000
rip = 0x00007eff4892a428
Found by: given as instruction pointer in context
……
17 libc-2.23.so + 0x2dc82
rsp = 0x00007fffe172e500 rip = 0x00007eff48922c82
Found by: stack scanning
18 wechat_server!wechat::open::openapi::get_template_msg_id_for_auther(std::shared_ptr const&, std::__cxx11::basic_string, std::allocator >&) [open.cpp : 420 + 0x46]
rsp = 0x00007fffe172e530 rip = 0x00000000005a36c2
Found by: stack scanning
19 wechat_server!_fini + 0x29178
rsp = 0x00007fffe172e558 rip = 0x00000000005e733c
Found by: stack scanning可以清楚的看到崩潰發生在了open.cpp文件的420行，函數爲get_template_msg_id_for_auther。
至於Linux發送郵件的配置，可以參考Ubuntu下部署MediaWiKi小記。