本文鏈接地址:調研內核調用棧方便的工具 kmalloc-top
我們在研究內核的時候，看了內核代碼後，就想着某個函數被誰誰調用。 調用路徑有很多條，有熱門的，有偏門的，但從代碼不大容易看出。 如果我們能和gdb那樣在函數上設個斷點，看下內核函數的調用棧就清楚了。 但是如何統計熱門路線呢？用systemtap就可以，參看這裏，這裏。
但是用systemtap寫統計的時候，用到統計功能的話，如果你的採樣點非常多，超過systemtap規定的上線，systemtap會選擇罷工，直接推出，很不爽。
kmalloc-top就是爲了解決這個問題寫的一個perl腳本，原本用來調查內核中kmalloc的使用情況的，在一個繁忙的內核中,kmallo每秒會被調用成千上萬次，明顯會超過處理的上限。 所以kmalloc-top的方法是stap部分只負責收集堆棧信息，收集一個就寫到標準輸出一個，然後由perl腳本來進一步分析統計。
腳本位於：/usr/local/share/doc/systemtap/examples/memory/kmalloc-top
#The systemtap script that instruments the kmalloc
$script=”
global kmalloc_stack
probe kernel.function(\”__kmalloc\”) { kmalloc_stack[backtrace()]++ }
probe timer.ms(100), end
{
foreach (stack in kmalloc_stack) {
printf(\”\\n\”)
print_stack(stack)
printf(\”\\n\”)
printf(\”%d\\n\”, kmalloc_stack[stack])
}
delete kmalloc_stack
}
“;
我們使用的時候把probe點用自己感興趣的替換掉就好。
它支持的功能見註釋部分：
#!/usr/bin/perl
#
# This script accumulates the execution paths of all calls to kmalloc
# in the kernel. On Ctrl-C (or exit of the command provided by -c option),
# it sorts, filters and displays them on
# stdout.
#
# The -e (exclude) option can be used to specify a comma-separated list
# – any stack with contents matching any of the items in the list will
# be excluded from the output.
#
# The -m (min) option can be used to specify the minimum number of
# occurrences a stack needs to be included in the output.
#
# The -t (top) option can be used to specify printing only the
# top N stack traces.
#
# The -c (command) option runs starts the command and script
# only runs for the duration of the command.
#
# The -o (options) option passes the options to systemap.
#
# Usage: ./kmalloc-top [-m min] [-e exclude list] [-t top_n] [-c command]
# [-o options]
# Ctrl-c
在實際使用中我們會加入 -o –all-modules, 用於打印全調用棧信息，詳細見這裏
典型使用如下：
./kmalloc-top -o –all-modules -m 128 -t 10
取top 10, 每個至少被調用10次。
演示下：
$uname -r 2.6.18-164.el5 $sudo /usr/local/share/doc/systemtap/examples/memory/kmalloc-top -o --all-modules -m 10 -t 3 Will print only the top 3 stacks. Will print stacks with counts >= 10. Press Ctrl-C to stop. ERROR: Skipped too many probes, check MAXSKIPPED or try again with stap -t for more details. WARNING: Number of errors: 0, skipped probes: 101 This path seen 31 times: 0xffffffff800d9d3b : __kmalloc+0x0/0x9f [kernel] 0xffffffff8002e1e9 : __alloc_skb+0x5c/0x12d [kernel] 0xffffffff802214a3 : sock_alloc_send_pskb+0x7d/0x282 [kernel] 0xffffffff8004a71d : unix_stream_sendmsg+0x15f/0x346 [kernel] 0xffffffff80055261 : sock_sendmsg+0xf8/0x14a [kernel] 0xffffffff80220056 : sys_sendto+0x11c/0x14f [kernel] 0xffffffff8005d116 : system_call+0x7e/0x83 [kernel] 0x246 : __mod_error_mask1type56+0x16/0x30 [bnx2i] 0x2 : __mod_version79+0x2/0x10 [cnic] 0x4000000000000000 0x40000000000000 0x400000000000 0x4000000000 0x40000000 0x400000 0x4000 0x40 : __mod_description77+0x20/0x40 [cnic] 0x0 : __mod_version79+0x0/0x10 [cnic] 0x0 : __mod_version79+0x0/0x10 [cnic] 0x0 : __mod_version79+0x0/0x10 [cnic] 0x0 : __mod_version79+0x0/0x10 [cnic] This path seen 22 times: 0xffffffff800d9d3b : __kmalloc+0x0/0x9f [kernel] 0xffffffff8002e1e9 : __alloc_skb+0x5c/0x12d [kernel] 0xffffffff80025c28 : tcp_sendmsg+0x184/0xb0e [kernel] 0xffffffff80055261 : sock_sendmsg+0xf8/0x14a [kernel] 0xffffffff8021fb48 : sys_sendmsg+0x217/0x28a [kernel] 0xffffffff8005d116 : system_call+0x7e/0x83 [kernel] 0x202 : __mod_error_mask157+0x2/0x30 [bnx2i] 0x2 : __mod_version79+0x2/0x10 [cnic] 0x0 : __mod_version79+0x0/0x10 [cnic] 0x0 : __mod_version79+0x0/0x10 [cnic] 0x0 : __mod_version79+0x0/0x10 [cnic] 0x0 : __mod_version79+0x0/0x10 [cnic] 0x0 : __mod_version79+0x0/0x10 [cnic] 0x0 : __mod_version79+0x0/0x10 [cnic] 0x0 : __mod_version79+0x0/0x10 [cnic] 0x700000000000000 0x7000000000000 0x70000000000 0x700000000 0x7000000 0x70000 This path seen 14 times: 0xffffffff800d9d3b : __kmalloc+0x0/0x9f [kernel] 0xffffffff8002e1e9 : __alloc_skb+0x5c/0x12d [kernel] 0xffffffff8003a7e8 : tcp_send_ack+0x20/0xf1 [kernel] 0xffffffff80252178 : tcp_delack_timer+0x180/0x1ed [kernel] 0xffffffff800968be : run_timer_softirq+0x133/0x1af [kernel] 0xffffffff8001235a : __do_softirq+0x89/0x133 [kernel] 0xffffffff8005e2fc : call_softirq+0x1c/0x28 [kernel] 0xffff81034ab93f98 0x20ffff81034ab93f 0xfe20ffff81034ab9 0xb8fe20ffff81034a 0x4ab8fe20ffff8103 0x34ab8fe20ffff81 0x81034ab8fe20ffff 0xff81034ab8fe20ff 0xffff81034ab8fe20 0x14ffff81034ab8fe 0xcb14ffff81034ab8 0x6cb14ffff81034a 0x8006cb14ffff8103 0xff8006cb14ffff81 Num stacks before filtering: 271 Num stacks after filtering: 3 Total kmallocs (before filtering): 536 Total kmallocs (after filtering): 67 The filter stacks have 12.5 of the total kmallocs