storm源碼分析-日誌服務

本文爲作者原創，轉載請標明出處。原作者：Tony_老七

原文鏈接：http://blog.csdn.net/tonylee0329/article/details/40081723

以下基於storm的版本: 0.9.0.1

storm中各種日誌是排查問題的有效工具，但是這些日誌是怎麼形成的呢？

storm使用logback作爲日誌服務插件，配置文件見$STORM_HOME/logback/cluster.xml

我們目前主要關心worker、nimbus、ui、supervisor日誌，METRICS、ACCESS這些比較簡單

worker-xxxx.log

nimbus.log

supervisor.log

ui.log

這些日誌都是使用了A1這個appender

<appender name="A1" class="ch.qos.logback.core.rolling.RollingFileAppender">
    <file>${storm.home}/logs/${logfile.name}</file>
    <rollingPolicy class="ch.qos.logback.core.rolling.FixedWindowRollingPolicy">
      <fileNamePattern>${storm.home}/logs/${logfile.name}.%i</fileNamePattern>
      <minIndex>1</minIndex>
      <maxIndex>9</maxIndex>
    </rollingPolicy>

    <triggeringPolicy class="ch.qos.logback.core.rolling.SizeBasedTriggeringPolicy">
      <maxFileSize>100MB</maxFileSize>
    </triggeringPolicy>

    <encoder>
      <pattern>%d{yyyy-MM-dd HH:mm:ss} %c{1} [%p] %m%n</pattern>
    </encoder>
</appender>

採用了logback的日誌輪轉功能，但是這裏的${storm.home}、${logfile.name}這個是在哪定義的呢？

從bin/storm腳本中，可以看到nimbus、ui、supervisor是在啓動的時候傳入了變量logfile.name

如nimbus啓動的時候

def nimbus(klass="backtype.storm.daemon.nimbus"):
    """Syntax: [storm nimbus]

    Launches the nimbus daemon. This command should be run under
    supervision with a tool like daemontools or monit.

    See Setting up a Storm cluster for more information.
    (https://github.com/nathanmarz/storm/wiki/Setting-up-a-Storm-cluster)
    """
    cppaths = [CLUSTER_CONF_DIR]
    jvmopts = parse_args(confvalue("nimbus.childopts", cppaths)) + [
       <span style="color:#ff0000;"> "-Dlogfile.name=nimbus.log",</span>
        "-Dlogback.configurationFile=" + STORM_DIR + "/logback/cluster.xml",
    ]  
    exec_storm_class(
        klass,
        jvmtype="-server",
        extrajars=cppaths,
        jvmopts=jvmopts)

worker的日誌因爲涉及到當前使用端口，並且只有在集羣有topology運行的時候纔會生成

閱讀源碼，可以看到在supervisor.clj的launch-worker方法中生成了logfile.name

在啓動worker進程的時候，根據worker所在port，形成變量名logfile.name,logback中據此引用變量,形成名稱worker-6700.log

(defmethod launch-worker
    :distributed [supervisor storm-id port worker-id]
    (let [conf (:conf supervisor)
          storm-home (System/getProperty "storm.home")
          stormroot (supervisor-stormdist-root conf storm-id)
          stormjar (supervisor-stormjar-path stormroot)
          storm-conf (read-supervisor-storm-conf conf storm-id)
          classpath (add-to-classpath (current-classpath) [stormjar])
          childopts (.replaceAll (str (conf WORKER-CHILDOPTS) " " (storm-conf TOPOLOGY-WORKER-CHILDOPTS))
                                 "%ID%"
                                 (str port))
          <span style="color:#ff0000;">logfilename (str "worker-" port ".log")</span>
          command (str "java -server " childopts
                       " -Djava.library.path=" (conf JAVA-LIBRARY-PATH)
                       " -Dlogfile.name=" logfilename
                       " -Dstorm.home=" storm-home
                       " -Dlogback.configurationFile=" storm-home "/logback/cluster.xml"
                       " -cp " classpath " backtype.storm.daemon.worker "
                       (java.net.URLEncoder/encode storm-id) " " (:assignment-id supervisor)
                       " " port " " worker-id)]
      (log-message "Launching worker with command: " command)
      (launch-process command :environment {"LD_LIBRARY_PATH" (conf JAVA-LIBRARY-PATH)})
      ))

以上便是strom的幾種日誌生成方式。

在storm的開發中，我們通常會自定義日誌路徑，方便管理

如定義路徑爲${storm.home}/logs/${topo_name}/${logfile.name}