spring cloud 服務重啓 Exception caught (might be ok if at shutdown) 異常日誌

 spring cloud 服務重啓中遇到異常 info級別日誌, 如下:

[INFO ] - [c.n.u.c.ShutdownEnabledTimer:59] - Exception caught (might be ok if at shutdown) [TraceInfo:-] 
java.lang.IllegalStateException: Shutdown in progress
	at java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:82)
	at java.lang.Runtime.removeShutdownHook(Runtime.java:239)
	at com.netflix.util.concurrent.ShutdownEnabledTimer.cancel(ShutdownEnabledTimer.java:57)
	at com.netflix.loadbalancer.BaseLoadBalancer.cancelPingTask(BaseLoadBalancer.java:632)
	at com.netflix.loadbalancer.BaseLoadBalancer.shutdown(BaseLoadBalancer.java:883)
	at com.netflix.loadbalancer.DynamicServerListLoadBalancer.shutdown(DynamicServerListLoadBalancer.java:285)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.beans.factory.support.DisposableBeanAdapter.invokeCustomDestroyMethod(DisposableBeanAdapter.java:337)
	at org.springframework.beans.factory.support.DisposableBeanAdapter.destroy(DisposableBeanAdapter.java:271)
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroyBean(DefaultSingletonBeanRegistry.java:571)
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingleton(DefaultSingletonBeanRegistry.java:543)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingleton(DefaultListableBeanFactory.java:1052)
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingletons(DefaultSingletonBeanRegistry.java:504)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingletons(DefaultListableBeanFactory.java:1059)
	at org.springframework.context.support.AbstractApplicationContext.destroyBeans(AbstractApplicationContext.java:1035)
	at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:1011)
	at org.springframework.context.support.AbstractApplicationContext.close(AbstractApplicationContext.java:961)
	at org.springframework.cloud.context.named.NamedContextFactory.destroy(NamedContextFactory.java:92)
	at org.springframework.beans.factory.support.DisposableBeanAdapter.destroy(DisposableBeanAdapter.java:256)
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroyBean(DefaultSingletonBeanRegistry.java:571)
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingleton(DefaultSingletonBeanRegistry.java:543)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingleton(DefaultListableBeanFactory.java:1052)
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingletons(DefaultSingletonBeanRegistry.java:504)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingletons(DefaultListableBeanFactory.java:1059)
	at org.springframework.context.support.AbstractApplicationContext.destroyBeans(AbstractApplicationContext.java:1035)
	at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:1011)
	at org.springframework.context.support.AbstractApplicationContext$1.run(AbstractApplicationContext.java:931)

根據日誌追蹤, 最終定位問題發生是在DynamicServerListLoadBalancer調用 shutdown 方法, 導致最終調用 ShutdownEnabledTimer 的cancel方法,

 

    public void cancel() {
        super.cancel();

        LOGGER.info("Shutdown hook removed for: {}", this.name);

        try {
            Runtime.getRuntime().removeShutdownHook(this.cancelThread);
        } catch (IllegalStateException ise) {
            LOGGER.info("Exception caught (might be ok if at shutdown)", ise);
        }

     }

執行完父類cancel方法後,執行removeShutdownHook這一步拋出異常, 最終會調用到 ApplicationShutdownHooks的remove方法 

    static synchronized boolean remove(Thread hook) {
        if(hooks == null)
            throw new IllegalStateException("Shutdown in progress");

        if (hook == null)
            throw new NullPointerException();

        return hooks.remove(hook) != null;
    }

在第一不驗證hooks時 拋出異常.

猜測原因: 服務重啓過程中, ApplicationShutdownHooks 已執行runHooks方法, 執行完成之後,  hooks置爲null;

    static void runHooks() {
        Collection<Thread> threads;
        synchronized(ApplicationShutdownHooks.class) {
            threads = hooks.keySet();
            hooks = null;
        }

        for (Thread hook : threads) {
            hook.start();
        }
        for (Thread hook : threads) {
            try {
                hook.join();
            } catch (InterruptedException x) { }
        }
    }

添加hook的代碼如下:

 

    public ShutdownEnabledTimer(String name, boolean daemon) {
        super(name, daemon);

        this.name = name;

        this.cancelThread = new Thread(new Runnable() {
            public void run() {
                ShutdownEnabledTimer.super.cancel();
            }
        });

        LOGGER.info("Shutdown hook installed for: {}", this.name);

        Runtime.getRuntime().addShutdownHook(this.cancelThread);
    }

執行調用了 ShutdownEnabledTimer 的cancelThread線程

 

驗證: 工程新建 com.netflix.util.concurrent, 複製原有的ShutdownEnabledTimer 類, cancelThread 構造時添加日誌

public ShutdownEnabledTimer(String name, boolean daemon) {
        super(name, daemon);

        this.name = name;

        this.cancelThread = new Thread(new Runnable() {
            public void run() {
                //添加日誌
                log.info(Thread.currentThread().getName() + " is executed");
                ShutdownEnabledTimer.super.cancel();
            }
        });

        LOGGER.info("Shutdown hook installed for: {}", this.name);

        Runtime.getRuntime().addShutdownHook(this.cancelThread);
    }

部署服務, 重新啓動.

觀察日誌, 發現在DynamicServerListLoadBalancer調用 shutdown 方法前, shutdownHook已執行.

 

綜上, netflix中並未針對服務停止時劃分專用的shutdown方法, 沒有兼容timer hook線程先於其執行的情況, 但是其在日誌中做了說明, 且針對該種異常進行了捕獲, 雖有堆棧日誌打出,但是並未影響服務的終止,可作爲優化點, 但其實並不影響服務質量.

 

另外, 社區關於該問題的討論(https://github.com/spring-cloud/spring-cloud-commons/issues/111),目前已經終止, 並未有修復的計劃.