MultiThreadedHttpConnectionManager遇到的坑

先說背景,使用的是commons-httpclient 3.1版本封裝的HttpUtils,請求一個失效的url。設置了重試大小3次,

因爲是失效的url,所以應該重試3次直接退出。但是現象是重試了兩次後,第三次一直等不到結果,並且請求不會超時,不會中斷,任務一直卡着。

 

附有問題的代碼

初始化client部分

private static HttpClient client = new HttpClient(new MultiThreadedHttpConnectionManager());

static {

    client.getHttpConnectionManager().getParams().setConnectionTimeout(5000);

    client.getHttpConnectionManager().getParams().setSoTimeout(60000);

    client.getParams().setParameter("http.method.retry-handler"new DefaultHttpMethodRetryHandler() {

 

        public boolean retryMethod(HttpMethod method, IOException exception, int executionCount) {

            return executionCount < 3;

        }

    });

}

 

 

請求部分

public static String postJson(String url, String body, ClientType clientType) {

    PostMethod method = new PostMethod(url);

    try {

        method.setRequestEntity(new StringRequestEntity(body, "application/json""UTF-8"));

 

        int code = 0;

        // 重試3次

        for (int retryIndex = 0; retryIndex < 3; retryIndex++) {

            switch (clientType) {

            case FAST:

                code = fast_client.executeMethod(method);

                break;

            case NORMAL:

                code = client.executeMethod(method);

                break;

            default:

                logger.error("error ClientType:" clientType.name());

                return null;

            }

            if (code == 200) {

                long contentLength = method.getResponseContentLength();

                if (contentLength == -1) {

                    return method.getResponseBodyAsString(Integer.MAX_VALUE);

                }

                return method.getResponseBodyAsString();

            }

            logger.error("第{}次重試,code={}", retryIndex, code);

        }

    catch (Exception e) {

        logger.error("push fail: url=" + url + ", body=" + body + ", ClientType=" clientType.name(), e);

    finally {

        method.releaseConnection();

    }

    return null;

}

 

MultiThreadedHttpConnectionManager 是HTTP Client中用來複用連接的連接管理類,可以通過這樣的方式去 創建一個Client,類似於線程池的作用。

創建後,每當執行 int statusCode = client.executeMethod(postMethod);時 http client 委託ConnectionManager創建連接,其實是先委託HttpMethodDirector 執行excute方法, 再通過它委託ConnectionManager 創建連接,HttpMethodDirector 中包含了一下host,請求參數等信息。

 

 我們按照這個流程分開說,先是HttpMethodDirector 創建連接代碼

 

// get a connection, if we need one

if (this.conn == null) {

    this.conn = connectionManager.getConnectionWithTimeout(

        hostConfiguration,

        this.params.getConnectionManagerTimeout()

    );

    this.conn.setLocked(true);

    if (this.params.isAuthenticationPreemptive()

     || this.state.isAuthenticationPreemptive())

    {

        LOG.debug("Preemptively sending default basic credentials");

        method.getHostAuthState().setPreemptive();

        method.getHostAuthState().setAuthAttempted(true);

        if (this.conn.isProxied() && !this.conn.isSecure()) {

            method.getProxyAuthState().setPreemptive();

            method.getProxyAuthState().setAuthAttempted(true);

        }

    }

}

 

connectionManager.getConnectionWithTimeout繼續深入去看 ,會發現如下重要源碼

private HttpConnection doGetConnection(HostConfiguration hostConfiguration,

    long timeout) throws ConnectionPoolTimeoutException {

 

    HttpConnection connection = null;

 

    int maxHostConnections = this.params.getMaxConnectionsPerHost(hostConfiguration);  //注意這裏

    int maxTotalConnections = this.params.getMaxTotalConnections();  //注意這裏

     

    synchronized (connectionPool) {

 

        // we clone the hostConfiguration

        // so that it cannot be changed once the connection has been retrieved

        hostConfiguration = new HostConfiguration(hostConfiguration);

        HostConnectionPool hostPool = connectionPool.getHostPool(hostConfiguration, true);

        WaitingThread waitingThread = null;

 

        boolean useTimeout = (timeout > 0);

        long timeToWait = timeout;

        long startWait = 0;

        long endWait = 0;

 

        while (connection == null) {

 

            if (shutdown) {

                throw new IllegalStateException("Connection factory has been shutdown.");

            }

             

            // happen to have a free connection with the right specs

            //

            if (hostPool.freeConnections.size() > 0) {

                connection = connectionPool.getFreeConnection(hostConfiguration);

 

            // have room to make more

            //

            else if ((hostPool.numConnections < maxHostConnections)

                && (connectionPool.numConnections < maxTotalConnections)) {

 

                connection = connectionPool.createConnection(hostConfiguration);

 

            // have room to add host connection, and there is at least one free

            // connection that can be liberated to make overall room

            //

            else if ((hostPool.numConnections < maxHostConnections)

                && (connectionPool.freeConnections.size() > 0)) {

 

                connectionPool.deleteLeastUsedConnection();

                connection = connectionPool.createConnection(hostConfiguration);

 

            // otherwise, we have to wait for one of the above conditions to

            // become true

            //

            else {

                // TODO: keep track of which hostConfigurations have waiting

                // threads, so they avoid being sacrificed before necessary

 

                try {

                     

                    if (useTimeout && timeToWait <= 0) {

                        throw new ConnectionPoolTimeoutException("Timeout waiting for connection");

                    }

                     

                    if (LOG.isDebugEnabled()) {

                        LOG.debug("Unable to get a connection, waiting..., hostConfig=" + hostConfiguration);

                    }

                     

                    if (waitingThread == null) {

                        waitingThread = new WaitingThread();

                        waitingThread.hostConnectionPool = hostPool;

                        waitingThread.thread = Thread.currentThread();

                    else {

                        waitingThread.interruptedByConnectionPool = false;

                    }

                                 

                    if (useTimeout) {

                        startWait = System.currentTimeMillis();

                    }

                     

                    hostPool.waitingThreads.addLast(waitingThread);

                    connectionPool.waitingThreads.addLast(waitingThread);

                    connectionPool.wait(timeToWait);

                catch (InterruptedException e) {

                    if (!waitingThread.interruptedByConnectionPool) {

                        LOG.debug("Interrupted while waiting for connection", e);

                        throw new IllegalThreadStateException(

                            "Interrupted while waiting in MultiThreadedHttpConnectionManager");

                    }

                    // Else, do nothing, we were interrupted by the connection pool

                    // and should now have a connection waiting for us, continue

                    // in the loop and let's get it.

                finally {

                    if (!waitingThread.interruptedByConnectionPool) {

                        // Either we timed out, experienced a "spurious wakeup", or were

                        // interrupted by an external thread.  Regardless we need to

                        // cleanup for ourselves in the wait queue.

                        hostPool.waitingThreads.remove(waitingThread);

                        connectionPool.waitingThreads.remove(waitingThread);

                    }

                     

                    if (useTimeout) {

                        endWait = System.currentTimeMillis();

                        timeToWait -= (endWait - startWait);

                    }

                }

            }

        }

    }

    return connection;

}

重點關注這幾行代碼

 int maxHostConnections = this.params.getMaxConnectionsPerHost(hostConfiguration);  
 int maxTotalConnections = this.params.getMaxTotalConnections();  
 connectionPool.wait(timeToWait);

看到這裏其實我們對整個獲取鏈接請求的過程就比較明確了。當從連接池中獲取連接時,會比較host 的connection和全局connection個數是否滿足條件,如果不滿足就會阻塞,等待其他請求鏈接釋放後的通知。

其中maxHostConnections 默認值是2,maxTotalConnections 默認值是20,timeToWait默認值是0,0的意思是永遠不會超時。由於我們的代碼都沒有設置,所以全是默認值。

當兩次請求完成後,就用完了maxHostConnections ,並且我們是在最後finally裏進行連接釋放的,導致請求無效的鏈接並沒有及時釋放,所以第三次的時候就拿不到connection,當執行到connectionPool.wait(0)時,就會一直等下去,也就是我們看到的現象

 

所以在初始化的時候我們應該這樣做

private static HttpClient client= new HttpClient(new MultiThreadedHttpConnectionManager());

 

static {

    client.getHttpConnectionManager().getParams().setConnectionTimeout(5000);

    client.getHttpConnectionManager().getParams().setSoTimeout(60000);

    client.getHttpConnectionManager().getParams().setDefaultMaxConnectionsPerHost(50);

    client.getHttpConnectionManager().getParams().setMaxTotalConnections(200);

    client.getParams().setConnectionManagerTimeout(10000);

    client.getParams().setParameter("http.method.retry-handler"new DefaultHttpMethodRetryHandler() {

 

        public boolean retryMethod(HttpMethod method, IOException exception, int executionCount) {

            return executionCount < 3;

        }

    });

 

}

總結一下,除了經常看到的connectionTimeout和soTimeout,connections和鏈接池獲取超時時間都要設置,不能依賴於默認值。

3.x版本的commons-httpclient 不是很好用,並且已經不再維護和更新了,建議升級到4.x httpComponent下的httpclient。在HttpComponent的網頁上,看到了這樣的文字:

HttpComponents Client is a successor of and replacement for Commons HttpClient 3.x. Users of Commons HttpClient are strongly encouraged to upgrade。

發佈了11 篇原創文章 · 獲贊 2 · 訪問量 3萬+
發表評論
所有評論
還沒有人評論,想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.
相關文章