mina高併發短連接導致java.io.IOException: Too many open files解決方案
這幾天在解決一個用mina開發的高併發通信過程中產生的一個bug。模擬場景爲:
通過定時觸發啓動線程模擬高併發短連接測試,測試的服務端有2個,一個是服務有起,一個沒起,客戶端和服務端均在同一服務器上。執行一段時間後linux主機上通過lsof命令查看,發現有遞增的文件句柄,pipe和eventpoll。
拋出的異常如下:
Failed to create a new instance of org.apache.mina.transport.socket.nio.NioProcessor: null
java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedConstructorAccessor110.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27 )
at java.lang.reflect.Constructor.newInstance(Constructor.java:513 )
at org.apache.mina.core.service.SimpleIoProcessorPool.<init>(SimpleIoProcessorPool.java:180 )
at org.apache.mina.core.service.SimpleIoProcessorPool.<init>(SimpleIoProcessorPool.java:112 )
at org.apache.mina.core.polling.AbstractPollingIoConnector.<init>(AbstractPollingIoConnector.java:93 )
at org.apache.mina.transport.socket.nio.NioSocketConnector.<init>(NioSocketConnector.java:56 )
at com.develop.webplatform.funnel.client.JobClient.sendMessage(JobClient.java:39 )
at com.develop.webplatform.funnel.client.JobClient.sendJob(JobClient.java:126 )
at com.develop.webplatform.funnel.extend.JobExecRemotelyBySocket.execJobByTask(JobExecRemotelyBySocket.java:66 )
at com.develop.webplatform.funnel.JobManager.execJobByTask(JobManager.java:27 )
at com.develop.webplatform.quartz.job.TaskJob.executeInternal(TaskJob.java:38 )
at org.springframework.scheduling.quartz.QuartzJobBean.execute(QuartzJobBean.java:86 )
at org.quartz.core.JobRunShell.run(JobRunShell.java:223 )
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:549 )
Caused by: org.apache.mina.core.RuntimeIoException: Failed to open a selector.
at org.apache.mina.transport.socket.nio.NioProcessor.<init>(NioProcessor.java:61 )
... 15 more
Caused by: java.io.IOException: Too many open files
at sun.nio.ch.IOUtil.initPipe(Native Method)
at sun.nio.ch.EPollSelectorImpl.<init>(EPollSelectorImpl.java:49 )
at sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:18 )
at java.nio.channels.Selector.open(Selector.java:209 )
at org.apache.mina.transport.socket.nio.NioProcessor.<init>(NioProcessor.java:59 )
... 15 more
原代碼中,關於客戶端連接的代碼如下:
final NioSocketConnector connector = new NioSocketConnector();
final String[] result = new String[ 1 ];
connector.getFilterChain().addLast("codec" ,
new ProtocolCodecFilter( new ObjectSerializationCodecFactory()));
connector.setHandler(handler);
//設置超時
connector.setConnectTimeoutMillis(defaultConnectTimeOut);
ConnectFuture connectFuture = connector.connect(address);
connectFuture.awaitUninterruptibly(); //同步,等待,直到連接完成
if (connectFuture.isDone()) {
if (!connectFuture.isConnected()) { //若在指定時間內沒連接成功,則拋出異常
logger.info("fail to connect " + logInfo);
throw new Exception();
}
}
經過分析,導致主機文件句柄泄露的原因爲,客戶端發起服務端連接時,會請求系統分配相關的文件句柄,在原代碼中,僅僅判斷是否連接成功,而未對連接失敗進 行資源釋放,從而造成文件句柄泄露。當總的文件句柄數超過系統設置值(ulimit -n 查看同一個進程允許的最大文件句柄數),則拋出異常“java.io.IOException: Too many open files”,導致無法創建新的連接,服務器掛掉。
更改後的代碼如下:
final NioSocketConnector connector = new NioSocketConnector();
final String[] result = new String[ 1 ];
connector.getFilterChain().addLast("codec" ,
new ProtocolCodecFilter( new ObjectSerializationCodecFactory()));
connector.setHandler(handler);
//設置超時
connector.setConnectTimeoutMillis(defaultConnectTimeOut);
ConnectFuture connectFuture = connector.connect(address);
connectFuture.awaitUninterruptibly(); //同步,等待,直到連接完成
if (connectFuture.isDone()) {
if (!connectFuture.isConnected()) { //若在指定時間內沒連接成功,則拋出異常
logger.info("fail to connect " + logInfo);
connector.dispose(); //不關閉的話會運行一段時間後拋出,too many open files異常,導致無法連接
throw new Exception();
}
}