解決Python下urllib3報錯requests.packages.urllib3.connectionpool:Connection pool is full
Python問題復現
本地新建request.Session, 多線程(或線程池)打開url(或高併發不斷訪問同個站點時)併發較高時, 出現以下報錯:
requests.packages.urllib3.connectionpool:Connection pool is full, discarding connection
測試代碼
def test_pool_max_size():
# 測試 pool_maxsize 對多線程訪問的影響
def thread_get(url):
s.get(url)
s = requests.Session()
s.mount('https://', HTTPAdapter(pool_connections=1, pool_maxsize=1))
ts = []
for _ in range(2):
t = Thread(target=thread_get, args=('https://www.ask.com',))
ts.append(t)
t.start()
for t in ts:
t.join()
代碼分析
HTTPAdapter用於對指定網址的連接管理, 參數包含pool_connections=1, pool_maxsize=1.
在HTTPAdapter中的init中會新建poolmanager
def __init__(self, pool_connections=DEFAULT_POOLSIZE,pool_maxsize=DEFAULT_POOLSIZE ...):
...
self.init_poolmanager(pool_connections, pool_maxsize, block=pool_block)
def init_poolmanager(self, connections, maxsize ...):
...
self.poolmanager = PoolManager(num_pools=connections, maxsize=maxsize,
block=block, strict=True, **pool_kwargs)
在PoolManager中num_pools也就是pool_connections用於構造RecentlyUsedContainer容器, 該容器用於保存最近使用的HTTPConnectionPool/HTTPSConnectionPool. HTTPConnectionPool/HTTPSConnectionPool管理指定(url, port)的所有連接.
def __init__(self, num_pools=10 ...):
...
self.pools = RecentlyUsedContainer(num_pools,
dispose_func=lambda p: p.close())
HTTPConnectionPool/HTTPSConnectionPool也有一個pool, 是LifoQueue容器, 由pool_maxsize參數構造, 保存的是HTTPConnection. HTTPConnection保存與服務器的socket連接.
pool_connections, pool_maxsize解析
pool_connections在poolmanager中限制緩存中不同url對應的HTTPConnectionPool/HTTPSConnectionPool數目.
報錯解析:
pool_maxsize在HTTPConnectionPool/HTTPSConnectionPool中限制緩存中同一個url連接的數目. 單線程時運行時, 只會同時存在一個連接, 不會出現連接數過多的問題. 多線程時, 當同一網址的url請求數量大於pool_maxsize時, 發起url調用時不會報錯, 在請求返回時, 會將連接放入HTTPConnectionPool/HTTPSConnectionPool中的LifoQueue, 當併發請求數量大於pool_maxsize時, LifoQueue不夠放入所有的請求, 就會報錯Connection pool is full, discarding connection.
參考:
urllib3連接池參數pool_connections, pool_maxsize相關源碼解析 - 時時除草，時時耕耘 - CSDN博客
https://blog.csdn.net/bolun365/article/details/82955886
python - urllib3 connectionpool - 連接池已滿，丟棄連接 - Stack Overflow
https://stackoverflow.com/questions/53765366/urllib3-connectionpool-connection-pool-is-full-discarding-connection
////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
///////////////////////////////////////////////解決辦法////////////////////////////////////////////////////////////////////////////////
///////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
參考:
multithreading - example urllib3 and threading in python - Stack Overflow
https://stackoverflow.com/questions/3731379/example-urllib3-and-threading-in-python