Python核心編程——第4章多線程編程筆記

引言

通用概念

計算機程序：存儲在磁盤上的可執行二進制（或其他類型）文件。

進程（重量級進程）則是一個執行中程序，有生命週期，每個進程都擁有自己的地址空間、內存、數據棧以及其他用於跟蹤執行的輔助數據。進程間通過進程間通信（IPC）方式共享信息。

線程（輕量級進程）在同一個進程下執行，共享相同的上下文。

臨界區代碼一般在多線程代碼中，總會有一些特定的函數或代碼塊不希望（或不應該）被多個線程同時執行，通常包括修改數據庫、更新文件或其他會產生竟態條件的類似情況。

同步任意數量的線程可以訪問臨界區的代碼，但在給定的時刻只有一個線程可以通過時，就是使用同步的時候。

信號量最古老的同步原語之一。它是一個計數器，當資源消耗時遞減，當資源釋放時遞增。

多線程的目的相互獨立、無因果關係的任務同時進行，以顯著提高整個任務的性能。

適用的任務特點

本質是上是異步的
需要多個併發活動
每個活動的處理順序可能是不確定的，或者是隨機、不可預測的
這種任務可以被組織或劃分成多個執行流，其中每個執行流都有一個指定要完成的任務。根據應用的不同，這些子任務可能需要計算出中間結果，然後合併爲最終的輸出結果。

典型的兩類任務

計算密集型任務
單線程多個外部輸入源（劃分爲3個任務）
- UserRequestThread：負責讀取客戶端輸入，該輸入可能來自 I/O 通道。程序將創建多個線程，每個客戶端一個，客戶端的請求將會被放入隊列中。
- RequestProcessor：該線程負責從隊列中獲取請求並進行處理，爲第 3 個線程提供輸出。
- ReplyThread:負責向用戶輸出，將結果傳回給用戶(如果是網絡應用)，或者把數據寫到本地文件系統或數據庫中。

python相關概念

執行方式 Python代碼由Python虛擬機（又名解釋器主循環）進行控制的。在主循環中同時只能有一個控制線程執行，任意給定時刻只有一個線程會被解釋器執行。
全局解釋器鎖（GIL）控制對Python虛擬機的訪問。

GIL執行方式
1. 設置GIL。
2. 切換一個進程取執行。
3. 執行指定數量的字節碼指令/線程主動讓出控制權
4. 把線程設置輝睡眠狀態（切換出進程）。
5. 解鎖GIL。
6. 重複上述步驟

守護線程整個Python程序（主線程）將在所有非守護線程退出後才退出，即主線程結束後守護線程仍然可以工作。

多線程的實現方式

兩大模塊

實現方式	提供原語	守護線程	是否建議使用
thread模塊	acquire獲取、release釋放、locked狀態：較基礎	不支持	一般不建議
threading模塊	Lock鎖、Condition、Semaphore信號量等：較豐富	支持	建議使用

三種替代方案

名稱	用途
subprocess模塊	主要用於通過標準（stdin、stdout、stderr）進行進程間通信
multiprocessing模塊	允許爲多核或多CPU派生進程，接口與threading相似
concurrent.futures模塊	新的高級庫，在“任務”級別進行操作。線程池的使用。

使用Thread類主要的三種創建線程的方法

方式	建議
創建Thread的實例，傳給它一個函數	簡單直接，建議
創建Thread的實例，傳給它一個可調用的類實例	難以閱讀，不建議
派生Thread類的子類，並創建子類的實例	更符合面向對象的接口時使用，建議

模塊	描述
thread	基本的、低級別的線程模塊，python3中重命名爲_thread
threading	高級別的線程和同步對象
multiprocessing	使用“threading”接口派生、使用子進程
subprocess	完全跳過線程，使用進程來執行
Queue	供多線程使用的同步先入先出隊列
mutex	互斥對象，python3.0已移除
concurrent.futures	異步執行的高級別庫
SocketServer	創建、管理線程控制的TCP、UDP服務器

代碼實現部分（python3）

最簡單的線程，定時等待

#!/usr/bin/env python
import _thread as thread
from time import sleep, ctime

def loop0():
    print('start loop 0 at: {}'.format(ctime()))
    sleep(4)
    print('loop 0 done at: {}'.format(ctime()))

def loop1():
    print('start loop 1 at: {}'.format(ctime()))
    sleep(2)
    print('loop 1 done at: {}'.format(ctime()))

def main():
    print('starting at: {}'.format(ctime()))
    thread.start_new_thread(loop0, ())
    thread.start_new_thread(loop1, ())
    sleep(6) # 停止6秒，該句去掉則先輸出all DONE 再輸出loop0或loop1
    print('all DONE at: {}'.format(ctime()))

if __name__ == '__main__':
    main()

使用鎖來等待

#!/usr/bin/env python
import _thread as thread
from time import sleep, ctime

def loop0():
    print('start loop 0 at: {}'.format(ctime()))
    sleep(4)
    print('loop 0 done at: {}'.format(ctime()))

def loop1():
    print('start loop 1 at: {}'.format(ctime()))
    sleep(2)
    print('loop 1 done at: {}'.format(ctime()))

def main():
    print('starting at: {}'.format(ctime()))
    thread.start_new_thread(loop0, ())
    thread.start_new_thread(loop1, ())
    sleep(6)
    print('all DONE at: {}'.format(ctime()))
     
if __name__ == '__main__':
    main()

使用Thread類的三種創建線程的方法之一：創建Thread實例，傳函數

#!/usr/bin/env python
# coding:utf-8
import threading
from time import sleep, ctime

loops = [4, 2]  # 等待時間

def loop(nloop, nsec):
    print('start loop {} at: {}'.format(nloop, ctime()))
    sleep(nsec)
    print('loop {} done at: {}'.format(nloop, ctime()))

def main():
    print('starting at: {}'.format(ctime()))
    threads = []
    nloops = list(range(len(loops)))
    
    for i in nloops: # 生成Thread對象，函數+參數
        t = threading.Thread(target=loop, args=(i, loops[i])) 
        threads.append(t)
        
    for i in nloops:
        threads[i].start() # 開始啓動線程
        
    for i in nloops:    # 等待所有線程執行完畢
        threads[i].join()   # threads to finish
   
    print('all DONE at: {}'.format(ctime()))

if __name__ == '__main__':
    main()

運行結果

starting at: Sat May 25 11:37:11 2019
start loop 0 at: Sat May 25 11:37:11 2019
start loop 1 at: Sat May 25 11:37:11 2019
loop 1 done at: Sat May 25 11:37:13 2019
loop 0 done at: Sat May 25 11:37:15 2019
all DONE at: Sat May 25 11:37:15 2019

使用Thread類的三種創建線程的方法之二：創建Thread實例，傳可調用類

#!/usr/bin/env python
import threading
from time import sleep, ctime

loops = [4, 2]

class ThreadFunc(object):
    
    def __init__(self, func, args, name=''):
        self.name = name
        self.func = func
        self.args = args
        
    def __call__(self):
        self.func(*self.args)

def loop(nloop, nsec):
    print('start loop {} at: {}'.format(nloop, ctime()))
    sleep(nsec)
    print('loop {} done at: {}'.format(nloop, ctime()))
    

def main():
    print('starting at: {}'.format(ctime()))
    threads = []
    nloops = list(range(len(loops)))
    
    for i in nloops:
        t = threading.Thread(target=ThreadFunc(loop,(i, loops[i]), loop.__name__))
        threads.append(t)
        
    for i in nloops:
        threads[i].start() # start threads
        
    for i in nloops:    # wait for all
        threads[i].join()   # threads to finish
   
    print('all DONE at: {}'.format(ctime()))

    
if __name__ == '__main__':
    main()

使用Thread類的三種創建線程的方法之三：派生Thread子類，並創建子類的實例

#!/usr/bin/env python
# coding:utf-8
import threading
from time import sleep, ctime

loops = [4, 2]

class MyThread(threading.Thread): # 繼承Thread類
    
    def __init__(self, func, args, name=''):
        threading.Thread.__init__(self)
        self.name = name
        self.func = func
        self.args = args
        
    def run(self):
        self.func(*self.args)

def loop(nloop, nsec):
    print('start loop {} at: {}'.format(nloop, ctime()))
    sleep(nsec)
    print('loop {} done at: {}'.format(nloop, ctime()))
    
def main():
    print('starting at: {}'.format(ctime()))
    threads = []
    nloops = list(range(len(loops)))
    
    for i in nloops:
        t = MyThread(loop,(i, loops[i]), loop.__name__) # 生成派生子類實例
        threads.append(t)
        
    for i in nloops:
        threads[i].start() # start threads
        
    for i in nloops:    # wait for all
        threads[i].join()   # threads to finish
   
    print('all DONE at: {}'.format(ctime()))

if __name__ == '__main__':
    main()

多線程與單線程的比較之基礎定義

#!/usr/bin/env python
import threading
from time import ctime

class MyThread(threading.Thread):
    
    def __init__(self, func, args, name=''):
        threading.Thread.__init__(self)
        self.name = name
        self.func = func
        self.args = args
        
    def getResult(self):
        return self.res
        
    def run(self):
        print('staring {} at: {}'.format(self.name, ctime()))
        self.res = self.func(*self.args)
        print('{} finished at: {}'.format(self.name, ctime()))

多線程與單線程的比較

#!/usr/bin/env python 
from myThread import MyThread
from time import ctime, sleep

def fib(x):
    sleep(0.005)
    if x < 2: return 1
    return (fib(x-2) + fib(x-1))

def fac(x):
    sleep(0.1)
    if x < 2: return 1 
    return (x * fac(x-1))
The 
def sum(x):
    sleep(0.1)
    if x < 2: return 1 
    return (x + sum(x-1))

funcs = [fib, fac, sum]
n = 12

def main():
    nfuncs = list(range(len(funcs)))
    print('*** SINGLE THREAD') 
    for i in nfuncs:
        print('starting', funcs[i].__name__, 'at:', ctime())
        print(funcs[i](n))
        print(funcs[i].__name__, 'finished at:', ctime())
    print('\n*** MULTIPLE THREADS')
    threads = []
    for i in nfuncs:
        t = MyThread(funcs[i], (n,), funcs[i].__name__)
        threads.append(t)
    
    for i in nfuncs:
        threads[i].start()
    
    for i in nfuncs:
        threads[i].join()
        print(threads[i].getResult())
    
    print('all DONE')

if __name__ == '__main__':
    main()

多線程與單線程的比較之輸出結果 5s vs. 2s

*** SINGLE THREAD
starting fib at: Sat May 25 12:08:06 2019
233
fib finished at: Sat May 25 12:08:08 2019
starting fac at: Sat May 25 12:08:08 2019
479001600
fac finished at: Sat May 25 12:08:09 2019
starting sum at: Sat May 25 12:08:09 2019
78
sum finished at: Sat May 25 12:08:11 2019

*** MULTIPLE THREADS
staring fib at: Sat May 25 12:08:11 2019staring fac at: Sat May 25 12:08:11 2019
staring sum at: Sat May 25 12:08:11 2019
sum finished at: Sat May 25 12:08:12 2019
fac finished at: Sat May 25 12:08:12 2019
fib finished at: Sat May 25 12:08:13 2019
233
479001600
78
all DONE

IO密集型實例：亞馬遜書籍排名(bookrank.py)

#!/usr/bin/env python
# coding:utf-8
from atexit import register
from re import compile
from threading import Thread
from time import ctime

import requests

REGEX = compile('#([\d,]+) in Books') # 正則規則
AMZN = 'https://amazon.com/dp/'
ISBNs = {
    '0132269937': 'Core Python Programming',
    '0132356139': 'Python Web Development with Django',
    '0137143419': 'Python Fundamentals',
}

def getRanking(isbn):
    _url = '%s%s' % (AMZN, isbn)
    user_agent = 'Mozilla/4.0 (compatible; MSIE 5.5; Windows NT)'
    headers = {'User-Agent': user_agent}
    page = requests.get(_url, headers=headers) 
    data = page.text
    if page.status_code == 200:
        return REGEX.findall(data)[0]
    else:
        return "unknown"

def _showRanking(isbn):
    print('- %r randked %s' %(ISBNs[isbn], getRanking(isbn)))

def _main():
    print('At', ctime(), 'on Amazon...')
    for isbn in ISBNs:
       # _showRanking(isbn)
       Thread(target=_showRanking, args=(isbn,)).start()

@register # atexit模塊主要的作用就是在程序即將結束之前執行的代碼, atexit.register 註冊函數，注jupyter notebook運行時無效，原因未知
def _atexit():
    print('all DONE at:', ctime())

if __name__ == '__main__':
    _main()

運行結果

At Sun May 26 22:33:05 2019 on Amazon…
- ‘Python Web Development with Django’ randked 451,395
- ‘Python Fundamentals’ randked 5,301,299
- ‘Core Python Programming’ randked 794,988
all DONE at: Sun May 26 22:33:10 2019

使用 concurrent.futures中的線程池模塊的加強版亞馬遜書籍排名

#!/usr/bin/env python
from concurrent.futures import ThreadPoolExecutor
from re import compile
from time import ctime
from urllib.request import urlopen as uopen, Request

REGEX = compile(b'#([\d,]+) in Books ')
AMZN = 'http://amazon.com/dp/'
ISBNs={
    '0132269937': 'Core Python Programming',
    '0132356139': 'Python Web Development with Django',
    '0137143419': 'Python Fundamentals',
}
def getRanking(isbn):
    # 僞裝成瀏覽器訪問，直接訪問的話會拒絕
    myUrl = '{0}{1}'.format(AMZN, isbn)
    user_agent = 'Mozilla/4.0 (compatible; MSIE 5.5; Windows NT)'
    headers = {'User-Agent': user_agent}
    # 構造請求
    req = Request(myUrl, headers=headers)
    with uopen(req) as page:
        return str(REGEX.findall(page.read())[0], 'utf-8')

def _main():
    print('At', ctime(), 'on Amazon...')
    with ThreadPoolExecutor(3) as executor:
        for isbn, ranking in zip(
                ISBNs, executor.map(getRanking, ISBNs)):
            print('- %r ranked %s' % (ISBNs[isbn], ranking))
    print('all DONE at:', ctime())

if __name__ == '__main__':
    _main()

輸出結果

At Sun May 26 22:36:40 2019 on Amazon…
- ‘Core Python Programming’ ranked 794,988
- ‘Python Web Development with Django’ ranked 451,395
- ‘Python Fundamentals’ ranked 5,301,299
all DONE at: Sun May 26 22:36:57 2019

使用信號量的實例

#!/usr/bin/env python

from atexit import register
from random import randrange
from threading import BoundedSemaphore, Lock, Thread
from time import sleep, ctime

lock = Lock()
MAX = 5
candytray = BoundedSemaphore(MAX)

def refill():
    lock.acquire()
    print('Refilling candy...', end=' ')
    try:
        candytray.release()
    except ValueError:
        print('full, skipping')
    else:
        print('OK')
    lock.release()

def buy():
    lock.acquire()
    print('Buying candy...')
    if candytray.acquire(False):
        print('OK')
    else:
        print('empty, skipping')
    lock.release()

def producer(loops):
    for i in range(loops):
        refill()
        sleep(randrange(3))

def consumer(loops):
    for i in range(loops):
        buy()
        sleep(randrange(3))

def _main():
    print('starting at:', ctime())
    nloops = randrange(2, 6)
    print('THE CANDY MACHINE (full with %d bars)!' % MAX)
    Thread(target=consumer, args=(randrange(
        nloops, nloops+MAX+2),)).start() # buyer
    Thread(target=producer, args=(nloops,)).start() #vndr

@register
def _atexit():
    print('all DONE at:', ctime())

if __name__ == '__main__':
    _main()

輸出結果

starting at: Sun May 26 22:39:31 2019
THE CANDY MACHINE (full with 5 bars)!
Buying candy…
OK
Buying candy…
OK
Refilling candy… OK
Refilling candy… OK
Buying candy…
OK
Refilling candy… OK
Refilling candy… full, skipping
Refilling candy… full, skipping
Buying candy…
OK
Buying candy…
OK
Buying candy…
OK
Buying candy…
OK
Buying candy…
OK
Buying candy…
empty, skipping
Buying candy…
empty, skipping
Buying candy…
empty, skipping
all DONE at: Sun May 26 22:39:45 2019

隊列+Thread: 生產者與消費者問題

#!/usr/bin/env python

from random import randint
from time import sleep
from queue import Queue
from myThread import MyThread

def writeQ(queue):
    print('producing object for Q…')
    queue.put('xxx', 1)
    print("size now", queue.qsize())

def readQ(queue):
    val = queue.get(1)
    print('consumed object from Q… size now', queue.qsize())

def writer(queue, loops):
    for i in range(loops):
        writeQ(queue)
        sleep(randint(1, 3))

def reader(queue, loops):
    for i in range(loops):
        readQ(queue)
        sleep(randint(2, 5))

funcs = [writer, reader]
nfuncs = list(range(len(funcs)))

def main():
    nloops = randint(2, 5)
    q = Queue(32)

    threads =[]
    for i in nfuncs:
        t = MyThread(funcs[i], (q, nloops),
             funcs[i].__name__)
        threads.append(t)

    for i in nfuncs:
        threads[i].start()

    for i in nfuncs:
        threads[i].join()

if __name__ == '__main__':
    main()

輸出結果

staring writer at: Sun May 26 22:41:42 2019
producing object for Q…
size now 1
staring reader at: Sun May 26 22:41:42 2019
consumed object from Q… size now 0
producing object for Q…
size now 1
producing object for Q…
size now 2
consumed object from Q… size now 1
writer finished at: Sun May 26 22:41:45 2019
consumed object from Q… size now 0
reader finished at: Sun May 26 22:41:52 2019

主要參考資料：

《Python核心編程》第三版
https://blog.csdn.net/zhou8201/article/details/72848452

Python核心編程——第4章多線程編程筆記