http://bbs.chinaunix.net/thread-4111379-1-1.html
import multiprocessing
class someClass(object):
def __init__(self):
pass
def f(self, x):
return x*x
def go(self):
pool = multiprocessing.Pool(processes=4)
#result = pool.apply_async(self.f, [10])
#print result.get(timeout=1)
print pool.map(self.f, range(10))
注意:如果出現
PicklingError: Can't pickle <type 'instancemethod'>: attribute lookup __builtin__.instancemethod failed
數據序列化:
https://zhidao.baidu.com/question/525917268.html
https://www.liaoxuefeng.com/wiki/001374738125095c955c1e6d8bb493182103fac9270762a000/00138683221577998e407bb309542d9b6a68d9276bc3dbe000
應該是說一個數據結構,比如二叉樹之類,序列化以後會變成一個char數組或者一個string字符串這樣,方便你存到文件裏面或者通過網絡傳輸。然後要恢復的時候就是“反序列化”,把文件裏讀出來/從網絡收到的char數組或者string恢復成一棵二叉樹或者其他什麼東西。主要就是方便保存
可以被序列化的類型有:
https://zhidao.baidu.com/question/619353578954760372.html
* None,True 和 False;
* 整數,浮點數,複數;
* 字符串,字節流,字節數組;
* 包含可pickle對象的tuples,lists,sets和dictionaries;
* 定義在module頂層的函數:
* 定義在module頂層的內置函數;
* 定義在module頂層的類;
* 擁有dict()或setstate()的自定義類型;
https://stackoverflow.com/questions/8804830/python-multiprocessing-pickling-error
The problem is that the pool methods all use a queue.Queue to pass tasks to the worker processes. Everything that goes through the queue.Queue must be pickable, and foo.work is not picklable since it is not defined at the top level of the module.
It can be fixed by defining a function at the top level
大意是類中的方法不能被序列化,而進程中的參數或者函數必須被序列化,所以報錯
解決:
import multiprocessing
def func(x):
return x*x
class someClass(object):
def __init__(self,func):
self.f = func
def go(self):
pool = multiprocessing.Pool(processes=4)
#result = pool.apply_async(self.f, [10])
#print result.get(timeout=1)
print pool.map(self.f, range(10))
a=someClass(func)
a.go()
defining a function at the top level
將進程需要調用的函數變成定義在module頂層的函數
解決過程中又出現的問題:
1. con = multiprocessing.Process(target=self.connect, args=(k, metrics, request, return_dict, ))
同樣是PicklingError: Can’t pickle錯誤
原因是:
type(metrics)
<type 'dict'>
雖然metrics的類型是dict但是裏面的具體內容是
'vmware_vm_net_transmitted_average': <prometheus_client.core.GaugeMetricFamily object at 0x7161650>
也會報PicklingError: Can’t pickle
可能是裏裏面的<prometheus_client.core.GaugeMetricFamily object at 0x7161650>
不能被序列化