感謝朋友支持本博客,歡迎共同探討交流,由於能力和時間有限,錯誤之處在所難免,歡迎指正!
如果轉載,請保留作者信息。
博客地址:http://blog.csdn.net/gaoxingnengjisuan
郵箱地址:[email protected]
PS:最近沒有登錄博客,很多朋友的留言沒有看見,這裏道歉!還有就是本人較少上QQ,可以郵件交流。
接續上篇:
轉到3.2,來看方法audit_container的實現:
def audit_container(self, account, name, recurse=False):
"""
指定container的審計驗證,並實現遞歸驗證container下每個object;
"""
if (account, name) in self.in_progress:
self.in_progress[(account, name)].wait()
if (account, name) in self.list_cache:
return self.list_cache[(account, name)]
self.in_progress[(account, name)] = Event()
print 'Auditing container "%s"' % name
# # 指定指定account下的容器具體路徑;
path = '/%s/%s' % (account, name)
# 獲取指定account下的容器列表;
account_listing = self.audit_account(account)
consistent = True
if name not in account_listing:
consistent = False
print " Container %s not in account listing!" % path
# 獲取指定name容器的所有副本的相關節點和分區號;
# 獲取account/container/object所對應的分區號和節點(可能是多個,因爲分區副本有多個,可能位於不同的節點上);
# 返回元組(分區,節點信息列表);
# 在節點信息列表中至少包含id、weight、zone、ip、port、device、meta;
part, nodes = self.container_ring.get_nodes(account, name.encode('utf-8'))
rec_d = {}
responses = {}
for node in nodes:
marker = ''
results = True
while results:
try:
conn = http_connect(node['ip'], node['port'],
node['device'], part, 'GET',
path.encode('utf-8'), {},
'format=json&marker=%s' %
quote(marker.encode('utf-8')))
# 獲取來自服務器的響應;
resp = conn.getresponse()
if resp.status // 100 != 2:
self.container_not_found += 1
consistent = False
print(' Bad status GETting container "%s" on %s/%s' %
(path, node['ip'], node['device']))
break
if node['id'] not in responses:
responses[node['id']] = dict(resp.getheaders())
results = simplejson.loads(resp.read())
except Exception:
self.container_exceptions += 1
consistent = False
print ' Exception GETting container "%s" on %s/%s' % \
(path, node['ip'], node['device'])
break
if results:
marker = results[-1]['name']
for obj in results:
obj_name = obj['name']
if obj_name not in rec_d:
rec_d[obj_name] = obj
if (obj['last_modified'] !=
rec_d[obj_name]['last_modified']):
self.container_obj_mismatch += 1
consistent = False
print(" Different versions of %s/%s "
"in container dbs." % (name, obj['name']))
if (obj['last_modified'] >
rec_d[obj_name]['last_modified']):
rec_d[obj_name] = obj
obj_counts = [int(header['x-container-object-count'])
for header in responses.values()]
if not obj_counts:
consistent = False
print " Failed to fetch container %s at all!" % path
else:
if len(set(obj_counts)) != 1:
self.container_count_mismatch += 1
consistent = False
print " Container databases don't agree on number of objects."
print " Max: %s, Min: %s" % (max(obj_counts), min(obj_counts))
self.containers_checked += 1
self.list_cache[(account, name)] = rec_d
self.in_progress[(account, name)].send(True)
del self.in_progress[(account, name)]
# 遞歸驗證container下每個object;
if recurse:
for obj in rec_d.keys():
self.pool.spawn_n(self.audit_object, account, name, obj)
if not consistent and self.error_file:
print >>open(self.error_file, 'a'), path
return rec_d
3.2.1 獲取指定account下的容器具體路徑;3.2.2 調用方法audit_account實現獲取指定account下的容器列表,驗證當前指定容器是否包含其中;
3.2.3 獲取指定name容器的所有副本的相關節點和分區號;
3.2.4 針對容器的所有副本相關節點,進行遍歷,對於每個節點執行以下操作:
通過HTTP應用GET方法遠程獲取節點的驗證響應信息,通過響應信息的狀態值,判斷遠程節點副本容器是否存在;
3.2.5 遞歸調用方法audit_object實現審計驗證容器下每個object;
轉到3.3,來看方法audit_account的實現:
def audit_account(self, account, recurse=False):
"""
指定account的審計驗證,並實現遞歸驗證account下每個container,並且進一步實現遞歸驗證container下每個object;
"""
if account in self.in_progress:
self.in_progress[account].wait()
if account in self.list_cache:
return self.list_cache[account]
self.in_progress[account] = Event()
print 'Auditing account "%s"' % account
consistent = True
path = '/%s' % account
# 獲取指定name賬戶的所有副本的相關節點和分區號;
# 獲取account所對應的分區號和節點(可能是多個,因爲分區副本有多個,可能位於不同的節點上);
# 返回元組(分區,節點信息列表);
# 在節點信息列表中至少包含id、weight、zone、ip、port、device、meta;
part, nodes = self.account_ring.get_nodes(account)
responses = {}
for node in nodes:
marker = ''
results = True
while results:
node_id = node['id']
try:
# 建立一個HTTPConnection類的對象;
# 並獲取來自服務器的響應信息;
conn = http_connect(node['ip'], node['port'],
node['device'], part, 'GET', path, {},
'format=json&marker=%s' %
quote(marker.encode('utf-8')))
resp = conn.getresponse()
if resp.status // 100 != 2:
self.account_not_found += 1
consistent = False
print(" Bad status GETting account '%s' "
" from %ss:%ss" %
(account, node['ip'], node['device']))
break
results = simplejson.loads(resp.read())
except Exception:
self.account_exceptions += 1
consistent = False
print(" Exception GETting account '%s' on %ss:%ss" %
(account, node['ip'], node['device']))
break
if node_id not in responses:
responses[node_id] = [dict(resp.getheaders()), []]
responses[node_id][1].extend(results)
if results:
marker = results[-1]['name']
headers = [resp[0] for resp in responses.values()]
cont_counts = [int(header['x-account-container-count'])
for header in headers]
if len(set(cont_counts)) != 1:
self.account_container_mismatch += 1
consistent = False
print(" Account databases for '%s' don't agree on"
" number of containers." % account)
if cont_counts:
print " Max: %s, Min: %s" % (max(cont_counts),
min(cont_counts))
obj_counts = [int(header['x-account-object-count'])
for header in headers]
if len(set(obj_counts)) != 1:
self.account_object_mismatch += 1
consistent = False
print(" Account databases for '%s' don't agree on"
" number of objects." % account)
if obj_counts:
print " Max: %s, Min: %s" % (max(obj_counts),
min(obj_counts))
containers = set()
for resp in responses.values():
containers.update(container['name'] for container in resp[1])
self.list_cache[account] = containers
self.in_progress[account].send(True)
del self.in_progress[account]
self.accounts_checked += 1
if recurse:
for container in containers:
self.pool.spawn_n(self.audit_container, account,
container, True)
if not consistent and self.error_file:
print >>open(self.error_file, 'a'), path
return containers
3.3.1 獲取指定賬戶的具體路徑;3.3.2 獲取指定name賬戶的所有副本的相關節點和分區號;
3.3.4 針對賬戶的所有副本相關節點,進行遍歷,對於每個節點執行以下操作:
通過HTTP應用GET方法遠程獲取節點的驗證響應信息,通過響應信息的狀態值,判斷遠程副本賬戶是否存在;
3.3.5 遞歸調用方法audit_container實現審計驗證賬戶下每個container;
至此,這個腳本的賬戶/容器/腳本的審計驗證流程分析完成,當然在具體審計驗證過程中還有很多其他細節,這裏不再贅述。