Openstack中的服務是分佈式部署，因此，各個服務的啓停狀態，決定了此係統的可用性。我們可以
通過openstack提供的接口來查看服務的運行狀態，比如：
nova service的相關配置
Nova相關的配置項都nova/conf這個目錄下，如果我們相查看service相關的配置項，可以看nova/conf/service.py。其中，注意一下這兩個配置項：
service_down_time
report_interval
具體作用可以查看註釋。大概意思就是說，超過service_down_time的時間沒有收到的心跳 的話，就認爲服務是掛 了的，report_interval是定義了心跳的週期，即多長時間
上報一次自己的心跳。
nova服務的啓動
nova-compute的服務的入口在nova/cmd/compute.py:main()
其中最重要的一行代碼就是高亮的部分。這裏首先會初始一個service.Service的類：
def main(): config.parse_args(sys.argv) logging.setup(CONF, 'nova') priv_context.init(root_helper=shlex.split(utils.get_root_helper())) utils.monkey_patch() objects.register_all() # Ensure os-vif objects are registered and plugins loaded os_vif.initialize() gmr.TextGuruMeditation.setup_autorun(version) cmd_common.block_db_access('nova-compute') objects_base.NovaObject.indirection_api = conductor_rpcapi.ConductorAPI() server = service.Service.create(binary='nova-compute', topic=CONF.compute_topic) service.serve(server) service.wait()
我們詳細看一下service.Service()這個類：
class Service(service.Service): """Service object for binaries running on hosts. A service takes a manager and enables rpc by listening to queues based on topic. It also periodically runs tasks on the manager and reports its state to the database services table. """ def __init__(self, host, binary, topic, manager, report_interval=None, periodic_enable=None, periodic_fuzzy_delay=None, periodic_interval_max=None, *args, **kwargs): super(Service, self).__init__() self.host = host self.binary = binary self.topic = topic self.manager_class_name = manager self.servicegroup_api = servicegroup.API() manager_class = importutils.import_class(self.manager_class_name) self.manager = manager_class(host=self.host, *args, **kwargs) self.rpcserver = None self.report_interval = report_interval self.periodic_enable = periodic_enable self.periodic_fuzzy_delay = periodic_fuzzy_delay self.periodic_interval_max = periodic_interval_max self.saved_args, self.saved_kwargs = args, kwargs self.backdoor_port = None if objects_base.NovaObject.indirection_api: conductor_api = conductor.API() conductor_api.wait_until_ready(context.get_admin_context()) setup_profiler(binary, self.host)這裏重點看一下高亮的部分，看到一個report_interval這個變量。還有一個servicegroup.API()。
接下來我們看一下serivce.Service的start()方法：
# Add service to the ServiceGroup membership group. self.servicegroup_api.join(self.host, self.topic, self)會看上面一行代碼，這裏調用了servicegroup_api中的join方法，具體路徑在nova/servicegroup/api.py:join()
def join(self, member, group, service=None): """Add a new member to a service group. :param member: the joined member ID/name :param group: the group ID/name, of the joined member :param service: a `nova.service.Service` object """ return self._driver.join(member, group, service)
從servicegroup.API()的初始化函數我們看到
class API(object): def __init__(self, *args, **kwargs): '''Create an instance of the servicegroup API. args and kwargs are passed down to the servicegroup driver when it gets created. ''' # Make sure report interval is less than service down time report_interval = CONF.report_interval if CONF.service_down_time <= report_interval: new_service_down_time = int(report_interval * 2.5) LOG.warning(_LW("Report interval must be less than service down " "time. Current config: <service_down_time: " "%(service_down_time)s, report_interval: " "%(report_interval)s>. Setting service_down_time " "to: %(new_service_down_time)s"), {'service_down_time': CONF.service_down_time, 'report_interval': report_interval, 'new_service_down_time': new_service_down_time}) CONF.set_override('service_down_time', new_service_down_time) driver_class = _driver_name_class_mapping[CONF.servicegroup_driver] self._driver = importutils.import_object(driver_class, *args, **kwargs)回到nova/conf/servicegrouup.py中我們可以看到，CONF.servicegroup_driver目前只支持兩個類型：
SERVICEGROUP_OPTS = [ cfg.StrOpt('servicegroup_driver', default='db', choices=['db', 'mc'], help=""" This option specifies the driver to be used for the servicegroup service. ServiceGroup API in nova enables checking status of a compute node. When a compute worker running the nova-compute daemon starts, it calls the join API to join the compute group. Services like nova scheduler can query the ServiceGroup API to check if a node is alive. Internally, the ServiceGroup client driver automatically updates the compute worker status. There are multiple backend implementations for this service: Database ServiceGroup driver and Memcache ServiceGroup driver. Possible Values: * db : Database ServiceGroup driver * mc : Memcache ServiceGroup driver Related Options: * service_down_time (maximum time since last check-in for up service) """), ]我們假定用的是db這種方式，因此，我們要找的join方法應該在nova/securitygroup/drivers/db.py中：
def join(self, member, group, service=None): """Add a new member to a service group. :param member: the joined member ID/name :param group: the group ID/name, of the joined member :param service: a `nova.service.Service` object """ LOG.debug('DB_Driver: join new ServiceGroup member %(member)s to ' 'the %(group)s group, service = %(service)s', {'member': member, 'group': group, 'service': service}) if service is None: raise RuntimeError(_('service is a mandatory argument for DB based' ' ServiceGroup driver')) report_interval = service.report_interval if report_interval: service.tg.add_timer(report_interval, self._report_state, api.INITIAL_REPORTING_DELAY, service)在這裏我們很容易發現了他的回調函數：self._report_state()
def _report_state(self, service): """Update the state of this service in the datastore.""" try: service.service_ref.report_count += 1 service.service_ref.save() # TODO(termie): make this pattern be more elegant. if getattr(service, 'model_disconnected', False): service.model_disconnected = False LOG.info( _LI('Recovered from being unable to report status.'))
其中service是從數據庫中取得的最新service數據，該函數只是僅僅把report_count加一，然後調用save方法保存到數據庫中。這裏需要注意的是，save方法每次都會記錄更新的時間,在數據庫的字段爲updated_at。
由此，nova服務的心跳機制，本質就是每隔一段時間往數據庫更新report_count值，並記錄最後更新時間作爲接收到的最新心跳時間戳。
這個值可以從數據庫中查到，如下：
服務狀態監控機制
我們從nova-api的分析入手。其中與service相關的代碼位於： nova/api/openstack/compute/services.py。當我們調用nova service-list的命令時，實際是調用：
1. nova/api/openstack/compute/serivces.py: index()
@extensions.expected_errors(()) def index(self, req): """Return a list of all running services. Filter by host & service name """ if api_version_request.is_supported(req, min_version='2.11'): _services = self._get_services_list(req, ['forced_down']) else: _services = self._get_services_list(req) return {'services': _services}2. nova/api/openstack/compute/serivces.py: _get_services_list():
def _get_services_list(self, req, additional_fields=()): _services = self._get_services(req) return [self._get_service_detail(svc, additional_fields) for svc in _services]3. nova/api/openstack/compute/serivces.py: _get_services_detail():
def _get_service_detail(self, svc, additional_fields): alive = self.servicegroup_api.service_is_up(svc) state = (alive and "up") or "down" active = 'enabled' if svc['disabled']: active = 'disabled' service_detail = {'binary': svc['binary'], 'host': svc['host'], 'id': svc['id'], 'zone': svc['availability_zone'], 'status': active, 'state': state, 'updated_at': svc['updated_at'], 'disabled_reason': svc['disabled_reason']} for field in additional_fields: service_detail[field] = svc[field] return service_detail
nova/servicegroup/api.py
def service_is_up(self, member): """Check if the given member is up.""" # NOTE(johngarbutt) no logging in this method, # so this doesn't slow down the scheduler if member.get('forced_down'): return False return self._driver.is_up(member)nova/serivcegroup/drivers/db.py
def is_up(self, service_ref): """Moved from nova.utils Check whether a service is up based on last heartbeat. """ last_heartbeat = (service_ref.get('last_seen_up') or service_ref['created_at']) if isinstance(last_heartbeat, six.string_types): # NOTE(russellb) If this service_ref came in over rpc via # conductor, then the timestamp will be a string and needs to be # converted back to a datetime. last_heartbeat = timeutils.parse_strtime(last_heartbeat) else: # Objects have proper UTC timezones, but the timeutils comparison # below does not (and will fail) last_heartbeat = last_heartbeat.replace(tzinfo=None) # Timestamps in DB are UTC. elapsed = timeutils.delta_seconds(last_heartbeat, timeutils.utcnow()) is_up = abs(elapsed) <= self.service_down_time if not is_up: LOG.debug('Seems service %(binary)s on host %(host)s is down. ' 'Last heartbeat was %(lhb)s. Elapsed time is %(el)s', {'binary': service_ref.get('binary'), 'host': service_ref.get('host'), 'lhb': str(last_heartbeat), 'el': str(elapsed)}) return is_up
看到高亮的部分，我們就很容易明白，nova是如何判斷一個服務是up還是down的了：
首先獲取service實例的最後更新時間戳，即最後心跳時間，然後計算最後心跳時間距離現在時間的間隔，如果小於等於service_down_time的值，則認爲服務是up的，否則是down。
比如假設我們設置的report_interval時間爲10秒，正常的話檢查最後心跳到當前時間一定小於10秒，不幸的是可能中間丟了2個心跳，那檢查的最後心跳距離當前時間可能爲20多秒，由於小於我們的service_down_time（假設爲60秒)，因此還是認爲服務是up的。如果連續丟掉超過6個心跳包，則服務就會返回down了。
問題分析
當Openstack不正常工作時，首先查看下服務狀態，比如執行nova service-list命令查看Nova相關的服務狀態。
如果服務狀態爲down，根據Openstack服務的心跳機制和狀態監控原理，可能有以下幾種故障情形：
1. 數據庫訪問錯誤導致心跳更新失敗，這種情況看日誌就能發現錯誤日誌。
2. Rabbitmq連接失敗，nova-compute不能直接訪問數據庫，更新時是通過RPC調用nova-conductor完成的，如果rabbitmq連接失敗，RPC將無法執行，導致心跳發送失敗。
3. nova-conductor故障，原因同上，不過這種情況概率很低，除非人爲關閉了該服務。
4. 時間不同步。這種情況排查非常困難，因爲你在日誌中是發現不了任何錯誤信息的，我們知道數據庫操作由nova-conductor組件完成的，而計算心跳間隔是在nova-api服務完成的，假如這兩個服務所在的主機時間不同步，將可能導致服務誤判爲down。對於多API節點部署時尤其容易出現這種情況，所有節點務必保證時間同步，NTP服務必須能夠正常工作，否則將影響Openstack服務的心跳機制和狀態監控。
nova的這種上報方式其實這種方法效率是非常低的，並且當服務衆多時，數據庫的壓力將會非常大，因此有人提出引入Zookeeper服務發現機制維護Openstack服務狀態，參考Services Heartbeat with ZooKeeper。
