Patch功能說明
StarlingX系統具有升級的能力，這個特性叫做“patching”，提供從2個版本之間升級的能力，主要用戶bug修復、安全補丁和特性增強等等。
Patching支持兩種補丁, In-Service補丁和Reboot-required補丁。In-Service補丁不需要主機節點重啓，只需要服務進程重啓即可。Reboot-required補丁需要重啓主機以實現補丁生效。在升級Reboot-required補丁時，需要先對主機進行lock操作，等待補丁applied，再unlock使其生效。
這篇介紹文檔，主要面向開發人員使用補丁功能，而不是產品用戶指南。它更着重介紹補丁修復功能，而不是包含補丁的各個方面。
簡要的說，補丁修復包含2個階段，創建補丁和應用補丁。下面進行詳細介紹這兩部分。
創建補丁
一個StarlingX補丁包括一個或多個系統升級所需要的rpm包。在開始創建補丁之前需要驗證rpm包已經在已部署的StarlingX上安裝了。以下步驟可以幫助我們確認。
-
確定已部署系統的軟件版本，有兩種方式
• 在horizon界面上 Admin -> Platform
-> System Configuration -> Systems
• 使用命令行 system show
controller-0:~$ . /etc/platform/openrc
[sysadmin@controller-0 ~(keystone_admin)]$ system show
+----------------------+--------------------------------------+
| Property | Value |
+----------------------+--------------------------------------+
| contact | None |
| created_at | 2019-10-14T03:10:50.862114+00:00 |
| description | None |
| https_enabled | False |
| location | None |
| name | 608dfe48-9a05-4b21-afc1-ea122574caa7 |
| region_name | RegionOne |
| sdn_enabled | False |
| security_feature | spectre_meltdown_v1 |
| service_project_name | services |
| software_version | 19.09 |
| system_mode | duplex |
| system_type | All-in-one |
| timezone | UTC |
| updated_at | 2019-10-14T03:12:41.983029+00:00 |
| uuid | 2639ad15-08a7-4f1b-a372-f927a5e4ab31 |
| vswitch_type | none |
+----------------------+--------------------------------------+ - 檢查最新構建，找到針對這個版本需要升級的rpm包，選擇需要的rpm生成補丁
一旦確定需要升級/安裝的rpm包，下一步就是準備補丁構建環境。作爲StarlingX開發人員，最簡單的辦法是使用StarlingX Building 容器，我們只需要對容器進行小小的修改就可以了。StarlingX Building 容器可以使用構建教程生成。
現在假設StarlingX的源碼已經下載好了，需要升級安裝的rpm包也準備好了，現在我們開始構造補丁構建環境。再次聲明，這個教程主要針對開發人員，而不是產品。
-
安裝2個cgcs-patch依賴包，crypto和pycrypto
sudo pip install crypto pycrypto - 使用腳本$MY_REPO/stx/stx-update/extras/scripts/patch_build.sh創建補丁。
在這個腳本中，它從release-info.inc 文件中獲取PLATFORM_RELEASE參數，並把PYTHONPATH指向repo中的cgcs-patch包，避免了安裝cgcs-patch和手動指定PLATFORM_RELEASE參數。可以使用下面命令查看構建腳本的使用說明。
$ $MY_REPO/stx/stx-update/cgcs-patch/bin/patch_build --help
Usage: patch_build [ <args> ] ... <rpm list>
Options:
--id <id> Patch ID
--release <version> Platform release version
--status <status> Patch Status Code (ie. O, R, V)
--unremovable Marks patch as unremovable
--reboot-required <Y|N> Marks patch as reboot-required (default=Y)
--summary <summary> Patch Summary
--desc <description> Patch Description
--warn <warnings> Patch Warnings
--inst <instructions> Patch Install Instructions
--req <patch_id> Required Patch
--controller <rpm> New package for controller
--worker <rpm> New package for worker node
--worker-lowlatency <rpm> New package for worker-lowlatency node
--storage <rpm> New package for storage node
--controller-worker <rpm> New package for combined node
--controller-worker-lowlatency <rpm> New package for lowlatency
combined node
--all-nodes <rpm> New package for all node types使用這個腳本可以指定patch id、reboot required、depended patches、rpm list等等，如果系統上沒有的，需要新安裝的包需要指定節點，比如 --controller 指定是在控制節點上新裝包。腳本執行完後，可以得到名字爲“<patch-id>.patch”的文件。
下面深入研究下這個補丁文件。
-
首先，這個補丁文件是個gzip壓縮包。我們可以通過file命令去檢查下。
$ file 001.patch
001.patch: gzip compressed data, was "001.patch", last modified:
Fri Aug 16 05:56:59 2019, max compression -
解壓出來後，可以看到以下文件
$ tar -xf 001.patch
$ tree
├── 001.patch
├── metadata.tar
├── signature
├── signature.v2
└── software.tar - 解壓 software.tar，可以發現它包含了所有需要安裝的rpm包。注意：所有的rpm包在補丁構建時用下面的key進行簽名。
$MY_REPO/build-tools/signing/ima_signing_key.priv
-
在metadata.tar中只有一個文件metadata.xml，包含補丁構建的所有信息。StarlingX集羣系統會讀取這個文件信息。
-
signature文件包含software.tar和metadata.tar的MD5的組合。
- signature.v2是爲software.tar和metadata.tar的簽名文件，在當前環境中，它由$MY_REPO/build-tools/signing/dev-private-key.pemkey文件生成。
安裝補丁
補丁生成後，可以手動安裝補丁到指定的StarlingX系統，同時支持界面和命令行安裝操作。補丁的生命週期包括四個狀態： Available,Partial-Apply, Applied 和 Partial-Remove.
• Available：表示補丁已經上傳到補丁存儲庫裏了，但是還沒有到軟件升級的倉庫，同樣也沒有在任何主機上安裝。
• Partial-Apply：表示補丁升級程序已經被觸發（sw-patch apply），已經在部分主機上安裝，但是還沒有在所有需要安裝的主機上安裝。
• Applied：表示已經在所有需要安裝的主機上安裝完成。
• Partial-Remove：表示補丁正在被移除，通過命令觸發(sw-patch remove),正在移除，但是還沒完全移除。
如果需要用命令行安裝補丁，需要把補丁拷貝到active的控制節點上。StarlingX集羣提供客戶端命令sw-patch。補丁操作都是通過這個命令完成，這個命令提供了很多功能，包括upload, apply, query,host-install, delete, remove等等。
controller-0:~$ sw-patch --help
usage: sw-patch [--debug]
<subcommand> ...
Subcomands:
upload: Upload one or more patches to the patching system.
upload-dir: Upload patches from one or more directories to the
patching system.
apply: Apply one or more patches. This adds the specified
patches to the repository, making the update(s)
available to the hosts in the system. Use --all to
apply all available patches.
Patches are specified as a space-separated list of
patch IDs.
remove: Remove one or more patches. This removes the specified
patches from the repository.
Patches are specified as a space-separated list of
patch IDs.
delete: Delete one or more patches from the patching system.
Patches are specified as a space-separated list of
patch IDs.
query: Query system patches. Optionally, specify 'query
applied' to query only those patches that are applied,
or 'query available' to query those that are not.
show: Show details for specified patches.
what-requires: List patches that require the specified patches.
query-hosts: Query patch states for hosts in the system.
host-install: Trigger patch install/remove on specified host. To
force install on unlocked node, use the --force option.
host-install-async: Trigger patch install/remove on specified host. To
force install on unlocked node, use the --force option.
Note: This command returns immediately upon dispatching
installation request.
install-local: Trigger patch install/remove on the local host. This
command can only be used for patch installation prior
to initial configuration.
drop-host: Drop specified host from table.
query-dependencies: List dependencies for specified patch. Use
--recursive for recursive query.
is-applied: Query Applied state for list of patches. Returns True
if all are Applied, False otherwise.
report-app-dependencies: Report application patch dependencies,
specifying application name with --app option, plus a
list of patches. Reported dependencies can be dropped
by specifying app with no patch list.
query-app-dependencies: Display set of reported application patch
dependencies.
commit: Commit patches to free disk space. WARNING: This
action is irreversible!
--os-region-name: Send the request to a specified region下面演示如何使用這個命令去安裝補丁。演示用的補丁是需要安裝在所有主機上的In-Service補丁，需要升級的StarlingX環境是 2+2+2的標準環境。
- 上傳補丁文件
controller-0:~$ sudo sw-patch upload 001.patch
001 is now available
檢查補丁狀態
controller-0:~$ sudo sw-patch query
Patch ID RR Release Patch State
======== == ======= ===========
001 N 19.09 Available
檢查所有主機的的升級狀態
controller-0:/$ sudo sw-patch query-hosts
Hostname IP Address Patch Current Reboot Required Release State
============ ============== ============= =============== ====== =====
compute-0 192.178.204.7 Yes No 19.09 idle
compute-1 192.178.204.9 Yes No 19.09 idle
controller-0 192.178.204.3 Yes No 19.09 idle
controller-1 192.178.204.4 Yes No 19.09 idle
storage-0 192.178.204.12 Yes No 19.09 idle
storage-1 192.178.204.11 Yes No 19.09 idle
Patch Current 表示當前主機是否有補丁安裝，Yes表示沒有安裝補丁，No表示至少有一個補丁在安裝
-
當補丁狀態available後，可以觸發補丁安裝
controller-0:/$ sudo sw-patch apply 001
001 is now in the repo
檢查補丁狀態
controller-0:~$ sudo sw-patch query
Patch ID RR Release Patch State
======== == ======= =============
001 N 19.09 Partial-Apply
檢查節點狀態
controller-0:~$ sudo sw-patch query-hosts
Hostname IP Address Patch Current Reboot Required Release State
============ ============== ============= =============== ====== =====
compute-0 192.178.204.7 No No 19.09 idle
compute-1 192.178.204.9 No No 19.09 idle
controller-0 192.178.204.3 No No 19.09 idle
controller-1 192.178.204.4 No No 19.09 idle
storage-0 192.178.204.12 No No 19.09 idle
storage-1 192.178.204.11 No No 19.09 idle -
在每個節點上安裝補丁，由於是in-service 補丁，所以不需要執行lock操作。
controller-0:~$ sudo sw-patch host-install controller-0
...
Installation was successful.
檢查主機升級狀態
controller-0:~$ sudo sw-patch query-hosts
Hostname IP Address Patch Current Reboot Required Release State
============ ============== ============= =============== ====== =====
compute-0 192.178.204.7 No No 19.09 idle
compute-1 192.178.204.9 No No 19.09 idle
controller-0 192.178.204.3 Yes No 19.09 idle
controller-1 192.178.204.4 No No 19.09 idle
storage-0 192.178.204.12 No No 19.09 idle
storage-1 192.178.204.11 No No 19.09 idle在所有節點上安裝補丁，需要爲每個節點執行命令
controller-0:~$ sudo sw-patch host-install controller-1
....
Installation was successful.
controller-0:~$ sudo sw-patch host-install compute-0
....
Installation was successful.
controller-0:~$ sudo sw-patch host-install compute-1
....
Installation was successful.
controller-0:~$ sudo sw-patch host-install storage-0
...
Installation was successful.
controller-0:~$ sudo sw-patch host-install storage-1
...
Installation was successful. - 所有節點按照完畢後，可以看到下面狀態
controller-0:~$ sudo sw-patch query
Patch ID RR Release Patch State
======== == ======= ===========
001 N 19.09 Applied
controller-0:~$ sudo sw-patch query-hosts
Hostname IP Address Patch Current Reboot Required Release State
============ ============== ============ =============== ======= =====
compute-0 192.178.204.7 Yes No 19.09 idle
compute-1 192.178.204.9 Yes No 19.09 idle
controller-0 192.178.204.3 Yes No 19.09 idle
controller-1 192.178.204.4 Yes No 19.09 idle
storage-0 192.178.204.12 Yes No 19.09 idle
storage-1 192.178.204.11 Yes No 19.09 idle
此時補丁升級程序完成
除了補丁升級，StarlingX還支持補丁回退和刪除，通過下面兩個命令實現sw-patch remove和sw-patch host-install，和補丁安裝有點類似。
補丁編排
在上面的例子中，演示了在集羣中補丁升級的功能。但是在大規模集羣中，整個升級過程會持續很長的時間。特別是reboot-required補丁，這個方案會很糟糕，效率很低而且給管理員帶來很多工作。因此StarlingX提供了另一個高級特性“補丁編排”。它支持集羣通過一些簡單的操作達到升級的目的，極大減少管理員的工作負擔和較少出錯。這個功能有三種方式使用，客戶端CLI、界面Horizon和VIM Restful API。
-
客戶端CLI。StarlingX提供客戶端工具sw-manager， 可以用於補丁編排。如下所示，可以通過創建和應用補丁策略來升級整個集羣
controller-0:~$ sw-manager patch-strategy -h
usage: sw-manager patch-strategy [-h] ...
optional arguments:
-h, --help show this help message and exit
Software Patch Commands:
create Create a strategy
delete Delete a strategy
apply Apply a strategy
abort Abort a strategy
show Show a strategy
controller-0:~$ sw-manager patch-strategy create -h
usage: sw-manager patch-strategy create [-h]
[--controller-apply-type {serial,ignore}]
[--storage-apply-type {serial,parallel,ignore}]
[--worker-apply-type {serial,parallel,ignore}]
[--max-parallel-worker-hosts {2,3,4,5,6,7,8,9,10,
11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,
28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,
45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,
62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,
79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,
96,97,98,99,100}]
[--instance-action {migrate,stop-start}]
[--alarm-restrictions {strict,relaxed}]
optional arguments:
-h, --help show this help message and exit
--controller-apply-type {serial,ignore}
defaults to serial
--storage-apply-type {serial,parallel,ignore}
defaults to serial
--worker-apply-type {serial,parallel,ignore}
defaults to serial
--max-parallel-worker-hosts {2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,
17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,
37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,
57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,
77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,
97,98,99,100}
maximum worker hosts to patch in parallel
--instance-action {migrate,stop-start}
defaults to stop-start
--alarm-restrictions {strict,relaxed}
defaults to strict -
界面Horizon。打開Admin -> Platform
-> Software Management -> Patch Orchestration 標籤 - VIM API。<http://<oam_ip>:4545>
+--------+---------------------------------------+----------------------------+
| Method | URI | Description |
+========+=======================================+============================+
| Post | /api/orchestration/sw-update/strategy | Create a patch strategy |
+--------+---------------------------------------+----------------------------+
| Delete | /api/orchestration/sw-update/strategy | Delete current patch |
| | | strategy |
+--------+---------------------------------------+----------------------------+
| Get | /api/orchestration/sw-update/strategy | Get detailed information of|
| | | current patch strategy |
+--------+---------------------------------------+----------------------------+
| Post | /api/orchestration/sw-update/strategy/| Apply or abort a patch |
| | actions | strategy |
+--------+---------------------------------------+----------------------------+
在補丁安裝時，補丁編排要求集羣處於一個良好的狀態。
• 所有主機必須處於unlocked-enabled-available狀態
• 系統沒有告警
• 足夠的空間用於VM遷移
當前開發狀態
• 所有的源碼都在StarlingX倉庫裏開源，包括“update”和“nfv”
• in-service補丁和reboot-required補丁的生成和安裝已經經過驗證
• 補丁編排還沒經過驗證