聯繫:手機/微信(+86 17813235971) QQ(107644445)
標題:PostgreSQL恢復系列:pg_filedump恢復字典構造
作者:惜分飛©版權所有[未經本人同意,不得以任何形式轉載,否則有進一步追究法律責任的權利.]
pg_filedump是在pg數據庫極端情況下直接解析數據庫文件的利器,但是由於是開源軟件,本身難以實現批量處理,通過對底層基表分析,可以實現批量處理功能
分析PostgreSQL庫中數據庫信息
--數據庫查詢結果 postgres= # select oid,datname,datdba,dattablespace from pg_database; oid | datname | datdba | dattablespace -------+-------------+--------+--------------- 14187 | postgres | 10 | 1663 16403 | db_xff | 10 | 1663 1 | template1 | 10 | 1663 14186 | template0 | 10 | 1663 16407 | db_xifenfei | 16405 | 16406 (5 rows) --通過dump 該文件解析數據 <Data> ----- Item 1 -- Length: 0 Offset: 5 (0x0005) Flags: REDIRECT Item 2 -- Length: 0 Offset: 6 (0x0006) Flags: REDIRECT Item 3 -- Length: 260 Offset: 7320 (0x1c98) Flags: NORMAL COPY: 14187 postgres Item 4 -- Length: 260 Offset: 7056 (0x1b90) Flags: NORMAL COPY: 16403 db_xff Item 5 -- Length: 297 Offset: 7888 (0x1ed0) Flags: NORMAL COPY: 1 template1 Item 6 -- Length: 297 Offset: 7584 (0x1da0) Flags: NORMAL COPY: 14186 template0 Item 7 -- Length: 260 Offset: 6792 (0x1a88) Flags: NORMAL COPY: 16407 db_xifenfei |
分析PostgreSQL 表空間信息
--sql查詢表空間信息 postgres= # select * from pg_tablespace; oid | spcname | spcowner | spcacl | spcoptions -------+--------------+----------+--------+------------ 1663 | pg_default | 10 | | 1664 | pg_global | 10 | | 16406 | tbs_xifenfei | 16405 | | (3 rows) --通過dump 該文件解析數據 <Data> ----- Item 1 -- Length: 96 Offset: 8096 (0x1fa0) Flags: NORMAL COPY: 1663 pg_default Item 2 -- Length: 96 Offset: 8000 (0x1f40) Flags: NORMAL COPY: 1664 pg_global Item 3 -- Length: 96 Offset: 7904 (0x1ee0) Flags: NORMAL COPY: 16406 tbs_xifenfei |
分析PostgreSQL 對象id、name、path對應關係
--對象信息查詢 postgres= # select oid ,relname,relnamespace,reltype,reloftype,relowner,relam,relfilenode, reltablespace from pg_class where relname like 't_t%' or relname like 't_x%' ; oid | relname | relnamespace | reltype | reloftype | relowner | relam | relfilenode | reltablespace -------+------------+--------------+---------+-----------+----------+-------+-------------+--------------- 16387 | t_xifenfei | 2200 | 16389 | 0 | 10 | 2 | 16390 | 0 16391 | t_xff | 2200 | 16393 | 0 | 10 | 2 | 16391 | 0 16394 | t_xff2 | 2200 | 16396 | 0 | 10 | 2 | 16394 | 0 16397 | t_xff3 | 2200 | 16399 | 0 | 10 | 2 | 16397 | 0 16400 | t_xff4 | 2200 | 16402 | 0 | 10 | 2 | 16400 | 0 16408 | t_tbs | 2200 | 16410 | 0 | 10 | 2 | 16408 | 16406 (6 rows) --通過dump 該文件解析數據(顯示部分) COPY: 16394 t_xff2 2200 16396 0 10 2 16394 0 0 0.000000000000 0 0 f f p r Item 29 -- Length: 0 Offset: 31 (0x001f) Flags: REDIRECT Item 30 -- Length: 172 Offset: 2592 (0x0a20) Flags: NORMAL COPY: 16397 t_xff3 2200 16399 0 10 2 16397 0 0 0.000000000000 0 0 f f p r Item 31 -- Length: 205 Offset: 3376 (0x0d30) Flags: NORMAL COPY: 12093 pg_shadow 11 12094 0 10 0 0 0 0 0.000000000000 0 0 f f p v Item 32 -- Length: 172 Offset: 2416 (0x0970) Flags: NORMAL COPY: 16400 t_xff4 2200 16402 0 10 2 16400 0 0 0.000000000000 0 0 f f p r |
這個裏面獲取到pg_class.reltablespace是表空間的id值,根據自定義表空間的規則:在pgdata/pg_tblspc創建link指向創建表空間時候的文件夾路徑
- bash -4.2$ pwd /var/lib/pgsql/12/data/pg_tblspc - bash -4.2$ ls -ltr total 0 lrwxrwxrwx 1 postgres postgres 30 Apr 15 20:13 16406 -> /var/lib/pgsql/12/data/tbs_xff |
結合上述的pg_database,pg_tablespace,pg_class信息,可以獲取到每個表對應實際的存儲路徑
分析PostgreSQL 模式信息
--sql查詢模式信息 postgres= # select * from pg_namespace; oid | nspname | nspowner | nspacl -------+--------------------+----------+------------------------------------- 99 | pg_toast | 10 | 12314 | pg_temp_1 | 10 | 12315 | pg_toast_temp_1 | 10 | 11 | pg_catalog | 10 | {postgres=UC /postgres ,=U /postgres } 2200 | public | 10 | {postgres=UC /postgres ,=UC /postgres } 13887 | information_schema | 10 | {postgres=UC /postgres ,=U /postgres } 16404 | u_xifenfei | 10 | (7 rows) --通過dump 該文件解析數據 <Data> ----- Item 1 -- Length: 0 Offset: 6 (0x0006) Flags: REDIRECT Item 2 -- Length: 96 Offset: 8096 (0x1fa0) Flags: NORMAL COPY: 99 pg_toast Item 3 -- Length: 0 Offset: 7 (0x0007) Flags: REDIRECT Item 4 -- Length: 96 Offset: 8000 (0x1f40) Flags: NORMAL COPY: 12314 pg_temp_1 Item 5 -- Length: 96 Offset: 7904 (0x1ee0) Flags: NORMAL COPY: 12315 pg_toast_temp_1 Item 6 -- Length: 141 Offset: 7760 (0x1e50) Flags: NORMAL COPY: 11 pg_catalog Item 7 -- Length: 141 Offset: 7616 (0x1dc0) Flags: NORMAL COPY: 2200 public Item 8 -- Length: 0 Offset: 9 (0x0009) Flags: REDIRECT Item 9 -- Length: 141 Offset: 7472 (0x1d30) Flags: NORMAL COPY: 13887 information_schema Item 10 -- Length: 96 Offset: 7376 (0x1cd0) Flags: NORMAL COPY: 16404 u_xifenfei |
通過pg_namespace,pg_class信息,可以獲取到對象所屬的模式關係,基於上述彙總,可以獲取到某個模式下面,所有表id和實際存儲路徑,現在使用pg_filedump進行恢復,還缺少表的列類型信息,通過pg_type和pg_attribute來獲取。
獲取PostgreSQL表的列名稱和類型[編號]信息
--sql查詢列信息 postgres= # \d t_tbs Table "public.t_tbs" Column | Type | Collation | Nullable | Default ----------+------+-----------+----------+--------- oid | oid | | | spcname | name | | | spcowner | oid | | | Tablespace: "tbs_xifenfei" postgres= # select attrelid,attname,atttypid,attstattarget,attlen,attnum from pg_attribute where attrelid=16408; attrelid | attname | atttypid | attstattarget | attlen | attnum ----------+----------+----------+---------------+--------+-------- 16408 | tableoid | 26 | 0 | 4 | -6 16408 | cmax | 29 | 0 | 4 | -5 16408 | xmax | 28 | 0 | 4 | -4 16408 | cmin | 29 | 0 | 4 | -3 16408 | xmin | 28 | 0 | 4 | -2 16408 | ctid | 27 | 0 | 6 | -1 16408 | oid | 26 | -1 | 4 | 1 16408 | spcname | 19 | -1 | 64 | 2 16408 | spcowner | 26 | -1 | 4 | 3 (9 rows) --dump 內容(截取部分) Item 11 -- Length: 144 Offset: 1424 (0x0590) Flags: NORMAL COPY: 16408 oid 26 -1 4 1 Item 12 -- Length: 144 Offset: 1280 (0x0500) Flags: NORMAL COPY: 16408 spcname 19 -1 64 2 Item 13 -- Length: 144 Offset: 1136 (0x0470) Flags: NORMAL COPY: 16408 spcowner 26 -1 4 3 Item 14 -- Length: 144 Offset: 992 (0x03e0) Flags: NORMAL COPY: 16408 ctid 27 0 6 -1 Item 15 -- Length: 144 Offset: 848 (0x0350) Flags: NORMAL COPY: 16408 xmin 28 0 4 -2 Item 16 -- Length: 144 Offset: 704 (0x02c0) Flags: NORMAL COPY: 16408 cmin 29 0 4 -3 Item 17 -- Length: 144 Offset: 560 (0x0230) Flags: NORMAL COPY: 16408 xmax 28 0 4 -4 Item 18 -- Length: 144 Offset: 416 (0x01a0) Flags: NORMAL COPY: 16408 cmax 29 0 4 -5 |
PostgreSQL獲取類型編號和實際類型名稱對應關係
--查詢類型編號和實際類型關係 postgres= # select oid,typname from pg_type; oid | typname -------+--------------------------------------- 16 | bool 17 | bytea 18 | char 19 | name 20 | int8 21 | int2 22 | int2vector 23 | int4 24 | regproc 25 | text 26 | oid 27 | tid 28 | xid 29 | cid …… --dump 內容(截取部分) Item 1 -- Length: 176 Offset: 8016 (0x1f50) Flags: NORMAL COPY: 16 bool Item 2 -- Length: 176 Offset: 7840 (0x1ea0) Flags: NORMAL COPY: 17 bytea Item 3 -- Length: 176 Offset: 7664 (0x1df0) Flags: NORMAL COPY: 18 char Item 4 -- Length: 176 Offset: 7488 (0x1d40) Flags: NORMAL COPY: 19 name Item 5 -- Length: 176 Offset: 7312 (0x1c90) Flags: NORMAL COPY: 20 int8 Item 6 -- Length: 176 Offset: 7136 (0x1be0) Flags: NORMAL COPY: 21 int2 Item 7 -- Length: 176 Offset: 6960 (0x1b30) Flags: NORMAL COPY: 22 int2vector Item 8 -- Length: 176 Offset: 6784 (0x1a80) Flags: NORMAL COPY: 23 int4 Item 9 -- Length: 176 Offset: 6608 (0x19d0) Flags: NORMAL COPY: 24 regproc Item 10 -- Length: 176 Offset: 6432 (0x1920) Flags: NORMAL COPY: 25 text Item 11 -- Length: 176 Offset: 6256 (0x1870) Flags: NORMAL COPY: 26 oid Item 12 -- Length: 176 Offset: 6080 (0x17c0) Flags: NORMAL COPY: 27 tid Item 13 -- Length: 176 Offset: 5904 (0x1710) Flags: NORMAL COPY: 28 xid Item 14 -- Length: 176 Offset: 5728 (0x1660) Flags: NORMAL COPY: 29 cid Item 15 -- Length: 176 Offset: 5552 (0x15b0) Flags: NORMAL COPY: 30 oidvector |
通過pg_class、pg_type和pg_attribute可以獲取對象的表的列名稱,數據類型等信息。通過以上幾個對象,即可獲取到pg_filedmp處理所需要的所有信息,然後可以實現批量處理