PostgreSQL性能優化綜合案例講解 - 1

Postgres2015全國用戶大會將於11月20至21日在北京麗亭華苑酒店召開。本次大會嘉賓陣容強大，國內頂級PostgreSQL數據庫專家將悉數到場，並特邀歐洲、俄羅斯、日本、美國等國家和地區的數據庫方面專家助陣:

Postgres-XC項目的發起人鈴木市一(SUZUKI Koichi)
Postgres-XL的項目發起人Mason Sharp
pgpool的作者石井達夫(Tatsuo Ishii)
PG-Strom的作者海外浩平(Kaigai Kohei)
Greenplum研發總監姚延棟
周正中(德哥), PostgreSQL中國用戶會創始人之一
汪洋，平安科技數據庫技術部經理
……


2015年度PG大象會報名地址：http://postgres2015.eventdove.com/
PostgreSQL中國社區： http://postgres.cn/
PostgreSQL專業1羣： 3336901（已滿）
PostgreSQL專業2羣： 100910388
PostgreSQL專業3羣： 150657323

【聲明】

歡迎轉載, 註明出處.

【正文】

【軟件環境】

CentOS 5 x64

PostgreSQL 9.1.3

plproxy 2.3

pgbouncer 1.4.2

【測試模型】

設計一個包含INSERT, UPDATE, SELECT語句的業務模型用於本優化案例.

業務邏輯 :

PostgreSQL性能優化綜合案例講解 - 1 - 德哥@Digoal - The Heart,The World.

【測試表】


create table user_info
(userid int,
engname text,
cnname text,
occupation text,
birthday date,
signname text,
email text,
qq numeric,
crt_time timestamp without time zone,
mod_time timestamp without time zone
);

create table user_session
(userid int,
logintime timestamp(0) without time zone,
login_count bigint default 0,
logouttime timestamp(0) without time zone,
online_interval interval default interval '0'
);

create table user_login_rec
(userid int,
login_time timestamp without time zone,
ip inet
);

create table user_logout_rec
(userid int,
logout_time timestamp without time zone,
ip inet
);

【初始化數據】


insert into user_info (userid,engname,cnname,occupation,birthday,signname,email,qq,crt_time,mod_time)
select generate_series(1,20000000),
'digoal.zhou',
'德哥',
'DBA',
'1970-01-01'
,E'公益是一輩子的事, I\'m Digoal.Zhou, Just do it!',
'[email protected]',
276732431,
clock_timestamp(),
NULL;

insert into user_session (userid) select generate_series(1,20000000);

set work_mem='2048MB';
set maintenance_work_mem='2048MB';
alter table user_info add constraint pk_user_info primary key (userid);
alter table user_session add constraint pk_user_session primary key (userid);

【業務函數】


-- 模擬用戶登錄的函數
create or replace function f_user_login 
(i_userid int,
OUT o_userid int,
OUT o_engname text,
OUT o_cnname text,
OUT o_occupation text,
OUT o_birthday date,
OUT o_signname text,
OUT o_email text,
OUT o_qq numeric
)
as $BODY$
declare
begin
select userid,engname,cnname,occupation,birthday,signname,email,qq
into o_userid,o_engname,o_cnname,o_occupation,o_birthday,o_signname,o_email,o_qq
from user_info where userid=i_userid;
insert into user_login_rec (userid,login_time,ip) values (i_userid,now(),inet_client_addr());
update user_session set logintime=now(),login_count=login_count+1 where userid=i_userid;
return;
end;
$BODY$
language plpgsql;
-- 模擬用戶退出的函數
create or replace function f_user_logout
(i_userid int,
OUT o_result int
)
as $BODY$
declare
begin
insert into user_logout_rec (userid,logout_time,ip) values (i_userid,now(),inet_client_addr());
update user_session set logouttime=now(),online_interval=online_interval+(now()-logintime) where userid=i_userid;
o_result := 0;
return;
exception 
when others then
o_result := 1;
return;
end;
$BODY$
language plpgsql;

【搭建測試模型】

1. 搭建環境, 安裝PostgreSQL9.1.3數據庫軟件並初始化數據庫(略).

2. 調整數據庫postgresql.conf參數. 打開日誌, SQL統計, 跟蹤, 以及性能參數, 便於優化過程中取證.

監聽IPv4的所有IP.

listen_addresses = '0.0.0.0'

最大允許1000個連接.

max_connections = 1000

爲超級用戶保留3個可用連接.

superuser_reserved_connections = 3

默認的unix socket文件放在/tmp, 修改爲$PGDATA, 以確保安全.

unix_socket_directory = '/pgdata/digoal/1921/data02/pg_root'

默認的訪問權限是0777, 修改爲0700更安全.

unix_socket_permissions = 0700

Linux下面默認是2小時.tcp的keepalives包發送間隔以及重試次數, 如果客戶端沒有響應, 將主動釋放對應的SOCKET.

tcp_keepalives_idle = 60

tcp_keepalives_interval = 10

tcp_keepalives_count = 6

大的shared_buffers需要大的checkpoint_segments,同時需要申請更多的System V共享內存資源.

這個值不需要設的太大, 因爲PostgreSQL還依賴操作系統的cache來提高讀性能, 另外, 寫操作頻繁的數據庫這個設太大反而會增加checkpoint壓力.

shared_buffers = 512MB

這個值越大, VACUUM, CREATE INDEX的操作越快, 當然大到一定程度瓶頸就不在內存了, 可能是CPU例如創建索引.

這個值是一個操作的內存使用上限, 而不是一次性分配出去的. 並且需要注意如果開啓了autovacuum, 最大可能有autovacuum_max_workers*maintenance_work_mem的內存被系統消耗掉.

maintenance_work_mem = 512MB

一般設置爲比系統限制的略少,ulimit -a : stack size (kbytes, -s) 10240

max_stack_depth = 8MB

手動執行vacuum操作時, 默認是沒有停頓執行到底的, 爲了防止VACUUM操作消耗太多數據庫服務器硬件資源, 這個值是指vacuum在消耗多少資源後停頓多少時間,以便其他的操作可以使用更多的硬件資源.

vacuum_cost_delay = 10ms

#vacuum_cost_page_hit = 1 # 0-10000 credits

#vacuum_cost_page_miss = 10 # 0-10000 credits

#vacuum_cost_page_dirty = 20 # 0-10000 credits

#vacuum_cost_limit = 200 # 1-10000 credits

默認bgwriter進程執行一次後會停頓200ms再被喚醒執行下一次操作, 當數據庫的寫操作很頻繁的時候, 200ms可能太長, 導致其他進程需要花費過多的時間來進行bgwriter的操作.

bgwriter_delay = 10ms

如果需要做數據庫WAL日誌備份的話至少需要設置成archive級別, 如果需要做hot_standby那麼需要設置成hot_standby, 由於這個值修改需要重啓數據庫, 所以先設置成hot_standby比較好. 當然hot_standby意味着WAL記錄得更詳細, 如果沒有打算做hot_standby設置得越低性能越好.

wal_level = hot_standby

wal buffers默認是-1 根據shared_buffers的設置自動調整shared_buffers*3% .最大限制是XLOG的segment_size.

wal_buffers = 16384kB

多少個xlog file產生後開始checkpoint操作, 這個值越大, 允許shared_buffer中的被頻繁訪問的髒數據存儲得更久. 一定程度上可以提高數據庫性能. 但是太大的話會導致在數據庫發生checkpoint的時候需要處理更多的髒數據帶來長時間的IO開銷. 太小的話會導致產生更多的WAL文件(因爲full page writes=on,CHECKPOINT後的第一次塊的改變要寫全塊, checkpoint越頻繁, 越多的數據更新要寫全塊導致產生更多WAL).

checkpoint_segments = 64

這個和checkpoint_segments的效果是一樣的, 只是觸發的條件是時間條件.

checkpoint_timeout = 5min

歸檔參數的修改也需要重啓數據庫, 所以就先打開吧.

archive_mode = on

這個是歸檔調用的命令, 我這裏用date代替, 所以歸檔的時候調用的是輸出時間而不是拷貝wal文件.

archive_command = '/bin/date'

如果要做hot standby這個必須大於0, 並且修改之後要重啓數據庫所以先設置爲32.

max_wal_senders = 32

這是個standby 數據庫參數, 爲了方便角色切換, 我一般是所有的數據庫都把他設置爲on 的.

hot_standby = on

這個參數是說數據庫中隨機的PAGE訪問的開銷佔seq_page_cost的多少倍 , seq_page_cost默認是1. 其他的開銷都是seq_page_cost的倍數. 這些都用於基於成本的執行計劃選擇.

random_page_cost = 2.0

和上一個參數一樣, 用於基於成本的執行計劃選擇. 不是說會用多少cache, 它只是個度量值. 表示系統有多少內存可以作爲操作系統的cache. 越大的話, 數據庫越傾向使用index這種適合random訪問的執行計劃.

effective_cache_size = 12000MB

下面是日誌輸出的配置.

log_destination = 'csvlog'

logging_collector = on

log_directory = '/var/applog/pg_log/digoal/1921'

log_truncate_on_rotation = on

log_rotation_age = 1d

log_rotation_size = 10MB

這個參數調整的是記錄執行時間超過1秒的SQL到日誌中, 一般用於跟蹤哪些SQL執行時間長.

log_min_duration_statement = 1000ms

記錄每一次checkpoint到日誌中.

log_checkpoints = on

記錄鎖等待超過1秒的操作, 一般用於排查業務邏輯上的問題.

log_lock_waits = on

deadlock_timeout = 1s

記錄DDL語句, 一般用於跟蹤數據庫中的危險操作.

log_statement = 'ddl'

這個原本是1024表示跟蹤的SQL在1024的地方截斷, 超過1024將無法顯示全SQL. 修改爲2048會消耗更多的內存(基本可以忽略), 不過可以顯示更長的SQL.

track_activity_query_size = 2048

默認autovacuum就是打開的, log_autovacuum_min_duration = 0記錄所有的autovacuum操作.

autovacuum = on

log_autovacuum_min_duration = 0

這個模塊用於記錄數據庫中的最近的1000條SQL以及這些SQL的統計信息, 如執行了多少次, 總共耗時是多少. 一般用於發現業務上最頻繁調用的SQL是什麼, 有針對性的進行SQL優化.

shared_preload_libraries = 'pg_stat_statements'

custom_variable_classes = 'pg_stat_statements'

pg_stat_statements.max = 1000

pg_stat_statements.track = all

其他參數值默認.

這些參數的詳細解釋如有疑問請參考PostgreSQL官方文檔.

3. 新建數據庫用戶digoal, 庫digoal. 並使用前面的測試模型新建表以及函數, 初始化數據.

下面的測試過程中只測登陸部分, 未測試退出部分, 因爲登陸過程已經包含了INSERT, UPDATE, SELECT. 基本上可以反映整個調優過程了.

【調優階段1】

使用pgbench進行壓力測試, 發現瓶頸併合理優化.

1. pgbench用到的登陸腳本

cat login.sql


\setrandom userid 1 20000000
select userid,engname,cnname,occupation,birthday,signname,email,qq from user_info where userid=:userid;
insert into user_login_rec (userid,login_time,ip) values (:userid,now(),inet_client_addr());
update user_session set logintime=now(),login_count=login_count+1 where userid=:userid;

2. pgbench用到的退出腳本

cat logout.sql


\setrandom userid 1 20000000
insert into user_logout_rec (userid,logout_time,ip) values (:userid,now(),inet_client_addr());
update user_session set logouttime=now(),online_interval=online_interval+(now()-logintime) where userid=:userid;

3. 壓力測試

pgbench -M simple -r -c 8 -f /home/postgres/test/login.sql -j 8 -n -T 180 -h 172.16.3.33 -p 1921 -U digoal digoal

4. 壓力測試結果


transaction type: Custom query
scaling factor: 1
query mode: simple
number of clients: 8
number of threads: 8
duration: 180 s
number of transactions actually processed: 62675
tps = 348.084647 (including connections establishing)
tps = 348.100337 (excluding connections establishing)
statement latencies in milliseconds:
        0.004577        \setrandom userid 1 20000000
        12.963789       select userid,engname,cnname,occupation,birthday,signname,email,qq from user_info where userid=:userid;
        5.540750        insert into user_login_rec (userid,login_time,ip) values (:userid,now(),inet_client_addr());
        4.457834        update user_session set logintime=now(),login_count=login_count+1 where userid=:userid;

5. 瓶頸分析與優化

壓力測試中查看數據庫服務器的iostat -x

avg-cpu: %user %nice %system %iowait %steal %idle

0.69 0.00 0.25 24.11 0.00 74.95

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util

cciss/c0d0 0.00 6.00 0.00 1.50 0.00 60.00 40.00 0.01 6.67 6.67 1.00

cciss/c0d0p1 0.00 6.00 0.00 1.50 0.00 60.00 40.00 0.01 6.67 6.67 1.00

cciss/c0d0p2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

cciss/c0d0p3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

cciss/c0d1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

cciss/c0d2 0.00 638.50 10.00 217.50 160.00 6444.00 29.03 152.58 707.89 4.40 100.10

cciss/c0d3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

cciss/c0d4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

cciss/c0d5 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

dm-1 0.00 0.00 10.00 866.50 160.00 6932.00 8.09 446.26 510.49 1.14 100.10

dm-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

dm-3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

dm-4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

操作系統的平均IO請求等待700多毫秒, PostgreSQL數據文件所處的塊設備使用率100%. 存在嚴重的IO性能瓶頸.

使用pgfincore降低讀的物理IO請求.

pgfincore的相關文章可參考如下,

《use posix_fadvise pre-cache frequency data》

http://blog.163.com/digoal@126/blog/static/163877040201062944945126/

《a powerful upgrade from pgfincore 1.0》

http://blog.163.com/digoal@126/blog/static/1638770402011630102117658/

《TOAST table with pgfincore》

http://blog.163.com/digoal@126/blog/static/16387704020120524144140/

pgfincore所起的作用類似EnterpriseDB的InfiniteCache或者熟悉Oracle的朋友可能更易於接受的KEEP BUFFER POOL.


-- 載入os cache
digoal=> select reltoastrelid from pg_class where relname='user_info';
 reltoastrelid 
---------------
         16424
(1 row)

digoal=> select relname from pg_class where oid=16424;
    relname     
----------------
 pg_toast_16421
(1 row)

digoal=> \c digoal postgres
seYou are now connected to database "digoal" as user "postgres".
digoal=# select * from pgfadvise_willneed('pg_toast.pg_toast_16421');
                   relpath                    | os_page_size | rel_os_pages | os_pages_free 
----------------------------------------------+--------------+--------------+---------------
 pg_tblspc/16385/PG_9.1_201105231/16386/16424 |         4096 |            0 |        243865
(1 row)

digoal=# select * from pgfadvise_willneed('digoal.user_info');
                    relpath                     | os_page_size | rel_os_pages | os_pages_free 
------------------------------------------------+--------------+--------------+---------------
 pg_tblspc/16385/PG_9.1_201105231/16386/16421   |         4096 |       262144 |        243834
 pg_tblspc/16385/PG_9.1_201105231/16386/16421.1 |         4096 |       262144 |        243834
 pg_tblspc/16385/PG_9.1_201105231/16386/16421.2 |         4096 |       244944 |        243834
(3 rows)

digoal=# select * from pgfadvise_willneed('digoal.user_session');
                    relpath                     | os_page_size | rel_os_pages | os_pages_free 
------------------------------------------------+--------------+--------------+---------------
 pg_tblspc/16385/PG_9.1_201105231/16386/16431   |         4096 |       262144 |        243834
 pg_tblspc/16385/PG_9.1_201105231/16386/16431.1 |         4096 |        33640 |        243834
(2 rows)

digoal=# select reltoastrelid from pg_class where relname='user_session';
 reltoastrelid 
---------------
             0
(1 row)

digoal=# select * from pgfadvise_willneed('digoal.pk_user_session');
                   relpath                    | os_page_size | rel_os_pages | os_pages_free 
----------------------------------------------+--------------+--------------+---------------
 pg_tblspc/16385/PG_9.1_201105231/16386/16438 |         4096 |       109680 |        243865
(1 row)

digoal=# select * from pgfadvise_willneed('digoal.pk_user_info');
                   relpath                    | os_page_size | rel_os_pages | os_pages_free 
----------------------------------------------+--------------+--------------+---------------
 pg_tblspc/16385/PG_9.1_201105231/16386/16436 |         4096 |       109680 |        235567
(1 row)

【調優階段2】

1. 壓力測試

pgbench -M simple -r -c 8 -f /home/postgres/test/login.sql -j 8 -n -T 180 -h 172.16.3.33 -p 1921 -U digoal digoal

2. 測試結果


transaction type: Custom query
scaling factor: 1
query mode: simple
number of clients: 8
number of threads: 8
duration: 180 s
number of transactions actually processed: 264895
tps = 1471.517096 (including connections establishing)
tps = 1471.585818 (excluding connections establishing)
statement latencies in milliseconds:
        0.004226        \setrandom userid 1 20000000
        0.459824        select userid,engname,cnname,occupation,birthday,signname,email,qq from user_info where userid=:userid;
        2.457797        insert into user_login_rec (userid,login_time,ip) values (:userid,now(),inet_client_addr());
        2.501684        update user_session set logintime=now(),login_count=login_count+1 where userid=:userid;

3. 瓶頸分析與優化

SELECT語句的延時已經通過上一個優化階段下降到了0.45毫秒, INSERT和UPDATE語句的平均耗時也從原來的5.5和4.45下降到了2.5.原因是select的請求在內存中命中了, 因此update和insert不需要和select爭搶物理io請求, 處理效率自然有一定的提高.

但是INSERT和UPDATE的語句延時還有2.5毫秒存在很大的可優化空間.

開啓PostgreSQL的異步提交日誌.


synchronous_commit = off
wal_writer_delay = 10ms

與Oracle的異步日誌差別請參考 :

《PostgreSQL and Oracle's async commit》

http://blog.163.com/digoal@126/blog/static/16387704020121229223072/

【調優階段3】

1. 壓力測試

pgbench -M simple -r -c 8 -f /home/postgres/test/login.sql -j 8 -n -T 180 -h 172.16.3.33 -p 1921 -U digoal digoal

2. 測試結果


transaction type: Custom query
scaling factor: 1
query mode: simple
number of clients: 8
number of threads: 8
duration: 180 s
number of transactions actually processed: 685344
tps = 3751.377919 (including connections establishing)
tps = 3751.568948 (excluding connections establishing)
statement latencies in milliseconds:
        0.003474        \setrandom userid 1 20000000
        0.418716        select userid,engname,cnname,occupation,birthday,signname,email,qq from user_info where userid=:userid;
        0.511601        insert into user_login_rec (userid,login_time,ip) values (:userid,now(),inet_client_addr());
        1.188277        update user_session set logintime=now(),login_count=login_count+1 where userid=:userid;

3. 瓶頸分析與優化

客戶端連接使用simple協議, 存在一定的可優化空間.

修改協議爲extended, 查看性能提升多少.

【調優階段4】

1. 壓力測試

pgbench -M extended -r -c 8 -f /home/postgres/test/login.sql -j 8 -n -T 180 -h 172.16.3.33 -p 1921 -U digoal digoal

2. 測試結果


transaction type: Custom query
scaling factor: 1
query mode: extended
number of clients: 8
number of threads: 8
duration: 180 s
number of transactions actually processed: 970981
tps = 5394.015368 (including connections establishing)
tps = 5394.215477 (excluding connections establishing)
statement latencies in milliseconds:
        0.003345        \setrandom userid 1 20000000
        0.381675        select userid,engname,cnname,occupation,birthday,signname,email,qq from user_info where userid=:userid;
        0.296300        insert into user_login_rec (userid,login_time,ip) values (:userid,now(),inet_client_addr());
        0.792592        update user_session set logintime=now(),login_count=login_count+1 where userid=:userid;

3. 瓶頸分析與優化

客戶端連接使用extended協議, 存在一定的可優化空間.

修改協議爲prepared, 查看性能提升多少.

參見 :

《PostgreSQL prepared statement: SPI_prepare, prepare|execute COMMAND, PL/pgsql STYLE: custom & generic plan cache》

http://blog.163.com/digoal@126/blog/static/1638770402012112452432251/

【調優階段5】

1. 壓力測試

pgbench -M prepared -r -c 8 -f /home/postgres/test/login.sql -j 8 -n -T 180 -h 172.16.3.33 -p 1921 -U digoal digoal

2. 測試結果


transaction type: Custom query
scaling factor: 1
query mode: prepared
number of clients: 8
number of threads: 8
duration: 180 s
number of transactions actually processed: 1044186
tps = 5800.589330 (including connections establishing)
tps = 5800.902491 (excluding connections establishing)
statement latencies in milliseconds:
        0.003465        \setrandom userid 1 20000000
        0.319665        select userid,engname,cnname,occupation,birthday,signname,email,qq from user_info where userid=:userid;
        0.266931        insert into user_login_rec (userid,login_time,ip) values (:userid,now(),inet_client_addr());
        0.777822        update user_session set logintime=now(),login_count=login_count+1 where userid=:userid;

3. 瓶頸分析與優化

壓力測試的腳本中使用的是普通的SQL語句, 未使用初始化時用到的登陸函數和退出函數. 使用普通SQL顯然比使用函數多了交互的次數以及每次發送的數據包的大小.

使用函數看看性能能提升多少.

【調優階段6】

1. 登陸腳本

cat login.sql


\setrandom userid 1 20000000
SELECT f_user_login(:userid);

2. 退出腳本

cat logout.sql


\setrandom userid 1 20000000
SELECT f_user_logout(:userid);

3. 壓力測試

pgbench -M prepared -r -c 8 -f /home/postgres/test/login.sql -j 8 -n -T 180 -h 172.16.3.33 -p 1921 -U digoal digoal

4. 測試結果


transaction type: Custom query
scaling factor: 1
query mode: prepared
number of clients: 8
number of threads: 8
duration: 180 s
number of transactions actually processed: 1616746
tps = 8981.596290 (including connections establishing)
tps = 8981.995800 (excluding connections establishing)
statement latencies in milliseconds:
        0.004012        \setrandom userid 1 20000000
        0.881060        SELECT f_user_login(:userid);

5. 瓶頸分析與優化

到這個時候看起來好像沒什麼好優化的了, 其實不然.

我們知道在整個登陸過程中用到了SELECT, UPDATE, INSERT.

其中UPDATE的表有一個PK索引, 每次更新需要修改數據表的同時還需要更新索引. 所以理論上這個更新操作表越小性能越高.

我們可以通過拆表來提升性能. 如下 :

拆表 :


create table user_info_0 (like user_info including all);
create table user_info_1 (like user_info including all);
create table user_info_2 (like user_info including all);
create table user_info_3 (like user_info including all);
create table user_info_4 (like user_info including all);

create table user_session_0 (like user_session including all);
create table user_session_1 (like user_session including all);
create table user_session_2 (like user_session including all);
create table user_session_3 (like user_session including all);
create table user_session_4 (like user_session including all);

插入初始化數據 :


insert into user_info_0 (userid,engname,cnname,occupation,birthday,signname,email,qq,crt_time,mod_time)
select generate_series(1,4000000),
'digoal.zhou',
'德哥',
'DBA',
'1970-01-01'
,E'公益是一輩子的事, I\'m Digoal.Zhou, Just do it!',
'[email protected]',
276732431,
clock_timestamp(),
NULL;

insert into user_info_1 (userid,engname,cnname,occupation,birthday,signname,email,qq,crt_time,mod_time)
select generate_series(4000001,8000000),
'digoal.zhou',
'德哥',
'DBA',
'1970-01-01'
,E'公益是一輩子的事, I\'m Digoal.Zhou, Just do it!',
'[email protected]',
276732431,
clock_timestamp(),
NULL;

insert into user_info_2 (userid,engname,cnname,occupation,birthday,signname,email,qq,crt_time,mod_time)
select generate_series(8000001,12000000),
'digoal.zhou',
'德哥',
'DBA',
'1970-01-01'
,E'公益是一輩子的事, I\'m Digoal.Zhou, Just do it!',
'[email protected]',
276732431,
clock_timestamp(),
NULL;

insert into user_info_3 (userid,engname,cnname,occupation,birthday,signname,email,qq,crt_time,mod_time)
select generate_series(12000001,16000000),
'digoal.zhou',
'德哥',
'DBA',
'1970-01-01'
,E'公益是一輩子的事, I\'m Digoal.Zhou, Just do it!',
'[email protected]',
276732431,
clock_timestamp(),
NULL;

insert into user_info_4 (userid,engname,cnname,occupation,birthday,signname,email,qq,crt_time,mod_time)
select generate_series(16000001,20000000),
'digoal.zhou',
'德哥',
'DBA',
'1970-01-01'
,E'公益是一輩子的事, I\'m Digoal.Zhou, Just do it!',
'[email protected]',
276732431,
clock_timestamp(),
NULL;

insert into user_session_0 (userid) select generate_series(1,4000000);
insert into user_session_1 (userid) select generate_series(4000001,8000000);
insert into user_session_2 (userid) select generate_series(8000001,12000000);
insert into user_session_3 (userid) select generate_series(12000001,16000000);
insert into user_session_4 (userid) select generate_series(16000001,20000000);

set work_mem='2048MB';
set maintenance_work_mem='2048MB';
alter table user_info_0 add primary key (userid);
alter table user_info_1 add primary key (userid);
alter table user_info_2 add primary key (userid);
alter table user_info_3 add primary key (userid);
alter table user_info_4 add primary key (userid);
alter table user_session_0 add primary key (userid);
alter table user_session_1 add primary key (userid);
alter table user_session_2 add primary key (userid);
alter table user_session_3 add primary key (userid);
alter table user_session_4 add primary key (userid);

同樣通過pgfincore把他們加載到內存中, 這裏不詳細描述.

新建登陸和退出函數


create or replace function f_user_login_0
(i_userid int,
OUT o_userid int,
OUT o_engname text,
OUT o_cnname text,
OUT o_occupation text,
OUT o_birthday date,
OUT o_signname text,
OUT o_email text,
OUT o_qq numeric
)
as $BODY$
declare
begin
select userid,engname,cnname,occupation,birthday,signname,email,qq
into o_userid,o_engname,o_cnname,o_occupation,o_birthday,o_signname,o_email,o_qq
from user_info_0 where userid=i_userid;
insert into user_login_rec (userid,login_time,ip) values (i_userid,now(),inet_client_addr());
update user_session_0 set logintime=now(),login_count=login_count+1 where userid=i_userid;
return;
end;
$BODY$
language plpgsql;

create or replace function f_user_login_1
(i_userid int,
OUT o_userid int,
OUT o_engname text,
OUT o_cnname text,
OUT o_occupation text,
OUT o_birthday date,
OUT o_signname text,
OUT o_email text,
OUT o_qq numeric
)
as $BODY$
declare
begin
select userid,engname,cnname,occupation,birthday,signname,email,qq
into o_userid,o_engname,o_cnname,o_occupation,o_birthday,o_signname,o_email,o_qq
from user_info_1 where userid=i_userid;
insert into user_login_rec (userid,login_time,ip) values (i_userid,now(),inet_client_addr());
update user_session_1 set logintime=now(),login_count=login_count+1 where userid=i_userid;
return;
end;
$BODY$
language plpgsql;

create or replace function f_user_login_2
(i_userid int,
OUT o_userid int,
OUT o_engname text,
OUT o_cnname text,
OUT o_occupation text,
OUT o_birthday date,
OUT o_signname text,
OUT o_email text,
OUT o_qq numeric
)
as $BODY$
declare
begin
select userid,engname,cnname,occupation,birthday,signname,email,qq
into o_userid,o_engname,o_cnname,o_occupation,o_birthday,o_signname,o_email,o_qq
from user_info_2 where userid=i_userid;
insert into user_login_rec (userid,login_time,ip) values (i_userid,now(),inet_client_addr());
update user_session_2 set logintime=now(),login_count=login_count+1 where userid=i_userid;
return;
end;
$BODY$
language plpgsql;

create or replace function f_user_login_3
(i_userid int,
OUT o_userid int,
OUT o_engname text,
OUT o_cnname text,
OUT o_occupation text,
OUT o_birthday date,
OUT o_signname text,
OUT o_email text,
OUT o_qq numeric
)
as $BODY$
declare
begin
select userid,engname,cnname,occupation,birthday,signname,email,qq
into o_userid,o_engname,o_cnname,o_occupation,o_birthday,o_signname,o_email,o_qq
from user_info_3 where userid=i_userid;
insert into user_login_rec (userid,login_time,ip) values (i_userid,now(),inet_client_addr());
update user_session_3 set logintime=now(),login_count=login_count+1 where userid=i_userid;
return;
end;
$BODY$
language plpgsql;

create or replace function f_user_login_4
(i_userid int,
OUT o_userid int,
OUT o_engname text,
OUT o_cnname text,
OUT o_occupation text,
OUT o_birthday date,
OUT o_signname text,
OUT o_email text,
OUT o_qq numeric
)
as $BODY$
declare
begin
select userid,engname,cnname,occupation,birthday,signname,email,qq
into o_userid,o_engname,o_cnname,o_occupation,o_birthday,o_signname,o_email,o_qq
from user_info_4 where userid=i_userid;
insert into user_login_rec (userid,login_time,ip) values (i_userid,now(),inet_client_addr());
update user_session_4 set logintime=now(),login_count=login_count+1 where userid=i_userid;
return;
end;
$BODY$
language plpgsql;

create or replace function f_user_logout_0
(i_userid int,
OUT o_result int
)
as $BODY$
declare
begin
insert into user_logout_rec (userid,logout_time,ip) values (i_userid,now(),inet_client_addr());
update user_session_0 set logouttime=now(),online_interval=online_interval+(now()-logintime) where userid=i_userid;
o_result := 0;
return;
exception 
when others then
o_result := 1;
return;
end;
$BODY$
language plpgsql;

create or replace function f_user_logout_1
(i_userid int,
OUT o_result int
)
as $BODY$
declare
begin
insert into user_logout_rec (userid,logout_time,ip) values (i_userid,now(),inet_client_addr());
update user_session_1 set logouttime=now(),online_interval=online_interval+(now()-logintime) where userid=i_userid;
o_result := 0;
return;
exception 
when others then
o_result := 1;
return;
end;
$BODY$
language plpgsql;

create or replace function f_user_logout_2
(i_userid int,
OUT o_result int
)
as $BODY$
declare
begin
insert into user_logout_rec (userid,logout_time,ip) values (i_userid,now(),inet_client_addr());
update user_session_2 set logouttime=now(),online_interval=online_interval+(now()-logintime) where userid=i_userid;
o_result := 0;
return;
exception 
when others then
o_result := 1;
return;
end;
$BODY$
language plpgsql;

create or replace function f_user_logout_3
(i_userid int,
OUT o_result int
)
as $BODY$
declare
begin
insert into user_logout_rec (userid,logout_time,ip) values (i_userid,now(),inet_client_addr());
update user_session_3 set logouttime=now(),online_interval=online_interval+(now()-logintime) where userid=i_userid;
o_result := 0;
return;
exception 
when others then
o_result := 1;
return;
end;
$BODY$
language plpgsql;

create or replace function f_user_logout_4
(i_userid int,
OUT o_result int
)
as $BODY$
declare
begin
insert into user_logout_rec (userid,logout_time,ip) values (i_userid,now(),inet_client_addr());
update user_session_4 set logouttime=now(),online_interval=online_interval+(now()-logintime) where userid=i_userid;
o_result := 0;
return;
exception 
when others then
o_result := 1;
return;
end;
$BODY$
language plpgsql;

【調優階段7】

1. 登陸腳本

cat login*.sql


\setrandom userid 1 4000000
SELECT f_user_login_0(:userid);
\setrandom userid 4000001 8000000
SELECT f_user_login_1(:userid);
\setrandom userid 8000001 12000000
SELECT f_user_login_2(:userid);
\setrandom userid 12000001 16000000
SELECT f_user_login_3(:userid);
\setrandom userid 16000001 20000000
SELECT f_user_login_4(:userid);

2. 退出腳本

cat logout*.sql


\setrandom userid 1 4000000
SELECT f_user_logout_0(:userid);
\setrandom userid 4000001 8000000
SELECT f_user_logout_1(:userid);
\setrandom userid 8000001 12000000
SELECT f_user_logout_2(:userid);
\setrandom userid 12000001 16000000
SELECT f_user_logout_3(:userid);
\setrandom userid 16000001 20000000
SELECT f_user_logout_4(:userid);

3. 壓力測試

pgbench -M prepared -r -c 1 -f /home/postgres/test/login0.sql -j 1 -n -T 180 -h 172.16.3.33 -p 1921 -U digoal digoal >./log.login0 &

pgbench -M prepared -r -c 1 -f /home/postgres/test/login1.sql -j 1 -n -T 180 -h 172.16.3.33 -p 1921 -U digoal digoal >./log.login1 &

pgbench -M prepared -r -c 2 -f /home/postgres/test/login2.sql -j 2 -n -T 180 -h 172.16.3.33 -p 1921 -U digoal digoal >./log.login2 &

pgbench -M prepared -r -c 2 -f /home/postgres/test/login3.sql -j 2 -n -T 180 -h 172.16.3.33 -p 1921 -U digoal digoal >./log.login3 &

pgbench -M prepared -r -c 2 -f /home/postgres/test/login4.sql -j 2 -n -T 180 -h 172.16.3.33 -p 1921 -U digoal digoal >./log.login4 &

4. 測試結果

cat log.log*


transaction type: Custom query
scaling factor: 1
query mode: prepared
number of clients: 1
number of threads: 1
duration: 180 s
number of transactions actually processed: 233348
tps = 1281.818097 (including connections establishing)
tps = 1281.837109 (excluding connections establishing)
statement latencies in milliseconds:
        0.003492        \setrandom userid 1 4000000
        0.771932        SELECT f_user_login_0(:userid);
transaction type: Custom query
scaling factor: 1
query mode: prepared
number of clients: 1
number of threads: 1
duration: 180 s
number of transactions actually processed: 233466
tps = 1282.514774 (including connections establishing)
tps = 1282.573500 (excluding connections establishing)
statement latencies in milliseconds:
        0.003546        \setrandom userid 4000001 8000000
        0.771399        SELECT f_user_login_1(:userid);
transaction type: Custom query
scaling factor: 1
query mode: prepared
number of clients: 2
number of threads: 2
duration: 180 s
number of transactions actually processed: 475466
tps = 2612.200783 (including connections establishing)
tps = 2612.281526 (excluding connections establishing)
statement latencies in milliseconds:
        0.003605        \setrandom userid 8000001 12000000
        0.757312        SELECT f_user_login_2(:userid);
transaction type: Custom query
scaling factor: 1
query mode: prepared
number of clients: 2
number of threads: 2
duration: 180 s
number of transactions actually processed: 468904
tps = 2576.380443 (including connections establishing)
tps = 2576.488485 (excluding connections establishing)
statement latencies in milliseconds:
        0.003587        \setrandom userid 12000001 16000000
        0.767869        SELECT f_user_login_3(:userid);
transaction type: Custom query
scaling factor: 1
query mode: prepared
number of clients: 2
number of threads: 2
duration: 180 s
number of transactions actually processed: 439381
tps = 2414.347086 (including connections establishing)
tps = 2414.425600 (excluding connections establishing)
statement latencies in milliseconds:
        0.004431        \setrandom userid 16000001 20000000
        0.817879        SELECT f_user_login_4(:userid);
總計 : 
tps = 10167.261183 (including connections establishing)
tps = 10167.261183 (excluding connections establishing)

5. 瓶頸分析與優化

到這裏我們還沒有關注過表空間, 其實這些表拆分後它們還在同一個表空間裏面. 把它們放在不同的表空間可以擴展它們整體的IO吞吐能力.


postgres=# \db+
                                           List of tablespaces
    Name    |  Owner   |                  Location                   |  Access privileges  | Description 
------------+----------+---------------------------------------------+---------------------+-------------
 digoal     | postgres | /pgdata/digoal/1921/data02/pg_tbs/digoal    | postgres=C/postgres+| 
            |          |                                             | digoal=C/postgres   | 
 digoal_01  | postgres | /pgdata/digoal/1921/data03/pg_tbs/digoal_01 | postgres=C/postgres+| 
            |          |                                             | digoal=C/postgres   | 
 digoal_02  | postgres | /pgdata/digoal/1921/data04/pg_tbs/digoal_02 | postgres=C/postgres+| 
            |          |                                             | digoal=C/postgres   | 
 digoal_03  | postgres | /pgdata/digoal/1921/data05/pg_tbs/digoal_03 | postgres=C/postgres+| 
            |          |                                             | digoal=C/postgres   | 
 digoal_04  | postgres | /pgdata/digoal/1921/data06/pg_tbs/digoal_04 | postgres=C/postgres+| 
            |          |                                             | digoal=C/postgres   | 
digoal=> alter table user_info_0 set tablespace digoal_04;
ALTER TABLE
digoal=> alter table user_info_2 set tablespace digoal_01;
ALTER TABLE
digoal=> alter table user_info_3 set tablespace digoal_02;
ALTER TABLE
digoal=> alter table user_info_4 set tablespace digoal_03;

digoal=> alter index user_info_0_pkey set tablespace digoal_04;
ALTER INDEX
digoal=> alter index user_info_2_pkey set tablespace digoal_01;
ALTER INDEX
digoal=> alter index user_info_3_pkey set tablespace digoal_02;
ALTER INDEX
digoal=> alter index user_info_4_pkey set tablespace digoal_03;

digoal=> alter table user_session_0 set tablespace digoal_04;
ALTER TABLE
digoal=> alter table user_session_2 set tablespace digoal_01;
ALTER TABLE
digoal=> alter table user_session_3 set tablespace digoal_02;
ALTER TABLE
digoal=> alter table user_session_4 set tablespace digoal_03;

digoal=> alter index user_session_0_pkey set tablespace digoal_04;
ALTER INDEX
digoal=> alter index user_session_2_pkey set tablespace digoal_01;
ALTER INDEX
digoal=> alter index user_session_3_pkey set tablespace digoal_02;
ALTER INDEX
digoal=> alter index user_session_4_pkey set tablespace digoal_03;

重新把它們加載到內存.

下節 :

http://blog.163.com/digoal@126/blog/static/163877040201221333411196/

【附】

pgbench simple|extended|prepare 部分源碼 :

1. pgbench.c


        if (commands[st->state]->type == SQL_COMMAND)
        {
                const Command *command = commands[st->state];
                int                     r;

                if (querymode == QUERY_SIMPLE)
                {
                        char       *sql;

                        sql = xstrdup(command->argv[0]);
                        sql = assignVariables(st, sql);

                        if (debug)
                                fprintf(stderr, "client %d sending %s\n", st->id, sql);
                        r = PQsendQuery(st->con, sql);
                        free(sql);
                }
                else if (querymode == QUERY_EXTENDED)
                {
                        const char *sql = command->argv[0];
                        const char *params[MAX_ARGS];

                        getQueryParams(st, command, params);

                        if (debug)
                                fprintf(stderr, "client %d sending %s\n", st->id, sql);
                        r = PQsendQueryParams(st->con, sql, command->argc - 1,
                                                                  NULL, params, NULL, NULL, 0);
                }
                else if (querymode == QUERY_PREPARED)
                {
                        char            name[MAX_PREPARE_NAME];
                        const char *params[MAX_ARGS];

                        if (!st->prepared[st->use_file])
                        {
                                int                     j;
                                for (j = 0; commands[j] != NULL; j++)
                                {
                                        PGresult   *res;
                                        char            name[MAX_PREPARE_NAME];

                                        if (commands[j]->type != SQL_COMMAND)
                                                continue;
                                        preparedStatementName(name, st->use_file, j);
                                        res = PQprepare(st->con, name,
                                                  commands[j]->argv[0], commands[j]->argc - 1, NULL);
                                        if (PQresultStatus(res) != PGRES_COMMAND_OK)
                                                fprintf(stderr, "%s", PQerrorMessage(st->con));
                                        PQclear(res);
                                }
                                st->prepared[st->use_file] = true;
                        }

                        getQueryParams(st, command, params);
                        preparedStatementName(name, st->use_file, st->state);

                        if (debug)
                                fprintf(stderr, "client %d sending %s\n", st->id, name);
                        r = PQsendQueryPrepared(st->con, name, command->argc - 1,
                                                                        params, NULL, NULL, 0);
                }

2. src/interfaces/libpq/fe-exec.c

PostgreSQL性能優化綜合案例講解 - 1

performance tuning case: array search & date order by , data updated daily (use cursor solve it)

PostgreSQL 秒殺場景優化

A Smart PostgreSQL extension plproxy 2.2 practices

useful function & operator & custom operator for Row and Array Comparisons

Oracle index by table(Associative array) used in PostgreSQL

https://yachay.unat.edu.pe/blog/index.php?comment_area=format_blog&comment_component=blog&comment_co

linux以太網驅動總結