利用系統緩存提高PostgreSQL操作效率 原 薦

原創 PGSmith 2018-12-02 21:20

環境介紹：

                OS：Centos 6.4 64bit
                Database：PostgreSQL9.4
                Memory：2G
                CPU：1核

下載安裝：

    在pgfoundry下載pgfincore-v1.1.1.tar.gz，，將源碼解壓到數據庫源碼下的contrib下。不要在其github上下載，目前應該有一些bug，最新版本爲1.1.1，1.1.2在我試用的時候發現大部分函數無法使用。

    其次在我編譯時一直出現如下錯誤： 

[postgres@localhost pgfincore-1.1.1]$ make
/bin/sh: pg_config: command not found
make: pg_config: Command not found
cp pgfincore.sql pgfincore--1.1.1.sql
cp: cannot create regular file `pgfincore--1.1.1.sql': Permission denied
make: *** [pgfincore--1.1.1.sql] Error 1
    所以在此我修改了一下Makefile文件，內容如下： 
# contrib/pgfincore/Makefile

MODULE_big = pgfincore
OBJS = pgfincore.o

EXTENSION = pgfincore
DATA = pgfincore--1.1.1.sql pgfincore--unpackaged--1.1.1.sql

ifdef USE_PGXS
PG_CONFIG = pg_config
PGXS := $(shell $(PG_CONFIG) --pgxs)
include $(PGXS)
else
subdir = contrib/pgfincore
top_builddir = ../..
include $(top_builddir)/src/Makefile.global
include $(top_srcdir)/contrib/contrib-global.mk
    現在進行make，即可： 
[postgres@localhost pgfincore-1.1.1]$ make
gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -g -O2 -fpic -I. -I. -I../../src/include -D_GNU_SOURCE   -c -o pgfincore.o pgfincore.c
pgfincore.c: In function ‘pgsysconf’:
pgfincore.c:227: warning: implicit declaration of function ‘heap_form_tuple’
pgfincore.c:227: warning: assignment makes pointer from integer without a cast
pgfincore.c: In function ‘pgfadvise’:
pgfincore.c:477: warning: assignment makes pointer from integer without a cast
pgfincore.c: In function ‘pgfadvise_loader’:
pgfincore.c:710: warning: assignment makes pointer from integer without a cast
pgfincore.c: In function ‘pgfincore’:
pgfincore.c:1016: warning: assignment makes pointer from integer without a cast
gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -g -O2 -fpic -shared -o pgfincore.so pgfincore.o -L../../src/port -L../../src/common -Wl,--as-needed -Wl,-rpath,'/opt/hg3.0/lib',--enable-new-dtags  
[postgres@localhost pgfincore-1.1.1]$ make install
/bin/mkdir -p '/opt/hg3.0/lib/postgresql'
/bin/mkdir -p '/opt/hg3.0/share/postgresql/extension'
/bin/mkdir -p '/opt/hg3.0/share/postgresql/extension'
/usr/bin/install -c -m 755  pgfincore.so '/opt/hg3.0/lib/postgresql/pgfincore.so'
/usr/bin/install -c -m 644 pgfincore.control '/opt/hg3.0/share/postgresql/extension/'
/usr/bin/install -c -m 644 pgfincore--1.1.1.sql pgfincore--unpackaged--1.1.1.sql '/opt/hg3.0/share/postgresql/extension/'
[postgres@localhost pgfincore-1.1.1]$
實驗記錄：

    1、安裝： 

[postgres@localhost bin]$ ./psql 
psql (9.4.5)
Type "help" for help.

postgres=# create extension pgfincore ;
CREATE EXTENSION
postgres=# select * from pg_extension ;
  extname  | extowner | extnamespace | extrelocatable | extversion | extconfig |
 extcondition 
-----------+----------+--------------+----------------+------------+-----------+
--------------
 plpgsql   |       10 |           11 | f              | 1.0        |           |
 
 pgfincore |       10 |         2200 | t              | 1.1.1      |           |
 
(2 rows)

postgres=#

    2、準備工作：記錄比較使用pgfincore前後的性能差，使用插件pg_stat_statments記錄每條sql的使用時間。
其次將shared_buffer改爲16mb，減少其對pgfincore的影響。 

postgres=# create extension pg_stat_statements ;
CREATE EXTENSION
postgres=# select * from pg_extension ;
      extname       | extowner | extnamespace | extrelocatable | extversion | ex
tconfig | extcondition 
--------------------+----------+--------------+----------------+------------+---
--------+--------------
 plpgsql            |       10 |           11 | f              | 1.0        |   
        | 
 pgfincore          |       10 |         2200 | t              | 1.1.1      |   
        | 
 pg_stat_statements |       10 |         2200 | t              | 1.2        |   
        | 
(3 rows)

    3、建立表testmem，分別在使用和不適用pgfincore情況下進行select, update, delete的操作，並對比執行時間，在這裏爲了更好的去作比較，建立三張表testmem1，testmem2，testmem3，分別進行三種操作進行對比。並插入相同的數據。 

[postgres@localhost bin]$ ./psql 
psql (9.4.5)
Type "help" for help.

postgres=# \d
No relations found.
postgres=# create table testmem1(t1 int, t2 text, t3 text, t4 bigint, t5 text, t6 text, t7 varchar, t8 char(22), t9 numeric);CREATE TABLE
postgres=# create table testmem2(t1 int, t2 text, t3 text, t4 bigint, t5 text, t6 text, t7 varchar, t8 char(22), t9 numeric);
CREATE TABLE
postgres=# create table testmem3(t1 int, t2 text, t3 text, t4 bigint, t5 text, t6 text, t7 varchar, t8 char(22), t9 numeric);
CREATE TABLE
postgres=# insert into testmem1 select generate_series(1,999999), 'wangshuo'||generate_series(1,999999)::text||random()::text, generate_series(1,999999)::text||'sure',generate_series(1,999999), random()::text,random()::text,random()::text,'liuyuanyuan', generate_series(1,999999);
INSERT 0 999999
postgres=# select pg_relation_size('testmem1')/1024/1024||'MB';
 ?column? 
----------
 166MB
(1 row)

postgres=# insert into testmem2 select generate_series(1,999999), 'wangshuo'||generate_series(1,999999)::text||random()::text, generate_series(1,999999)::text||'sure',generate_series(1,999999), random()::text,random()::text,random()::text,'liuyuanyuan', generate_series(1,999999);
INSERT 0 999999
postgres=# insert into testmem3 select generate_series(1,999999), 'wangshuo'||generate_series(1,999999)::text||random()::text, generate_series(1,999999)::text||'sure',generate_series(1,999999), random()::text,random()::text,random()::text,'liuyuanyuan', generate_series(1,999999);
INSERT 0 999999
postgres=# select pg_relation_size('testmem2')/1024/1024||'MB'; ?column? 
----------
 166MB
(1 row)

postgres=# select pg_relation_size('testmem3')/1024/1024||'MB';
 ?column? 
----------
 166MB
(1 row)

postgres=#

    三種操作性能對比： 

postgres=# select * from testmem1 where t1 < 789432 and t1 > 1208;
   t1   |                 t2                 |     t3     |   t4   |          t5
          |          t6          |          t7          |           t8          
 |   t9   
--------+------------------------------------+------------+--------+------------
----------+----------------------+----------------------+-----------------------
-+--------
   1209 | wangshuo12090.964191045146435      | 1209sure   |   1209 | 0.075409890
152514    | 0.468206159770489    | 0.692714618053287    | liuyuanyuan           
 |   1209
   1210 | wangshuo12100.652063825167716      | 1210sure   |   1210 | 0.026932121
720165    | 0.802233531605452    | 0.706556385848671    | liuyuanyuan           
 |   1210
   1211 | wangshuo12110.724962076637894      | 1211sure   |   1211 | 0.732285636
477172    | 0.816857317462564    | 0.868489125277847    | liuyuanyuan           
 |   1211
   1212 | wangshuo12120.0478062951005995     | 1212sure   |   1212 | 0.190716865
006834    | 0.898683389648795    | 0.537546805106103    | liuyuanyuan           
 |   1212
   1213 | wangshuo12130.582098880317062      | 1213sure   |   1213 | 0.161297460
086644    | 0.395338968373835    | 0.684920639265329    | liuyuanyuan           
 |   1213
   1214 | wangshuo12140.53120110463351       | 1214sure   |   1214 | 0.253457680
810243    | 0.428548218682408    | 0.671272001229227    | liuyuanyuan           

postgres=# update testmem2 set t7 = random()::text where t1 < 789432 and t1 > 1208;
UPDATE 788223
postgres=# delete from testmem3 where t1 < 789432 and t1 > 1208;
DELETE 788223

    使用pgfincore前：通過查詢表pg_stat_statments查看三種操作時間： 

postgres=# SELECT query, calls, total_time, rows, 100.0 * shared_blks_hit /                
               nullif(shared_blks_hit + shared_blks_read, 0) AS hit_percent
          FROM pg_stat_statements ORDER BY total_time DESC;
-[ RECORD 1 ]------------------------------------------------------
query       | update testmem2 set t7 = random()::text where t1 < ? and t1 > ?;
calls       | 1
total_time  | 14303.53
rows        | 788223
hit_percent | 97.7162513440240383
-[ RECORD 2 ]------------------------------------------------------
query       | delete from testmem3 where t1 < ? and t1 > ?;
calls       | 1
total_time  | 5256.305
rows        | 788223
hit_percent | 97.4004941321803582
-[ RECORD 6 ]------------------------------------------------------
query       | select * from testmem1 where t1 < ? and t1 > ?;
calls       | 1
total_time  | 2397.866
rows        | 788223
hit_percent | 0.15039714245429336843

    使用pgfincore後：通過查詢表pg_stat_statments查看三種操作時間（重啓機器後，重新建庫，重新建表、插數據）： 

SELECT query, calls, total_time, rows, 100.0 * shared_blks_hit /
               nullif(shared_blks_hit + shared_blks_read, 0) AS hit_percent
          FROM pg_stat_statements ORDER BY total_time DESC;
-[ RECORD 1 ]------------------------------------------------------
query       | update testmem2 set t7 = random()::text where t1 < ? and t1 > ?;
calls       | 1
total_time  | 6800.237
rows        | 788223
hit_percent | 97.6908892215405358
-[ RECORD 2 ]------------------------------------------------------
query       | delete from testmem3 where t1 < ? and t1 > ?;
calls       | 1
total_time  | 3295.312
rows        | 788223
hit_percent | 97.3715873996294009
-[ RECORD 3 ]------------------------------------------------------
query       | select * from testmem1 where t1 < ? and t1 > ?;
calls       | 1
total_time  | 891.002
rows        | 788223
hit_percent | 0.000000000000000000000000

    明顯看出當執行select時速度提升明顯，其他的時間提升並不明顯，這是因爲數據還需要進行IO操作，所以在這裏速度提升應該僅僅是讀入的時間節省掉了。 

postgres=# select pgfadvise_willneed('testmem1');
-[ RECORD 1 ]------+-------------------------------------
pgfadvise_willneed | (base/13003/16384,4096,42554,279311)

postgres=# select * from pgfincore('testmem1');
-[ RECORD 1 ]-+-----------------
relpath       | base/13003/16384
segment       | 0
os_page_size  | 4096
rel_os_pages  | 42554
pages_mem     | 42554
group_mem     | 1
os_pages_free | 279317
databit       | 

postgres=# select count(*) from testmem1 where t1 < 99999 and t1 > 12;
-[ RECORD 1 ]
count | 99986

postgres=# insert into testmem1 select generate_series(1,9999), 'wangshuo'||generate_series(1,9999)::text||random()::text, generate_series(1,9999)::text||'sure',generate_series(1,9999), random()::text,random()::text,random()::text,'liuyuanyuan', generate_series(1,9999);
INSERT 0 9999
postgres=# select * from pgfincore('testmem1');-[ RECORD 1 ]-+-----------------
relpath       | base/13003/16384
segment       | 0
os_page_size  | 4096
rel_os_pages  | 42980
pages_mem     | 42980
group_mem     | 1
os_pages_free | 277433
databit       | 

postgres=# select pg_relation_size('testmem1');
-[ RECORD 1 ]----+----------
pg_relation_size | 176046080

postgres=# select 42980*4096;
-[ RECORD 1 ]-------
?column? | 176046080

postgres=# insert into testmem1 select generate_series(1,999999), 'wangshuo'||generate_series(1,999999)::text||random()::text, generate_series(1,999999)::text||'sure',generate_series(1,999999), random()::text,random()::text,random()::text,'liuyuanyuan', generate_series(1,999999);
INSERT 0 999999
postgres=# select * from pgfincore('testmem1');
-[ RECORD 1 ]-+-----------------
relpath       | base/13003/16384
segment       | 0
os_page_size  | 4096
rel_os_pages  | 85532
pages_mem     | 6188
group_mem     | 22
os_pages_free | 403050
databit       | 

postgres=#
    上面這個實驗驗證了willneed會將小部分新加入的數據加載到cache中，能夠快速的定位查找，但是當插入數據量較大時，就不會及時的寫入到cache中了。

總結：pgfincore目前適用於數據量非常大、更新不頻繁、更新量較小的表去加載到緩存中，有助於提高應用效率。

    插件函數介紹：

    這裏會對函數進行介紹，並對所有函數進行操作實驗。

    1、pgsysconf：
         這個函數輸出OS block的大小（os_page_size），OS中剩餘的page數(os_pages_free)和OS擁有的page總數(os_total_pages)。
 

postgres=# select * from pgsysconf()；
 os_page_size | os_pages_free | os_total_pages 
--------------+---------------+----------------
         4096 |        167445 |         476688
(1 row)
    2、pgsysconf_pretty：
        這個函數和上面的一樣，僅僅是輸出更易懂些。 

postgres=# select * from pgsysconf_pretty();
 os_page_size | os_pages_free | os_total_pages 
--------------+---------------+----------------
 4096 bytes   | 654 MB        | 1862 MB
(1 row)

    3、pgfadvise_willneed：

    這個函數是
    輸出文件名（relpath），OS block大小（os_page_size），對象佔用系統page數（rel_os_pages）,OS剩餘的page數（os_pages_free）。 

postgres=# select * from pgfadvise_willneed('testmem1');
     relpath      | os_page_size | rel_os_pages | os_pages_free 
------------------+--------------+--------------+---------------
 base/13003/16384 |         4096 |        42554 |        167294
(1 row)
    4、pgfadvise_dontneed：

    這個函數對當前對象設置dontneed標記。dontneed標記的意思就是當操作系統需要釋放內存時優先釋放標記爲dontneed的pages。輸出的意義和上面一致，就不多做介紹了。 

postgres=# select * from pgfadvise_dontneed('testmem1');
     relpath      | os_page_size | rel_os_pages | os_pages_free 
------------------+--------------+--------------+---------------
 base/13003/16384 |         4096 |        42554 |        332798
(1 row)
    5、pgfadvise_NORMAL、pgfadvise_SEQUENTIAL、pgfadvise_RANDOM：

    和pgfadvise_dontneed大同小異，分別將對象標記爲normal、sequential、random，意思按照字面意思理解就可以了。
    其實pgfadvise是一個單獨的函數，參數是relname,fork,action,含義分別爲對象名、文件分支名以及行爲id，上面的3、4、5函數都是輸入不同參數而設置的函數，其中willneed、dontneed、normal、sequential、random的值分別是10、20、30、40、50，且默認fork爲main，即表文件。
    這裏的pgfadvise主要調用了Linux下的函數posix_fadvise，標記值也是posix_fadvise所需要的。

    6、pgfincore：

    這個函數是是提供對象在操作系統緩存中的信息的。
    它分爲三個函數，參數分別爲（relname, fork, getdatabit），（relname, getdatabit），（relname），三個參數意思爲對象名，進程名（這個地方默認是main），是否要顯示databit（很長，注意顯示），第一個函數需要全部輸入，第二個函數默認fork爲main，第三個函數默認fork爲main，getdatabit爲false。
    它輸出的是文件位置及名稱（relpath），文件順序（segment），OS page或block大小（os_page_size），對象佔用系統緩存需要的頁面個數（rel_os_pages），對象已經佔用緩存頁面個數（pages_mem），在緩存中連續的頁面組的個數（group_mem），OS剩餘的page數（os_pages_free），加載信息的位圖（databit）。 

postgres=# select * from pgfincore('testmem1', 'main', false);
-[ RECORD 1 ]-+-----------------
relpath       | base/13003/16384
segment       | 0
os_page_size  | 4096
rel_os_pages  | 42554
pages_mem     | 2
group_mem     | 1
os_pages_free | 325475
databit       | 

postgres=# select * from pgfincore('testmem1', false);
-[ RECORD 1 ]-+-----------------
relpath       | base/13003/16384
segment       | 0
os_page_size  | 4096
rel_os_pages  | 42554
pages_mem     | 2
group_mem     | 1
os_pages_free | 325475
databit       | 

postgres=# select * from pgfincore('testmem1');
-[ RECORD 1 ]-+-----------------
relpath       | base/13003/16384
segment       | 0
os_page_size  | 4096
rel_os_pages  | 42554
pages_mem     | 2
group_mem     | 1
os_pages_free | 325475
databit       |

    7、pgfadvise_loader：

    這個函數可以對頁面緩存直接進行兩方面的作用。因此，它能通過頁面的位圖在緩存中來對頁面進行加載或卸載。
    它分爲兩個函數和上邊的類似，就是設置缺省值，的輸入參數是（relname, fork, segment, load, unload, databit）和（relname, segment, load, unload, databit），分別是對象名，文件分支名，文件序號，是否加載，是否卸載，位圖信息。第二個函數默認fork爲main。
它輸出的是物理文件名及path（relpath），OS page或block大小（os_page_size）, OS中剩餘的page數(os_pages_free)，加載的page數（pages_load），卸載的page數（pages_unload）。 

postgres=# select * from pgfincore('testmem1');
-[ RECORD 1 ]-+-----------------
relpath       | base/13003/16384
segment       | 0
os_page_size  | 4096
rel_os_pages  | 42554
pages_mem     | 0
group_mem     | 0
os_pages_free | 331290
databit       | 

postgres=# select * from pgfadvise_loader('testmem1', 0, true, true, B'1100');
-[ RECORD 1 ]--+-----------------
relpath        | base/13003/16384
os_page_size   | 4096
os_pages_free  | 331294
pages_loaded   | 2
pages_unloaded | 2

postgres=# select * from pgfincore('testmem1');
-[ RECORD 1 ]-+-----------------
relpath       | base/13003/16384
segment       | 0
os_page_size  | 4096
rel_os_pages  | 42554
pages_mem     | 2
group_mem     | 1
os_pages_free | 331290
databit       |
發表評論
所有評論
還沒有人評論，想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.
相關文章

PostgreSQL數據庫角色(五)

IdealReality 2019-02-22 23:18:07

PostgreSQL --序列與自增id的祕密

朱飛東 2019-02-22 21:21:41

進程管理工具的使用

王慶典 2019-02-23 00:39:19

linux out of memory分析(OOM)

一見_ 2019-02-23 00:16:35

WeakReference和內存泄漏有什麼樣的關係？

冰冰水晶 2019-02-23 00:08:25

linux系統調優

zhu_12311 2019-02-23 00:05:45

Redis學習筆記五——啓動警告問題的解決

喵鮮生 2019-02-23 00:00:55

nagios下配置監控系統內存的使用情況的腳本，帶預警功能

劉艤翔 2019-02-22 23:14:17

簡單內存泄漏檢測方法，解決Detected memory leaks!問題

xuzhiming 2019-02-22 23:00:21

pg_data_t數據結構linux2.6.37

xjgwwseu 2019-02-22 22:40:47

PHP文件上傳大小限制修改

Alexsname 2019-02-22 22:18:06

Android避免內存泄露：合理使用getContext()和getApplication()

cfy10 2019-02-22 21:13:40

LruCache大小的定義

IT學無止境 2019-02-22 21:04:18

Java多線程之內存可見性

mingyongyao_cto 2019-02-22 20:29:29

Java多線程之內存可見性

mingyongyao_cto 2019-02-22 20:29:29