postgresql 數據庫使用 flexible-freeze.py 處理 vacuum freeze

os: centos 7.4
db: postgresql 10.10

看 github 的更新日期，又是一個被遺棄的項目。

版本

# cat /etc/centos-release
CentOS Linux release 7.4.1708 (Core) 
# 
# su - postgres
Last login: Sat Oct 26 22:55:25 CST 2019 on pts/0
$
$ psql -c "select version();"
                                                 version                                                  
----------------------------------------------------------------------------------------------------------
 PostgreSQL 10.10 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-36), 64-bit
(1 row)

setup

有些依賴需要安裝 psycopg2-binary==2.7.4

# yum install -y python2-psycopg2 python2-argparse-manpage python2-configargparse

# su - postgres
$ git clone https://github.com/pgexperts/flexible-freeze.git
Cloning into 'flexible-freeze'...
remote: Enumerating objects: 176, done.
remote: Total 176 (delta 0), reused 0 (delta 0), pack-reused 176
Receiving objects: 100% (176/176), 32.66 KiB | 3.00 KiB/s, done.
Resolving deltas: 100% (80/80), done.

$ cd flexible-freeze
$ cat README.md

Usage example:

::
    python flexible_freeze.py -m 120 --dblist="prod,queue" --pause 5 -U postgres

Arguments:

* -m, --minutes : number of minutes to run for (see note below)
* -d, --databases : comma-delimited list of databases to vacuum
* --vacuum : do a VACUUM ANALYZE instead of a VACUUM FREEZE
* --pause : seconds to pause between vacuums (10)
* --freezeage : minimum XID age for freezing (10000)
* --costdelay : vacuum_cost_delay in ms (20)
* --costlimit : vacuum_cost_limit (2000)
* --enforce-time : enforce ending time through statement_timeout
* -l, --log : log file for all output (optional)
* -v, --verbose
* -U, --user : database user
* -H, --host : database host
* -p, --port : database port
* -w, --password : database password

先手動執行下

$ cd scripts
$ chmod u+x ./flexible_freeze.py
$ python /var/lib/pgsql/flexible-freeze/scripts/flexible_freeze.py -v -m 120 -d pgbenchdb --pause 5 -U postgres

Flexible Freeze run starting
Processing 1 database (list of databases is pgbenchdb)
working on database pgbenchdb
getting list of tables
All tables vacuumed.
0 tables in 1 databases
Flexible Freeze run complete

查看 flexible_freeze.py 腳本核心部分

    cur = conn.cursor()
    cur.execute("SET vacuum_cost_delay = {0}".format(args.costdelay))
    cur.execute("SET vacuum_cost_limit = {0}".format(args.costlimit))
    
    # if vacuuming, get list of top tables to vacuum
    if args.vacuum:
        tabquery = """WITH deadrow_tables AS (
                SELECT relid::regclass as full_table_name,
                    ((n_dead_tup::numeric) / ( n_live_tup + 1 )) as dead_pct,
                    pg_relation_size(relid) as table_bytes
                FROM pg_stat_user_tables
                WHERE n_dead_tup > 100
                AND ( (now() - last_autovacuum) > INTERVAL '1 hour'
                    OR last_autovacuum IS NULL )
                AND ( (now() - last_vacuum) > INTERVAL '1 hour'
                    OR last_vacuum IS NULL )
            )
            SELECT full_table_name
            FROM deadrow_tables
            WHERE dead_pct > 0.05
            AND table_bytes > 1000000
            ORDER BY dead_pct DESC, table_bytes DESC;"""
    else:
    # if freezing, get list of top tables to freeze
    # includes TOAST tables in case the toast table has older rows
        tabquery = """WITH tabfreeze AS (
                SELECT pg_class.oid::regclass AS full_table_name,
                greatest(age(pg_class.relfrozenxid), age(toast.relfrozenxid)) as freeze_age,
                pg_relation_size(pg_class.oid)
            FROM pg_class JOIN pg_namespace ON pg_class.relnamespace = pg_namespace.oid
                LEFT OUTER JOIN pg_class as toast
                    ON pg_class.reltoastrelid = toast.oid
            WHERE nspname not in ('pg_catalog', 'information_schema')
                AND nspname NOT LIKE 'pg_temp%'
                AND pg_class.relkind = 'r'
            )
            SELECT full_table_name
            FROM tabfreeze
            WHERE freeze_age > {0}
            ORDER BY freeze_age DESC
            LIMIT 1000;""".format(args.freezeage)
			

    # if not, vacuum or freeze
        if args.vacuum:
            exquery = """VACUUM ANALYZE %s""" % table
        else:
            exquery = """VACUUM FREEZE ANALYZE %s""" % table

參考:
https://github.com/pgexperts/flexible-freeze
http://www.databasesoup.com/2014/10/introducing-flexible-freeze.html