數據大挪移之ora2pg使用指南

ora2pg是一款免費的小工具,用於將Oracle數據庫遷移到PostgreSQL。ora2pg通過自動掃描oracle數據庫獲取表結構以及數據,然後生成PostgreSQL數據庫的SQL腳本或者直接在PostgreSQL上創建表並插入數據。而且它還支持視圖、序列以及分區表,其功能非常強大。這裏有一個120G左右的oracle 11g測試庫,因業務需求將其遷移至PostgreSQL 10.6。

一、安裝ora2pg

1.1 安裝OS依賴軟件包

[root@pg07 ~]# yum install perl-DBD-Pg perl perl-devel perl-DBI  perl-CPAN bzip2 perl-ExtUtils-CBuilder perl-ExtUtils-MakeMaker

1.2 安裝Oracle瘦客戶端

從Oracle官方網站下載oracle 12c的瘦客戶端,然後進行安裝並配置。

[root@pg07 u02]# la /u02/software/oracle/
total 55M
drwxr-xr-x 2 root root 4.0K Dec  3 15:04 .
drwxr-xr-x 7 root root   76 Dec  3 15:04 ..
-rw-r--r-- 1 root root  51M Nov 28 14:34 oracle-instantclient12.2-basic-12.2.0.1.0-1.x86_64.rpm
-rw-r--r-- 1 root root 593K Nov 28 14:32 oracle-instantclient12.2-devel-12.2.0.1.0-1.x86_64.rpm
-rw-r--r-- 1 root root 1.5M Nov 28 14:32 oracle-instantclient12.2-jdbc-12.2.0.1.0-1.x86_64.rpm
-rw-r--r-- 1 root root 692K Nov 28 14:32 oracle-instantclient12.2-sqlplus-12.2.0.1.0-1.x86_64.rpm
-rw-r--r-- 1 root root 922K Nov 28 14:35 oracle-instantclient12.2-tools-12.2.0.1.0-1.x86_64.rpm
[root@pg07 ~]# yum -y localinstall /u02/software/oracle/oracle-instantclient12.2-*.rpm
[root@pg07 ~]# mkdir -p /usr/lib/oracle/12.2/client64/network/admin

編輯root環境變量,加入以下內容:

[root@pg07 ~]# vi .bash_profile 
export LD_LIBRARY_PATH=/usr/lib/oracle/12.2/client64/lib
export NLS_LANG=AMERICAN_AMERICA.ZHS16GBK
export TNS_ADMIN=/usr/lib/oracle/12.2/client64/network/admin
export ORACLE_HOME=/usr/lib/oracle/12.2/client64/
export PATH=$PATH:$ORACLE_HOME/bin:$HOME/bin
[root@pg07 ~]# source .bash_profile

1.3 安裝ora2pg

[root@pg07 ~]# wget https://sourceforge.net/projects/ora2pg/files/19.1/ora2pg-19.1.tar.bz2
[root@pg07 ~]# tar -xjf ora2pg-19.1.tar.bz2 
[root@pg07 ~]# cd ora2pg-19.1
[root@pg07 ora2pg-19.1]# perl Makefile.PL 
Checking if your kit is complete...
Looks good
Writing Makefile for Ora2Pg

Done...
------------------------------------------------------------------------------
Please read documentation at http://ora2pg.darold.net/ before asking for help
------------------------------------------------------------------------------
Now type: make && make install
[root@pg07 ora2pg-19.1]# make && make install
cp lib/Ora2Pg.pm blib/lib/Ora2Pg.pm
cp lib/Ora2Pg/GEOM.pm blib/lib/Ora2Pg/GEOM.pm
cp lib/Ora2Pg/PLSQL.pm blib/lib/Ora2Pg/PLSQL.pm
cp lib/Ora2Pg/MySQL.pm blib/lib/Ora2Pg/MySQL.pm
cp scripts/ora2pg blib/script/ora2pg
/usr/bin/perl -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/ora2pg
cp scripts/ora2pg_scanner blib/script/ora2pg_scanner
/usr/bin/perl -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/ora2pg_scanner
Manifying blib/man3/ora2pg.3
Installing /usr/local/share/perl5/Ora2Pg.pm
Installing /usr/local/share/perl5/Ora2Pg/GEOM.pm
Installing /usr/local/share/perl5/Ora2Pg/PLSQL.pm
Installing /usr/local/share/perl5/Ora2Pg/MySQL.pm
Installing /usr/local/share/man/man3/ora2pg.3
Installing /usr/local/bin/ora2pg
Installing /usr/local/bin/ora2pg_scanner
Installing default configuration file (ora2pg.conf.dist) to /etc/ora2pg
Appending installation info to /usr/lib64/perl5/perllocal.pod

1.4 安裝DBD::Oracle

[root@pg07 ~]# perl -MCPAN -e shell
Terminal does not support AddHistory.

cpan shell -- CPAN exploration and modules installation (v1.9800)
Enter 'h' for help.
cpan[1]> get DBD::Oracle
Fetching with HTTP::Tiny:
http://mirror.truenetwork.ru/CPAN/authors/01mailrc.txt.gz
Reading '/root/.cpan/sources/authors/01mailrc.txt.gz'
............................................................................DONE
Fetching with HTTP::Tiny:
http://mirror.truenetwork.ru/CPAN/modules/02packages.details.txt.gz
Reading '/root/.cpan/sources/modules/02packages.details.txt.gz'
  Database was generated on Wed, 28 Nov 2018 23:54:25 GMT
  HTTP::Date not available
.............
  New CPAN.pm version (v2.16) available.
  [Currently running version is v1.9800]
  You might want to try
    install CPAN
    reload cpan
  to both upgrade CPAN.pm and run the new version without leaving
  the current session.

...............................................................DONE
Fetching with HTTP::Tiny:
http://mirror.truenetwork.ru/CPAN/modules/03modlist.data.gz
Reading '/root/.cpan/sources/modules/03modlist.data.gz'
DONE
Writing /root/.cpan/Metadata
Running get for module 'DBD::Oracle'
Fetching with HTTP::Tiny:
http://mirror.truenetwork.ru/CPAN/authors/id/Z/ZA/ZARQUON/DBD-Oracle-1.76.tar.gz
Fetching with HTTP::Tiny:
http://mirror.truenetwork.ru/CPAN/authors/id/Z/ZA/ZARQUON/CHECKSUMS
Checksum for /root/.cpan/sources/authors/id/Z/ZA/ZARQUON/DBD-Oracle-1.76.tar.gz ok
Scanning cache /root/.cpan/build for sizes
DONE
cpan[2]> quit
Terminal does not support GetHistory.
Lockfile removed.
[root@pg07 ~]# cd .cpan/build/DBD-Oracle-1.76-sU5yu5/
[root@pg07 DBD-Oracle-1.76-sU5yu5]# perl Makefile.PL
[root@pg07 DBD-Oracle-1.76-sU5yu5]# make && make install

1.5 創建ora2pg配置文件

首先從示例文件複製一份,然後根據實際情況更改配置文件相關參數:

[root@pg07 ~]# cp /etc/ora2pg/ora2pg.conf.dist /etc/ora2pg/ora2pg.conf
[root@pg07 ~]#  vi /etc/ora2pg/ora2pg.conf
--Oracle連接參數
ORACLE_HOME     /usr/lib/oracle/12.2/client64
ORACLE_DSN      dbi:Oracle:host=172.16.100.19;sid=kkdb;port=1521
ORACLE_USER     system --如果這裏的用戶是sys、system,則user_grants選項必須設置爲1
ORACLE_PWD      abcABC12
USER_GRANTS     1
ORA_INITIAL_COMMAND commit
--schema參數
EXPORT_SCHEMA   1
SCHEMA  XXKIN
CREATE_SCHEMA   0
COMPILE_SCHEMA  0
PG_SCHEMA   public
--導出對象(根據實際情況選擇)
TYPE    TABLE COPY  VIEW TYPE 
DISABLE_COMMENT         0
EXTERNAL_TO_FDW         1
TRUNCATE_TABLE  0
USE_TABLESPACE          0
REORDERING_COLUMNS      0
--約束參數
KEEP_PKEY_NAMES         0
PKEY_IN_CREATE          0
FKEY_ADD_UPDATE         never
FKEY_DEFERRABLE 0
DEFER_FKEY      0
DROP_FKEY       1
--觸發器參數
DISABLE_SEQUENCE        1
DISABLE_TRIGGERS 1
--對象修改參數
PRESERVE_CASE   0
INDEXES_RENAMING        0
USE_INDEX_OPCLASS       0
PREFIX_PARTITION        0
DISABLE_PARTITION       1
WITH_OID                0
ORA_RESERVED_WORDS      audit,comment
USE_RESERVED_WORDS      1
DISABLE_UNLOGGED        0
--PostgreSQL參數
PG_DSN          dbi:Pg:dbname=kkdb;host=192.168.120.149;port=5432
PG_USER         xxkin
PG_PWD          redhat
OUTPUT          xxkin.sql
OUTPUT_DIR     /tmp
BZIP2
FILE_PER_CONSTRAINT     0
FILE_PER_INDEX          0
FILE_PER_FKEYS          0
FILE_PER_TABLE  0
FILE_PER_FUNCTION       0
STOP_ON_ERROR           1
COPY_FREEZE             0
CREATE_OR_REPLACE       1
--TYPE參數
PG_NUMERIC_TYPE 1
PG_INTEGER_TYPE 1
DEFAULT_NUMERIC bigint
ENABLE_MICROSECOND      1
TO_NUMBER_CONVERSION    numeric
GEN_USER_PWD    0
FORCE_OWNER     0
FORCE_SECURITY_INVOKER  0
--數據參數
DATA_LIMIT      10000
NOESCAPE        0
TRANSACTION     serializable
STANDARD_CONFORMING_STRINGS     1
NO_LOB_LOCATOR  1
XML_PRETTY      0
LOG_ON_ERROR            0
TRIM_TYPE               BOTH
INTERNAL_DATE_MAX       49
FUNCTION_CHECK          1
--Export/Import性能參數
JOBS            1
ORACLE_COPIES   1
PARALLEL_TABLES 1
DEFAULT_PARALLELISM_DEGREE      0
PARALLEL_MIN_ROWS               100000
DROP_INDEXES    0
SYNCHRONOUS_COMMIT      0
--PLSQL參數
EXPORT_INVALID  0
PLSQL_PGSQL     1
NULL_EQUAL_EMPTY        0
EMPTY_LOB_NULL          0
PACKAGE_AS_SCHEMA       1
REWRITE_OUTER_JOIN      1
FUNCTION_STABLE         1
COMMENT_COMMIT_ROLLBACK 0
COMMENT_SAVEPOINT       0
USE_ORAFCE      0
AUTONOMOUS_TRANSACTION  1
--ASSESSMENT參數
ESTIMATE_COST           0
COST_UNIT_VALUE         5
DUMP_AS_HTML            0
TOP_MAX                 10
HUMAN_DAYS_LIMIT        5
--PostgreSQL特性參數
PG_SUPPORTS_WHEN        1
PG_SUPPORTS_INSTEADOF   1
PG_SUPPORTS_MVIEW       1
PG_SUPPORTS_CHECKOPTION 1
PG_SUPPORTS_IFEXISTS    1
PG_SUPPORTS_LATERAL     1
PG_SUPPORTS_PARTITION   1
BITMAP_AS_GIN           1
PG_BACKGROUND           0
PG_SUPPORTS_SUBSTR      1
PG_SUPPORTS_NAMED_OPERATOR      1
PG_SUPPORTS_IDENTITY    1
PG_SUPPORTS_PROCEDURE   0
--SPATIAL參數
AUTODETECT_SPATIAL_TYPE 1
CONVERT_SRID            1
DEFAULT_SRID            4326
GEOMETRY_EXTRACT_TYPE   INTERNAL
--FDW參數
FDW_SERVER      orcl
--MySQL參數
MYSQL_PIPES_AS_CONCAT           0
MYSQL_INTERNAL_EXTRACT_FORMAT   0

到此,整個ora2pg的安裝配置已完成。

二、使用ora2pg

2.1 導出全部表結構

如果在配置文件裏指定的type包含了COPY,導完表結構後,表的數據也會隨之導出,加-t參數即可只導出表結構不導出數據。

[root@pg07 ~]# ora2pg -t table view
[========================>] 777/777 tables (100.0%) end of scanning.                 
[========================>] 777/777 tables (100.0%) end of table export.                  
[========================>] 0/0 views (100.0%) end of output. 

導出完成後,在PostgreSQL中創建數據庫以及對應的表。

-bash-4.2$ psql
psql (10.6)
Type "help" for help.

postgres=# CREATE USER xxkin WITH password 'redhat' CREATEDB SUPERUSER replication bypa***ls createrole login;
postgres=# create database kkdb;
postgres=# alter database kkdb owner to xxkin;
postgres=# \c kkdb
You are now connected to database "kkdb" as user "postgres".
kkdb=# \i /tmp/output.sql

ora2pg有很多參數可供使用,具體請參考其幫助文件。

2.2 通過ora2pg遷移數據

之前已創建好數據庫以及相關表結構,現在只需通過ora2pg將數據同步到PostgreSQL數據庫上即可。

[root@pg07 ~]# ora2pg -d -t copy
[2018-11-30 11:38:51] Ora2Pg version: 19.1
[2018-11-30 11:38:51] Trying to connect to database: dbi:Oracle:host=172.16.100.19;sid=kkdb;port=1521
[2018-11-30 11:38:51] Isolation level: SET TRANSACTION ISOLATION LEVEL SERIALIZABLE
[2018-11-30 11:38:51] DEBUG: executing initial command to Oracle: commit
[2018-11-30 11:38:51] Retrieving table information...
[2018-11-30 11:40:55] [1] Scanning table ABC_DEL_CLASS (1 rows)...
[2018-11-30 11:40:55] [2] Scanning table ACTIVITY_DATA (41 rows)...
[2018-11-30 11:40:55] [3] Scanning table ACTIVITY_GBEAN_STAT (243897 rows)...
......

如果在遷移過程中,需要排除某些表,加-e參數,如下:

[root@pg07 ~]# ora2pg -d -t copy -e usr_dx_tbs,hr_info_pg,op_log

總結:

以下是在使用過程遇到的問題,提出的建議:
1、先導出表結構,然後在同步數據;
2、在導出表結構的時候禁用分區表;
3、注意PostgreSQL的保留的關鍵字是否爲oracle表字段名;
4、刪除所有外鍵約束。
5、直接通過ora2pg遷移數據,不要將oracle表數據導出爲csv格式,然後再導入PostgreSQL數據庫。
參考文獻:ora2pg DOCUMENTATION

發表評論
所有評論
還沒有人評論,想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.
相關文章