【SparkSQL筆記】SarkSQL高併發讀取數據庫和存儲數據到數據庫（三）

1. SparkSql 高併發讀取數據庫

SparkSql連接數據庫讀取數據給了三個API：

//Construct a DataFrame representing the database table accessible via JDBC URL url named table and connection properties.
Dataset<Row> 	jdbc(String url, String table, java.util.Properties properties)
//Construct a DataFrame representing the database table accessible via JDBC URL url named table using connection properties.
Dataset<Row> 	jdbc(String url, String table, String[] predicates, java.util.Properties connectionProperties)
//Construct a DataFrame representing the database table accessible via JDBC URL url named table.
Dataset<Row> 	jdbc(String url, String table, String columnName, long lowerBound, long upperBound, int numPartitions, java.util.Properties connectionProperties)

三個API介紹：

1. 單個分區，單個task執行，無併發

   遇到數據量很大的表，抽取速度慢。

   實例：

   SparkSession sparkSession = SparkSession.builder().appName("SPARK_FENGDING_TASK1").master("local").config("spark.testing.memory", 471859200).getOrCreate();
   // 配置連接屬性
   Properties dbProps = new Properties();
   dbProps.put("user","user");
   dbProps.put("password","pwd");
   dbProps.put("driver","oracle.jdbc.driver.OracleDriver");
   // 連接數據庫 獲取數據 要使用自己的數據庫連接串
   Dataset<Row> tableDf = sparkSession.read().jdbc("jdbc:oracle:thin:@IP:1521:DEMO", "TABLE_DEMO", dbProps);
   // 返回1
   tableDf.rdd().getPartitions();
   

   該API的併發數爲1，單分區，不管你留給該任務節點多少資源，都只有一個task執行任務

2. 任意字段分區

   該API是第二個API，根據設置的分層條件設置併發度：

   def jdbc(
       url: String,
       table: String,
       predicates: Array[String], #這個是分層的條件，一個數組
       connectionProperties: Properties): DataFrame = {
       val parts: Array[Partition] = predicates.zipWithIndex.map { case (part, i) =>
           JDBCPartition(part, i) : Partition
       }
       jdbc(url, table, parts, connectionProperties)
   }
   

   實例：

   // 設置分區條件 通過入庫時間 把 10月和11月 的數據 分兩個分區
   String[] patitions = {"rksj >= '1569859200' and rksj < '1572537600'","rksj >= '1572537600' and rksj < '1575129600'"};
   // 根據StudentId 分15個分區，就會有15個task抽取數據
   Dataset<Row> tableDf3 = sparkSession.read().jdbc("jdbc:oracle:thin:@IP:1521:DEMO", "TABLE_DEMO",patitions,dbProps);
   // 返回2
   tableDf3.rdd().getPartitions();
   

   該API操作相對自由，就是設置分區條件麻煩一點。

3. 根據Long類型字段分區
   該API是第三個API，根據設置的分區數併發抽取數據：

   def jdbc(
       url: String,
       table: String,
       columnName: String,    # 根據該字段分區，需要爲整形，比如id等
       lowerBound: Long,      # 分區的下界
       upperBound: Long,      # 分區的上界
       numPartitions: Int,    # 分區的個數
       connectionProperties: Properties): DataFrame = {
       val partitioning = JDBCPartitioningInfo(columnName, lowerBound, upperBound, numPartitions)
       val parts = JDBCRelation.columnPartition(partitioning)
       jdbc(url, table, parts, connectionProperties)
   }
   

   實例：

   // 根據StudentId 分15個分區，就會有15個task抽取數據
   Dataset<Row> tableDf2 = sparkSession.read().jdbc("jdbc:oracle:thin:@IP:1521:DEMO", "TABLE_DEMO", "studentId",0,1500,15,dbProps);
   // 返回10
   tableDf2.rdd().getPartitions();
   

   該操作根據分區數設置併發度，缺點是隻能用於Long類型字段。

2. 存儲數據到數據庫

存儲數據庫API給了Class DataFrameWriter<T>類，該類有存儲到文本，Hive，數據庫的API。這裏只說數據庫的API，提一句，如果保存到Text格式，只支持保存一列。。。就很難受。

實例：

有三種寫法

// 第一張寫法，指定format類型，使用save方法存儲數據庫
jdbcDF.write()
  .format("jdbc")
  .option("url", "jdbc:postgresql:dbserver")
  .option("dbtable", "schema.tablename")
  .option("user", "username")
  .option("password", "password")
  .save();
// 第二種寫法 使用jdbc寫入數據庫
jdbcDF2.write()
  .jdbc("jdbc:postgresql:dbserver", "schema.tablename", connectionProperties);

// 第三種寫法,也是使用jdbc，只不過添加createTableColumnTypes，創建表的時候使用該屬性字段創建表字段
jdbcDF.write()
  .option("createTableColumnTypes", "name CHAR(64), comments VARCHAR(1024)")
  .jdbc("jdbc:postgresql:dbserver", "schema.tablename", connectionProperties);

當我們的表已經存在的時候，使用上面的語句就會報錯表已存在，這是因爲我們沒有指定存儲模式，默認是ErrorIfExists

保存模式：

SaveMode	實例	含義
Append	SaveMode.Append	當保存DF到數據庫，如果表已經存在，我們則會在表中追加數據
Overwrite	SaveMode.Overwrite	當保存DF到數據庫，如果表已經存在，我們則會重寫表的數據，和truncate搭配使用
ErrorIfExists	SaveMode.ErrorIfExists	當保存DF到數據庫，如果表已經存在，報錯，提示表已經存在
Ignore	SaveMode.Ignore	當保存DF到數據庫，如果表已經存在，不做任何操作

所以一般都是這樣用：

tableDf3.write().mode(SaveMode.Append).jdbc("jdbc:oracle:thin:@IP:1521:DEMO", "TABLE_DEMO", connectionProperties);

對於connectionProperties還有很多其他選項：

Property Name	Meaning
url	The JDBC URL to connect to. The source-specific connection properties may be specified in the URL. e.g., jdbc:postgresql://localhost/test?user=fred&password=secret
dbtable	The JDBC table that should be read from or written into. Note that when using it in the read path anything that is valid in a FROM clause of a SQL query can be used. For example, instead of a full table you could also use a subquery in parentheses. It is not allowed to specify dbtable and query options at the same time.
query	A query that will be used to read data into Spark. The specified query will be parenthesized and used as a subquery in the FROM clause. Spark will also assign an alias to the subquery clause. As an example, spark will issue a query of the following form to the JDBC Source. SELECT FROM () spark_gen_alias Below are couple of restrictions while using this option. It is not allowed to specify dbtable and query options at the same time. It is not allowed to specify query and partitionColumn options at the same time. When specifying partitionColumn option is required, the subquery can be specified using dbtable option instead and partition columns can be qualified using the subquery alias provided as part of dbtable. Example: spark.read.format("jdbc") .option("url", jdbcUrl) .option("query", "select c1, c2 from t1") .load()
driver	The class name of the JDBC driver to use to connect to this URL.
partitionColumn, lowerBound, upperBound	These options must all be specified if any of them is specified. In addition, numPartitions must be specified. They describe how to partition the table when reading in parallel from multiple workers. partitionColumn must be a numeric, date, or timestamp column from the table in question. Notice that lowerBound and upperBound are just used to decide the partition stride, not for filtering the rows in table. So all rows in the table will be partitioned and returned. This option applies only to reading.
numPartitions	The maximum number of partitions that can be used for parallelism in table reading and writing. This also determines the maximum number of concurrent JDBC connections. If the number of partitions to write exceeds this limit, we decrease it to this limit by calling coalesce(numPartitions) before writing.
queryTimeout	The number of seconds the driver will wait for a Statement object to execute to the given number of seconds. Zero means there is no limit. In the write path, this option depends on how JDBC drivers implement the API setQueryTimeout, e.g., the h2 JDBC driver checks the timeout of each query instead of an entire JDBC batch. It defaults to 0.
fetchsize	The JDBC fetch size, which determines how many rows to fetch per round trip. This can help performance on JDBC drivers which default to low fetch size (eg. Oracle with 10 rows). This option applies only to reading.
batchsize	The JDBC batch size, which determines how many rows to insert per round trip. This can help performance on JDBC drivers. This option applies only to writing. It defaults to 1000.
isolationLevel	The transaction isolation level, which applies to current connection. It can be one of NONE, READ_COMMITTED, READ_UNCOMMITTED, REPEATABLE_READ, or SERIALIZABLE, corresponding to standard transaction isolation levels defined by JDBC’s Connection object, with default of READ_UNCOMMITTED. This option applies only to writing. Please refer the documentation in java.sql.Connection.
sessionInitStatement	After each database session is opened to the remote DB and before starting to read data, this option executes a custom SQL statement (or a PL/SQL block). Use this to implement session initialization code. Example: option("sessionInitStatement", """BEGIN execute immediate 'alter session set "_serial_direct_read"=true'; END;""")
truncate	This is a JDBC writer related option. When SaveMode.Overwrite is enabled, this option causes Spark to truncate an existing table instead of dropping and recreating it. This can be more efficient, and prevents the table metadata (e.g., indices) from being removed. However, it will not work in some cases, such as when the new data has a different schema. It defaults to false. This option applies only to writing.
cascadeTruncate	This is a JDBC writer related option. If enabled and supported by the JDBC database (PostgreSQL and Oracle at the moment), this options allows execution of a TRUNCATE TABLE t CASCADE (in the case of PostgreSQL a TRUNCATE TABLE ONLY t CASCADE is executed to prevent inadvertently truncating descendant tables). This will affect other tables, and thus should be used with care. This option applies only to writing. It defaults to the default cascading truncate behaviour of the JDBC database in question, specified in the isCascadeTruncate in each JDBCDialect.
createTableOptions	This is a JDBC writer related option. If specified, this option allows setting of database-specific table and partition options when creating a table (e.g., CREATE TABLE t (name string) ENGINE=InnoDB.). This option applies only to writing.
createTableColumnTypes	The database column data types to use instead of the defaults, when creating the table. Data type information should be specified in the same format as CREATE TABLE columns syntax (e.g: "name CHAR(64), comments VARCHAR(1024)"). The specified types should be valid spark sql data types. This option applies only to writing.
customSchema	The custom schema to use for reading data from JDBC connectors. For example, "id DECIMAL(38, 0), name STRING". You can also specify partial fields, and the others use the default type mapping. For example, "id DECIMAL(38, 0)". The column names should be identical to the corresponding column names of JDBC table. Users can specify the corresponding data types of Spark SQL instead of using the defaults. This option applies only to reading.
pushDownPredicate	The option to enable or disable predicate push-down into the JDBC data source. The default value is true, in which case Spark will push down filters to the JDBC data source as much as possible. Otherwise, if set to false, no filter will be pushed down to the JDBC data source and thus all filters will be handled by Spark. Predicate push-down is usually turned off when the predicate filtering is performed faster by Spark than by the JDBC data source.

這裏面的truncate就是說當使用SaveMode.Overwrite的時候，設置truncate爲true，就會對錶進行truncate語句清理表，不再是刪除表在重建表的操作。