Oozie 4.3.0

說明

完整更新日誌：release log
注意：更新日誌裏面只有 jira 編號和標題，詳細信息需要到 jira 網站上依次查找。

Oozie 4.3.0 更新日誌

OOZIE-2613 hive 版本從 0.13.1 升級到 1.2.0
OOZIE-2676 構建 oozie 時默認 profile 從 hadoop-1 變爲 hadoop-2
OOZIE-1978 修復 ForkJoin 過多時，驗證 Forkjoin 是否有效速度過慢的問題
(引入了新的bug（OOZIE-3345）：Multiple “ok to” transitions to the same node are not allowed)

     A1--->A2--->A3    A7--->A8--->A9 
     |           |     |            |
S--> F1          J1-->F2            J2--->E
     |           |     |            |
     A4--->A5--->A6    A10-->A11-->A12
例如，第一個遞歸選擇路徑 F1-> A1-> A2-> A3-> J1，選擇 F2-> A7-> A8-> A9-> J2；然後返回檢查 F2-> A10-> A11-> A12-> J2。
然後，返回到F1，選擇第二條路徑：F1-> A4-> A5-> A6。但是，它再次重新走了F2-> J2的兩條路徑。

例如，現在有10個ForkJoin對，每對有4個分支，則總共有4^10條路徑。

OOZIE-2037 禁用SSL3，支持TLSv1.1，TLSv1.2
OOZIE-2552 ActiveMQ 版本從 5.8.0 升級至 5.10.2
OOZIE-2571 添加 spark.scala.binary.version 屬性以支持Scala 2.11
OOZIE-2488 OpenJPA 版本從 2.2.2 升級到 2.4.1
OOZIE-2648 子作業不應該發送回調給 Oozie Server，而是由Launcher Job完成（MapReduce Action 除外）
OOZIE-2584 在 TestMemoryLocks 類中消除Thread.sleep() 調用，用CountDownLatch替換
OOZIE-2632 提供數據庫導入導出功能，便於數據庫遷移（oozie-setup.sh）
OOZIE-2243 修復 kill 只殺死 launcher job 而未殺死由它啓動的子作業
OOZIE-2649 修復子工作流不能覆蓋父工作流定義的配置屬性問題
OOZIE-1173 null值校驗應統一用參數檢查形式

/**
 *
if (conf == null) {
	throw new IllegalArgumentException("conf cannot be null");
}
if (scriptFile == null) {
	throw new IllegalArgumentException("scriptFile cannot be null");
}
*/
OozieClient.notNull(conf, "conf");
OozieClient.notNull(scriptFile, "scriptFile");

OOZIE-2657 去除接口冗餘修飾符

 public interface CoordInputDependency {
 
    // public static final String INTERNAL_VERSION_ID = "V=1";
    String INTERNAL_VERSION_ID = "V=1";
 
    // public void addInputInstanceList(String inputEventName, List<CoordInputInstance> inputInstanceList);
    void addInputInstanceList(String inputEventName, List<CoordInputInstance> inputInstanceList);

OOZIE-2517 coord 和 bundles 支持 startCreatedTime 和 endCreatedTime 過濾條件

curl "http://host:11000/oozie/v2/jobs?filter=startCreatedTime%3D2016-04-28T00%3A00Z&jobtype=bundle"

curl "http://host:11000/oozie/v2/jobs?filter=startCreatedTime%3D2016-04-28T00%3A00Z&jobtype=coordinator"

curl "http://host:11000/oozie/v2/jobs?filter=startCreatedTime%3D2016-04-28T00%3A00Z&jobtype=wf"

OOZIE-2036 構建 Oozie 時 JDK 版本最低要求爲 1.7
OOZIE-2602 pig 版本從 0.12.1 升級至 0.16.0
OOZIE-2574 支持 Mysql replication 模式 (abhishekbafna via rkanter)

jdbc:mysql:replication://master_host:port,slave_host:port/database

OOZIE-2440 workflow 重試策略支持週期型（periodic ）與指數型（exponential）

全局配置參數（oozie.site.xml）：oozie.service.LiteWorkflowStoreService.user.retry.policy
工作流配置：<action name="a" retry-max="2" retry-interval="1" retry-policy=" exponential">

OOZIE-2507 通過 Oozie 中的JMX beans 暴露監控信息（Zabbix）

<property>
    <name>oozie.jmx_monitoring.enable</name>
    <value>false</value>
    <description>
        If the oozie functional metrics needs to be exposed via JMX interface, 
        set it to true.
    </description>
</property>

OOZIE-2603 給線程池（SchedulerService 與 CallableQueueService）中的線程賦予有意義的名字

// executor = new ThreadPoolExecutor(threads, threads, 10, TimeUnit.SECONDS, (BlockingQueue) queue);
executor = new ThreadPoolExecutor(threads, threads, 10, TimeUnit.SECONDS, (BlockingQueue) queue, new NamedThreadFactory("CallableQueue"));

public class NamedThreadFactory implements ThreadFactory {
    private final AtomicInteger counter = new AtomicInteger();
    private final String threadPrefix;

    public NamedThreadFactory(String threadPrefix) {
        this.threadPrefix = ParamChecker.notEmpty(threadPrefix, "threadPrefix");
    }

    @Override
    public Thread newThread(Runnable r) {
        Thread t = new Thread(r);
        t.setName(threadPrefix + "-" + counter.getAndIncrement());
        return t;
    }
}

OOZIE-2362 修復 BulkJPAExecutor 中存在的 SQL 注入問題
OOZIE-2567 修復 HCat 連接未關閉的問題
OOZIE-2541 修復未關閉 hive 連接可能導致的內存泄露問題
OOZIE-2551 增加轉時間戳函數 coord:epochTime(String timestamp, String millis)

示例：timeStamp 爲 2009-01-01T00:00Z
	millis 爲 false, 返回 ‘1230768000’
	millis 爲 ‘true’，返回 ‘1230768000000’

OOZIE-2542 添加選項以禁用 OpenJPA BrokerImpl finalization，減少GC時間

<property>
    <name>oozie.service.JPAService.openjpa.BrokerImpl</name>
    <value>non-finalizing</value>
    <description>
      	The default OpenJPAEntityManager implementation automatically closes 
        itself during instance finalization.
      	This guards against accidental resource leaks that may occur if a 
        developer fails to explicitly close EntityManagers when finished with 
        them, but it also incurs a scalability bottleneck, since the JVM must
      	perform synchronization during instance creation, and since the finalizer 
        thread will have more instances to monitor.
      	To avoid this overhead, set the openjpa.BrokerImpl configuration property
        to non-finalizing.
      	To use default implementation set it to empty space.
   </description>
</property>

OOZIE-2546 OozieDBCLI中在finally塊中關閉資源
OOZIE-2476 修復一個fork 分支中的操作因短暫錯誤而失敗（但在重試幾次後成功），可能永遠不會匯聚到join的問題
OOZIE-2475 修復殺死 action 後未刪除 action 目錄的問題
OOZIE-2467 修復因GC時間過長 Oozie 殺死自己的情況

<property>   
    <name>oozie.zookeeper.connection.timeout</name>
    <value>180</value>
    <description>
   		 Default ZK connection timeout (in sec).
    </description>
</property>
<property>
    <name>oozie.zookeeper.session.timeout</name>
    <value>300</value>
    <description>
        Default ZK session timeout (in sec). If connection is lost even 
        after retry, then Oozie server will shutdown
        itself if oozie.zookeeper.server.shutdown.ontimeout is true.
    </description>
</property>
<property>
    <name>oozie.zookeeper.max.retries</name>
    <value>10</value>
    <description>
        Maximum number of times to retry.
    </description>
</property>

OOZIE-2312 修復未清理 audit 與 error 日誌的問題
OOZIE-2431 支持使用 hive.metastore.uris 作爲 hcat.metastore.uri 的次要選項
OOZIE-1976 coordinator 輸入 datasets 之前只支持 and 邏輯，現在支持 and、or、combine 邏輯

<!-- 若dataset A與B可用，或者dataset C與D可用，action則會啓動；注意檢查的順序由xml中定義的順序決定 -->
<input-logic>
    <or>
        <and name="AorB">
              <data-in dataset="A"/>
              <data-in dataset="B"/>
        </and>
        <and name="CorD">
              <data-in dataset="C"/>
              <data-in dataset="D"/>
        </and>
    </or>
</input-logic>

<!-- 首先從A中檢查，缺少的instance將從B中補充 -->
<input-logic>
       <combine name="AorB">
            <data-in dataset="A"/>
            <data-in dataset="B"/>
       </combine>
</input-logic>

<!-- dataset A的instance數最少爲5，達到5後會等待10分鐘，期間新增的instance都會被加進來；或者dataset B的instance數最少爲5，達到5後會等待10分鐘，期間新增的instance都會被加進來-->
<input-logic>
    <or name="AorB" min="5" wait="10">
        <data-in dataset="A"/>
        <data-in dataset="B"/>
    </or>
</input-logic>

OOZIE-2444 在 bundle xml 中添加 enabled 標記，以支持是否啓用某個 coordinator
OOZIE-2380 解決 hive action 中調用 “show tables" 或者 "show databases“ 報錯的問題

22257 [uber-SubtaskRunner] ERROR org.apache.hadoop.hive.ql.Driver - FAILED: IllegalArgumentException java.net.URISyntaxException: Relative path in absolute URI: file:./tmp/mapr/b79b1ef5-c8a4-4da9-a307-b4f9045c7bb9/hive_2015-10-01_11-30$
java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: file:./tmp/mapr/b79b1ef5-c8a4-4da9-a307-b4f9045c7bb9/hive_2015-10-01_11-30-46_978_7012892802177357645-1
at org.apache.hadoop.fs.Path.initialize(Path.java:217)
at org.apache.hadoop.fs.Path.<init>(Path.java:208)
at org.apache.hadoop.hive.ql.Context.getScratchDir(Context.java:284)
at org.apache.hadoop.hive.ql.Context.getLocalScratchDir(Context.java:319)
at org.apache.hadoop.hive.ql.Context.getLocalTmpPath(Context.java:414)
...

OOZIE-2402 修復在大規模集羣上通過 oozie-setup.sh 創建 sharelib 過慢的問題

sharelib create -fs FS_URI [-locallib SHARED_LIBRARY] [-concurrency CONCURRENCY]

private void concurrentCopyFromLocal(final FileSystem fs, int threadPoolSize,
        						File srcFile, final Path dstPath) throws IOException {
    List<Future<Void>> futures = Collections.emptyList();
    ExecutorService threadPool = Executors.newFixedThreadPool(threadPoolSize);
    try {
        futures = copyFolderRecursively(fs, threadPool, srcFile, dstPath);
        System.out.println("Running " + futures.size() + " copy tasks on " + threadPoolSize + " threads");
    } finally {
        try {
            threadPool.shutdown();
        } finally {
            checkCopyResults(futures);
        }
    }
}

OOZIE-2185 oozie cli 支持在 oozie-client-env.sh 文件中定義變量
OOZIE-2397 解決 LAST_ONLY 與 NONE 無法正確處理處於 READY 狀態的 action 的問題
OOZIE-2168 將 action 名字長度限制從 50 個字符增加到128
OOZIE-2251 暴露 instrumental 監控信息給某些第三方圖形化監控工具（graphite 或者 ganglia）

<property>
    <name>oozie.external_monitoring.enable</name>
    <value>false</value>
    <description>
        If the oozie functional metrics needs to be exposed to the metrics-server 
        backend, set it to true
        If set to true, the following properties has to be specified : 
        oozie.metrics.server.name, oozie.metrics.host, oozie.metrics.prefix, 	
        oozie.metrics.report.interval.sec, oozie.metrics.port
    </description>
</property>

<property>
    <name>oozie.external_monitoring.type</name>
    <value>graphite</value>
    <description>
        The name of the server to which we want to send the metrics, 
        would be graphite or ganglia.
    </description>
</property>

<property>
    <name>oozie.external_monitoring.address</name>
    <value>http://localhost:2020</value>
</property>

<property>
    <name>oozie.external_monitoring.metricPrefix</name>
    <value>oozie</value>
</property>

<property>
    <name>oozie.external_monitoring.reporterIntervalSecs</name>
    <value>60</value>
</property>

OOZIE-2377 修復 Hive2 Action 將 oozie.hive2.* properties 傳遞給 Beeline 的問題
OOZIE-2345 fork 中的 action 由串行提交改爲並行提交
OOZIE-2356 增加屬性決定是否跳過獲取 credentials

<property>
    <name>oozie.credentials.skip</name>
    <value>false</value>
    <description>
        This determines if Oozie should skip getting credentials from the 
        credential providers.  
        This can be overwritten at a job-level or action-level.
    </description>
</property>

OOZIE-2347 移除不必要的對象創建（new Configuration()、new jobConf()）
OOZIE-2322 修復 Oozie Web UI 啓用 Kerberos 後與 Internet Explorer 10/11 兼容問題及 curl 無法正常工作問題
OOZIE-2332 Hive 和 Hive 2 Action 支持在 workflow.xml 文件中內嵌查詢（<query>標籤替換<script>標籤）
OOZIE-2325 修復當用戶覆蓋 oozie.launcher.mapreduce.map.env 變量時 Shell action 運行失敗問題
OOZIE-2324 修復在 kill 節點發生語法錯誤時導致工作流卡住的問題
OOZIE-2308 增加函數 bundle:conf()
OOZIE-1837 修復因時鐘偏移導致作業失敗的問題

long endTime = System.currentTimeMillis();
if (startTime > endTime) {
    System.out.println("WARNING: Clock skew between the Oozie server host and this host detected.  Please fix this.  " +
            "Attempting to work around...");
    // We don't know which one is wrong (relative to the RM), so to be safe, let's assume they're both wrong and add an
    // offset in both directions
    long diff = 2 * (startTime - endTime);
    startTime = startTime - diff;
    endTime = endTime + diff;
}

OOZIE-2187 支持在 Oozie 配置文件中指定默認 JobTracker 和 NameNode

<property>
    <name>oozie.actions.default.name-node</name>
    <value> </value>
    <description>
        The default value to use for the <name-node> element in applicable action types. 
        This value will be used when neither the action itself nor the global section
        specifies a <name-node>.  
        As expected, it should be of the form "hdfs://HOST:PORT".
    </description>
</property>

<property>
    <name>oozie.actions.default.job-tracker</name>
    <value> </value>
    <description>
        The default value to use for the <job-tracker> element in applicable action
        types.  
        This value will be used when neither the action itself nor the global section 
        specifies a <job-tracker>.  
        As expected, it should be of the form "HOST:PORT".
    </description>
</property>

OOZIE-2159 將 ‘oozie validate’ 命令移至服務端，支持本地文件與 HDFS 文件

oozie validate hdfs://localhost:8020/user/test/myApp/bundle.xml
oozie validate /home/test/myApp/coordinator.xml

OOZIE-2271 Tomcat 版本從 6.0.43 升級到 6.0.44
OOZIE-2178 修復 Java 8 不允許 self-closing element 的問題

Oozie 4.3.0 更新日誌

Oozie 4.3.0

說明

Oozie 4.3.0 更新日誌

DAPPER 事務 TRANSACTION

Oozie 4.3.0 更新日誌

編譯原理 - 文法（一）

編譯原理 - 文法（二）

編譯原理 - 推導與歸約（二）

編譯原理 - 推導與歸約（一）

https://yachay.unat.edu.pe/blog/index.php?comment_area=format_blog&comment_component=blog&comment_co

linux以太網驅動總結