Oozie定時任務/循環任務

1. 目標：Coordinator週期性調度任務
2. 檢查系統當前時區：
    命令：date -R
    注意：如果顯示的時區不是+0800，刪除localtime文件夾後，再關聯一個正確時區的鏈接過去
    
    
    注意：檢測ntp是否安裝，命令：rpm -qa | grep ntp,使用 root 用戶制定計劃任務,週期性同步集羣時間
    
3. 修改/usr/local/module/oozie-4.0.0-cdh5.3.6/conf/ oozie-site.xml 文件 
    
4. 修改 js 框架中的關於時間設置的代碼 
    
    
5. 重啓 oozie 服務，並重啓瀏覽器（一定要注意清除緩存） 
    
6. 效果
    
7. 拷貝官方模板配置定時任務
     
    
8. 修改coordinator.xml 文件
    

   <!--
   	修改頻率爲5分鐘，這是最小值：frequency="${coord:minutes(5)
       timezone="GMT+0800"
   -->
   <coordinator-app name="cron-coord" frequency="${coord:minutes(5)}" start="${start}" end="${end}" timezone="GMT+0800"
                    xmlns="uri:oozie:coordinator:0.2">
           <action>
           <workflow>
               <app-path>${workflowAppUri}</app-path>
               <configuration>
                   <property>
                       <name>jobTracker</name>
                       <value>${jobTracker}</value>
                   </property>
                   <property>
                       <name>nameNode</name>
                       <value>${nameNode}</value>
                   </property>
                   <property>
                       <name>queueName</name>
                       <value>${queueName}</value>
                   </property>
               </configuration>
           </workflow>
       </action>
   </coordinator-app>
   
   

    

9. 修改workflow.xml 文件
    

   <workflow-app xmlns="uri:oozie:workflow:0.4" name="shell-wf">
   	<start to="p1-shell-node"/> 
   	  <action name="p1-shell-node"> 
   		  <shell xmlns="uri:oozie:shell-action:0.2"> 
   			  <job-tracker>${jobTracker}</job-tracker> 
   			  <name-node>${nameNode}</name-node> 
   			  <configuration> 
   				  <property> 
   					  <name>mapred.job.queue.name</name> 
   					  <value>${queueName}</value> 
   				  </property> 
   			  </configuration> 
   			  <!--要執行的腳本-->
   			  <exec>p1.sh</exec> 
   			  <!--腳本路徑，注意：這個是HDFS上路徑-->
   			  <file>/user/kgf/oozie-apps/cron/p1.sh</file> 
   			  <capture-output/> 
   		  </shell> 
   		  <ok to="end"/> 
   		  <error to="fail"/> 
   	  </action> 
   <kill name="fail"> 
       <message>Shell action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message> 
   </kill> 
   <kill name="fail-output"> 
       <message>Incorrect output, expected [Hello Oozie] but was [${wf:actionData('shell-node')['my_output']}]</message> 
   </kill> 
   <end name="end"/> 
   </workflow-app>
   
   

   編輯p1.sh腳本
   

10. 編輯job.properties文件
     

    nameNode=hdfs://hadoop111:8020
    jobTracker=hadoop112:8032
    queueName=default
    examplesRoot=oozie-apps
    
    oozie.coord.application.path=${nameNode}/user/${user.name}/${examplesRoot}/cron
    #注意：start必須設置爲未來時間，否則任務失敗
    start=2019-11-10T13:28+0800
    end=2019-11-10T13:33+0800
    workflowAppUri=${nameNode}/user/${user.name}/${examplesRoot}/cron
    
    
    

     

11. 提交任務
     
12. 效果
     a：出錯
           

    2019-11-10 12:15:00,579  INFO ActionStartXCommand:541 - SERVER[hadoop111] USER[kgf] GROUP[-] TOKEN[] APP[shell-wf] JOB[0000001-191110121143517-oozie-kgf-W] ACTION[0000001-191110121143517-oozie-kgf-W@:start:] Start action [0000001-191110121143517-oozie-kgf-W@:start:] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
    2019-11-10 12:15:00,579  INFO ActionStartXCommand:541 - SERVER[hadoop111] USER[kgf] GROUP[-] TOKEN[] APP[shell-wf] JOB[0000001-191110121143517-oozie-kgf-W] ACTION[0000001-191110121143517-oozie-kgf-W@:start:] [***0000001-191110121143517-oozie-kgf-W@:start:***]Action status=DONE
    2019-11-10 12:15:00,580  INFO ActionStartXCommand:541 - SERVER[hadoop111] USER[kgf] GROUP[-] TOKEN[] APP[shell-wf] JOB[0000001-191110121143517-oozie-kgf-W] ACTION[0000001-191110121143517-oozie-kgf-W@:start:] [***0000001-191110121143517-oozie-kgf-W@:start:***]Action updated in DB!
    2019-11-10 12:15:02,358  INFO ActionStartXCommand:541 - SERVER[hadoop111] USER[kgf] GROUP[-] TOKEN[] APP[shell-wf] JOB[0000001-191110121143517-oozie-kgf-W] ACTION[0000001-191110121143517-oozie-kgf-W@p1-shell-node] Start action [0000001-191110121143517-oozie-kgf-W@p1-shell-node] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
    2019-11-10 12:15:04,840  INFO ShellActionExecutor:541 - SERVER[hadoop111] USER[kgf] GROUP[-] TOKEN[] APP[shell-wf] JOB[0000001-191110121143517-oozie-kgf-W] ACTION[0000001-191110121143517-oozie-kgf-W@p1-shell-node] checking action, hadoop job ID [job_1573438272511_0003] status [RUNNING]
    2019-11-10 12:15:04,841  INFO ActionStartXCommand:541 - SERVER[hadoop111] USER[kgf] GROUP[-] TOKEN[] APP[shell-wf] JOB[0000001-191110121143517-oozie-kgf-W] ACTION[0000001-191110121143517-oozie-kgf-W@p1-shell-node] [***0000001-191110121143517-oozie-kgf-W@p1-shell-node***]Action status=RUNNING
    2019-11-10 12:15:04,842  INFO ActionStartXCommand:541 - SERVER[hadoop111] USER[kgf] GROUP[-] TOKEN[] APP[shell-wf] JOB[0000001-191110121143517-oozie-kgf-W] ACTION[0000001-191110121143517-oozie-kgf-W@p1-shell-node] [***0000001-191110121143517-oozie-kgf-W@p1-shell-node***]Action updated in DB!
    2019-11-10 12:15:31,915  INFO CallbackServlet:541 - SERVER[hadoop111] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000001-191110121143517-oozie-kgf-W] ACTION[0000001-191110121143517-oozie-kgf-W@p1-shell-node] callback for action [0000001-191110121143517-oozie-kgf-W@p1-shell-node]
    2019-11-10 12:15:32,063  WARN ShellActionExecutor:544 - SERVER[hadoop111] USER[kgf] GROUP[-] TOKEN[] APP[shell-wf] JOB[0000001-191110121143517-oozie-kgf-W] ACTION[0000001-191110121143517-oozie-kgf-W@p1-shell-node] Exception in check(). Message[JA017: Unknown hadoop job [job_1573438272511_0003] associated with action [0000001-191110121143517-oozie-kgf-W@p1-shell-node].  Failing this action!]
    org.apache.oozie.action.ActionExecutorException: JA017: Unknown hadoop job [job_1573438272511_0003] associated with action [0000001-191110121143517-oozie-kgf-W@p1-shell-node].  Failing this action!
    	at org.apache.oozie.action.hadoop.JavaActionExecutor.check(JavaActionExecutor.java:1210)
    	at org.apache.oozie.command.wf.ActionCheckXCommand.execute(ActionCheckXCommand.java:181)
    	at org.apache.oozie.command.wf.ActionCheckXCommand.execute(ActionCheckXCommand.java:55)
    	at org.apache.oozie.command.XCommand.call(XCommand.java:281)
    	at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:174)
    	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    	at java.lang.Thread.run(Thread.java:748)
    2019-11-10 12:15:32,066  WARN ActionCheckXCommand:544 - SERVER[hadoop111] USER[kgf] GROUP[-] TOKEN[] APP[shell-wf] JOB[0000001-191110121143517-oozie-kgf-W] ACTION[0000001-191110121143517-oozie-kgf-W@p1-shell-node] Exception while executing check(). Error Code [JA017], Message[JA017: Unknown hadoop job [job_1573438272511_0003] associated with action [0000001-191110121143517-oozie-kgf-W@p1-shell-node].  Failing this action!]
    org.apache.oozie.action.ActionExecutorException: JA017: Unknown hadoop job [job_1573438272511_0003] associated with action [0000001-191110121143517-oozie-kgf-W@p1-shell-node].  Failing this action!
    	at org.apache.oozie.action.hadoop.JavaActionExecutor.check(JavaActionExecutor.java:1210)
    	at org.apache.oozie.command.wf.ActionCheckXCommand.execute(ActionCheckXCommand.java:181)
    	at org.apache.oozie.command.wf.ActionCheckXCommand.execute(ActionCheckXCommand.java:55)
    	at org.apache.oozie.command.XCommand.call(XCommand.java:281)
    	at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:174)
    	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    	at java.lang.Thread.run(Thread.java:748)
    2019-11-10 12:15:32,067  WARN ActionCheckXCommand:544 - SERVER[hadoop111] USER[kgf] GROUP[-] TOKEN[] APP[shell-wf] JOB[0000001-191110121143517-oozie-kgf-W] ACTION[0000001-191110121143517-oozie-kgf-W@p1-shell-node] Failing Job due to failed action [p1-shell-node]
    2019-11-10 12:15:32,068  WARN LiteWorkflowInstance:544 - SERVER[hadoop111] USER[kgf] GROUP[-] TOKEN[] APP[shell-wf] JOB[0000001-191110121143517-oozie-kgf-W] ACTION[0000001-191110121143517-oozie-kgf-W@p1-shell-node] Workflow Failed. Failing node [p1-shell-node]
    2019-11-10 12:15:33,740  INFO KillXCommand:541 - SERVER[hadoop111] USER[kgf] GROUP[-] TOKEN[] APP[shell-wf] JOB[0000001-191110121143517-oozie-kgf-W] ACTION[-] STARTED WorkflowKillXCommand for jobId=0000001-191110121143517-oozie-kgf-W
    2019-11-10 12:15:35,218  INFO KillXCommand:541 - SERVER[hadoop111] USER[kgf] GROUP[-] TOKEN[] APP[shell-wf] JOB[0000001-191110121143517-oozie-kgf-W] ACTION[-] ENDED WorkflowKillXCommand for jobId=0000001-191110121143517-oozie-kgf-W
    

    b：解決方案：
          ⑴在配置歷史服務器時，/usr/local/module/hadoop-cdh5.3.6/hadoop-2.5.0-cdh5.3.6/etc/hadoop/mapred-site.xml要有
              如下配置
               
         ⑵在/usr/local/module/oozie-4.0.0-cdh5.3.6/conf/oozie-site.xml要如下配置
              

13. 最後效果