Container exited with a non-zero exit code 134錯誤原因查找

問題:spark在yarn執行任務遇到如下的錯誤:

19/12/03 17:19:50 ERROR YarnScheduler: Lost executor 8 on hd060.corp.yodao.com: Container marked as failed: container_e84_1574035425015_31355_01_000009 on host: hd060.corp.yodao.com. Exit status: 134. Diagnostics: Exception from container-launch.
Container id: container_e84_1574035425015_31355_01_000009
Exit code: 134
Exception message: /bin/bash: line 1: 122080 Aborted                 /disk1/eadop/jdk1.8.0_77/bin/java -server -XX:OnOutOfMemoryError='kill %p' -Xms10240m -Xmx10240m '-Dodis.home=/global/exec/zhanghl/corpus-spark-temp1' '-Djava.io.tmpdir=/disk2/zhanghl/tmp' -Djava.io.tmpdir=/disk1/eadop/hadoop-tmp/nm-local-dir/usercache/zhanghl/appcache/application_1574035425015_31355/container_e84_1574035425015_31355_01_000009/tmp '-Dspark.driver.port=22789' -Dspark.yarn.app.container.log.dir=/disk7/eadop/hadoop-tmp/userlogs/application_1574035425015_31355/container_e84_1574035425015_31355_01_000009 org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url spark://[email protected]:22789 --executor-id 8 --hostname hd060.corp.yodao.com --cores 2 --app-id application_1574035425015_31355 --user-class-path file:/disk1/eadop/hadoop-tmp/nm-local-dir/usercache/zhanghl/appcache/application_1574035425015_31355/container_e84_1574035425015_31355_01_000009/__app__.jar > /disk7/eadop/hadoop-tmp/userlogs/application_1574035425015_31355/container_e84_1574035425015_31355_01_000009/stdout 2> /disk7/eadop/hadoop-tmp/userlogs/application_1574035425015_31355/container_e84_1574035425015_31355_01_000009/stderr

Stack trace: ExitCodeException exitCode=134: /bin/bash: line 1: 122080 Aborted                 /disk1/eadop/jdk1.8.0_77/bin/java -server -XX:OnOutOfMemoryError='kill %p' -Xms10240m -Xmx10240m '-Dodis.home=/global/exec/zhanghl/corpus-spark-temp1' '-Djava.io.tmpdir=/disk2/zhanghl/tmp' -Djava.io.tmpdir=/disk1/eadop/hadoop-tmp/nm-local-dir/usercache/zhanghl/appcache/application_1574035425015_31355/container_e84_1574035425015_31355_01_000009/tmp '-Dspark.driver.port=22789' -Dspark.yarn.app.container.log.dir=/disk7/eadop/hadoop-tmp/userlogs/application_1574035425015_31355/container_e84_1574035425015_31355_01_000009 org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url spark://[email protected]:22789 --executor-id 8 --hostname hd060.corp.yodao.com --cores 2 --app-id application_1574035425015_31355 --user-class-path file:/disk1/eadop/hadoop-tmp/nm-local-dir/usercache/zhanghl/appcache/application_1574035425015_31355/container_e84_1574035425015_31355_01_000009/__app__.jar > /disk7/eadop/hadoop-tmp/userlogs/application_1574035425015_31355/container_e84_1574035425015_31355_01_000009/stdout 2> /disk7/eadop/hadoop-tmp/userlogs/application_1574035425015_31355/container_e84_1574035425015_31355_01_000009/stderr

	at org.apache.hadoop.util.Shell.runCommand(Shell.java:545)
	at org.apache.hadoop.util.Shell.run(Shell.java:456)
	at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:722)
	at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)


Container exited with a non-zero exit code 134

1.在這段錯誤附近找到applicationId

類似這樣的 id: application_1574035425015_31345

2.執行:

yarn logs -applicationId application_1574035425015_31345

搜索Error,你會在打印的日誌中看到Error的原因。

或者spark提交任務遇到錯誤執行不成功,而又找不到錯誤,或者打印的日誌中想輸出錯誤的示例,但是打印不出來,也同樣的,在命令行顯示的日誌或者自己的輸出nohup.txt中找application先關的id,執行上訴命令 即可。

其他參考:
ExitCodeException exitCode=1
此篇博客最下面
Container exited with a non-zero exit code 134
yarn報錯ExitCodeException exitCode=127

發表評論
所有評論
還沒有人評論,想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.
相關文章