我有一个带有spark-submit命令的shell脚本,可以使用oozie shell动作运行。
Oozie能够从shell脚本运行spark-submit命令,但是在部署纱线时作业失败。
…
通过添加两个系统参数解决了该问题。 HADOOP_CONF_DIR =的/ etc / hadoop的/ CONF YARN_CONF_DIR =的/ etc / hadoop的/ CONF
更新的workflow.xml:
<workflow-app name="test-job" xmlns="uri:oozie:workflow:0.5"> <credentials> <credential name="hcat" type="hcat"> <property> <name>hcat.metastore.uri</name> <value>${hcatMetastoreUri}</value> </property> <property> <name>hcat.metastore.principal</name> <value>${hcatMetastorePrincipal}</value> </property> </credential> <credential name="hive2" type="hive2"> <property> <name>hive2.jdbc.url</name> <value>${hive2JdbcUrl}</value> </property> <property> <name>hive2.server.principal</name> <value>${hive2ServerPrincipal}</value> </property> </credential> </credentials> <start to="shell-node"/> <action name="shell-node" cred="hcat, hive2"> <shell xmlns="uri:oozie:shell-action:0.2"> <job-tracker>${jobTracker}</job-tracker> <name-node>${nameNode}</name-node> <configuration> <property> <name>mapred.job.queue.name</name> <value>${queueName}</value> </property> </configuration> <exec>${myscript}</exec> <env-var>HADOOP_CONF_DIR=/etc/hadoop/conf</env-var> <env-var>YARN_CONF_DIR=/etc/hadoop/conf</env-var> <file>${myscriptPath}</file><file>hdfs://<HDFS_PATH>/application.properties#application.properties</file><file>hdfs://<HDFS_PATH>/test-job.jar#test-job.jar</file><file>hdfs://<HDFS_PATH>/myusr.keytab#myusr.keytab</file> <capture-output/> </shell> <ok to="end"/> <error to="fail"/> </action> <kill name="fail"> <message>Shell action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message> </kill> <kill name="fail-output"> <message>Incorrect output, expected [Hello Oozie] but was [${wf:actionData('shell-node')['my_output']}]</message>