Skip to content

Commit

Permalink
Add README.md and update tag
Browse files Browse the repository at this point in the history
  • Loading branch information
Baohe Zhang committed Jun 2, 2020
1 parent 1a3e963 commit 6b00df2
Show file tree
Hide file tree
Showing 3 changed files with 21 additions and 3 deletions.
17 changes: 17 additions & 0 deletions src/main/resources/oozie/spark_hdfs_lr/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
Instructions for running this oozie application:

- create a directory `spark_hdfs_lr/` in HDFS for the oozie application.

- upload `workflow.xml` to `spark_hdfs_lr/apps/spark/`.

- use `mvn clean package` to create the jar package of spark-starter if you haven't done so.

- upload the jar package `spark-starter/target/spark-starter-2.0-SNAPSHOT-jar-with-dependencies.jar` to `spark_hdfs_lr/apps/lib/`.

- upload resource files `spark-starter/src/main/resources/data/lr_data.txt` to `spark_hdfs_lr/data/`.

- update `nameNode` and `jobTracker` in `job.properties` if you are running on the cluster other than AR.

- export OOZIE_URL, for example, `export OOZIE_URL=https://axonitered-oozie.red.ygrid.yahoo.com:4443/oozie/`.

- submit the oozie job using `oozie job -run -config job.properties -auth KERBEROS`
1 change: 1 addition & 0 deletions src/main/resources/oozie/spark_hdfs_lr/job.properties
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
nameNode=hdfs://axonitered-nn1.red.ygrid.yahoo.com:8020
jobTracker=axonitered-jt1.red.ygrid.yahoo.com:8032
wfRoot=spark_hdfs_lr
sparkTag=spark_latest
oozie.libpath=/user/${user.name}/${wfRoot}/apps/lib
oozie.wf.application.path=${nameNode}/user/${user.name}/${wfRoot}/apps/spark
6 changes: 3 additions & 3 deletions src/main/resources/oozie/spark_hdfs_lr/workflow.xml
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@
<configuration>
<property>
<name>oozie.action.sharelib.for.spark</name>
<value>spark_latest</value>
<value>${sparkTag}</value>
</property>
</configuration>
<master>yarn</master>
Expand All @@ -32,7 +32,7 @@
<configuration>
<property>
<name>oozie.action.sharelib.for.spark</name>
<value>spark_latest</value>
<value>${sparkTag}</value>
</property>
</configuration>
<master>yarn</master>
Expand All @@ -53,7 +53,7 @@
<configuration>
<property>
<name>oozie.action.sharelib.for.spark</name>
<value>spark_latest</value>
<value>${sparkTag}</value>
</property>
</configuration>
<master>yarn</master>
Expand Down

0 comments on commit 6b00df2

Please sign in to comment.