Skip to content

Commit

Permalink
remove export PYSPARK_PYTHON in spark submit
Browse files Browse the repository at this point in the history
  • Loading branch information
giwa committed Aug 18, 2014
1 parent 8efa266 commit 3a671cc
Showing 1 changed file with 0 additions and 16 deletions.
16 changes: 0 additions & 16 deletions bin/spark-submit
Original file line number Diff line number Diff line change
Expand Up @@ -37,22 +37,6 @@ done

DEPLOY_MODE=${DEPLOY_MODE:-"client"}


# This is a hack to make DStream.pyprint work.
# This will be removed after pyprint is moved to PythonDStream.
# Problem is that print function is in (Scala)DStream.
# Whenever python code is executed, we call PythonDStream which passes
# pythonExec(which python Spark should execute). pythonExec is used to call python.
# Since pyprint is located in DStream, Spark does not know which python should use.
# In that case, get python path from PYSPARK_PYTHON, environmental variable.

# Figure out which Python executable to use
if [[ -z "$PYSPARK_PYTHON" ]]; then
PYSPARK_PYTHON="python"
fi
export PYSPARK_PYTHON


if [ -n "$DRIVER_MEMORY" ] && [ $DEPLOY_MODE == "client" ]; then
export SPARK_DRIVER_MEMORY=$DRIVER_MEMORY
fi
Expand Down

0 comments on commit 3a671cc

Please sign in to comment.