Prevent module shadowing in pyspark_runner.py

[adaitche]

Motivation

PySparkTask uses spark-submit to run the script pyspark_runner.py. Because it is run as a script the modules and packages from its directory (luigi/contrib/) shadow the global modules. In particular the package hdfs which is used by webhdfs_client.py is shadowed by the luigi.contrib.hdfs package. Thus PySparkTask does not work together with webhdfs.

An example task which highlights the problem can be found below:

import luigi
import luigi.contrib.hdfs
import luigi.contrib.spark


class MyTask(luigi.contrib.spark.PySparkTask):
    def output(self):
        return luigi.LocalTarget("not_present.txt")

    def main(self, sc):
        client = luigi.contrib.hdfs.webhdfs_client.WebHdfsClient("some_host")
        print(client.client)

Description

The problem is resolved by putting the current directory at the end of the path list sys.path in pyspark_runner.py.

Testing

I tested manually that this resolves the problem (for several weeks in production). I thought about adding a unit test for this but did not come with a good solution. The difficulty is that the problem occurs only when pyspark_runner.py is run as a script. But running the script necessarily tries to start spark by creating a SparkContext, which I guess is not desired for the unit tests. Suggestions on how to test this are welcome.

[dlstadther]

Seems reasonable to me. Mind blaming another pyspark contributor for confirmed review?

Thanks!

[adaitche]

@jthi3rry Could you take a quick look at the patch?

[jthi3rry]

LGTM @adaitche @dlstadther

[dlstadther]

Thanks @adaitche !