Commits on Sep 11, 2017

1. Prevent module shadowing in pyspark_runner.py …

   PySparkTask uses spark-submit to run the script pyspark_runner.py.
   Because it is run as a script the modules and packages from its
   directory (luigi/contrib/) shadow the global modules. In particular the
   package `hdfs` which is used by webhdfs_client.py is shadowed by the
   `luigi.contrib.hdfs` package. Thus PySparkTask does not work together
   with webhdfs.
   
   The problem is resolved by putting the current directory at the end of
   the path list `sys.path`.

   

   adaitche committed Sep 11, 2017
   Configuration menu

   • View commit details
   • Copy full SHA for 3744700
   • Browse repository at this point

   Copy the full SHA

   3744700 View commit details

   Browse the repository at this point in the history