--module-only --force messes with the reprod folder

[mboisson]

When running eb with the options --module-only --force, it recreates the easyconfig stored in the reprod folder. Unfortunately, this means that this easyconfig may actually no longer be reproducible.

This happened to me with GDAL, when the start_dir that is listed in the EasyConfig in the reprod folder went from

start_dir = '%(builddir)s/%(namelower)s-%(version)s/'

to

start_dir = '%(builddir)s/'

This is because the EasyBlock updates its config, and the full EasyBlock is not being run.

[mboisson]

@boegel would it make sense to you to not over-write the reprod/*.eb if --module-only is used ?

[boegel]

@mboisson It probably does, but I guess it also depends: you may have made a change in the easyconfig file to make the sanity check pass for example. Not including that in the easyconfig file in reprod is also problematic...

cc @ocaisa

[mboisson]

Investigating more related problems, I am thinking that saving start_dir in the reprod file is in itself problematic, independent of --modules-only.

Grepping start_dir through easyconfigs, they are always relative paths. Yet, when saving the easyconfig into the reprod folder, this has become an absolute path. This is bound to cause trouble.

I think the concept of start_dir needs to be split into two :

1. The easyconfig, relative start directory
2. The absolute start directory where things are starting to get built from

The two concepts should be saved separately in the easyconfig.

[mboisson]

I would say we should also avoid dumping exts_filter

When installing Platypus, I get the following error :

== sanity checking...
ERROR: Traceback (most recent call last):
  File "/cvmfs/soft.computecanada.ca/easybuild/lib/python2.7/site-packages/easybuild-framework/easybuild/main.py", line 126, in build_and_install_software
    (ec_res['success'], app_log, err) = build_and_install_one(ec, init_env)
  File "/cvmfs/soft.computecanada.ca/easybuild/lib/python2.7/site-packages/easybuild-framework/easybuild/framework/easyblock.py", line 2860, in build_and_install_one
    result = app.run_all_steps(run_test_cases=run_test_cases)
  File "/cvmfs/soft.computecanada.ca/easybuild/lib/python2.7/site-packages/easybuild-framework/easybuild/framework/easyblock.py", line 2776, in run_all_steps
    self.run_step(step_name, step_methods)
  File "/cvmfs/soft.computecanada.ca/easybuild/lib/python2.7/site-packages/easybuild-framework/easybuild/framework/easyblock.py", line 2642, in run_step
    step_method(self)()
  File "/cvmfs/soft.computecanada.ca/easybuild/lib/python2.7/site-packages/easybuild-easyblocks/easybuild/easyblocks/generic/pythonpackage.py", line 538, in sanity_check_step
    (success, fail_msg) = super(PythonPackage, self).sanity_check_step(*args, **kwargs)
  File "/cvmfs/soft.computecanada.ca/easybuild/lib/python2.7/site-packages/easybuild-framework/easybuild/framework/extensioneasyblock.py", line 131, in sanity_check_step
    (sanity_check_ok, fail_msg) = Extension.sanity_check_step(self)
  File "/cvmfs/soft.computecanada.ca/easybuild/lib/python2.7/site-packages/easybuild-framework/easybuild/framework/extension.py", line 173, in sanity_check_step
    cmd = cmd % template
KeyError: 'pyver'

on this bit :

exts_filter = (
    '/cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/python/%(pyver)s/bin/python -c "import %(ext_name)s"',
    '',
)

[boegel]

If I'm not mistaken #2664 takes care of these issues... Is that right @ocaisa?

[ocaisa]

Yes #2664 will address all of these problems by dumping the easyconfig after any modifications from the configuration but before any of the easyblock steps. It will also dump the easyblocks and the hooks alongside it.

[ocaisa]

@mboisson Can you clarify this:

Grepping start_dir through easyconfigs, they are always relative paths. Yet, when saving the easyconfig into the reprod folder, this has become an absolute path. This is bound to cause trouble.

In my case I'm not seeing any absolute paths in the reprod easyconfigs, they are still relative. Is there a case where they are not?

[boegel]

@ocaisa See guess_start_dir in easyblock.py, when --module-only is used, start_dir is set to '' initially, which eventually lead to it being set to self.builddir which is an absolute path.

[mboisson]

As Kenneth said, this is when you use --module-only. This happened to me with GDAL.

[ocaisa]

Ok, understood, after cleaning up a develop branch merge in #2664 this should now be fixed there.

[boegel]

Closing this since #2664 is merged, should be fixed (I checked quickly myself with --module-only, worked fine).