Customize shebang 739 #940

lekah · 2017-11-20T13:31:59Z

This pull request resolves #739 by enforcing the customization of the shebang line for each computer.
For existing computers, the previous default is taken (#!/bin/bash)

…bang in the metadata of the computer for both backends

…from the computer instance as an attribute

sphuber

minor fix

sphuber · 2017-11-21T15:23:39Z

aiida/backends/sqlalchemy/models/computer.py

+        try:
+            return self._metadata['shebang']
+        except KeyError:
+            raise ConfigurationError('No workdir found for DbComputer {} '.format(


I think you forgot to change the exception message

giovannipizzi

A few additional comments:

we need to update the docs, telling what the shebang is, and what the default is
also in the docs we should say that the only supported ones are currently #!/bin/bash and #!/bin/bash -l; other shells might not work (e.g. for the way we escape commands, the way we set environment variables etc.)
once we merge we need to update also external repos relying on the number of inputs (there are a few docker files around, like aiida_docker_base, aiida_docker_compose, the ones in materials cloud, in the Quantum Mobile VM, ...)
I don't know if @DropD has been already working to a new verdi computer setup command with cmdline options, this has also to be updated. Probably at least these should be merged in develop so the external packages can use the command line options instead of a text file, and new (optional) options will not break external dependencies.

giovannipizzi · 2017-12-01T11:57:39Z

aiida/orm/implementation/django/computer.py

+
+    def set_shebang(self, val):
+        metadata = self._get_metadata()
+        # TODO: What are good checks?


A good check is that the first two characters are "#!" (they have to be); this would also implicitly check it's a string

giovannipizzi · 2017-12-01T11:59:17Z

aiida/orm/implementation/sqlalchemy/computer.py

-        #else:
-        #    raise ModificationNotAllowed("Cannot set a property after having stored the entry")
+
+    def get_shebang(self):


Can't this logic be put in the abstract class? It's a pity that it's copy-pasted, and also even more dangerously the default is duplicated. I think the function can just stay in the base class instead.

Don't think it should. The fact that there is a member called dbcomputer is a choice of the implementation. Some other backend could call it _dbcomputer or whatever, it's not enforced. As well as the function get_shebang, also not enforced. I would leave it where it is.

giovannipizzi · 2017-12-01T11:59:45Z

aiida/orm/implementation/sqlalchemy/computer.py

+            # This happens the first time: I provide a reasonable default value
+            return "#!/bin/bash"
+
+    def set_shebang(self, val):


Probably similarly for the other functions that are independent of the implementation, I would use the occasion to move them in the base class.

giovannipizzi · 2017-12-01T12:02:02Z

aiida/scheduler/__init__.py


        # I fill the list with the lines, and finally join them and return
        script_lines = []
-        script_lines.append(shebang)
+        script_lines.append(job_tmpl.shebang)


Can we put here a default? Something like: if job_tmpl.shebang is None: ... else: ...? I see you changed all plugins, which is good, but I think there are a few that are developed as external plugins and might not be aware of this change.

(maybe verifying that the default is None, and leaving the option for an empty string to be dealt differently, if a user does not want the shebang line at all? I don't know if there is ever this need, but one never knows). Something like elif job_tmpl == "": #dont print anything, not even an empty line

giovannipizzi · 2017-12-01T12:04:16Z

aiida/scheduler/plugins/test_pbspro.py

@@ -908,6 +908,7 @@ def test_submit_script(self):
        s = PbsproScheduler()

        job_tmpl = JobTemplate()
+        job_tmpl.shebang = '#!/bin/bash -l'


You put a -l as a way to verify the current implementation?
Maybe if you implement also the case with shebang not specified, you can have a copy of one of these tests to verify that even if you don't set it, you get the default value?

…nto customize_shebang_739

…er to provide a default if it's None or an empty string

… after the scheduler, which makes more sense than where it was before

…e. This is respected

sphuber · 2017-12-18T08:43:35Z

aiida/orm/implementation/general/computer.py

        if not val.startswith('!#'):
-            raise ValueError("A shebang line has to start with #!")
+            raise ValueError("{} is invalid. A shebang line has to start with #!".format(val))


Not sure if this was intentional irony, but you actually swapped the # and ! around in the conditional 😄

I think that if this is fixed, we still need @sphuber to accept the changes before merging?

giovannipizzi

I think you fixed Sebastiaan's comment, so it's good to merge for me

lekah added 5 commits November 17, 2017 10:27

First work on aiidateam#739, in which I make it possible to store she…

004c3b2

…bang in the metadata of the computer for both backends

Updated tests that were failing for aiidateam#739

8178cdb

for aiidateam#739 the JobTemplate instance now gets the shebang line …

3799315

…from the computer instance as an attribute

Updated the travis-test for the daemon for aiidateam#739

ccdc450

Updated scheduler test (aiidateam#739)

d9a5bf5

lekah requested a review from giovannipizzi November 20, 2017 13:32

Merge branch 'develop' into customize_shebang_739

ed612c1

sphuber reviewed Nov 21, 2017

View reviewed changes

lekah added 4 commits November 21, 2017 18:27

Minor fix for Configuration Error message in sqlalchemy's DbComputer

4a27bce

Merge branch 'develop' into customize_shebang_739

b224ee7

Merge branch 'develop' into customize_shebang_739

5050ccc

Merge branch 'develop' into customize_shebang_739

b2001da

giovannipizzi requested changes Dec 1, 2017

View reviewed changes

lekah added 12 commits December 15, 2017 15:43

Merge branch 'develop' into customize_shebang_739

e308ad0

Merge branch 'customize_shebang_739' of github.com:lekah/aiida_core i…

7206e89

…nto customize_shebang_739

For aiidateam#739 moved the set_shebang into the abstract class

6fc7daa

For aiidateam#739 I check the jobTemplate's shebang attribute, in ord…

0390233

…er to provide a default if it's None or an empty string

For aiidateam#739 I test the default shebang line

f112369

Added documentation for aiidateam#739

8c7c9b9

Changed the order of the computer setup so that shebang line is asked…

8ca5c64

… after the scheduler, which makes more sense than where it was before

Added new check if user specified an empty string for the shebang lin…

ff96a5c

…e. This is respected

Improved exception message when setting shebabgn

af7e3b6

Corrected small bag in check for shebang line

68758fc

Still related to changing the order of shebang and workdir

de8e5ef

Fixed docstring

6b08696

sphuber requested changes Dec 18, 2017

View reviewed changes

lekah and others added 2 commits December 18, 2017 19:14

Merge branch 'develop' into customize_shebang_739

49633da

Merge branch 'develop' into customize_shebang_739

1278e4d

giovannipizzi approved these changes Dec 20, 2017

View reviewed changes

sphuber approved these changes Dec 21, 2017

View reviewed changes

sphuber merged commit da5a653 into aiidateam:develop Dec 21, 2017

lekah deleted the customize_shebang_739 branch December 22, 2017 14:59

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Customize shebang 739 #940

Customize shebang 739 #940

lekah commented Nov 20, 2017

sphuber left a comment

sphuber Nov 21, 2017

giovannipizzi left a comment

giovannipizzi Dec 1, 2017

giovannipizzi Dec 1, 2017

lekah Dec 15, 2017

giovannipizzi Dec 1, 2017

giovannipizzi Dec 1, 2017

giovannipizzi Dec 1, 2017

giovannipizzi Dec 1, 2017

sphuber Dec 18, 2017

giovannipizzi Dec 20, 2017

giovannipizzi left a comment

Customize shebang 739 #940

Customize shebang 739 #940

Conversation

lekah commented Nov 20, 2017

sphuber left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

giovannipizzi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

giovannipizzi left a comment

Choose a reason for hiding this comment