Is it possible to except a workflow when one element from the context fails?

[JPchico]

Hello!

I'm wondering, I have built a workflow in which several calculations are submitted internally via a loop. It is basically a custom EOS plugin with some bells and whistles.

However, in the code I'm using there is the possibility that one calculation (out of the many sent by the loop) ends in an irrecoverable failure state (i.e. the BaseRestartWorkchain cannot handle the issue) which in turn will result in the whole workchain failing (when collecting the results the workchain will try to find something that is not there).

So I wonder is it possible to kill the workchain in case of one of these calculations submitted to the context fails, before waiting for all the other ones to finish?

Or would you solve the problem in a different manner? How do you normally handle cases like this?

Thanks!

[mbercx]

Hi @JPchico! Thanks for your question, and happy to see you've found your way to the GitHub Discussions. 😃

Would it be possible to share the work chain you are working on somehow? It's probably easier if I can have a look. To be honest though, @sphuber is probably best suited to answer your question 😅, and he's AFK at the moment.

[sphuber]

There is nothing built into aiida-core that let's you instruct the workchain to terminate itself as soon as one of the "awaitables" excepts, is killed or terminates with a specific exit code. One could think of adding such a thing, but it would require a change in the current interface. One option would be to allow specifying a callback when adding an awaitable to the context, for example:

def submit_sub(self):
    for i in range(5):
        sub = self.submit(SomeProcess, **inputs)
        self.to_context_new(f'sub_{i}', sub, callback=self.check_sub)


def check_sub(self, sub_node):
    if sub_node.is_excepted or sub_node.is_killed or sub_node.exit_code.status < 400:
        return self.exit_codes.SUB_PROCESS_FAILED

The new method to_context_new would save the callback check_sub and as soon as a sub process terminates, that callback is called, which then gives the opportunity to check the result of the sub process and abort the workchain if needed.

The one thing missing here is that the other subprocesses are not killed. It is not clear if this should be done automatically (maybe in some cases you actually want to keep them running) so maybe the user should do this manually in the callback. But in that case you need the Process instance, and we only have the ProcessNode reference in the context. Currently there is no clear way in the API to get a Process from the ProcessNode and one can only kill a process through its Process instance.

TLDR: there is nothing built-in, but if thought to be useful, some support could be built into aiida-core but it would require some work and new API

[JPchico]

Thanks @sphuber and @mbercx! Sorry for not answering before I did not see the answer.

That sounds quite interesting, I think that it is something that could be useful, as sometimes this is something that can happen particularly in GF based codes, where you could run into a situation in which a very specific set of parameters will not work for the whole volume range and this could result in the entire workchain failing. I wonder if there are any other situations where this kind of problem could pop up.