Implement the concept of 'finish status' for Calculations #1189
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
sphuber
force-pushed
the
fix_1065_process_finish_status
branch
from
1a19a9e
to
710dc02
Compare
Also cleaned the code a lot and used class variables for static key values, such as the updatable attribute keys
The _del_checkpoint unrightfully expected an argument while it does not need one. Fixed the problem and added tests for the AiiDAPersister which writes yaml'ed process bundles to the attribute table of the calculation node and is currently the default persister used
In the new Process paradigm, all Calculations function simply as records of what happened during execution of the process. The state of the process is inscribed on the calculation at every state change, so that it can be reached by proxy through the node. The terminal process state FINISHED means that it executed nominally and was not killed nor did an exception occur. However, calculations may want to still distinguish in this case between a successful and unsuccessful result. This is now facilitated by the 'finish status'. If a process finishes nominally, the return value will be interpreted as an exit code and set as the finish status, a new attribute for the calculation classes. The properties 'is_finished_ok' and 'is_failed' now allow a user to distinguish between a successful and an unsuccessful finished process. This is currently implemented for InlineCalculation and WorkCalculations. Since inline and workfuntions either return successfully or except, their finish status will always be 0, so if they are FINISHED they will always necessarily be FINISHED_OK. For WorkChains however, this new schema allows users to exit the workchain with a specific non-zero error code to indicate that it failed, while still running through nominally. This can be achieved by simply returning a non zero value from within an outline function. It can also be done in the outline directly by using the 'return_' instruction. By default this will pass a zero return value, but a non-zero value can be set by calling it, e.g. as 'return_(255)'. This will set the finish status of the work calculation node to 255 and will cause 'is_failed' to return false
With the new unified process state attributes for all Calculations the old attribute keys that are excluded in the hashing procedure need to be updated. Since all attributes that are in the _updatable_attributes tuple will already be considered very often no explicit ignored hash attributes have to be added.
sphuber
force-pushed
the
fix_1065_process_finish_status
branch
2 times, most recently
from
6c96a72
to
f7966bb
Compare
The JobCalculation already has a sort of finish status, the terminal states of the DbCalcState table, i.e.: * SUBMISSIONFAILED * RETRIEVALFAILED * PARSINGFAILED * FAILED The idea is to map these calc states to a fixed finish status integer and have the JobProcess layer set it according to the calculation state, which remains leading. The FAILED status was set when the whole calc execution was successful, from submission, through retrieval, up until successful parsing, but the result of the calculation itself was not considered to be successful, which will vary per calculation. We extend this paradigm to allow the parsing function to return any non-zero positive integer to mean a specific calculation failure mode, which can then be defined on a per calculation plugin basis and used as error codes. Support for extending the basic enumeration for JobCalculations the JobCalculationFinishStatus is not yet implemented, and therefore until that is done, any non standard integers cannot be mapped onto a error label but instead will show UNKNOWN.
sphuber
force-pushed
the
fix_1065_process_finish_status
branch
from
f7966bb
to
88cb869
Compare
The updating of the scheduler state was predicated on a previous state being known if it was not yet finished, which meant that for jobs that would finish after the first query, would never have their scheduler state updated
muhrin
approved these changes
Feb 27, 2018
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #1065
In the new Process paradigm, all Calculations function simply as records
of what happened during execution of the process. The state of the process
is inscribed on the calculation at every state change, so that it can
be reached by proxy through the node. The terminal process state FINISHED
means that it executed nominally and was not killed nor did an exception
occur. However, calculations may want to still distinguish in this case
between a successful and unsuccessful result.
This is now facilitated by the 'finish status'. If a process finishes
nominally, the return value will be interpreted as an exit code and set
as the finish status, a new attribute for the calculation classes.
The properties 'is_finished_ok' and 'is_failed' now allow a user to
distinguish between a successful and an unsuccessful finished process.
This is currently implemented for InlineCalculation and WorkCalculations.
Since inline and workfuntions either return successfully or except, their
finish status will always be 0, so if they are FINISHED they will always
necessarily be FINISHED_OK.
For WorkChains however, this new schema allows users to exit the workchain
with a specific non-zero error code to indicate that it failed, while still
running through nominally. This can be achieved by simply returning a non
zero value from within an outline function. It can also be done in the
outline directly by using the 'return_' instruction. By default this will
pass a zero return value, but a non-zero value can be set by calling it,
e.g. as 'return_(255)'. This will set the finish status of the work
calculation node to 255 and will cause 'is_failed' to return false