Proposal: Lazy statepoint loading

[bdice]

Feature description

I am experimenting with refactoring statepoint loading to be lazy. Currently, project.open_job() is called on every job during a flow run command, which causes the statepoints to be loaded. However, the statepoint information isn't even used in many cases. This may be a case where signac is executing unnecessary I/O (which is costly for large projects like the ones I'm working with).

Proposed solution

The property job.statepoint currently just returns job._statepoint, meaning that the internal state is effectively identical to the publicly-accessible state. I am investigating making the property job.statepoint load lazily, with job._statepoint = None until the statepoint is requested.

I see a HUGE performance boost (from 30 seconds to <0.5 seconds for a simple operation on 3000 jobs) since the I/O is dropped dramatically, but some tests fail because job statepoint corruption (which raises a JobsCorruptedError) is not detected at the time of project.open_job(). I think that lazy loading is generally safe except in cases where the statepoint hash and job id don't match (corrupted jobs).

I just wanted to open this for discussion - I am not sure where we stand on approaches to handling corrupted data. Specifically, I want to know whether we think that validating hash(statepoint) == id (and thus the I/O cost of a statepoint load) is always necessary when opening a job by id.

Additional context

Maybe investigate async as an alternative...? Still expensive but maybe less so.

[bdice]

We discussed this offline (I can't remember who was a part of the discussion). Our conclusion was that loading a job by id should not require validating that hash(statepoint) == id. A user (or I/O error) may corrupt the data space by breaking that invariant, and the user will still be allowed to open the job by id. In the proposed lazy-load of statepoints in #239, job corruption will be checked on accessing job.statepoint. (Note that this may not reflect the current design in that PR.) @glotzerlab/signac-committers Was this a conclusion you recall and/or agree with?

[csadorf]

I am ok with lazy loading and only validating the state point metadata when it is accessed.

[b-butler]

@bdice I don't remember being a part of the conversation, but I agree with the approach.

[vyasr]

@bdice and I did discuss and agree on this approach. I need to post my pseudo-prototype for #249 along with a class diagram so that we can progress on that front. IMO that serves as the cleanest path forward for implementing lazy loading by isolating the logic for synchronization and ensuring that a clear set of invariants are well-defined and well-tested for the different possible use-cases of nesting and buffering.

[bdice]

@vyasr I think (but am not sure) that my intended implementation of lazy loading can be clearly separated from your work on #249. I may need to work on #239 a little more to be sure, but I believe my implementation will rely more heavily on the Project class's caching and opening of jobs than the specifics of how job state points / job documents are synchronized. You should feel free to go ahead and post your "pseudo-prototype" (😉) in the meantime.