Data Staging

[andre-merzky]

Jobs frequently require input data to be staged along with jobs. For local submission this is not as much of an issue as data can be pre-staged to the shared file systems by the user. For remote submission however, data staging should be an integral part of the job submission process. This thread can be used to discuss possible approaches for data staging support.

[andre-merzky]

API

Support for data staging could be rendered as a separate set of API calls:

jex = psij.JobExecutor.get_instance(schema)
jex.stage_in(src, tgt, flags)
jex.stage_out(src, tgt, flags)

It could also be rendered as part of the job specification, thus combining data staging and job submission into a single operation (from the API perspective):

spec = psij.JobSpec()
spec.executable = jd.executable
spec.arguments  = jd.arguments
spec.stage_in   = [[src, tgt, flags], ...]
spec.stage_out  = [[src, tgt, flags], ...]

Separate calls give better control to the application but also require more application logic (when to stage, coordinate with job sandboxes, react on job state changes, ...).
Staging as part of the job spec moves that logic into PSIJ (don't stage on cancellation or error, names relative to job sandbox, etc), but it limits the set of supported use cases (it's not possible to just stage data w/o job submission). Do we care about those use cases?

Both approaches could be supported, but that runs against a minimalistic API. Is minimalism important to us?

Implementation

In both cases, the implementation could be added to the implementation of the PSIJ executor. For all local executors (which all executor are in the moment), the implementation could likely be added to the executor base class or provided as auxiliary helper methods, using native python file and directory manipulation calls.

Semantics

The semantics we intent to support with data staging should be derived from the target use cases. Speaking for RCT, we would like the following set of operations to be supported:

• copy(src, tgt)
• link(src, tgt)
• support for files and directories
• target directories should get created on the fly

[hategan]

Both approaches could be supported, but that runs against a minimalistic API. Is minimalism important to us?

We had both in Swift/CoG. But they weren't quite different. There were job, transfer, and I/O providers. The job providers were similar to psi/j executors, whereas transfer providers handled "transfer jobs" and I/O providers were similar on top of a stateful I/O layer (e.g., a GridFTP session). You could use the I/O providers with jobs by adding stage-in and stage-out directives to the job spec.

That wasn't fully sufficient for Swift, since we had caching that avoided staging-in files that were already there, space management, which could put a cap on the total size of files that were stored remotely, etc. I imagine a realistic high-performance scenario will need the kind of flexibility that can only be achieved with a remote I/O layer similar to FTP.

But I don't think we should worry about that in the first two layers.

[andre-merzky]

We had both in Swift/CoG. But they weren't quite different.

Do you mean "were quite different"?

Either way, RCT also supports both approaches, even though our semantics is less sophisticated than Swift's (no caching, no caps).

But I don't think we should worry about that in the first two layers.

What exactly is 'about that' in this sentence - an FTP like layer or staging in general? If the first I agree. Simple staging calls w/o additional policies are relatively easy to implement (both as separate calls and as part of the job spec) and should likely cover a large part of use cases already.

[andre-merzky]

We had both in Swift/CoG. But they weren't quite different.

Do you mean "were quite different"?

Either way, RCT also supports both approaches, even though our semantics is less sophisticated than Swift's (no caching, no caps).

But I don't think we should worry about that in the first two layers.

What exactly is 'about that' in this sentence - an FTP like layer or staging in general? If the first I agree. Simple staging calls w/o additional policies are relatively easy to implement (both as separate calls and as part of the job spec) and should likely cover a large part of use cases already.

[hategan]

We had both in Swift/CoG. But they weren't quite different.

Do you mean "were quite different"?

No. You could put staging directives in the job, but, internally, it would end up using the transfer API.

Either way, RCT also supports both approaches, even though our semantics is less sophisticated than Swift's (no caching, no caps).

Fancy stuff was done by swift, not at the "Grid API" level. The point is that it might be hard to make a jobspec based staging scheme that can support random advanced stuff.

But I don't think we should worry about that in the first two layers.

What exactly is 'about that' in this sentence - an FTP like layer or staging in general? If the first I agree. Simple staging calls w/o additional policies are relatively easy to implement (both as separate calls and as part of the job spec) and should likely cover a large part of use cases already.

I'm referring to high performance, although what I really mean is high troughput.

[andre-merzky]

[...] You could put staging directives in the job, but, internally, it would end up using the transfer API.

Ah, I see - that makes sense.

Either way, RCT also supports both approaches, even though our semantics is less sophisticated than Swift's (no caching, no caps).

Fancy stuff was done by swift, not at the "Grid API" level. The point is that it might be hard to make a jobspec based staging scheme that can support random advanced stuff.

I agree. We still could cover the simple and common case via job spec staging directives and add more advanced stuff via dedicated subsystem.

But I don't think we should worry about that in the first two layers. [...] I'm referring to high performance, although what I really mean is high throughput.

I agree.

So my proposal would be to continue with the simplistic and limited approach as outlined above while being conscious that this will need expansion as our use cases advance. I am trying to get the basic case covered now for the following reason: PSIJ's REST and ZMQ service endpoints will get more use in RCT as we throw SAGA out over the next months. It is rather trivial to add the basic data staging to those service endpoints, and doing so would simplify our SAGA removal process.