path in batch/secondary service logs

[soxofaan]

GET /services/{service_id}/logs:

the log records have a path field, which is required and should reference process graph node_ids (and optionally process_ids). But what about logging from places that are not directly tied to a process graph node? For example: general secondary service aspects (start up, request handling, ...), the logic that transforms service parameters to process graph parameters, ...

[m-mohr]

An empty path would work, right? But indeed, it could also just be left out.

[soxofaan]

Indeed an empty path would work as workaround, but I think it should also be possible to provide an alternative way to describe the "location" of the log, e.g. just with a simple string (instead of a list of structured node_id objects).

In general (so also in the case of batch job logs), from the standpoint of the VITO backend implementation there is quite a mismatch between this structured node_id listing and what is feasible/possible for us. I guess it is also quite challenging for other backends to achieve that level of location granularity at the logging layer. Especially because the log handling can be very implementation specific and might involve attempts to parse out-of-band log files. So I am wondering if it wouldn't be better to keep the "location" property of a log (currently the path field) simple, like a single string

[m-mohr]

Indeed an empty path would work as workaround

I'll make the path optional.

but I think it should also be possible to provide an alternative way to describe the "location" of the log, e.g. just with a simple string (instead of a list of structured node_id objects).

The path is aiming to allow a precise mapping so that clients could actually point to the code/block where the error happend. It's a bit like a stack trace. The JS parser is able to track the (parent) node IDs, I'm curious why that's a problem in Python? All others can probably just put it in the message.

[soxofaan]

well, I'm not talking about errors, but logging in general. For example with the VITO geopyspark backend, we (try to) fetch the logs of Spark/YARN, which are loosely structured log files. So there is no real guarantee that we can extract some kind of process graph path from them; let alone the logging messages are related to a process graph in some way.

So making the path field optional would be indeed useful in this context.

Another possible improvement would be adding a field to describe some kind of location/origin/source, so we can flag for example that a log entry is coming from Spark, YARN logs or from our own Python/Scala/Java logic. Again, this is probably highly implementation specific, and thus hard to standardize

[m-mohr]

As of ddfde37 the path is no longer required.

There's probably room for improvement, but that feels like a API 1.1 addition once we have more implementations. I'm not sure yet how to best expose other information commonly across back-ends.