Should we allow users provide custom session_id

[noklam]

Description

This was originally a thread from Discord

Is your feature request related to a problem? A clear and concise description of what the problem is: "I'm always frustrated when ..."

When using an orchestrator like Airflow, kedro nodes are run with multiple sessions and it is hard to organise the output dataset since all of them have different timestamps.

Context

(Edited on Jun16)

Use Case

1. A id to organize for multiple runs with Orchestrator, a "logical run" id for all KedroSession. Logging of all I/O related to a particular session ID
2. Add metadata for the session_id for various logging purpose

Possible Implementation

1. Do we need session_id in KedroSession.create()?
2. Do we allow user to customize session_id other than the default timestamp? (Joel has a suggestion that allow user to add suffix so it won't change the order but give more flexibility for user to add metadata.)
3. Is save_version = session_id still valid? Parametrized session_id #1571
4. (Bonus) - KedroSession.reset() to create a new session easily? - this can potentially make the Jupyter workflow nicer. Instead of asking user to create their session with lots of details, they can just take the global session and do session.reset() Parametrized session_id #1571

(Optional) Suggest an idea for implementing the addition or change.
workarounds or changes in kedro framework side.

Possible Alternatives

(Optional) Describe any alternative solutions or features you've considered.

[datajoely]

So it looks simple enough to make this a parameter of the create method...

Is the question that we need to ensure that custom session_ids are sorted correctly as otherwise it will ruin versioning?

I think this has a lot of user value, both Airflow and Prefect users have raised this.

[noklam]

@datajoely Implementation-wise should be simple, but I remember the team discussed related topics when we have the 1 session = 1 run changes 2 months ago. There are probably more discussions before that, so I don't have the full context.

Personally, I think this is a very practical use case.

Is the question that we need to ensure that custom session_ids are sorted correctly as otherwise it will ruin versioning?

Yes for this too, we need to ensure they are monotonic to make sure versioning works.

[datajoely]

I wonder if there is any merit in a prefix and suffix structure where the prefix is Kedro generated, but the suffix can be provided by the user to manage groups etc.

[noklam]

Description

Hello! I know, that you have In a backlog discussion related to parametrized session_id and as a user I would like to add mine extra requirements.

1. As a user I want to specify (entire or prefix or suffix) of session_id as parameter in Kedro CLI
2. As a user I want to log all data I/O operations within scope of particular session_id

(entire or prefix or suffix) - This part depends on your decision about how you will manage it

Why as user I would like to have this feature

1. This feature would help to link node & pipeline logs with logs produced from other components in my system when I execute pipelines using kedro run command
2. Awareness about session_id inside of functions of DatasetSpecs class would help to distinguish to which kedro session I/O operation does belong inside of logs. But I don't about degree of coupling kedro striving between KedroSession and DataCatalog classes, so I think safest is to make session_id optional to keep lowest

Importance

1. Not critical, nice-to-have: I consider feature as nice-to-have because, I can store extra details of KedroSession as environment variable, and refer to them when I log something.
2. Not critical, nice to have: I consider feature as nice-to-have because, I the same as above I can refer to env variable

Thanks for your support

@atmosone I have copied your comments here so we can track it in 1 issue, hope this is fine.

[noklam]

Updated the issue to facilitate future tech design session.

[antonymilne]

Just so we have all this in one place - this is very relevant: #1273

[merelcht]

Discussed in Technical Design on 27/7

Discussion

As part of the discussion we talked about the following points:

• The purpose of the session_id in Kedro at the moment is mainly to link session data (run metadata) to versioned datasets for the experiment tracking view in Viz.
• The session_id==save_version and we want to keep it that way.
• It needs to be made very clear to users that when they choose a custom session_id it needs to be chronologically sortable, because this id will also be used for versioning of the datasets (see point above).
• Does this influence how experiment tracking works on distributed versions Kedro? Probably not, and the suspicion is that this currently doesn't work properly anyway.

Conclusion

The outcome of the discussion was that we will allow users to provide a custom session_id.

Implementation

• Make session_id customisable. Add a function to generate the session_id that users can then overwrite. Also update the docs to make it very clear what the recommended format and properties of this id are.
• Update kedro-airflow to take in the custom session_id

[merelcht]

Closing this issue in favour of the action points mentioned above.

[datajoely]

I would add that we should update the Prefect deployment guide too