Getting a 404 error from Lighter on expired sessions but we expect it to start a new session instead when we run a query .

[HarshithaRmAkamai]

We are using Z2JH with sparkmagic kernel and Lighter to create Spark sessions.

We have configured LIGHTER_SESSION_TIMEOUT_INTERVAL with a value of "30m", which causes the Spark driver pod to be terminated by lighter after 30 minutes of inactivity which it is doing as expected.

However, when we attempt to return to the same session once the session is killed by lighter and run a query, we encounter the following error
An error was encountered:
Invalid status code '404' from with error payload: {"message":"Not Found","path":"/lighter/api/sessions/24d35846-0abf-4ea2-ba1b-2558e021ea5b/log?from=0","errors":[{"message":"Page Not Found","title":null,"path":null}]}

Our expectation is that Lighter should clear all caches and start a new session when user comes back and tries to run a query in notebook to ensure the user experience is smooth, without requiring manual intervention to start a new session after timeout.

So Could you please provide some insights on this issue?