Separate training from inference more cleanly in Spaceflights?

[astrojuanlu]

Today I wanted to do a inference-only run with a Spaceflights project (see #3856) and I had to resort to this:

1. Declare X_test, y_test in the catalog (and found a scikit-learn bug along the way Investigate why Spaceflights project failing with ParallelRunner  #3674 (comment))
2. Training: kedro run --to-outputs=X_test,y_test,regressor
3. Actual inference: kedro run --from-nodes=evaluate_model_node

It would be good to rework our Spaceflights starters so that they follow an architecture that more closely resembles Hopsworks' FTI Pipeline model, which would in turn make it easier for users (and for ourselves) to see how could they

• Easily transition to a production-like setting
• Perform inference with another set of data that was not the training or evaluation data
• Serve the model they trained, with Kedro or something else (MLflow, Seldon Core, a custom FastAPI endpoint)

[DimedS]

Nice article, I agree with that