Discuss how we organise steps

[lewisjared]

The problem

In downstream packages, notebooks/config/steps are spread across different parts of the codebase. This makes it harder to find related code and relies heavily on naming to guide developers around the codebase. Naming stuff is hard and will probably change over a project's lifetime, which is hard to resolve with the current suggested folder structure.

Organising by steps (rather than by type) might make sense. I think it is closer to the mental model of the problem that library users are trying to solve. I've found that for application code, you are more likely to reuse a whole step (feature) in another project and this is particularly true for notebooks. By lumping all of the required notebooks/config into a single location (directory structure TBD) it makes it easier to reuse these components.

I don't think the folder structure has a direct impact on code in this project as I don't think we enforce any structure, so this is more related to "best practice".

Definition of "done"

• Try with a basic example
• Discuss

Additional information

Django structures things by feature as well. It isn't universally loved, but it does make it easier to find/reuse code.

Django project structure

my_project/
├── manage.py
├── core/
│   ├── __init__.py
│   ├── asgi.py
│   ├─ settings.py
│   ├─ urls.py
│   ├── wsgi.py
├── my_app/  << - a project might contain many "apps" (one per feature)
    ├── migrations/
    │   └── __init__.py
    ├── __init__.py
    ├── admin.py
    ├── apps.py
    ├── models.py
    ├── tests.py
    ├── urls.py
    └── views.py