Create statistics

[davide-f]

Closes # (if applicable).

Changes proposed in this Pull Request

This PR aims at creating a new rule (make_statistics) that create statistics on the overall workflow.
This PR requires changes from #543 so first the #543 must be reviewed and merged and then this one can follow.

Checklist

• I tested my contribution locally and it seems to work fine.
• Code and workflow changes are sufficiently documented.
• Newly introduced dependencies are added to envs/environment.yaml and envs/environment.docs.yaml.
• Changes in configuration options are added in all of config.default.yaml and config.tutorial.yaml.
• Add a test config or line additions to test/ (note tests are changing the config.tutorial.yaml)
• Changes in configuration options are also documented in doc/configtables/*.csv and line references are adjusted in doc/configuration.rst and doc/tutorial.rst.
• A note for the release notes doc/release_notes.rst is amended in the format of previous release notes, including reference to the requested PR.

[davide-f]

This PR is ready for preliminary reviews but not for complete merge as it requires #543 to be merged first (some commits seen here are those from #543)

[davide-f]

@pz-max I think this PR is ready for review.
It contains several improvements that may be better merged and work on them later, yet appropriate review is needed

[pz-max]

@davide-f happy to do that. I see several good improvements and great code (as always)
Do you think we could create a small notebook in the documentation repo to demonstrate the make_statistic stuff?

I am thinking about a super small notebook:

• instructions how to modify the config tutorial/ test case for Nigeria
• instructions what command to use to run the whole stats stuff
• some tables with the stats maybe with little explanations

This would also help review a lot

[davide-f]

I'd leave that for a subsequent PR maybe later on when this scenario generation and statistics is fully stable and validated with the paper.
What do you think?

[pz-max]

We agreed:

• I do a code review
• @davide-f adds a docstring on use
• we add an issue in the documentation repo that aims to add a jupyter notebook on the make statistics stuff

[pz-max]

Looks good to me. Main comments:

• check if new created _mock_snakemake function can be avoided by passing snakemake objects over the function
• PEP 257
• Optional, little os to Pathlib switch at some locations

[davide-f]

@pz-max if you like to test the complete run_all_scenarios.
You make clone this PR and run:
snakemake -j1 run_all_scenarios
I've added a test case for nigeria in the configs/scenarios so to be used as an example

Be aware that this will run the default configuration though

[pz-max]

@pz-max if you like to test the complete run_all_scenarios. You make clone this PR and run: snakemake -j1 run_all_scenarios I've added a test case for nigeria in the configs/scenarios so to be used as an example

Be aware that this will run the default configuration though

• Probably nice to make it run the tutorial or not? (for testing etc)
• Checking the CI, the test needs to be adapted

Ready to merge if all tests are passing

[davide-f]

@pz-max if you like to test the complete run_all_scenarios. You make clone this PR and run: snakemake -j1 run_all_scenarios I've added a test case for nigeria in the configs/scenarios so to be used as an example
Be aware that this will run the default configuration though

• Probably nice to make it run the tutorial or not? (for testing etc)
• Checking the CI, the test needs to be adapted

Ready to merge if all tests are passing

You may try locally to run it with the tutorial. I never tested it but it may work.
Just:

1. copy the tutorial config
2. run snakemake -j1 run_all_scenarios

the default configuration file that is used as a base case for the run_scenarios is automatically chosen using the tutorial flag.
So it may work as is now.

Currently, I have the folder filled with world data in default configuration, I cannot mess it up...

[davide-f]

The CI works and there the run_scenario is already implemented