Setup #1
Setup #1
Conversation
There was a problem hiding this comment.
Awesome stuff. This is looking very professional. pydoit-nb is starting to get conceptually busy so clarifying some of the terms that we use will be pretty important soon. I still don't have a clear definition of Branch in my head (is it simply a collection of notebooks or is it more in line with running multiple scenarios?).
I like the splitting of tasks and config
I could checkout and run successfully
| – put notebook number at start of saved data file, makes it easier to find where it came from later | ||
| – make clean data and plots in same notebook first (fast iteration), but make sure you go back and split in follow up step (fast iteration for plots) | ||
| - config gets hydrated and written to disk so put sensitive information in environment variables | ||
| - config class plus cattrs is way of getting around awkwardness of passing things in memory (pass everything via serialising to disk which makes things a bit slower and more IO intense, but also way simpler, put note that if your setup does heaps of IO for the config, this solution may not work for you/you may need to work out how to pull things out of the config into somewhere else) |
There was a problem hiding this comment.
I'd be super surprised if this was a limitation unless you were running each magicc instance as a separate notebook step. Have this been an issue in the past?
There was a problem hiding this comment.
I think it's helpful to have in your head so you realise that you probably don't want everything in the config object. In the carbon budget crunching stuff, I was just serialising ScmData objects to yaml but I don't think that's very smart. I think having a note somewhere that the config class gets serialised as yaml, and that might not always be the smartest choice (e.g. it's often better to serialise ScmData to netCDF and then just pass the name of the config file around via yaml rather than the full data itself), can be helpful.
|
@JGuetschow tagging you here, I can add you as a reviewer once you join the CR GitHub group |
|
@lewisjared @JGuetschow @mikapfl last chance for any comments or follow up here. Otherwise I'll merge Monday and we can deal with issues/refactorings in a more narrow context. |
I think you showing it and our discussion based on it was super useful for me. This PR here is a lot of code, so I didn't actually look into it. Any more detailed feedback from me would probably happen after I worked with this setup. |
Very fair. I hope to get you working with it in the week of December 5 just as fyi (only checking things run though, not actually doing development) |
Sets up the repository and adds a basic pipeline that can be used for testing, refactoring etc.