Create "base dataset" with default values

[marliekeverweij]

Currently when you add new interface elements to ETLocal, you have to add values for these new interface elements to ALL datasets through a migration. In many cases these values are the same for all datasets.

@michieldenhaan and I had the idea to add a "base dataset" to ETLocal where we could add these values to. The values in the "base dataset" will be used as default values in datasets when no custom value is given / the fields are empty.

Advantages:

• this would make updates a lot easier: then we only have to add the values to one dataset
• probably also saves some space in the database

Characteristics of this base dataset:

• it can be one dataset, doesn't have to differ between different countries -> right @michieldenhaan ?
• should be able to adjust for modellers, but doesn't have to be visible in the front-end of the ETM Dataset Manager to all users

[michieldenhaan]

it can be one dataset, doesn't have to differ between different countries

I think it could be useful to have different base datasets per country (in line with the parent dataset attribute currently already in the database for each region). This is useful when setting country-specific (but not subregion specific) data such as plant efficiencies, cost attributes etc.

but doesn't have to be visible in the front-end of the ETM Dataset Manager to all users

To clarify: Users don't necessarily have to be able to 'open' the base dataset, but users should be able to see when a derived dataset uses a default value from the base dataset (instead of showing an empty field)

[marliekeverweij]

@ChaelKruip @noracato @antw it would be nice if this issue could be picked up. For example: we will be adding efficiencies for many power and heat plants to all datasets on ETlocal in two weeks. It would be great if we would have the base dataset before then :)

[antw]

I do not think adding a base dataset would be too difficult, but agree with @michieldenhaan that a single global base dataset is probably not the way to go. Supporting a couple of base datasets – at least one per country – would be a better approach.

I think we'd be better off doing this carefully and thoughtfully, with consideration for how we might want to support country datasets in ETLocal, and at the same time simplify existing datasets by removing any values that are actually not customised. I worry that half-assing a "base dataset" feature will come back to bite us later when we want to add full datasets to the app.

in two weeks

Though this is not ultimately my call, I think this is unlikely. To me it has the air of a nice-to-have.

[ChaelKruip]

I do not think adding a base dataset would be too difficult, but agree with @michieldenhaan that a single global base dataset is probably not the way to go. Supporting a couple of base datasets – at least one per country – would be a better approach

I agree

I think we'd be better off doing this carefully and thoughtfully, with consideration for how we might want to support country datasets in ETLocal, and at the same time simplify existing datasets by removing any values that are actually not customised. I worry that half-assing a "base dataset" feature will come back to bite us later when we want to add full datasets to the app.

Agree!

Though this is not ultimately my call, I think this is unlikely. To me it has the air of a nice-to-have.

I'm not the authority here. @AlexanderWirtz can you chip in?

[AlexanderWirtz]

As far as @antw 's considerations are concerned, I defer to his and Chael's better judgements. I do not yet understand, however, why one base dataset per country would be a better approach. Having said that, I defer to your better judgments.

I'm not the authority here. @AlexanderWirtz can you chip in?

We have a project due for Northern Ireland in a few weeks in which we enable the power plants, CHP and heat plant efficiencies to be set via de ETM Dataset Manager. I don't know if this means this will then have to be done for all ETMDM datasets or just NI. If the former, then it would be really nice to have some default global option, at some point. We cannot possibly maintain it for each dataset otherwise.

[marliekeverweij]

Thanks for all your input!

@antw: I think we'd be better off doing this carefully and thoughtfully, with consideration for how we might want to support country datasets in ETLocal, and at the same time simplify existing datasets by removing any values that are actually not customised. I worry that half-assing a "base dataset" feature will come back to bite us later when we want to add full datasets to the app.

I agree with doing it carefully, we should not rush it.

@michieldenhaan: I think it could be useful to have different base datasets per country (in line with the parent dataset attribute currently already in the database for each region). This is useful when setting country-specific (but not subregion specific) data such as plant efficiencies, cost attributes etc.

I think that having a base dataset for each country is a good approach and will not bite us later when supporting country datasets in ETLocal.

@AlexanderWirtz: We have a project due for Northern Ireland in a few weeks in which we enable the power plants, CHP and heat plant efficiencies to be set via de ETM Dataset Manager. I don't know if this means this will then have to be done for all ETMDM datasets or just NI.

This issue doesn't have to be fixed for this improvement, we could do it how we're used to do it (update all datasets!).

The reason why I raised this issue is that I heard that @noracato has some time on her hands the coming weeks and this would be really nice to be picked up! It streamlines and speeds up the modellers workflow

[noracato]

What would be the set values in these base datasets? Are these the same exact values as the eventual country dataset would have? Then, to me this issue sounds a lot like a first step in support for country datasets.

If that's the case, I propose the following. All regions will inherit missing values from their parent, the country dataset. The country dataset is viewable and editable in the Dataset Manager only by admins. We won't yet add any support for exporting to ETSource, or for layers etc, but we do keep all of this in mind and create possibilities to (easily) add these features later. We would need a clear view of what a country dataset should be, and which features will be neccessary in the future.

If we do want support for multiple base sets per country this plan of course doesn't go.

[michieldenhaan]

This sounds perfect @noracato!

If we do want support for multiple base sets per country this plan of course doesn't go.

We most likely do want this, but only one per country/year combination. So preferably the Dutch regional 2018 datasets are all linked to/inherit data from a nl2018 base dataset, the 2019 datasets from a nl2019 dataset etc.

[github-actions]

This issue has had no activity for 60 days and will be closed in 7 days. Removing the "Stale" label or posting a comment will prevent it from being closed automatically. You can also add the "Pinned" label to ensure it isn't marked as stale in the future.

[mabijkerk]

@noracato let's evaluate this idea within the upcoming datasets project.

[noracato]

Shall we create a label or milestone for the dataset project where we can tag all issues we want to remember for the upcoming project?