How do we communicate a warning/notice of eia early release data?

[cmgosnell]

If we are to integrate early release data (draft releases of full years) into pudl, how do we want to communicate the fact that the new year of data is early release? Some options:

• add a logger/warning (easy but will not help datasette users)
• add info into data set level metadata (this will show up in datasette web interface, but once - [ ] users pull the data down, they could very easily forget)
• add some flag in a/any table w/ early release data?

[cmgosnell]

There is a RIDICULOUSLY long-named column in the early release data (early_release_data_june_2022_not_fully_edited_use_with_caution_do_not_aggregate_to_state_regional_or_national_totals).

If we went with the latter option of adding a flag, I could imagine using this column or generating another column altogether. We could use this for the eia860m data as well. iirc we are not reeeeeally communicating that the newest 860 year is a monthly release. in part bc it is of annual-type data (i.e. capacity_mw not net_generaiton_mwh)

[zaneselvans]

data_label seems too generic. That could be like... tags or keywords or something.

What are we really trying to convey here? We have DataSource already defined in our little universe, but now it turns out there's a subdivision within DataSource, such that the same information is being provided in the same structure, but at a different level of maturity? and what is this new dimension? Data maturity? Data version? Release version?

What about data_source_version?

[zaneselvans]

Or if we wanted to try and enforce a uniform taxonomy across the board we could just have data_maturity or data_revision, not include the form, and instead have a controlled vocabulary that we define like...

• year_to_date
• early_release
• final

[cmgosnell]

I think I like data_maturity/ having a uniform taxonomy that does not actually include the data_source. I think these three options are good but I think we should add monthly_release for the eia860m. mostly because it's not reeeeally year_to_date like the 923 or cems YTD data is in that it is the monthly release of annual/static data.

[zaneselvans]

Hmm, to me monthly_release sounds much more like monthly data than annual data.

[zaneselvans]

But anyway, if we want to do a controlled vocab like this we should look at all the data sources we already know about and identify the existing universe of different kinds of maturity so we can make sure our categories can encompass them all clearly.