Pull DAC codes from source programattically

[Bjwebb]

This should keep withdrawn codes on the list, see the approach for other codelists at #51

Except where stated, these are taken orginally from the spreadsheet linked at http://www.oecd.org/dac/stats/dacandcrscodelists.htm

These are now also available as XML, but there are still concerns about the XML/spreadsheet not matching. Historic data also doesn't seem to be available in a machine readable format.

• AidType
• CollaborationType
• FinanceType
• PolicySignificance - https://github.com/IATI/IATI-Codelists-NonEmbedded/blob/master/xml/PolicySignificance.xml
• FlowType
• Region
• Sector
• SectoryCategory

[andylolz]

In the short term at least, this could pull from this secondary (machine readable) source:
https://datahub.io/core/dac-and-crs-code-lists

…or alternatively, that scraper could be part of this codebase. But I think splitting into two steps makes sense:

1. maintain a faithful machine readable version of the DAC CRS codelists (that’s what this does)
2. maintain an IATI version, that includes all withdrawn codes (TODO)

…especially given that the OECD intends to do (1).

[amy-silcock]

This does not fit with current plans to keep IATI/DAC codelists in sync.

[andylolz]

This does not fit with current plans to keep IATI/DAC codelists in sync.

@amy-silcock Could you expand on this?

This post from @bill-anderson seems to suggest DAC codelists will be pulled from source programmatically (but from the source XML mentioned in the body of this ticket, rather than the XLS spreadsheet).

If that’s the case, this issue is still relevant and should be reopened.

[samuele-mattiuzzo]

Hi @andylolz , we're currently going through various repositories to do some cleanup; this issue will not be discarded, it just does not fit in this quarter's pipeline so we're closing it for now. Once we'll tackle it, we'll make sure this is re-created.

[andylolz]

Hi @samuele-mattiuzzo – the idiomatic approach would be to label this with “on hold” or similar, rather than closing (or use github projects and have an icebox column or something). Closing is usually reserved for when an issue is fixed or wontfix or invalid. That’s fair, isn’t it? My worry is that once it’s closed, it’ll disappear into the ether. I know I certainly won’t keep an eye on it anymore.

By the sounds of it, there’s no intention to address this during this quarter. That’s fine, but In that case, do you mind if I bump this post again to ask that the pull request to update DAC codelists is reviewed and merged, as a stopgap until the more sustainable solution can be completed? Bill said this work was “in the job queue” but if it’s not scheduled for the current quarter, then recent changes should be merged another way. Some of the updates in that PR are from January 2018. Fortunately, they’re all very minor, so it should be trivial to review and merge them.

Longer term, if the plan is to pull DAC codes programmatically from DAC XML (I’m not 100% clear whether that’s actually the plan?) then I’d be happy to send a pull request for that.