DAC Codelist update from 28-02-2020 #326

akmiller01 · 2020-03-05T15:56:50Z

No description provided.

…rratives

andylolz · 2020-03-10T16:02:57Z

There are lots of instances of withdrawn codes being removed completely in this PR. Withdrawn codes should not be removed

akmiller01 · 2020-03-10T16:33:16Z

Thank you, @andylolz ! Working with a bit of inherited conversion code and the above was literally the result of a misplaced indentation in a bit of code that was trying to add the "withdrawn" status where a code was omitted:

From:

for key, element in iati_codes.items():
        if key not in dac_codes.keys():
            if element.attrib['status'] != 'withdrawn':
                element.attrib['status'] = 'withdrawn'
                dac_codes[key] = element

To:

for key, element in iati_codes.items():
        if key not in dac_codes.keys():
            if element.attrib['status'] != 'withdrawn':
                element.attrib['status'] = 'withdrawn'
            dac_codes[key] = element

andylolz · 2020-03-10T16:50:16Z

Fix sounds good to me.

Working with a bit of inherited conversion code

Interesting – is the conversion code public, at all?

In case it’s of use, the code we are using to do this same task for codelists.codeforiati.org is here:
https://github.com/codeforIATI/codelist-updater/blob/master/importers/helpers/__init__.py

akmiller01 · 2020-03-10T17:25:56Z

Fix sounds good to me.

Working with a bit of inherited conversion code

Interesting – is the conversion code public, at all?

In case it’s of use, the code we are using to do this same task for codelists.codeforiati.org is here:
https://github.com/codeforIATI/codelist-updater/blob/master/importers/helpers/__init__.py

Yep absolutely. I've hosted it here https://github.com/akmiller01/DAC-Codelists but I don't know if it would be more appropriate to place it elsewhere.

It does seem like parts of it have already been sourced from your Codelist Updater.

andylolz · 2020-03-11T10:02:50Z

Great – many thanks for sharing this!

It does seem like parts of it have already been sourced from your Codelist Updater.

Seems very possible that they have a shared ancestor. The codelist-updater code is based on the code in this pull request, which in turn is based on the code in this pull request.

I don't know if it would be more appropriate to place it elsewhere.

I’d suggest moving to an IATI owned repo.

stevieflow · 2020-03-11T10:40:35Z

Don't forget all the previous work around this:

#51
#172

PetyaKangalova · 2020-03-12T11:21:02Z

xml/Sector.xml

@@ -81,7 +81,7 @@
            </description>
            <category>112</category>
        </codelist-item>
-        <codelist-item status="active">
+        <codelist-item status="withdrawn">


@akmiller01 let's check this. There are a number of codes that have been changed from active to withdrawn and this looks incorrect to me:

Is there a "withdrawn-date" that you can identify in the source XML?

This is a "voluntary code" in OECD DAC terms. In IATI we don't differentiate between voluntary and non-voluntary, so my thinking is there is some error there. I think the status should still be "active".

Let's cross-check this for other codes.

PetyaKangalova · 2020-03-12T11:23:35Z

xml/Sector.xml

@@ -249,6 +249,18 @@
            </description>
            <category>121</category>
        </codelist-item>
+        <codelist-item status="withdrawn" activation-date="2019-01-01">


@akmiller01 this looks like another one where codelist has been listed as "withdrawn", while it should be "active". Maybe there was an error in the code replacing "Vonlontary basis" with "withdrawn" which is incorrect?

Yep, I was recoding "Vonlontary basis" as withdrawn. This will be corrected in the next commit.

Thanks @akmiller01 !

PetyaKangalova · 2020-03-12T11:24:19Z

xml/FinanceType.xml

@@ -810,10 +602,6 @@
                <narrative>Purchase of securities from issuing agencies</narrative>
                <narrative xml:lang="fr">Titres et autres instruments émis par les agences multilatérales</narrative>
            </name>
-            <description>


@akmiller01 can you see a reason why this was removed?

There was code to remove descriptions where narratives are empty; would we rather not do that?

Ok, that makes logical sense. I just did not understand why they were removed.

Seems fine to remove this stuff, but I wonder if it should be a separate PR. It creates a lot of noise in this one, which also contains important changes.

Agreed. Reversed for now.

Ah – that’s much better! Thanks

…en code has been removed

andylolz · 2020-03-12T19:01:14Z

xml/CRSChannelCode.xml

@@ -1,4 +1,4 @@
-<codelist name="CRSChannelCode" xml:lang="en" complete="1" embedded="0">
+<codelist name="Channel-category" xml:lang="en" complete="1" embedded="0">


I don’t know if the name of the codelist should be changed.

andylolz · 2020-03-12T19:21:17Z

xml/CRSChannelCode.xml

        </codelist-item>
-        <codelist-item status="active" activation-date="2015-12-07">
+        <codelist-item status="withdrawn" activation-date="2010-01-01" withdrawal-date="2010-12-31">


I don’t think this codelist item has been withdrawn. There are two withdrawn entries for this item, but also one active one.

Looking at the source XML, code 21011 does indeed appear three times. Active from January 1, 2003, withdrawn on December 31, 2003; reactivated January 1, 2004 (without an in-line withdrawal); and then reactivated again on January 1, 2010, withdrawn December 31, 2010.

Maybe it's worth asking the OECD about the intended meaning, @PetyaKangalova ? I think it would be fair to concatenate codelist items when codes and narratives match verbatim (which would ultimately make this item withdrawn, with an activate date of 2003 and withdrawal date at the end of 2010?), but I don't want to make that determination for them. I'm also assuming that we don't want codelist items to have noncontiguous active time periods.

Thanks @akmiller01 ! Will check on this one.

which would ultimately make this item withdrawn, with an activate date of 2003 and withdrawal date at the end of 2010?

I don’t think that’s necessarily the case. It completely depends on the rules you decide to apply when merging codelist items.

It’s also worth noting that in this instance, this code is present in the XLS:

Anyway – thanks for checking with the DAC on this.

Thanks @andylolz this code has been reactivated.

andylolz · 2020-03-12T19:22:12Z

xml/CRSChannelCode.xml

        </codelist-item>
-        <codelist-item status="active" activation-date="2015-12-07">
+        <codelist-item status="withdrawn" activation-date="2010-01-01" withdrawal-date="2010-12-31">


I don’t think this codelist item has been withdrawn. There are two withdrawn entries for this item, but also one active one.

Reactivated.

andylolz · 2020-03-12T21:03:07Z

xml/SectorCategory.xml

@@ -458,7 +458,7 @@
                <narrative xml:lang="fr"/>
            </description>
        </codelist-item>
-        <codelist-item status="withdrawn">
+        <codelist-item status="withdrawn" withdrawal-date="2020-02-28">


I don’t think this withdrawal-date is correct. This code was withdrawn a long time ago – it wasn’t present in March 2017, so it was withdrawn earlier than that.

@andylolz I already emailed yesterday the OECD to check about all the withdrawn dates that were missing and ask them if they can specify the dates :) I'll update here once I hear back. Thanks for cross-checking the date for this one.

andylolz · 2020-03-12T22:31:05Z

xml/SectorCategory.xml

@@ -163,7 +163,7 @@
                <narrative xml:lang="fr"/>
            </description>
        </codelist-item>
-        <codelist-item status="withdrawn">
+        <codelist-item status="withdrawn" withdrawal-date="2020-02-28">


This code has been withdrawn since at least June 2017.

Thanks! Checking with the OECD and will be updating the dates once we hear back from them.

andylolz · 2020-03-12T22:36:10Z

It appears as though missing withdrawal dates have been set to 28th Feb 2020 throughout. I think that could potentially cause confusion. Many of these codes were withdrawn long before that date. Where the withdrawal date is not known, I think it’s probably safer to leave it blank.

PetyaKangalova · 2020-03-13T09:21:02Z

@andylolz thanks for all your comments. On the dates, we indeed discussed this with Alex and now waiting to hear back from the OECD if they can specify withdrawn dates where they were missing. Will provide an update here once I get a response.

…odes

akmiller01 · 2020-03-31T14:28:33Z

Attaching documents from the OECD. One spreadsheet which has activation/withdrawn years for specific sector codes. And two PDFs with withdraw decisions for the 151 and 920 series:

oldPurposecodes.xlsx

https://one.oecd.org/document/DCD/DAC/STAT(2010)6/en/pdf
https://one.oecd.org/document/DCD/DAC/STAT(2012)3/en/pdf

PetyaKangalova

@akmiller01 Thanks for the latest updates. I have now checked the updated withdrawn dates and looks good to me so happy to approve the full list of changes !

alexlydiate

All looks good from my perspective.

andylolz · 2020-04-13T21:47:39Z

This is great! Great work getting the activation and withdrawal dates from the DAC 🎉

It looks like those dates were added to the source XML by the DAC – is that correct?

With these latest improvements, it looks like the XML may have overtaken the XLS as the canonical source for DAC codelist data!

Worth noting that some codelists are still only present in the DAC XLS, and not the XML. I think the only one that IATI replicates is Region. It would be great to get this added to the XML.

akmiller01 · 2020-04-16T14:05:02Z

Hi Andy, the OECD did not have time to add the new dates to their source XML, so I manually went into the output and added them according to the XLSX and PDF documents attached in a comment above. We'll reach out to them about the French translation you noted.

You can preview some of the new codes in the SSOT here: http://dev.reference.iatistandard.org/203/codelists/Sector/

I believe all looks good on our end, so shortly we'll be deploying that to http://reference.iatistandard.org as well.

DAC Codelist update from 28-02-2020

1c78343

akmiller01 requested a review from PetyaKangalova March 5, 2020 15:56

akmiller01 added 2 commits March 6, 2020 12:02

Remove MCD, Remove vonlontary basis, trim whitespace, remove empty na…

8408d5e

…rratives

Remove whitespace in category

d989e99

Re-include previously withdrawn but not included codelist items

c50c9ca

PetyaKangalova reviewed Mar 12, 2020

View reviewed changes

Vonlontary basis to active, and code withdrawal date as 2020-02-28 wh…

88588a2

…en code has been removed

andylolz reviewed Mar 12, 2020

View reviewed changes

Remap DAC codelist name to IATI

298ce6d

andylolz reviewed Mar 12, 2020

View reviewed changes

Reverse removal of empty narratives for PR legibility

2a1be24

andylolz reviewed Mar 12, 2020

View reviewed changes

akmiller01 added 2 commits March 31, 2020 10:24

Withdrawn dates from OECD DAC, and reactivate code 21011 in channel c…

f986629

…odes

Reactivate code 21013 channel

f16567e

PetyaKangalova approved these changes Apr 9, 2020

View reviewed changes

alexlydiate approved these changes Apr 9, 2020

View reviewed changes

akmiller01 merged commit 394e449 into master Apr 9, 2020

akmiller01 deleted the Feb-28-CRS-Update branch April 9, 2020 15:50

		@@ -1,4 +1,4 @@
		<codelist name="CRSChannelCode" xml:lang="en" complete="1" embedded="0">
		<codelist name="Channel-category" xml:lang="en" complete="1" embedded="0">

DAC Codelist update from 28-02-2020 #326

DAC Codelist update from 28-02-2020 #326

Conversation

akmiller01 commented Mar 5, 2020

andylolz commented Mar 10, 2020

akmiller01 commented Mar 10, 2020

andylolz commented Mar 10, 2020 • edited Loading

akmiller01 commented Mar 10, 2020

andylolz commented Mar 11, 2020 • edited Loading

stevieflow commented Mar 11, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andylolz Mar 13, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andylolz commented Mar 12, 2020

PetyaKangalova commented Mar 13, 2020

akmiller01 commented Mar 31, 2020

PetyaKangalova left a comment

Choose a reason for hiding this comment

alexlydiate left a comment

Choose a reason for hiding this comment

andylolz commented Apr 13, 2020

akmiller01 commented Apr 16, 2020

andylolz commented Mar 10, 2020 •

edited

Loading

andylolz commented Mar 11, 2020 •

edited

Loading

andylolz Mar 13, 2020 •

edited

Loading