From f4489c68cbd4ae5c808146819cbd70946bbaf2df Mon Sep 17 00:00:00 2001 From: Monica Turner Date: Mon, 3 Jun 2024 23:26:07 +0200 Subject: [PATCH 1/8] HDXDSYS-731 start adding latest docs --- dev-requirements.txt | 1 + docs/data_usage_guides/affected_people.md | 71 ++++++++++++ .../coordination_and_context.md | 104 ++++++++++++++++++ .../conflict_events_details.yaml | 10 ++ .../subcategory_details/funding_details.yaml | 10 ++ .../humanitarian_needs_details.yaml | 10 ++ .../national_risk_details.yaml | 10 ++ .../operational_presence_details.yaml | 10 ++ .../subcategory_details/refugees_details.yaml | 10 ++ mkdocs.yml | 12 +- 10 files changed, 246 insertions(+), 2 deletions(-) create mode 100644 docs/data_usage_guides/affected_people.md create mode 100644 docs/data_usage_guides/coordination_and_context.md create mode 100644 docs/data_usage_guides/subcategory_details/conflict_events_details.yaml create mode 100644 docs/data_usage_guides/subcategory_details/funding_details.yaml create mode 100644 docs/data_usage_guides/subcategory_details/humanitarian_needs_details.yaml create mode 100644 docs/data_usage_guides/subcategory_details/national_risk_details.yaml create mode 100644 docs/data_usage_guides/subcategory_details/operational_presence_details.yaml create mode 100644 docs/data_usage_guides/subcategory_details/refugees_details.yaml diff --git a/dev-requirements.txt b/dev-requirements.txt index 50017c95..2a2ee41e 100644 --- a/dev-requirements.txt +++ b/dev-requirements.txt @@ -1,5 +1,6 @@ mkdocs~=1.5.2 mkdocs-material~=9.2.6 +mkdocs-table-reader-plugin~=2.2.2 pytest~=7.4.2 pytest-cov~=4.1.0 diff --git a/docs/data_usage_guides/affected_people.md b/docs/data_usage_guides/affected_people.md new file mode 100644 index 00000000..a85b24c7 --- /dev/null +++ b/docs/data_usage_guides/affected_people.md @@ -0,0 +1,71 @@ +# Affected People + +## Refugees & Persons of Concern + +This dataset, compiled by the [UNHCR](https://www.unhcr.org/), offers annual +age- and gender-disaggregated statistics on refugees and people of concern, +categorised by their country of origin and country of asylum. The data are +sourced primarily from governments hosting these populations, UNHCR's own +registration data, and occasionally data published by non-governmental +organisations. + +### Summary + +{{ read_yaml('data_usage_guides/subcategory_details/refugees_details.yaml') }} + +### Transformations applied + +* The table has been reshaped from wide to long: demographic-specific columns + have been cast to gender, age, and population +* Population group has been filtered for refugees (REF) and others of concern + (OOC) +* It is not possible to p-code based on the location information in the + original data, therefore population numbers are aggregated to the national + level +* The reference period is constructed using the full range of the year + presented in the “Year” column of the original data + +### Usage notes + +* An “all” value in the `gender` and `age_range` columns indicates no + disaggregation + +## Humanitarian Needs + +This Humanitarian Needs Overview (HNO) represents the shared understanding of +OCHA Humanitarian Country Teams of people's widespread emergency needs during +crises. It includes an estimate of the number of people by sector who require +assistance, often referred to as People in Need (PIN), which is derived using +the [Joint Intersectoral Analysis Framework (JIAF)](https://www.jiaf.info/). + +While the HNO data is +[directly available on HDX](https://data.humdata.org/dataset/?dataseries_name=Humanitarian+Needs+Overview), +it comes from different OCHA offices and is currently not standardised. Thus, +HDX HAPI obtains the PIN numbers from the +[HPC Tools API](https://api.hpc.tools/docs/v1/). +Datasets from the HPC API will soon be available on HDX. + +### Summary + +{{ read_yaml('data_usage_guides/subcategory_details/humanitarian_needs_details.yaml') }} + +### Transformations applied + +* The table has been reshaped from wide to long: the columns of population, in + need, targeted, affected, and reached have been cast to a single + `population_status` column +* Sector values of “ALL” have been converted to “intersectoral” +* Gender and disabled values of “a” have been converted to “all” +* The methodology in Yemen leads to negative population values in some admin 2 + level areas. Where negative values appear they have been omitted from the API. +* The reference period is obtained from the HDX dataset + +### Usage notes + +* The PIN should *not* be summed across sectors, as the same people can be + counted across multiple sectors. For the number of people affected across all + sectors, please use the PIN value where sector=intersectoral. +* An “all” value in the `gender`, `age_range`, `disable_marker`, and + `population_group` columns indicates no disaggregation +* A “population” value in the `population_status` column indicates no d + isaggregation diff --git a/docs/data_usage_guides/coordination_and_context.md b/docs/data_usage_guides/coordination_and_context.md new file mode 100644 index 00000000..2d35db1e --- /dev/null +++ b/docs/data_usage_guides/coordination_and_context.md @@ -0,0 +1,104 @@ +# Coordination & Context + +## 3W - Who is Doing What Where + +The [Who does What Where (3W)](https://3w.unocha.org/) is a core humanitarian +coordination dataset that contains the geographic and sectoral spread of +humanitarian activities and partners. It is critical to know where humanitarian +organisations are working and what they are doing in order to ensure +collaboration and efficient resource allocation, avoid duplication of efforts, +identify gaps, and plan for future humanitarian response. + +### Summary + +{{ read_yaml('data_usage_guides/subcategory_details/operational_presence_details.yaml') }} + +### Transformations applied + +* For consistency and interoperability, we aggregate to an + [operational presence](https://humanitarian.atlassian.net/wiki/spaces/imtoolbox/pages/214499412/Who+does+What+Where+3W) + level (3W:OP, per org, sector, and admin2), even if the original 3W data is + more detailed (e.g. the source lists individual activities) +* Countries that are not p-coded are aggregated to Admin 0 +* Organisation deduplication is a long-running challenge with this data, since + there are no unique identifiers, and organisation names may be spelled + different ways from different OCHA offices, or sometimes even within the same + 3W. See the [`org`](org) section below for more information on how we handle + these details. +* Organisations without an associated sector are skipped +* The reference period is obtained from the HDX dataset + +### Usage notes + +* Available at either Admin 0, 1, or 2 depending on the country. Please check + data coverage for further information +* We cannot guarantee org consistency over time; for example, if an IMO + changes, the 3W might change the spelling or the whole name of an org between + releases + +## Funding + +FTS publishes data on humanitarian funding flows as reported by donors and +recipient organisations. It presents all humanitarian funding to a country and +funding that is reported or that can be specifically mapped against funding +requirements stated in Humanitarian Response Plans. + +### Summary + +{{ read_yaml('data_usage_guides/subcategory_details/funding_details.yaml') }} + +### Transformations applied + +* The present version of the API currently captures only funding associated + with an appeal. Funding data without associated appeals will be added in a + future version. +* The reference period is taken from the start and end date provided in each + row front he original data + +## Conflict Events + +[ACLED](https://acleddata.com/) collects real-time data on the locations, +dates, actors, fatalities, and types of all reported political violence and +protest events around the world. HDX HAPI leverages ACLED’s publicly accessible +data, aggregated on a monthly basis and to administrative regions. + +### Summary + +{{ read_yaml('data_usage_guides/subcategory_details/conflict_events_details.yaml') }} + +### Transformations applied + +* The data for political violence events, civilian targeting events, and + demonstrations are in separate resource on HDX, but are combined into a + single endpoint in the API +* Any duplicate rows in the original data are removed +* The reference period is constructed using the full range of the month + presented in the “Month” and “Year” columns of the original data +* Usage notes +* Data is either national or disaggregated to admin 2, see individual resources + for more details + +## National Risk + +The [INFORM Risk Index](https://drmkc.jrc.ec.europa.eu/inform-index/INFORM-Risk) +is a global, open-source risk assessment for humanitarian crises and disasters. +It can support decisions about prevention, preparedness and response. For more +information on the methodology, see +[here](https://drmkc.jrc.ec.europa.eu/inform-index/INFORM-Risk/Methodology). + +### Summary + +{{ read_yaml('data_usage_guides/subcategory_details/national_risk_details.yaml') }} + +### Transformations applied + +* The reference period is obtained from the HDX dataset + +### Usage Notes + +* [To be moved to parameter table] +* Missing data: The total number of original indicators missing, including + any that have been estimated (e.g. HDI derived from GDP per capita). +* Recentness: The average of the total number of years older than the + reference year per indicator, to account for any older data used as a proxy + for the most recent year. diff --git a/docs/data_usage_guides/subcategory_details/conflict_events_details.yaml b/docs/data_usage_guides/subcategory_details/conflict_events_details.yaml new file mode 100644 index 00000000..08c0bd41 --- /dev/null +++ b/docs/data_usage_guides/subcategory_details/conflict_events_details.yaml @@ -0,0 +1,10 @@ +- Attribute: Data provider + Details: ACLED +- Attribute: Dataset + Details: "[ACLED Conflict Events Data series](https://data.humdata.org/dataset/?dataseries_name=ACLED+-+Conflict+Events)" +- Attribute: Time series + Details: Monthly +- Attribute: Update frequency + Details: Weekly +- Attribute: Related tables + Details: "[`EventType`](event-type)" diff --git a/docs/data_usage_guides/subcategory_details/funding_details.yaml b/docs/data_usage_guides/subcategory_details/funding_details.yaml new file mode 100644 index 00000000..d4cc939c --- /dev/null +++ b/docs/data_usage_guides/subcategory_details/funding_details.yaml @@ -0,0 +1,10 @@ +- Attribute: Data provider + Details: OCHA FTS +- Attribute: Dataset + Details: "[OCHA FTS - Requirements and Funding Data series](https://data.humdata.org/dataset/?dataseries_name=OCHA+FTS+-+Requirements+and+Funding+Data)" +- Attribute: Time series + Details: Data contains funding since 2006, but the timepoints are not regular +- Attribute: Update frequency + Details: Annually +- Attribute: Related tables + Details: None diff --git a/docs/data_usage_guides/subcategory_details/humanitarian_needs_details.yaml b/docs/data_usage_guides/subcategory_details/humanitarian_needs_details.yaml new file mode 100644 index 00000000..8e5e6533 --- /dev/null +++ b/docs/data_usage_guides/subcategory_details/humanitarian_needs_details.yaml @@ -0,0 +1,10 @@ +- Attribute: Data provider + Details: OCHA HPC +- Attribute: Dataset + Details: Coming soon to HDX +- Attribute: Time series + Details: Past datasets are available in the HPC API, but are sparse prior to 2023 +- Attribute: Update frequency + Details: Annually +- Attribute: Related tables + Details: "[ `DisabledMarker` ](disabled-marker), [`Gender`](gender), [`PopulationGroup`](population-group), [`PopulationStatus`](population-status)" diff --git a/docs/data_usage_guides/subcategory_details/national_risk_details.yaml b/docs/data_usage_guides/subcategory_details/national_risk_details.yaml new file mode 100644 index 00000000..93a0d621 --- /dev/null +++ b/docs/data_usage_guides/subcategory_details/national_risk_details.yaml @@ -0,0 +1,10 @@ +- Attribute: Data provider + Details: INFORM +- Attribute: Dataset + Details: "[INFORM Risk Index](https://data.humdata.org/dataset/inform-risk-index-2021)" +- Attribute: Time series + Details: Not available for this subcategory +- Attribute: Update frequency + Details: Annually +- Attribute: Related tables + Details: "[`RiskClass`](risk-class)" diff --git a/docs/data_usage_guides/subcategory_details/operational_presence_details.yaml b/docs/data_usage_guides/subcategory_details/operational_presence_details.yaml new file mode 100644 index 00000000..b708bdd5 --- /dev/null +++ b/docs/data_usage_guides/subcategory_details/operational_presence_details.yaml @@ -0,0 +1,10 @@ +- Attribute: Data provider + Details: OCHA country and regional offices +- Attribute: Dataset + Details: "[Operational Presence tag on HDX](https://data.humdata.org/dataset/?vocab_Topics=operational%20presence&q=&sort=last_modified%20desc&ext_page_size=25)" +- Attribute: Time series + Details: Not available for this subcategory. While older resources are available on HDX, HDX HAPI currently only supports the latest resource. +- Attribute: Update frequency + Details: Irregular, depending on the country. Common update frequencies include quarterly and annually. Please check individual resources for more information. +- Attribute: Related tables + Details: "[`Org`](org), [`OrgType`](org-type), [`Sector`](sector)" diff --git a/docs/data_usage_guides/subcategory_details/refugees_details.yaml b/docs/data_usage_guides/subcategory_details/refugees_details.yaml new file mode 100644 index 00000000..8f07575b --- /dev/null +++ b/docs/data_usage_guides/subcategory_details/refugees_details.yaml @@ -0,0 +1,10 @@ +- Attribute: Data provider + Details: UNHCR +- Attribute: Dataset + Details: "[Data on forcibly displaced populations and stateless persons (Global)](https://data.humdata.org/dataset/unhcr-population-data-for-world)" +- Attribute: Time series + Details: Annual, dating back to 2001 +- Attribute: Update frequency + Details: Annually +- Attribute: Related tables + Details: "[`Gender`](gender), [`PopulationGroup`](population-group)" diff --git a/mkdocs.yml b/mkdocs.yml index 449ee77c..b88b63af 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -4,8 +4,16 @@ nav: - Getting Started: getting-started.md - Data Coverage: data.md - Code Examples: examples.md - - Data Usage Guide: subcategory_details.md + - Data Usage Guide: + - Affected People: data_usage_guides/affected_people.md + - Coordination & Context : data_usage_guides/coordination_and_context.md - Geo Data: geo.md - Contact: contact.md theme: - name: "material" \ No newline at end of file + name: "material" +markdown_extensions: + - pymdownx.snippets +plugins: + - search + - table-reader: + base_path: "docs_dir" From d85f1928a6fbeb06a14156b1a525eefb19a59f6c Mon Sep 17 00:00:00 2001 From: Monica Turner Date: Tue, 4 Jun 2024 12:48:46 +0200 Subject: [PATCH 2/8] HDXDSYS-731 add rest of subcats --- docs/data_usage_guides/affected_people.md | 5 +- .../food_security_and_nutrition.md | 54 ++++++++++++++++++ .../population_and_socio-economy.md | 55 +++++++++++++++++++ .../_details_template.yaml | 10 ++++ .../baseline_population_details.yaml | 10 ++++ .../food_prices_details.yaml | 10 ++++ .../food_security_details.yaml | 10 ++++ .../poverty_rate_details.yaml | 10 ++++ mkdocs.yml | 2 + 9 files changed, 164 insertions(+), 2 deletions(-) create mode 100644 docs/data_usage_guides/food_security_and_nutrition.md create mode 100644 docs/data_usage_guides/population_and_socio-economy.md create mode 100644 docs/data_usage_guides/subcategory_details/_details_template.yaml create mode 100644 docs/data_usage_guides/subcategory_details/baseline_population_details.yaml create mode 100644 docs/data_usage_guides/subcategory_details/food_prices_details.yaml create mode 100644 docs/data_usage_guides/subcategory_details/food_security_details.yaml create mode 100644 docs/data_usage_guides/subcategory_details/poverty_rate_details.yaml diff --git a/docs/data_usage_guides/affected_people.md b/docs/data_usage_guides/affected_people.md index a85b24c7..e8a85e23 100644 --- a/docs/data_usage_guides/affected_people.md +++ b/docs/data_usage_guides/affected_people.md @@ -17,8 +17,6 @@ organisations. * The table has been reshaped from wide to long: demographic-specific columns have been cast to gender, age, and population -* Population group has been filtered for refugees (REF) and others of concern - (OOC) * It is not possible to p-code based on the location information in the original data, therefore population numbers are aggregated to the national level @@ -27,6 +25,9 @@ organisations. ### Usage notes +* UNHCR is the only source for returnees in some countries and is therefore + included in HDX HAPI, but if data is available from the IDP endpoint, that + data is preferable * An “all” value in the `gender` and `age_range` columns indicates no disaggregation diff --git a/docs/data_usage_guides/food_security_and_nutrition.md b/docs/data_usage_guides/food_security_and_nutrition.md new file mode 100644 index 00000000..e433fdbe --- /dev/null +++ b/docs/data_usage_guides/food_security_and_nutrition.md @@ -0,0 +1,54 @@ +# Food Security & Nutrition + +## Food Security + +The [IPC Acute Food Insecurity (IPC AFI) classification](https://www.ipcinfo.org/ipcinfo-website/ipc-overview-and-classification-system/ipc-acute-food-insecurity-classification/en/) +provides strategically relevant information to decision makers that focuses on +short-term objectives to prevent, mitigate or decrease severe food insecurity. + +The Beta release of HDX HAPI only contains data from the +[Cadre Harmonisé](https://www.cadreharmonise.org/en_GB), as it is p-coded. In +a future release we will p-code and expand coverage to other IPC datasets. + +### Summary + +{{ read_yaml('data_usage_guides/subcategory_details/food_security_details.yaml') }} + +### Transformations applied + +* The table has been reshaped from wide to long: the population and individual +* IPC phase columns have been cast to a combination of `ipc_phase` and + `population_in_phase` +* The IPC fraction is computed in the HDX HAPI API pipeline, by dividing the + population in that phase by the total population (ipc_phase=all). +* The reference period is computed from the “reference_label” and + “reference_year” columns in the original data. In the case of a projection, + it captures date range that the projection covers, not when the projection + was made +* In the `ipc_type` column, “projected” is converted to “first projection” + +### Usage Notes + +* The total population (ipc_phase=all) is not necessarily equal to the sum of + the populations in phases 1-5. The differences are usually small (due to + rounding errors), or because there is no IPC phase data +* Due to the above, the sum of the IPC fractions from phase 1-5 may not sum to + exactly 1 + +## Food Prices + +The World Food Programme Price Database covers foods such as maize, rice, +beans, fish, and sugar for 98 countries and some 3000 markets. It is updated +weekly but primarily contains data with a monthly update frequency. + +### Summary + +{{ read_yaml('data_usage_guides/subcategory_details/food_prices_details.yaml') }} + +### Transformations applied + +* The reference period is computed by converting date from the “date” column, + originally presented as 15th day of a particular month, into a range spanning + the entire month +* The source data is not p-coded, however we have used the admin 1 and 2 names + to p-code most markets. See [Markets](markets) for more details. diff --git a/docs/data_usage_guides/population_and_socio-economy.md b/docs/data_usage_guides/population_and_socio-economy.md new file mode 100644 index 00000000..8bc4a4de --- /dev/null +++ b/docs/data_usage_guides/population_and_socio-economy.md @@ -0,0 +1,55 @@ +# Population & Socio-economy + +## Baseline Population + +The population statistics presented here are sourced from the +[common operational datasets](https://cod.unocha.org/) (CODs), and are +typically disaggregated by age and/or gender, and extend to administrative +levels 1 or 2. Primary data providers include the UNFPA and OCHA country +offices. These data are projections based on demographic indicators, and +therefore caution should be used when comparing to population data from other +sources. + +### Summary + +{{ read_yaml('data_usage_guides/subcategory_details/baseline_population_details.yaml') }} + +### Transformations applied + +* The table has been reshaped from wide to long: demographic-specific columns + have been cast to gender, age, and population +* The reference period is obtained from the HDX dataset + +### Usage Notes + +* Age disaggregation ranges are not consistent across countries +* Any aggregation to a higher administrative level (e.g., admin 1 for a + country where admin 2 is also available) has been taken directly from the + data provided, and was not computed in the API pipeline + +## Poverty Rate + +The global [Oxford Multidimensional Poverty Index](https://ophi.org.uk/global-mpi) +(MPI) measures multidimensional poverty in over 100 developing countries, +using internationally comparable datasets. The MPI assesses poverty through +three main dimensions: health, education, and living standards, each of which +is represented by specific indicators. Please see the +[OPHI methodological note](https://ophi.org.uk/publications/MN-54) for more +details. + +### Summary + +{{ read_yaml('data_usage_guides/subcategory_details/poverty_rate_details.yaml') }} + +### Transformations applied + +* For rows in the original data with two timepoints, we take each timepoint as + single entry into HDX HAPI +* The reference period is constructed using the full range of the year or year + range presented in the “year” column, pertaining to the timepoint in + question, of the original data + +### Usage Notes + +* Please be aware of the metric definitions, provided in the parameter + summary table !!!! TODO add link and table diff --git a/docs/data_usage_guides/subcategory_details/_details_template.yaml b/docs/data_usage_guides/subcategory_details/_details_template.yaml new file mode 100644 index 00000000..bc356876 --- /dev/null +++ b/docs/data_usage_guides/subcategory_details/_details_template.yaml @@ -0,0 +1,10 @@ +- Attribute: Data provider + Details: +- Attribute: Dataset + Details: +- Attribute: Time series + Details: +- Attribute: Update frequency + Details: +- Attribute: Related tables + Details: diff --git a/docs/data_usage_guides/subcategory_details/baseline_population_details.yaml b/docs/data_usage_guides/subcategory_details/baseline_population_details.yaml new file mode 100644 index 00000000..54f9d15f --- /dev/null +++ b/docs/data_usage_guides/subcategory_details/baseline_population_details.yaml @@ -0,0 +1,10 @@ +- Attribute: Data provider + Details: UNFPA, OCHA country offices +- Attribute: Dataset + Details: "[COD - subnational population statistics data series](https://data.humdata.org/dataset/?dataseries_name=COD+-+Subnational+Population+Statistics)" +- Attribute: Time series + Details: No +- Attribute: Update frequency + Details: Annually +- Attribute: Related tables + Details: "[`Gender`](gender)" diff --git a/docs/data_usage_guides/subcategory_details/food_prices_details.yaml b/docs/data_usage_guides/subcategory_details/food_prices_details.yaml new file mode 100644 index 00000000..acb6a473 --- /dev/null +++ b/docs/data_usage_guides/subcategory_details/food_prices_details.yaml @@ -0,0 +1,10 @@ +- Attribute: Data provider + Details: The World Food Programme +- Attribute: Dataset + Details: "[WFP Food Prices data series](https://data.humdata.org/dataset/?dataseries_name=WFP+-+Food+Prices)" +- Attribute: Time series + Details: Yes, monthly +- Attribute: Update frequency + Details: Weekly +- Attribute: Related tables + Details: "[`Commodity`](commodity), [`Currency`](Currency), [`Market`](market)" diff --git a/docs/data_usage_guides/subcategory_details/food_security_details.yaml b/docs/data_usage_guides/subcategory_details/food_security_details.yaml new file mode 100644 index 00000000..492db4da --- /dev/null +++ b/docs/data_usage_guides/subcategory_details/food_security_details.yaml @@ -0,0 +1,10 @@ +- Attribute: Data provider + Details: Food Security and Nutrition Working Group, West and Central Africa +- Attribute: Dataset + Details: "[West & Central Africa Food Security Data - Cadre Harmonise (CH) and Integrated Food Security Phase Classification (IPC) data](https://data.humdata.org/dataset/cadre-harmonise)" +- Attribute: Time series + Details: Yes, with projections +- Attribute: Update frequency + Details: Annually +- Attribute: Related tables + Details: "[`IPCPhase`](ipc-phase), [`IPCType`](ipc-type)" diff --git a/docs/data_usage_guides/subcategory_details/poverty_rate_details.yaml b/docs/data_usage_guides/subcategory_details/poverty_rate_details.yaml new file mode 100644 index 00000000..71fe4b9f --- /dev/null +++ b/docs/data_usage_guides/subcategory_details/poverty_rate_details.yaml @@ -0,0 +1,10 @@ +- Attribute: Data provider + Details: Oxford Poverty and Human Development Initiative (OPHI) +- Attribute: Dataset + Details: "[Oxford Poverty and Human Development Initiative - Global Multidimensional Poverty Index data series](https://data.humdata.org/dataset/?dataseries_name=Oxford+Poverty+and+Human+Development+Initiative+-+Global+Multidimensional+Poverty+Index)" +- Attribute: Time series + Details: Annual, with some combined years +- Attribute: Update frequency + Details: Annually +- Attribute: Related tables + Details: None diff --git a/mkdocs.yml b/mkdocs.yml index b88b63af..8740d653 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -7,6 +7,8 @@ nav: - Data Usage Guide: - Affected People: data_usage_guides/affected_people.md - Coordination & Context : data_usage_guides/coordination_and_context.md + - Food Security & Nutrition : data_usage_guides/food_security_and_nutrition.md + - Population & Socio-economy : data_usage_guides/population_and_socio-economy.md - Geo Data: geo.md - Contact: contact.md theme: From 2ee4b387e3edc284ace662232c46b60dd7d0849b Mon Sep 17 00:00:00 2001 From: Monica Turner Date: Tue, 4 Jun 2024 15:30:25 +0200 Subject: [PATCH 3/8] HDXDSYS-731 interim commit --- .../coordination_and_context.md | 9 +- .../commodity_category_parameters.yaml | 16 +++ .../disabled_marker_parameters.yaml | 6 + .../event_type_parameters.yaml | 6 + .../enum_parameters/gender_parameters.yaml | 12 ++ .../enum_parameters/ipc_phase_parameters.yaml | 21 ++++ .../enum_parameters/ipc_type_parameters.yaml | 6 + .../population_group_parameters.yaml | 32 +++++ .../population_status_parameters.yaml | 12 ++ .../price_flag_parameters.yaml | 6 + .../price_type_parameters.yaml | 6 + .../risk_class_parameters.yaml | 10 ++ docs/data_usage_guides/enums.md | 116 ++++++++++++++++++ docs/data_usage_guides/metadata.md | 59 +++++++++ .../baseline_population_details.yaml | 2 +- .../conflict_events_details.yaml | 2 +- .../food_prices_details.yaml | 2 +- .../food_security_details.yaml | 2 +- .../humanitarian_needs_details.yaml | 2 +- .../national_risk_details.yaml | 2 +- .../operational_presence_details.yaml | 2 +- .../subcategory_details/refugees_details.yaml | 2 +- mkdocs.yml | 2 + 23 files changed, 325 insertions(+), 10 deletions(-) create mode 100644 docs/data_usage_guides/enum_parameters/commodity_category_parameters.yaml create mode 100644 docs/data_usage_guides/enum_parameters/disabled_marker_parameters.yaml create mode 100644 docs/data_usage_guides/enum_parameters/event_type_parameters.yaml create mode 100644 docs/data_usage_guides/enum_parameters/gender_parameters.yaml create mode 100644 docs/data_usage_guides/enum_parameters/ipc_phase_parameters.yaml create mode 100644 docs/data_usage_guides/enum_parameters/ipc_type_parameters.yaml create mode 100644 docs/data_usage_guides/enum_parameters/population_group_parameters.yaml create mode 100644 docs/data_usage_guides/enum_parameters/population_status_parameters.yaml create mode 100644 docs/data_usage_guides/enum_parameters/price_flag_parameters.yaml create mode 100644 docs/data_usage_guides/enum_parameters/price_type_parameters.yaml create mode 100644 docs/data_usage_guides/enum_parameters/risk_class_parameters.yaml create mode 100644 docs/data_usage_guides/enums.md create mode 100644 docs/data_usage_guides/metadata.md diff --git a/docs/data_usage_guides/coordination_and_context.md b/docs/data_usage_guides/coordination_and_context.md index 2d35db1e..51d426a1 100644 --- a/docs/data_usage_guides/coordination_and_context.md +++ b/docs/data_usage_guides/coordination_and_context.md @@ -61,6 +61,8 @@ requirements stated in Humanitarian Response Plans. dates, actors, fatalities, and types of all reported political violence and protest events around the world. HDX HAPI leverages ACLED’s publicly accessible data, aggregated on a monthly basis and to administrative regions. +To learn more about their methodology, please consult the +[ACLED Codebook](https://acleddata.com/knowledge-base/codebook/). ### Summary @@ -74,7 +76,11 @@ data, aggregated on a monthly basis and to administrative regions. * Any duplicate rows in the original data are removed * The reference period is constructed using the full range of the month presented in the “Month” and “Year” columns of the original data -* Usage notes + +### Usage notes + +* The three event categories are not mutually exclusive, see + [`Event Type`](enums.md#event-type) for more details * Data is either national or disaggregated to admin 2, see individual resources for more details @@ -101,4 +107,3 @@ information on the methodology, see any that have been estimated (e.g. HDI derived from GDP per capita). * Recentness: The average of the total number of years older than the reference year per indicator, to account for any older data used as a proxy - for the most recent year. diff --git a/docs/data_usage_guides/enum_parameters/commodity_category_parameters.yaml b/docs/data_usage_guides/enum_parameters/commodity_category_parameters.yaml new file mode 100644 index 00000000..7d203444 --- /dev/null +++ b/docs/data_usage_guides/enum_parameters/commodity_category_parameters.yaml @@ -0,0 +1,16 @@ +- commodity category: cereals and tubers + description: '' +- commodity category: meat, fish and eggs + description: '' +- commodity category: milk and dairy + description: '' +- commodity category: miscellaneous food + description: '' +- commodity category: non-food + description: '' +- commodity category: oil and fats + description: '' +- commodity category: pulses and nuts + description: '' +- commodity category: vegetables and fruits + description: '' diff --git a/docs/data_usage_guides/enum_parameters/disabled_marker_parameters.yaml b/docs/data_usage_guides/enum_parameters/disabled_marker_parameters.yaml new file mode 100644 index 00000000..2147c6b7 --- /dev/null +++ b/docs/data_usage_guides/enum_parameters/disabled_marker_parameters.yaml @@ -0,0 +1,6 @@ +- disabled marker: y + description: 'yes' +- disabled marker: n + description: 'no' +- disabled marker: all + description: 'all' diff --git a/docs/data_usage_guides/enum_parameters/event_type_parameters.yaml b/docs/data_usage_guides/enum_parameters/event_type_parameters.yaml new file mode 100644 index 00000000..e204b6f1 --- /dev/null +++ b/docs/data_usage_guides/enum_parameters/event_type_parameters.yaml @@ -0,0 +1,6 @@ +- event type: civilian_targeting + description: Civilian targeting events include violence against civilians events and explosions/remote violence events in which civilians were directly targeted. +- event type: demonstration + description: Demonstration events include ACLED’s protests and riots event types, with the exception of the mob violence sub-event type of the riots event type. +- event type: political_violence + description: Political violence events include ACLED’s battles, violence against civilians, and explosions/remote violence event types, as well as the mob violence sub-event type of the riots event type. diff --git a/docs/data_usage_guides/enum_parameters/gender_parameters.yaml b/docs/data_usage_guides/enum_parameters/gender_parameters.yaml new file mode 100644 index 00000000..fbbc2b41 --- /dev/null +++ b/docs/data_usage_guides/enum_parameters/gender_parameters.yaml @@ -0,0 +1,12 @@ +- gender: f + description: 'female' +- gender: m + description: 'male' +- gender: x + description: 'non-binary' +- gender: u + description: 'unspecified' +- gender: o + description: 'other' +- gender: all + description: 'all' diff --git a/docs/data_usage_guides/enum_parameters/ipc_phase_parameters.yaml b/docs/data_usage_guides/enum_parameters/ipc_phase_parameters.yaml new file mode 100644 index 00000000..5a4844d6 --- /dev/null +++ b/docs/data_usage_guides/enum_parameters/ipc_phase_parameters.yaml @@ -0,0 +1,21 @@ +- IPC Phase: 1 + Name: None/Minimal + Description: Households are able to meet essential food and non-food needs without engaging in atypical and unsustainable strategies to access food and income. +- IPC Phase: 2 + Name: Stressed + Description: Households have minimally adequate food consumption but are unable to afford some essential non-food expenditures without engaging in stress-coping strategies. +- IPC Phase: 3 + Name: Crisis + Description: Households either have food consumption gaps that are reflected by high or above-usual acute malnutrition, or are marginally able to meet minimum food needs but only by depleting essential livelihood assets or through crisis-coping strategies. +- IPC Phase: 4 + Name: Emergency + Description: Households either have large food consumption gaps which are reflected in very high acute malnutrition and excess mortality, or are able to mitigate large food consumption gaps but only by employing emergency livelihood strategies and asset liquidation. +- IPC Phase: 5 + Name: Catastrophe/Famine + Description: Households have an extreme lack of food and/or other basic needs even after full employment of coping strategies. Starvation, death, destitution and extremely critical acute malnutrition levels are evident. (For Famine Classification, an area needs to have extreme critical levels of acute malnutrition and mortality.) +- IPC Phase: 3+ + Name: In Need of Action + Description: Sum of population in phases 3, 4, and 5. The population in Phase 3+ does not necessarily reflect the full population in need of urgent action. This is because some households may be in Phase 2 or even 1 but only because of receipt of assistance, and thus, they may be in need of continued action. +- IPC Phase: all + Name: Population + Description: Total population diff --git a/docs/data_usage_guides/enum_parameters/ipc_type_parameters.yaml b/docs/data_usage_guides/enum_parameters/ipc_type_parameters.yaml new file mode 100644 index 00000000..db303754 --- /dev/null +++ b/docs/data_usage_guides/enum_parameters/ipc_type_parameters.yaml @@ -0,0 +1,6 @@ +- IPC type: current + Definition: Food insecurity that is occurring in the current analysis period. +- IPC type: first projection + Definition: Projected food insecurity occurring in the period immediately following the current analysis period. +- IPC type: second projection + Definition: Projected food insecurity occurring in the period immediately following the first projection period. diff --git a/docs/data_usage_guides/enum_parameters/population_group_parameters.yaml b/docs/data_usage_guides/enum_parameters/population_group_parameters.yaml new file mode 100644 index 00000000..7153f60a --- /dev/null +++ b/docs/data_usage_guides/enum_parameters/population_group_parameters.yaml @@ -0,0 +1,32 @@ +- population group: REF + description: '' +- population group: ROC + description: '' +- population group: ASY + description: '' +- population group: OIP + description: '' +- population group: IDP + description: '' +- population group: IOC + description: '' +- population group: STA + description: '' +- population group: OOC + description: '' +- population group: HST + description: '' +- population group: RET + description: '' +- population group: RST + description: '' +- population group: NAT + description: '' +- population group: POC + description: '' +- population group: RDP + description: '' +- population group: RRI + description: '' +- population group: all + description: '' diff --git a/docs/data_usage_guides/enum_parameters/population_status_parameters.yaml b/docs/data_usage_guides/enum_parameters/population_status_parameters.yaml new file mode 100644 index 00000000..b2db9b9d --- /dev/null +++ b/docs/data_usage_guides/enum_parameters/population_status_parameters.yaml @@ -0,0 +1,12 @@ +- population status: POP + description: '' +- population status: AFF + description: '' +- population status: INN + description: '' +- population status: TGT + description: '' +- population status: REA + description: '' +- population status: all + description: '' diff --git a/docs/data_usage_guides/enum_parameters/price_flag_parameters.yaml b/docs/data_usage_guides/enum_parameters/price_flag_parameters.yaml new file mode 100644 index 00000000..5995610a --- /dev/null +++ b/docs/data_usage_guides/enum_parameters/price_flag_parameters.yaml @@ -0,0 +1,6 @@ +- price flag: actual + description: '' +- price flag: aggregate + description: '' +- price flag: actual,aggregate + description: '' diff --git a/docs/data_usage_guides/enum_parameters/price_type_parameters.yaml b/docs/data_usage_guides/enum_parameters/price_type_parameters.yaml new file mode 100644 index 00000000..bb5adf95 --- /dev/null +++ b/docs/data_usage_guides/enum_parameters/price_type_parameters.yaml @@ -0,0 +1,6 @@ +- price type: Farm Gate + description: '' +- price type: Retail + description: '' +- price type: Wholesale + description: '' diff --git a/docs/data_usage_guides/enum_parameters/risk_class_parameters.yaml b/docs/data_usage_guides/enum_parameters/risk_class_parameters.yaml new file mode 100644 index 00000000..e0d255b9 --- /dev/null +++ b/docs/data_usage_guides/enum_parameters/risk_class_parameters.yaml @@ -0,0 +1,10 @@ +- risk class: 1 + description: '' +- risk class: 2 + description: '' +- risk class: 3 + description: '' +- risk class: 4 + description: '' +- risk class: 5 + description: '' diff --git a/docs/data_usage_guides/enums.md b/docs/data_usage_guides/enums.md new file mode 100644 index 00000000..a80419ac --- /dev/null +++ b/docs/data_usage_guides/enums.md @@ -0,0 +1,116 @@ +# Enums + +## Commodity Category + +**Used in:** +[`Food Prices`](food_security_and_nutrition.md#food-prices), +[`WFP Commodity`](metadata.md#wfp-commodity) + +The commodity categories are used in the WFP food prices data to +group together similar types of food items. + +### Allowed values + +{{ read_yaml('data_usage_guides/enum_parameters/commodity_category_parameters.yaml') }} + +## Disabled Marker + +**Used in:** [`Humanitarian Needs`](affected_people.md#humanitarian-needs) + +### Allowed values + +{{ read_yaml('data_usage_guides/enum_parameters/disabled_marker_parameters.yaml') }} + +## Event Type + +**Used in:** [`Conflict Events`](coordination_and_context.md#conflict-events) + +ACLED compiles information on political violence, demonstrations, and other +non-violent but politically important events. For their public dataset, +these events are aggregated into three, non-mutually exclusive categories, +outlined in the table below. +Please see the [ACLED Codebook](https://acleddata.com/knowledge-base/codebook/#acled-events) +for their methodology and more detailed descriptions of the sub-event types. + +### Allowed values + +{{ read_yaml('data_usage_guides/enum_parameters/event_type_parameters.yaml') }} + +## Gender + +**Used in:** +[`Baseline Population`](population_and_socio-economy.md#baseline-population), +[`Humanitarian Needs`](affected_people.md#humanitarian-needs), +[`Refugees & Persons of Concern`](refugees) + +### Allowed values + +{{ read_yaml('data_usage_guides/enum_parameters/gender_parameters.yaml') }} + +## IPC Code + +**Used in:** [`Food Security`](food_security_and_nutrition.md#food-security) + +The IPC classification includes 5 different phases of increasing severity, +described in detail on page 53 of +[the IPC technical manual version 3.1](https://www.ipcinfo.org/fileadmin/user_upload/ipcinfo/manual/IPC_Technical_Manual_3_Final.pdf), +and summarised in the table below. +This enum also includes Phase 3+, to highlight the population in need of action, +and total population, used to compute fractions. + +### Allowed values + +{{ read_yaml('data_usage_guides/enum_parameters/ipc_phase_parameters.yaml') }} + +## IPC Type + +**Used in:** [`Food Security`](food_security_and_nutrition.md#food-security) + +The IPC provides different projections to aid in planning and response efforts. + +### Allowed values + +{{ read_yaml('data_usage_guides/enum_parameters/ipc_type_parameters.yaml') }} + +## Population Group + +**Used in:** +[`Humanitarian Needs`](affected_people.md#humanitarian-needs), +[`Refugees & Persons of Concern`](refugees) + +### Allowed values + +{{ read_yaml('data_usage_guides/enum_parameters/population_group_parameters.yaml') }} + +## Population Status + +**Used in:** +[`Humanitarian Needs`](affected_people.md#humanitarian-needs), + +### Allowed values + +{{ read_yaml('data_usage_guides/enum_parameters/population_status_parameters.yaml') }} + +## Price Flag + +**Used in:** [`Food Prices`](food_security_and_nutrition.md#food-prices) + +### Allowed values + +{{ read_yaml('data_usage_guides/enum_parameters/price_flag_parameters.yaml') }} + +## Price Type + +**Used in:** [`Food Prices`](food_security_and_nutrition.md#food-prices) + +### Allowed values + +{{ read_yaml('data_usage_guides/enum_parameters/price_type_parameters.yaml') }} + +## Risk Class + +**Used in:** [`National Risk`](coordination_and_context.md#national-risk) + +### Allowed values + +{{ read_yaml('data_usage_guides/enum_parameters/risk_class_parameters.yaml') }} diff --git a/docs/data_usage_guides/metadata.md b/docs/data_usage_guides/metadata.md new file mode 100644 index 00000000..7dc46828 --- /dev/null +++ b/docs/data_usage_guides/metadata.md @@ -0,0 +1,59 @@ +# Metadata + +## Dataset + +## Resource + +## Location + +## Admin1 + +## Admin2 + +## Org + +* The table is populated using + [OCHA Digital Services organization types list](https://data.humdata.org/dataset/organization-types-beta), + with the addition of: + * Civil Society + * Observer + * Development Programme + * Local NGO +* Organisation types all have an associated name and code + +## Sector + +* This table is populated using the Global Coordination Groups, with the + following additional entries: + * cash + * humanitarian assistance + * multi-sector + * intersectoral +* The sector name strings in the 3W data are normalised and then aligned to the + sector table, using the “sector_map” section of + [this configuration file](https://github.com/OCHA-DAP/hapi-pipelines/blob/main/src/hapi/pipelines/configs/core.yaml) + if needed. + In the absence of a direct match, phonetic matching is used for + strings > 5 characters. + +## Currency + +## WFP Market + +The Markets data comes from the WFP Food Prices datasets. We p-code it by … + +## WFP Commodity diff --git a/docs/data_usage_guides/subcategory_details/baseline_population_details.yaml b/docs/data_usage_guides/subcategory_details/baseline_population_details.yaml index 54f9d15f..e8031ce8 100644 --- a/docs/data_usage_guides/subcategory_details/baseline_population_details.yaml +++ b/docs/data_usage_guides/subcategory_details/baseline_population_details.yaml @@ -7,4 +7,4 @@ - Attribute: Update frequency Details: Annually - Attribute: Related tables - Details: "[`Gender`](gender)" + Details: "[`Gender`](enums.md#gender)" diff --git a/docs/data_usage_guides/subcategory_details/conflict_events_details.yaml b/docs/data_usage_guides/subcategory_details/conflict_events_details.yaml index 08c0bd41..b8d7afd0 100644 --- a/docs/data_usage_guides/subcategory_details/conflict_events_details.yaml +++ b/docs/data_usage_guides/subcategory_details/conflict_events_details.yaml @@ -7,4 +7,4 @@ - Attribute: Update frequency Details: Weekly - Attribute: Related tables - Details: "[`EventType`](event-type)" + Details: "[`EventType`](enums.md#event-type)" diff --git a/docs/data_usage_guides/subcategory_details/food_prices_details.yaml b/docs/data_usage_guides/subcategory_details/food_prices_details.yaml index acb6a473..b6b6f581 100644 --- a/docs/data_usage_guides/subcategory_details/food_prices_details.yaml +++ b/docs/data_usage_guides/subcategory_details/food_prices_details.yaml @@ -7,4 +7,4 @@ - Attribute: Update frequency Details: Weekly - Attribute: Related tables - Details: "[`Commodity`](commodity), [`Currency`](Currency), [`Market`](market)" + Details: "[`Commodity Category`](enums.md#commodity-category), [`Currency`](metadata.md#currency), [`Price Flag`](enums.md#price-flag), [`Price Type`](enums.md#price-type), [`WFP Commodity`](metadata.md#wfp-commodity), [`WFP Market`](metadata.md#wfp-market)" diff --git a/docs/data_usage_guides/subcategory_details/food_security_details.yaml b/docs/data_usage_guides/subcategory_details/food_security_details.yaml index 492db4da..c98e3c89 100644 --- a/docs/data_usage_guides/subcategory_details/food_security_details.yaml +++ b/docs/data_usage_guides/subcategory_details/food_security_details.yaml @@ -7,4 +7,4 @@ - Attribute: Update frequency Details: Annually - Attribute: Related tables - Details: "[`IPCPhase`](ipc-phase), [`IPCType`](ipc-type)" + Details: "[`IPCPhase`](enums.md#ipc-phase), [`IPCType`](enums.md#ipc-type)" diff --git a/docs/data_usage_guides/subcategory_details/humanitarian_needs_details.yaml b/docs/data_usage_guides/subcategory_details/humanitarian_needs_details.yaml index 8e5e6533..3401b352 100644 --- a/docs/data_usage_guides/subcategory_details/humanitarian_needs_details.yaml +++ b/docs/data_usage_guides/subcategory_details/humanitarian_needs_details.yaml @@ -7,4 +7,4 @@ - Attribute: Update frequency Details: Annually - Attribute: Related tables - Details: "[ `DisabledMarker` ](disabled-marker), [`Gender`](gender), [`PopulationGroup`](population-group), [`PopulationStatus`](population-status)" + Details: "[ `DisabledMarker` ](enums.md#disabled-marker), [`Gender`](enums.md#gender), [`PopulationGroup`](enums.md#population-group), [`PopulationStatus`](enums.md#population-status)" diff --git a/docs/data_usage_guides/subcategory_details/national_risk_details.yaml b/docs/data_usage_guides/subcategory_details/national_risk_details.yaml index 93a0d621..4bf04917 100644 --- a/docs/data_usage_guides/subcategory_details/national_risk_details.yaml +++ b/docs/data_usage_guides/subcategory_details/national_risk_details.yaml @@ -7,4 +7,4 @@ - Attribute: Update frequency Details: Annually - Attribute: Related tables - Details: "[`RiskClass`](risk-class)" + Details: "[`RiskClass`](enums.md#risk-class)" diff --git a/docs/data_usage_guides/subcategory_details/operational_presence_details.yaml b/docs/data_usage_guides/subcategory_details/operational_presence_details.yaml index b708bdd5..f7e50e97 100644 --- a/docs/data_usage_guides/subcategory_details/operational_presence_details.yaml +++ b/docs/data_usage_guides/subcategory_details/operational_presence_details.yaml @@ -7,4 +7,4 @@ - Attribute: Update frequency Details: Irregular, depending on the country. Common update frequencies include quarterly and annually. Please check individual resources for more information. - Attribute: Related tables - Details: "[`Org`](org), [`OrgType`](org-type), [`Sector`](sector)" + Details: "[`Org`](metadata.md#org), [`OrgType`](metadata.md#org-type), [`Sector`](metadata.md#sector)" diff --git a/docs/data_usage_guides/subcategory_details/refugees_details.yaml b/docs/data_usage_guides/subcategory_details/refugees_details.yaml index 8f07575b..31cee25c 100644 --- a/docs/data_usage_guides/subcategory_details/refugees_details.yaml +++ b/docs/data_usage_guides/subcategory_details/refugees_details.yaml @@ -7,4 +7,4 @@ - Attribute: Update frequency Details: Annually - Attribute: Related tables - Details: "[`Gender`](gender), [`PopulationGroup`](population-group)" + Details: "[`Gender`](enums.md#gender), [`PopulationGroup`](metadata.md#population-group)" diff --git a/mkdocs.yml b/mkdocs.yml index 8740d653..c9623699 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -9,6 +9,8 @@ nav: - Coordination & Context : data_usage_guides/coordination_and_context.md - Food Security & Nutrition : data_usage_guides/food_security_and_nutrition.md - Population & Socio-economy : data_usage_guides/population_and_socio-economy.md + - Metadata: data_usage_guides/metadata.md + - Enums: data_usage_guides/enums.md - Geo Data: geo.md - Contact: contact.md theme: From c4081224095ea21ad7568c79f04f9605c632ac89 Mon Sep 17 00:00:00 2001 From: Monica Turner Date: Tue, 4 Jun 2024 22:34:30 +0200 Subject: [PATCH 4/8] HDXDSYS-731 remainder of text --- .../coordination_and_context.md | 2 +- .../commodity_category_parameters.yaml | 8 - .../population_group_parameters.yaml | 34 ++-- .../population_status_parameters.yaml | 12 +- .../price_flag_parameters.yaml | 3 - .../price_type_parameters.yaml | 3 - .../risk_class_parameters.yaml | 20 +- docs/data_usage_guides/enums.md | 38 ++-- .../food_security_and_nutrition.md | 4 +- docs/data_usage_guides/metadata.md | 175 ++++++++++++++---- .../population_and_socio-economy.md | 3 + docs/index.md | 34 ++-- mkdocs.yml | 6 + 13 files changed, 228 insertions(+), 114 deletions(-) diff --git a/docs/data_usage_guides/coordination_and_context.md b/docs/data_usage_guides/coordination_and_context.md index 51d426a1..ebed82ee 100644 --- a/docs/data_usage_guides/coordination_and_context.md +++ b/docs/data_usage_guides/coordination_and_context.md @@ -25,7 +25,7 @@ identify gaps, and plan for future humanitarian response. different ways from different OCHA offices, or sometimes even within the same 3W. See the [`org`](org) section below for more information on how we handle these details. -* Organisations without an associated sector are skipped +* Rows without an associated sector are skipped * The reference period is obtained from the HDX dataset ### Usage notes diff --git a/docs/data_usage_guides/enum_parameters/commodity_category_parameters.yaml b/docs/data_usage_guides/enum_parameters/commodity_category_parameters.yaml index 7d203444..202f8302 100644 --- a/docs/data_usage_guides/enum_parameters/commodity_category_parameters.yaml +++ b/docs/data_usage_guides/enum_parameters/commodity_category_parameters.yaml @@ -1,16 +1,8 @@ - commodity category: cereals and tubers - description: '' - commodity category: meat, fish and eggs - description: '' - commodity category: milk and dairy - description: '' - commodity category: miscellaneous food - description: '' - commodity category: non-food - description: '' - commodity category: oil and fats - description: '' - commodity category: pulses and nuts - description: '' - commodity category: vegetables and fruits - description: '' diff --git a/docs/data_usage_guides/enum_parameters/population_group_parameters.yaml b/docs/data_usage_guides/enum_parameters/population_group_parameters.yaml index 7153f60a..25e03bc7 100644 --- a/docs/data_usage_guides/enum_parameters/population_group_parameters.yaml +++ b/docs/data_usage_guides/enum_parameters/population_group_parameters.yaml @@ -1,32 +1,32 @@ - population group: REF - description: '' + description: 'refugees' - population group: ROC - description: '' + description: 'people in a refugee-like situation' - population group: ASY - description: '' + description: 'asylum seekers' - population group: OIP - description: '' + description: 'other people in need of international protection' - population group: IDP - description: '' + description: 'internally displaced persons ' - population group: IOC - description: '' + description: 'people in IDP-like situation' - population group: STA - description: '' + description: 'stateless people' - population group: OOC - description: '' + description: 'others of concern' - population group: HST - description: '' + description: 'host community' - population group: RET - description: '' + description: 'returned refugees' - population group: RST - description: '' + description: 'resettled refugees' - population group: NAT - description: '' -- population group: POC - description: '' + description: 'naturalized refugees' - population group: RDP - description: '' + description: 'returned IDPs' +- population group: POC + description: 'persons of concern' - population group: RRI - description: '' + description: 'TODO: what is this' - population group: all - description: '' + description: 'all' diff --git a/docs/data_usage_guides/enum_parameters/population_status_parameters.yaml b/docs/data_usage_guides/enum_parameters/population_status_parameters.yaml index b2db9b9d..5943e576 100644 --- a/docs/data_usage_guides/enum_parameters/population_status_parameters.yaml +++ b/docs/data_usage_guides/enum_parameters/population_status_parameters.yaml @@ -1,12 +1,12 @@ - population status: POP - description: '' + description: 'population' - population status: AFF - description: '' + description: 'affected' - population status: INN - description: '' + description: 'inneed' - population status: TGT - description: '' + description: 'targeted' - population status: REA - description: '' + description: 'reached' - population status: all - description: '' + description: 'all' diff --git a/docs/data_usage_guides/enum_parameters/price_flag_parameters.yaml b/docs/data_usage_guides/enum_parameters/price_flag_parameters.yaml index 5995610a..29a4e504 100644 --- a/docs/data_usage_guides/enum_parameters/price_flag_parameters.yaml +++ b/docs/data_usage_guides/enum_parameters/price_flag_parameters.yaml @@ -1,6 +1,3 @@ - price flag: actual - description: '' - price flag: aggregate - description: '' - price flag: actual,aggregate - description: '' diff --git a/docs/data_usage_guides/enum_parameters/price_type_parameters.yaml b/docs/data_usage_guides/enum_parameters/price_type_parameters.yaml index bb5adf95..941f3970 100644 --- a/docs/data_usage_guides/enum_parameters/price_type_parameters.yaml +++ b/docs/data_usage_guides/enum_parameters/price_type_parameters.yaml @@ -1,6 +1,3 @@ - price type: Farm Gate - description: '' - price type: Retail - description: '' - price type: Wholesale - description: '' diff --git a/docs/data_usage_guides/enum_parameters/risk_class_parameters.yaml b/docs/data_usage_guides/enum_parameters/risk_class_parameters.yaml index e0d255b9..d6a50388 100644 --- a/docs/data_usage_guides/enum_parameters/risk_class_parameters.yaml +++ b/docs/data_usage_guides/enum_parameters/risk_class_parameters.yaml @@ -1,10 +1,20 @@ - risk class: 1 - description: '' + description: very low + risk_score min: 0.0 + risk score max: 1.9 - risk class: 2 - description: '' + description: low + risk score min: 2.0 + risk score max: 3.4 - risk class: 3 - description: '' + description: medium + risk score min: 3.5 + risk score_max: 4.9 - risk class: 4 - description: '' + description: high + risk score min: 5.0 + risk score max: 6.4 - risk class: 5 - description: '' + description: very high + risk score min: 6.5 + risk score max: 10 diff --git a/docs/data_usage_guides/enums.md b/docs/data_usage_guides/enums.md index a80419ac..32fd0ad7 100644 --- a/docs/data_usage_guides/enums.md +++ b/docs/data_usage_guides/enums.md @@ -6,10 +6,8 @@ [`Food Prices`](food_security_and_nutrition.md#food-prices), [`WFP Commodity`](metadata.md#wfp-commodity) -The commodity categories are used in the WFP food prices data to -group together similar types of food items. - -### Allowed values +The commodity categories are used in the WFP food prices data to organize +foods into food groups. {{ read_yaml('data_usage_guides/enum_parameters/commodity_category_parameters.yaml') }} @@ -17,7 +15,9 @@ group together similar types of food items. **Used in:** [`Humanitarian Needs`](affected_people.md#humanitarian-needs) -### Allowed values +This marker is used to indicate any HNO PIN calculations that are disaggregated +by disability. For more information on disability classification, see the +[WHO International Classification of Functioning, Disability and Health (ICF)](https://www.who.int/standards/classifications/international-classification-of-functioning-disability-and-health). {{ read_yaml('data_usage_guides/enum_parameters/disabled_marker_parameters.yaml') }} @@ -32,8 +32,6 @@ outlined in the table below. Please see the [ACLED Codebook](https://acleddata.com/knowledge-base/codebook/#acled-events) for their methodology and more detailed descriptions of the sub-event types. -### Allowed values - {{ read_yaml('data_usage_guides/enum_parameters/event_type_parameters.yaml') }} ## Gender @@ -43,7 +41,7 @@ for their methodology and more detailed descriptions of the sub-event types. [`Humanitarian Needs`](affected_people.md#humanitarian-needs), [`Refugees & Persons of Concern`](refugees) -### Allowed values +Several sub-categories in HDX-HAPI are disaggregated by gender. {{ read_yaml('data_usage_guides/enum_parameters/gender_parameters.yaml') }} @@ -58,8 +56,6 @@ and summarised in the table below. This enum also includes Phase 3+, to highlight the population in need of action, and total population, used to compute fractions. -### Allowed values - {{ read_yaml('data_usage_guides/enum_parameters/ipc_phase_parameters.yaml') }} ## IPC Type @@ -68,17 +64,16 @@ and total population, used to compute fractions. The IPC provides different projections to aid in planning and response efforts. -### Allowed values - {{ read_yaml('data_usage_guides/enum_parameters/ipc_type_parameters.yaml') }} ## Population Group **Used in:** [`Humanitarian Needs`](affected_people.md#humanitarian-needs), -[`Refugees & Persons of Concern`](refugees) +[`Refugees & Persons of Concern`](affected_people.md#refugees) -### Allowed values +Most population group terms come from [UNHCR's glossary](https://www.unhcr.org/refugee-statistics/methodology/data-content/), +while RRI comes from the humanitarian needs data. {{ read_yaml('data_usage_guides/enum_parameters/population_group_parameters.yaml') }} @@ -87,7 +82,8 @@ The IPC provides different projections to aid in planning and response efforts. **Used in:** [`Humanitarian Needs`](affected_people.md#humanitarian-needs), -### Allowed values +The population status disaggregation provides a framework for the HNO to +determine how effectively people in need are being reached. {{ read_yaml('data_usage_guides/enum_parameters/population_status_parameters.yaml') }} @@ -95,22 +91,24 @@ The IPC provides different projections to aid in planning and response efforts. **Used in:** [`Food Prices`](food_security_and_nutrition.md#food-prices) -### Allowed values - {{ read_yaml('data_usage_guides/enum_parameters/price_flag_parameters.yaml') }} ## Price Type **Used in:** [`Food Prices`](food_security_and_nutrition.md#food-prices) -### Allowed values - {{ read_yaml('data_usage_guides/enum_parameters/price_type_parameters.yaml') }} ## Risk Class **Used in:** [`National Risk`](coordination_and_context.md#national-risk) -### Allowed values +In national risk, the final INFORM risk score is grouped into one of five +risk classes, with the objective of systemically identifying risk in a +consistent manner. Note that other categories and dimensions are also +grouped into classes but with different thresholds, and these are not included +in HDX HAPI. Please see the +[methodology](https://drmkc.jrc.ec.europa.eu/inform-index/INFORM-Risk/Methodology) +for more details. {{ read_yaml('data_usage_guides/enum_parameters/risk_class_parameters.yaml') }} diff --git a/docs/data_usage_guides/food_security_and_nutrition.md b/docs/data_usage_guides/food_security_and_nutrition.md index e433fdbe..4c376c8b 100644 --- a/docs/data_usage_guides/food_security_and_nutrition.md +++ b/docs/data_usage_guides/food_security_and_nutrition.md @@ -39,7 +39,9 @@ a future release we will p-code and expand coverage to other IPC datasets. The World Food Programme Price Database covers foods such as maize, rice, beans, fish, and sugar for 98 countries and some 3000 markets. It is updated -weekly but primarily contains data with a monthly update frequency. +weekly but primarily contains data with a monthly update frequency. For a +detailed methodology, see WFP's +[Market Analysis Guidelines](https://www.wfp.org/publications/market-analysis-guidelines). ### Summary diff --git a/docs/data_usage_guides/metadata.md b/docs/data_usage_guides/metadata.md index 7dc46828..8b3d0458 100644 --- a/docs/data_usage_guides/metadata.md +++ b/docs/data_usage_guides/metadata.md @@ -1,59 +1,166 @@ # Metadata -## Dataset +## HDX Metadata -## Resource +All of the data in HDX HAPI comes from publicly-available datasets on HDX, and +the HDX metadata can be used refer back to the original data to see what HAPI +has simplified or omitted. +The two primary HDX metadata tables are +[`dataset`](dataset) and [`resource`](resource), and they +contain a small subset of the metadata available through the CKAN API or +HDX library. + +The HAPI data records contain enough information to access the full records +from CKAN, see the parameters tables for more details. +All subcategory tables link back to a resource record, and all resources link +back to a dataset record. + +### Dataset + +**Used in:** [`Resource`](resource), all sub-category tables + +This table contains the HDX-specific metadata associated with all datasets +used to populate the HAPI sub-category tables. Every dataset has at least +one child [resource](resource). + +### Resource + +**Used in:** All sub-category tables + +This table contains the HDX-specific metadata associated with all resources +used to populate the HAPI sub-category tables. Every resource has one +parent [dataset](dataset). + +## Geographical Metadata + +HAPI supports three hierarchical levels of geographical metadata: +location (a country or country-like entity), admin 1, and admin 2. An entry in +the location table does not necessarily imply UN recognition of statehood. +The subcategory data tables link to the lowest administrative level used by +that data type; it will usually be admin2, but in some cases may be admin1 or +location. + +The names and p-codes are read in from a [global p-code list](https://data.humdata.org/dataset/global-pcodes) +taken from the common operational dataset (COD) gazetteers administrative +boundaries. +The p-code field is unique only in combination with reference_period_start, +since p-codes may be reused in different versions of geographical metadata. + +When bringing in data from sub-categories, HDX HAPI uses p-code matching +from [`hdx-python-country`](https://hdx-python-country.readthedocs.io/en/latest/). +Thie can involve .... TODO ## Location -## Admin1 +**Used in:** [`Admin 1`](admin1), +[`Refugees & Persons of Concern`](affected_people.md#refugees), +[`Funding`](coordination_and_context.md#funding), +[`National Risk`](coordination_and_context.md#national-risk) + +Country or country-like entities in HDX HAPI, from the CODs +[global p-code list](https://data.humdata.org/dataset/global-pcodes). + +## Admin 1 + +**Used in:** [`Admin 2](admin2), +[`Poverty Rate`](population_and_socio-economy.md#poverty-rate) + +Admin 1 level names and p-codes in HDX HAPI, from the CODs +[global p-code list](https://data.humdata.org/dataset/global-pcodes). + +## Admin 2 -## Admin2 +Admin 2 level names and p-codes in HDX HAPI, from the CODs +[global p-code list](https://data.humdata.org/dataset/global-pcodes). -## Org + +**Used in:** +[`3W - Who is Doing What Where`](coordination_and_context.md#operational-presence) + +The organisation table is populated from the 3W data, using the following +methodology: -* The organisation table is populated from the 3W data * Organisation name and acronym strings are normalised. If an acronym isn’t available, the first 32 characters of the name are used. * This [organisation mapping](https://docs.google.com/spreadsheets/d/e/2PACX-1vSfBWvSu3fKA743VvHtgf-pIGkYH7zhy-NP7DZgEV9_a6YU7vtCeWhbLM56aUL1iIfrfv5UBvvjVt7B/pub?gid=1040329566&single=true&output=csv) is used for common alternative names -* Organisations without a sector are excluded -* Organisations can have an associated organisation type. If available, the +* Organisations must have an associated organisation type. If available, the organisation type code is taken directly from the 3W data, otherwise the name string is normalised and matched to the org type names. In the absence of a direct match, phonetic matching is used for strings > 5 characters. If no match is found, the organisation is skipped. -## Org Type +### Org Type + +**Used in:** +[`Org`](org), +[`3W - Who is Doing What Where`](coordination_and_context.md#operational-presence) + +The table is populated using +[OCHA Digital Services organization types list](https://data.humdata.org/dataset/organization-types-beta), +with the addition of: + +* Civil Society +* Observer +* Development Programme +* Local NGO. + +Organisation types all have an associated name and code. + +### Sector + +**Used in:** +[`3W - Who is Doing What Where`](coordination_and_context.md#operational-presence), +[`Humanitarian Needs`](affected_people.md#humanitarian-needs) + +This table is populated using the Global Coordination Groups, with the +following additional entries: + +* cash +* humanitarian assistance +* multi-sector +* intersectoral. + +The sector name strings in the 3W data are normalised and then aligned to the +sector table, using the “sector_map” section of +[this configuration file](https://github.com/OCHA-DAP/hapi-pipelines/blob/main/src/hapi/pipelines/configs/core.yaml) +if needed. +In the absence of a direct match, phonetic matching is used for +strings > 5 characters. + +### Currency + +**Used in:** +[`Food Prices`](food_security_and_nutrition.md#food-prices) -* The table is populated using - [OCHA Digital Services organization types list](https://data.humdata.org/dataset/organization-types-beta), - with the addition of: - * Civil Society - * Observer - * Development Programme - * Local NGO -* Organisation types all have an associated name and code +The currency table is populated using the WFP VAM Data Bridges API. -## Sector +### WFP Market -* This table is populated using the Global Coordination Groups, with the - following additional entries: - * cash - * humanitarian assistance - * multi-sector - * intersectoral -* The sector name strings in the 3W data are normalised and then aligned to the - sector table, using the “sector_map” section of - [this configuration file](https://github.com/OCHA-DAP/hapi-pipelines/blob/main/src/hapi/pipelines/configs/core.yaml) - if needed. - In the absence of a direct match, phonetic matching is used for - strings > 5 characters. +**Used in:** +[`Food Prices`](food_security_and_nutrition.md#food-prices) -## Currency +Markets are defined as the physical locations where buyers and sellers +come together to trade goods and services. This table is populated from the +food prices data, which contains location names down to admin 2 as well as +a latitude and longitude, however is not p-coded. +We therefore p-code .... TODO -## WFP Market +### WFP Commodity -The Markets data comes from the WFP Food Prices datasets. We p-code it by … +**Used in:** +[`Food Prices`](food_security_and_nutrition.md#food-prices) -## WFP Commodity +The commodity table tracks all food items, and their associated +commodity category, present in the food prices data. diff --git a/docs/data_usage_guides/population_and_socio-economy.md b/docs/data_usage_guides/population_and_socio-economy.md index 8bc4a4de..663c7cf2 100644 --- a/docs/data_usage_guides/population_and_socio-economy.md +++ b/docs/data_usage_guides/population_and_socio-economy.md @@ -51,5 +51,8 @@ details. ### Usage Notes +* The data is disaggregated to admin 1, but not p-coded. We have kept the + admin 1 names in the data, but link only to national level p-codes. + We plan to p-code this data in a future release. * Please be aware of the metric definitions, provided in the parameter summary table !!!! TODO add link and table diff --git a/docs/index.md b/docs/index.md index 0e0f660a..69516b88 100644 --- a/docs/index.md +++ b/docs/index.md @@ -2,16 +2,14 @@ --- - -The [HDX Humanitarian API](https://dev.data-humdata-org.ahconu.org/hapi)(HDX HAPI) is a way to access standardised indicators from multiple sources to automate workflows and visualisations - +The [HDX Humanitarian API](https://dev.data-humdata-org.ahconu.org/hapi)(HDX HAPI) is a way to access standardised indicators from multiple sources to automate workflows and visualisations HDX HAPI is in beta phase, and we are seeking feedback. To share your thoughts or join our slack channel, send an email to [hdx@un.org](hdx@un.org). - The initial scope of HDX HAPI will be the data included in the [HDX Data Grids](https://data.humdata.org/dashboards/overview-of-data-grids). Work is ongoing to add more data. # App Identifier + To access HDX HAPI you need to generate an API identifier. This can be done via the the [sandbox interface encode_identifier endpoint](https://stage.hapi-humdata-org.ahconu.org/docs#/Utility/get_encoded_identifier_api_v1_encode_identifier_get). Enter your application name and email address and it will return the app identifier. The key must be included as a query string parameter e.g. ``` @@ -20,7 +18,6 @@ https://stage.hapi-humdata-org.ahconu.org/api/v1/themes/3w?app_identifier={your # The Structure of HDX HAPI - ## Data Subcategory Endpoints HDX HAPI is organised around a set of key humanitarian data subcategories like **Baseline Population** and **Conflict Events**. Each of these subcategories can be queried via its endpoint. @@ -28,25 +25,30 @@ HDX HAPI is organised around a set of key humanitarian data subcategories like * #### Affected People -- [Humanitarian Needs](https://stage.hapi-humdata-org.ahconu.org/docs#/Affected%20people/get_humanitarian_needs_api_v1_affected_people_humanitarian_needs_get) -- [Refugees](https://stage.hapi-humdata-org.ahconu.org/docs#/Affected%20people/get_refugees_api_v1_affected_people_refugees_get) +- [Refugees & Persons of Concern](https://stage.hapi-humdata-org.ahconu.org/docs#/Affected%20people/get_refugees_api_v1_affected_people_refugees_get) +- [Humanitarian Needs](https://stage.hapi-humdata-org.ahconu.org/docs#/Affected%20people/get_humanitarian_needs_api_v1_affected_people_humanitarian_needs_get) -#### Coordination and Context -- [Conflict Events](https://stage.hapi-humdata-org.ahconu.org/docs#/Conflict%20Events/get_conflict_events_api_v1_coordination_context_conflict_event_get) +#### Coordination & Context + +- [3W - Who is Doing What Where](https://stage.hapi-humdata-org.ahconu.org/docs#/3W%20Operational%20Presence/get_operational_presences_api_v1_coordination_context_operational_presence_get) - [Funding](https://stage.hapi-humdata-org.ahconu.org/docs#/Funding/get_fundings_api_v1_coordination_context_funding_get) -- [National risk](https://stage.hapi-humdata-org.ahconu.org/docs#/National%20Risk/get_national_risks_api_v1_coordination_context_national_risk_get) -- [Operational Presence (3W)](https://stage.hapi-humdata-org.ahconu.org/docs#/3W%20Operational%20Presence/get_operational_presences_api_v1_coordination_context_operational_presence_get) +- [Conflict Events](https://stage.hapi-humdata-org.ahconu.org/docs#/Conflict%20Events/get_conflict_events_api_v1_coordination_context_conflict_event_get) +- [National Risk](https://stage.hapi-humdata-org.ahconu.org/docs#/National%20Risk/get_national_risks_api_v1_coordination_context_national_risk_get) + +#### Food Security & Nutrition -#### Food -- [Food Prices](https://stage.hapi-humdata-org.ahconu.org/docs#/Food%20Security%20%26%20Nutrition/get_food_prices_api_v1_food_food_price_get) - [Food Security](https://stage.hapi-humdata-org.ahconu.org/docs#/Food%20Security%20%26%20Nutrition/get_food_security_api_v1_food_food_security_get) +- [Food Prices](https://stage.hapi-humdata-org.ahconu.org/docs#/Food%20Security%20%26%20Nutrition/get_food_prices_api_v1_food_food_price_get) -#### Population Social -- [Population](https://stage.hapi-humdata-org.ahconu.org/docs#/Baseline%20Population/get_populations_api_v1_population_social_population_get) +#### Population & Socio-economy + +- [Baseline Population](https://stage.hapi-humdata-org.ahconu.org/docs#/Baseline%20Population/get_populations_api_v1_population_social_population_get) - [Poverty Rate](https://stage.hapi-humdata-org.ahconu.org/docs#/Baseline%20Population/get_poverty_rates_api_v1_population_social_poverty_rate_get) # Terms Of Use + [The HDX Terms of Service](https://data.humdata.org/faqs/terms) ## FAQS -Please [refer to the landing page](https://dev.data-humdata-org.ahconu.org/hapi) for non-technical FAQs \ No newline at end of file + +Please [refer to the landing page](https://dev.data-humdata-org.ahconu.org/hapi) for non-technical FAQs diff --git a/mkdocs.yml b/mkdocs.yml index c9623699..de4dd6b1 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -16,7 +16,13 @@ nav: theme: name: "material" markdown_extensions: + - pymdownx.highlight: + anchor_linenums: true + line_spans: __span + pygments_lang_class: true + - pymdownx.inlinehilite - pymdownx.snippets + - pymdownx.superfences plugins: - search - table-reader: From cd7afe6c6a208a6f4918521d5304bbabf0addfa8 Mon Sep 17 00:00:00 2001 From: Monica Turner Date: Tue, 4 Jun 2024 23:21:00 +0200 Subject: [PATCH 5/8] HDXDSYS-731 formatting and link fixes --- .../event_type_parameters.yaml | 14 ++++++-- .../enum_parameters/ipc_phase_parameters.yaml | 34 +++++++++++++++---- .../enum_parameters/ipc_type_parameters.yaml | 11 ++++-- docs/data_usage_guides/enums.md | 2 +- .../population_and_socio-economy.md | 6 ++-- .../baseline_population_details.yaml | 10 ------ .../conflict_events_details.yaml | 6 ++-- .../food_prices_details.yaml | 11 ++++-- .../food_security_details.yaml | 6 ++-- .../subcategory_details/funding_details.yaml | 3 +- .../humanitarian_needs_details.yaml | 6 +++- .../national_risk_details.yaml | 6 ++-- .../operational_presence_details.yaml | 8 +++-- .../population_details.yaml | 12 +++++++ .../poverty_rate_details.yaml | 3 +- .../subcategory_details/refugees_details.yaml | 7 ++-- 16 files changed, 103 insertions(+), 42 deletions(-) delete mode 100644 docs/data_usage_guides/subcategory_details/baseline_population_details.yaml create mode 100644 docs/data_usage_guides/subcategory_details/population_details.yaml diff --git a/docs/data_usage_guides/enum_parameters/event_type_parameters.yaml b/docs/data_usage_guides/enum_parameters/event_type_parameters.yaml index e204b6f1..1e29cb5c 100644 --- a/docs/data_usage_guides/enum_parameters/event_type_parameters.yaml +++ b/docs/data_usage_guides/enum_parameters/event_type_parameters.yaml @@ -1,6 +1,14 @@ - event type: civilian_targeting - description: Civilian targeting events include violence against civilians events and explosions/remote violence events in which civilians were directly targeted. + description: > + Civilian targeting events include violence against civilians events and + explosions/remote violence events in which civilians were directly + targeted. - event type: demonstration - description: Demonstration events include ACLED’s protests and riots event types, with the exception of the mob violence sub-event type of the riots event type. + description: > + Demonstration events include ACLED’s protests and riots event types, with + the exception of the mob violence sub-event type of the riots event type. - event type: political_violence - description: Political violence events include ACLED’s battles, violence against civilians, and explosions/remote violence event types, as well as the mob violence sub-event type of the riots event type. + description: > + Political violence events include ACLED’s battles, violence against + civilians, and explosions/remote violence event types, as well as the mob + violence sub-event type of the riots event type. diff --git a/docs/data_usage_guides/enum_parameters/ipc_phase_parameters.yaml b/docs/data_usage_guides/enum_parameters/ipc_phase_parameters.yaml index 5a4844d6..0860603b 100644 --- a/docs/data_usage_guides/enum_parameters/ipc_phase_parameters.yaml +++ b/docs/data_usage_guides/enum_parameters/ipc_phase_parameters.yaml @@ -1,21 +1,43 @@ - IPC Phase: 1 Name: None/Minimal - Description: Households are able to meet essential food and non-food needs without engaging in atypical and unsustainable strategies to access food and income. + Description: > + Households are able to meet essential food and non-food needs without + engaging in atypical and unsustainable strategies to access food and income. - IPC Phase: 2 Name: Stressed - Description: Households have minimally adequate food consumption but are unable to afford some essential non-food expenditures without engaging in stress-coping strategies. + Description: > + Households have minimally adequate food consumption but are unable to + afford some essential non-food expenditures without engaging in + stress-coping strategies. - IPC Phase: 3 Name: Crisis - Description: Households either have food consumption gaps that are reflected by high or above-usual acute malnutrition, or are marginally able to meet minimum food needs but only by depleting essential livelihood assets or through crisis-coping strategies. + Description: > + Households either have food consumption gaps that are reflected by high or + above-usual acute malnutrition, or are marginally able to meet minimum food + needs but only by depleting essential livelihood assets or through + crisis-coping strategies. - IPC Phase: 4 Name: Emergency - Description: Households either have large food consumption gaps which are reflected in very high acute malnutrition and excess mortality, or are able to mitigate large food consumption gaps but only by employing emergency livelihood strategies and asset liquidation. + Description: > + Households either have large food consumption gaps which are reflected in + very high acute malnutrition and excess mortality, or are able to mitigate + large food consumption gaps but only by employing emergency livelihood + strategies and asset liquidation. - IPC Phase: 5 Name: Catastrophe/Famine - Description: Households have an extreme lack of food and/or other basic needs even after full employment of coping strategies. Starvation, death, destitution and extremely critical acute malnutrition levels are evident. (For Famine Classification, an area needs to have extreme critical levels of acute malnutrition and mortality.) + Description: > + Households have an extreme lack of food and/or other basic needs even after + full employment of coping strategies. Starvation, death, destitution and + extremely critical acute malnutrition levels are evident. (For Famine + Classification, an area needs to have extreme critical levels of acute + malnutrition and mortality.) - IPC Phase: 3+ Name: In Need of Action - Description: Sum of population in phases 3, 4, and 5. The population in Phase 3+ does not necessarily reflect the full population in need of urgent action. This is because some households may be in Phase 2 or even 1 but only because of receipt of assistance, and thus, they may be in need of continued action. + Description: > + Sum of population in phases 3, 4, and 5. The population in Phase 3+ does + not necessarily reflect the full population in need of urgent action. This + is because some households may be in Phase 2 or even 1 but only because of + receipt of assistance, and thus, they may be in need of continued action. - IPC Phase: all Name: Population Description: Total population diff --git a/docs/data_usage_guides/enum_parameters/ipc_type_parameters.yaml b/docs/data_usage_guides/enum_parameters/ipc_type_parameters.yaml index db303754..1eb96198 100644 --- a/docs/data_usage_guides/enum_parameters/ipc_type_parameters.yaml +++ b/docs/data_usage_guides/enum_parameters/ipc_type_parameters.yaml @@ -1,6 +1,11 @@ - IPC type: current - Definition: Food insecurity that is occurring in the current analysis period. + Definition: > + Food insecurity that is occurring in the current analysis period. - IPC type: first projection - Definition: Projected food insecurity occurring in the period immediately following the current analysis period. + Definition: > + Projected food insecurity occurring in the period immediately following the + current analysis period. - IPC type: second projection - Definition: Projected food insecurity occurring in the period immediately following the first projection period. + Definition: > + Projected food insecurity occurring in the period immediately following the + first projection period. diff --git a/docs/data_usage_guides/enums.md b/docs/data_usage_guides/enums.md index 32fd0ad7..a253c3cb 100644 --- a/docs/data_usage_guides/enums.md +++ b/docs/data_usage_guides/enums.md @@ -37,7 +37,7 @@ for their methodology and more detailed descriptions of the sub-event types. ## Gender **Used in:** -[`Baseline Population`](population_and_socio-economy.md#baseline-population), +[`Baseline Population`](population_and_socio-economy.md#population), [`Humanitarian Needs`](affected_people.md#humanitarian-needs), [`Refugees & Persons of Concern`](refugees) diff --git a/docs/data_usage_guides/population_and_socio-economy.md b/docs/data_usage_guides/population_and_socio-economy.md index 663c7cf2..09aca4a5 100644 --- a/docs/data_usage_guides/population_and_socio-economy.md +++ b/docs/data_usage_guides/population_and_socio-economy.md @@ -1,6 +1,6 @@ # Population & Socio-economy -## Baseline Population +## Baseline Population The population statistics presented here are sourced from the [common operational datasets](https://cod.unocha.org/) (CODs), and are @@ -12,7 +12,7 @@ sources. ### Summary -{{ read_yaml('data_usage_guides/subcategory_details/baseline_population_details.yaml') }} +{{ read_yaml('data_usage_guides/subcategory_details/population_details.yaml') }} ### Transformations applied @@ -54,5 +54,3 @@ details. * The data is disaggregated to admin 1, but not p-coded. We have kept the admin 1 names in the data, but link only to national level p-codes. We plan to p-code this data in a future release. -* Please be aware of the metric definitions, provided in the parameter - summary table !!!! TODO add link and table diff --git a/docs/data_usage_guides/subcategory_details/baseline_population_details.yaml b/docs/data_usage_guides/subcategory_details/baseline_population_details.yaml deleted file mode 100644 index e8031ce8..00000000 --- a/docs/data_usage_guides/subcategory_details/baseline_population_details.yaml +++ /dev/null @@ -1,10 +0,0 @@ -- Attribute: Data provider - Details: UNFPA, OCHA country offices -- Attribute: Dataset - Details: "[COD - subnational population statistics data series](https://data.humdata.org/dataset/?dataseries_name=COD+-+Subnational+Population+Statistics)" -- Attribute: Time series - Details: No -- Attribute: Update frequency - Details: Annually -- Attribute: Related tables - Details: "[`Gender`](enums.md#gender)" diff --git a/docs/data_usage_guides/subcategory_details/conflict_events_details.yaml b/docs/data_usage_guides/subcategory_details/conflict_events_details.yaml index b8d7afd0..be44497d 100644 --- a/docs/data_usage_guides/subcategory_details/conflict_events_details.yaml +++ b/docs/data_usage_guides/subcategory_details/conflict_events_details.yaml @@ -1,10 +1,12 @@ - Attribute: Data provider Details: ACLED - Attribute: Dataset - Details: "[ACLED Conflict Events Data series](https://data.humdata.org/dataset/?dataseries_name=ACLED+-+Conflict+Events)" + Details: > + [ACLED Conflict Events Data series](https://data.humdata.org/dataset/?dataseries_name=ACLED+-+Conflict+Events) - Attribute: Time series Details: Monthly - Attribute: Update frequency Details: Weekly - Attribute: Related tables - Details: "[`EventType`](enums.md#event-type)" + Details: > + [`EventType`](enums.md#event-type) diff --git a/docs/data_usage_guides/subcategory_details/food_prices_details.yaml b/docs/data_usage_guides/subcategory_details/food_prices_details.yaml index b6b6f581..1bb168b8 100644 --- a/docs/data_usage_guides/subcategory_details/food_prices_details.yaml +++ b/docs/data_usage_guides/subcategory_details/food_prices_details.yaml @@ -1,10 +1,17 @@ - Attribute: Data provider Details: The World Food Programme - Attribute: Dataset - Details: "[WFP Food Prices data series](https://data.humdata.org/dataset/?dataseries_name=WFP+-+Food+Prices)" + Details: > + [WFP Food Prices data series](https://data.humdata.org/dataset/?dataseries_name=WFP+-+Food+Prices) - Attribute: Time series Details: Yes, monthly - Attribute: Update frequency Details: Weekly - Attribute: Related tables - Details: "[`Commodity Category`](enums.md#commodity-category), [`Currency`](metadata.md#currency), [`Price Flag`](enums.md#price-flag), [`Price Type`](enums.md#price-type), [`WFP Commodity`](metadata.md#wfp-commodity), [`WFP Market`](metadata.md#wfp-market)" + Details: > + [`Commodity Category`](enums.md#commodity-category), + [`Currency`](metadata.md#currency), + [`Price Flag`](enums.md#price-flag), + [`Price Type`](enums.md#price-type), + [`WFP Commodity`](metadata.md#wfp-commodity), + [`WFP Market`](metadata.md#wfp-market) diff --git a/docs/data_usage_guides/subcategory_details/food_security_details.yaml b/docs/data_usage_guides/subcategory_details/food_security_details.yaml index c98e3c89..40b5c09d 100644 --- a/docs/data_usage_guides/subcategory_details/food_security_details.yaml +++ b/docs/data_usage_guides/subcategory_details/food_security_details.yaml @@ -1,10 +1,12 @@ - Attribute: Data provider Details: Food Security and Nutrition Working Group, West and Central Africa - Attribute: Dataset - Details: "[West & Central Africa Food Security Data - Cadre Harmonise (CH) and Integrated Food Security Phase Classification (IPC) data](https://data.humdata.org/dataset/cadre-harmonise)" + Details: > + [West & Central Africa Food Security Data - Cadre Harmonise (CH) and Integrated Food Security Phase Classification (IPC) data](https://data.humdata.org/dataset/cadre-harmonise) - Attribute: Time series Details: Yes, with projections - Attribute: Update frequency Details: Annually - Attribute: Related tables - Details: "[`IPCPhase`](enums.md#ipc-phase), [`IPCType`](enums.md#ipc-type)" + Details: > + [`IPCPhase`](enums.md#ipc-phase), [`IPCType`](enums.md#ipc-type) diff --git a/docs/data_usage_guides/subcategory_details/funding_details.yaml b/docs/data_usage_guides/subcategory_details/funding_details.yaml index d4cc939c..5da5ee52 100644 --- a/docs/data_usage_guides/subcategory_details/funding_details.yaml +++ b/docs/data_usage_guides/subcategory_details/funding_details.yaml @@ -1,7 +1,8 @@ - Attribute: Data provider Details: OCHA FTS - Attribute: Dataset - Details: "[OCHA FTS - Requirements and Funding Data series](https://data.humdata.org/dataset/?dataseries_name=OCHA+FTS+-+Requirements+and+Funding+Data)" + Details: > + [OCHA FTS - Requirements and Funding Data series](https://data.humdata.org/dataset/?dataseries_name=OCHA+FTS+-+Requirements+and+Funding+Data) - Attribute: Time series Details: Data contains funding since 2006, but the timepoints are not regular - Attribute: Update frequency diff --git a/docs/data_usage_guides/subcategory_details/humanitarian_needs_details.yaml b/docs/data_usage_guides/subcategory_details/humanitarian_needs_details.yaml index 3401b352..bd2f65cb 100644 --- a/docs/data_usage_guides/subcategory_details/humanitarian_needs_details.yaml +++ b/docs/data_usage_guides/subcategory_details/humanitarian_needs_details.yaml @@ -7,4 +7,8 @@ - Attribute: Update frequency Details: Annually - Attribute: Related tables - Details: "[ `DisabledMarker` ](enums.md#disabled-marker), [`Gender`](enums.md#gender), [`PopulationGroup`](enums.md#population-group), [`PopulationStatus`](enums.md#population-status)" + Details: > + [ `DisabledMarker` ](enums.md#disabled-marker), + [`Gender`](enums.md#gender), + [`PopulationGroup`](enums.md#population-group), + [`PopulationStatus`](enums.md#population-status) diff --git a/docs/data_usage_guides/subcategory_details/national_risk_details.yaml b/docs/data_usage_guides/subcategory_details/national_risk_details.yaml index 4bf04917..bb5b5373 100644 --- a/docs/data_usage_guides/subcategory_details/national_risk_details.yaml +++ b/docs/data_usage_guides/subcategory_details/national_risk_details.yaml @@ -1,10 +1,12 @@ - Attribute: Data provider Details: INFORM - Attribute: Dataset - Details: "[INFORM Risk Index](https://data.humdata.org/dataset/inform-risk-index-2021)" + Details: > + [INFORM Risk Index](https://data.humdata.org/dataset/inform-risk-index-2021) - Attribute: Time series Details: Not available for this subcategory - Attribute: Update frequency Details: Annually - Attribute: Related tables - Details: "[`RiskClass`](enums.md#risk-class)" + Details: > + [`RiskClass`](enums.md#risk-class) diff --git a/docs/data_usage_guides/subcategory_details/operational_presence_details.yaml b/docs/data_usage_guides/subcategory_details/operational_presence_details.yaml index f7e50e97..c2a463eb 100644 --- a/docs/data_usage_guides/subcategory_details/operational_presence_details.yaml +++ b/docs/data_usage_guides/subcategory_details/operational_presence_details.yaml @@ -1,10 +1,14 @@ - Attribute: Data provider Details: OCHA country and regional offices - Attribute: Dataset - Details: "[Operational Presence tag on HDX](https://data.humdata.org/dataset/?vocab_Topics=operational%20presence&q=&sort=last_modified%20desc&ext_page_size=25)" + Details: > + [Operational Presence tag on HDX](https://data.humdata.org/dataset/?vocab_Topics=operational%20presence&q=&sort=last_modified%20desc&ext_page_size=25) - Attribute: Time series Details: Not available for this subcategory. While older resources are available on HDX, HDX HAPI currently only supports the latest resource. - Attribute: Update frequency Details: Irregular, depending on the country. Common update frequencies include quarterly and annually. Please check individual resources for more information. - Attribute: Related tables - Details: "[`Org`](metadata.md#org), [`OrgType`](metadata.md#org-type), [`Sector`](metadata.md#sector)" + Details: > + [`Org`](metadata.md#org), + [`OrgType`](metadata.md#org-type), + [`Sector`](sector) diff --git a/docs/data_usage_guides/subcategory_details/population_details.yaml b/docs/data_usage_guides/subcategory_details/population_details.yaml new file mode 100644 index 00000000..c063c435 --- /dev/null +++ b/docs/data_usage_guides/subcategory_details/population_details.yaml @@ -0,0 +1,12 @@ +- Attribute: Data provider + Details: UNFPA, OCHA country offices +- Attribute: Dataset + Details: > + [COD - subnational population statistics data series](https://data.humdata.org/dataset/?dataseries_name=COD+-+Subnational+Population+Statistics) +- Attribute: Time series + Details: No +- Attribute: Update frequency + Details: Annually +- Attribute: Related tables + Details: > + [`Gender`](enums.md#gender) diff --git a/docs/data_usage_guides/subcategory_details/poverty_rate_details.yaml b/docs/data_usage_guides/subcategory_details/poverty_rate_details.yaml index 71fe4b9f..39096d5a 100644 --- a/docs/data_usage_guides/subcategory_details/poverty_rate_details.yaml +++ b/docs/data_usage_guides/subcategory_details/poverty_rate_details.yaml @@ -1,7 +1,8 @@ - Attribute: Data provider Details: Oxford Poverty and Human Development Initiative (OPHI) - Attribute: Dataset - Details: "[Oxford Poverty and Human Development Initiative - Global Multidimensional Poverty Index data series](https://data.humdata.org/dataset/?dataseries_name=Oxford+Poverty+and+Human+Development+Initiative+-+Global+Multidimensional+Poverty+Index)" + Details: > + [Oxford Poverty and Human Development Initiative - Global Multidimensional Poverty Index data series](https://data.humdata.org/dataset/?dataseries_name=Oxford+Poverty+and+Human+Development+Initiative+-+Global+Multidimensional+Poverty+Index)" - Attribute: Time series Details: Annual, with some combined years - Attribute: Update frequency diff --git a/docs/data_usage_guides/subcategory_details/refugees_details.yaml b/docs/data_usage_guides/subcategory_details/refugees_details.yaml index 31cee25c..f6c9b59c 100644 --- a/docs/data_usage_guides/subcategory_details/refugees_details.yaml +++ b/docs/data_usage_guides/subcategory_details/refugees_details.yaml @@ -1,10 +1,13 @@ - Attribute: Data provider Details: UNHCR - Attribute: Dataset - Details: "[Data on forcibly displaced populations and stateless persons (Global)](https://data.humdata.org/dataset/unhcr-population-data-for-world)" + Details: > + [Data on forcibly displaced populations and stateless persons (Global)](https://data.humdata.org/dataset/unhcr-population-data-for-world)" - Attribute: Time series Details: Annual, dating back to 2001 - Attribute: Update frequency Details: Annually - Attribute: Related tables - Details: "[`Gender`](enums.md#gender), [`PopulationGroup`](metadata.md#population-group)" + Details: > + [`Gender`](enums.md#gender), + [`PopulationGroup`](metadata.md#population-group)" From 85b818dec2537de023b88aadae134a4d11be5eb2 Mon Sep 17 00:00:00 2001 From: Monica Turner Date: Wed, 5 Jun 2024 10:27:16 +0200 Subject: [PATCH 6/8] HDXDSYS-731 Mike and Ian comments --- docs/data_usage_guides/affected_people.md | 3 +- .../coordination_and_context.md | 22 +++++-------- .../population_status_parameters.yaml | 2 +- .../price_flag_parameters.yaml | 5 +++ docs/data_usage_guides/enums.md | 16 ++++++++-- docs/data_usage_guides/metadata.md | 32 ++++++++++++++----- .../humanitarian_needs_details.yaml | 3 +- 7 files changed, 54 insertions(+), 29 deletions(-) diff --git a/docs/data_usage_guides/affected_people.md b/docs/data_usage_guides/affected_people.md index e8a85e23..ffa8f1e0 100644 --- a/docs/data_usage_guides/affected_people.md +++ b/docs/data_usage_guides/affected_people.md @@ -43,8 +43,7 @@ While the HNO data is [directly available on HDX](https://data.humdata.org/dataset/?dataseries_name=Humanitarian+Needs+Overview), it comes from different OCHA offices and is currently not standardised. Thus, HDX HAPI obtains the PIN numbers from the -[HPC Tools API](https://api.hpc.tools/docs/v1/). -Datasets from the HPC API will soon be available on HDX. +[HPC Tools API](https://api.hpc.tools/docs/v1/)-based datasets on HDX. ### Summary diff --git a/docs/data_usage_guides/coordination_and_context.md b/docs/data_usage_guides/coordination_and_context.md index ebed82ee..ca9844d1 100644 --- a/docs/data_usage_guides/coordination_and_context.md +++ b/docs/data_usage_guides/coordination_and_context.md @@ -22,7 +22,7 @@ identify gaps, and plan for future humanitarian response. * Countries that are not p-coded are aggregated to Admin 0 * Organisation deduplication is a long-running challenge with this data, since there are no unique identifiers, and organisation names may be spelled - different ways from different OCHA offices, or sometimes even within the same + different ways by different OCHA offices, or sometimes even within the same 3W. See the [`org`](org) section below for more information on how we handle these details. * Rows without an associated sector are skipped @@ -38,10 +38,12 @@ identify gaps, and plan for future humanitarian response. ## Funding -FTS publishes data on humanitarian funding flows as reported by donors and -recipient organisations. It presents all humanitarian funding to a country and -funding that is reported or that can be specifically mapped against funding -requirements stated in Humanitarian Response Plans. +[OCHA Financial Tracking Service](https://fts.unocha.org/home/2024/donors/view) +(FTS) publishes data on +humanitarian funding flows as reported by donors and +recipient organisations. It presents all humanitarian funding to a country +as well as funding that is reported or that can be specifically +mapped against funding requirements stated in Humanitarian Response Plans. ### Summary @@ -53,7 +55,7 @@ requirements stated in Humanitarian Response Plans. with an appeal. Funding data without associated appeals will be added in a future version. * The reference period is taken from the start and end date provided in each - row front he original data + row from the original data ## Conflict Events @@ -99,11 +101,3 @@ information on the methodology, see ### Transformations applied * The reference period is obtained from the HDX dataset - -### Usage Notes - -* [To be moved to parameter table] -* Missing data: The total number of original indicators missing, including - any that have been estimated (e.g. HDI derived from GDP per capita). -* Recentness: The average of the total number of years older than the - reference year per indicator, to account for any older data used as a proxy diff --git a/docs/data_usage_guides/enum_parameters/population_status_parameters.yaml b/docs/data_usage_guides/enum_parameters/population_status_parameters.yaml index 5943e576..2ab4a097 100644 --- a/docs/data_usage_guides/enum_parameters/population_status_parameters.yaml +++ b/docs/data_usage_guides/enum_parameters/population_status_parameters.yaml @@ -3,7 +3,7 @@ - population status: AFF description: 'affected' - population status: INN - description: 'inneed' + description: 'in-need' - population status: TGT description: 'targeted' - population status: REA diff --git a/docs/data_usage_guides/enum_parameters/price_flag_parameters.yaml b/docs/data_usage_guides/enum_parameters/price_flag_parameters.yaml index 29a4e504..1c5e5db2 100644 --- a/docs/data_usage_guides/enum_parameters/price_flag_parameters.yaml +++ b/docs/data_usage_guides/enum_parameters/price_flag_parameters.yaml @@ -1,3 +1,8 @@ - price flag: actual + description: Collected with monthly frequency - price flag: aggregate + description: > + Collected with greater than monthly frequency (daily or weekly), + and averaged through a weighted mean to monthly - price flag: actual,aggregate + description: diff --git a/docs/data_usage_guides/enums.md b/docs/data_usage_guides/enums.md index a253c3cb..af2e7104 100644 --- a/docs/data_usage_guides/enums.md +++ b/docs/data_usage_guides/enums.md @@ -39,7 +39,7 @@ for their methodology and more detailed descriptions of the sub-event types. **Used in:** [`Baseline Population`](population_and_socio-economy.md#population), [`Humanitarian Needs`](affected_people.md#humanitarian-needs), -[`Refugees & Persons of Concern`](refugees) +[`Refugees & Persons of Concern`](affected_people.md#refugees) Several sub-categories in HDX-HAPI are disaggregated by gender. @@ -72,8 +72,11 @@ The IPC provides different projections to aid in planning and response efforts. [`Humanitarian Needs`](affected_people.md#humanitarian-needs), [`Refugees & Persons of Concern`](affected_people.md#refugees) -Most population group terms come from [UNHCR's glossary](https://www.unhcr.org/refugee-statistics/methodology/data-content/), -while RRI comes from the humanitarian needs data. +Almost all population group terms come from [UNHCR's glossary](https://www.unhcr.org/refugee-statistics/methodology/data-content/), +except for "POC", "RRI", and "all". "RRI" was added as umbrella term +for returnees because unlike UNHCR, the humanitarian +needs data does not distinguish between returned refugees and returned +IDPs. {{ read_yaml('data_usage_guides/enum_parameters/population_group_parameters.yaml') }} @@ -91,12 +94,19 @@ determine how effectively people in need are being reached. **Used in:** [`Food Prices`](food_security_and_nutrition.md#food-prices) +Pre-processing characteristics of food prices. + {{ read_yaml('data_usage_guides/enum_parameters/price_flag_parameters.yaml') }} ## Price Type **Used in:** [`Food Prices`](food_security_and_nutrition.md#food-prices) +The point in the supply chain at which the price is determined. +See FAO's +[concepts on price data](https://www.fao.org/economic/the-statistics-division-ess/methodology/methodology-systems/price-statistics-and-index-numbers-of-agricultural-production-and-prices/4-concepts-on-price-data/en/) +for more information. + {{ read_yaml('data_usage_guides/enum_parameters/price_type_parameters.yaml') }} ## Risk Class diff --git a/docs/data_usage_guides/metadata.md b/docs/data_usage_guides/metadata.md index 8b3d0458..f335c011 100644 --- a/docs/data_usage_guides/metadata.md +++ b/docs/data_usage_guides/metadata.md @@ -11,7 +11,7 @@ contain a small subset of the metadata available through the CKAN API or HDX library. The HAPI data records contain enough information to access the full records -from CKAN, see the parameters tables for more details. +from CKAN (see the parameters tables for more details). All subcategory tables link back to a resource record, and all resources link back to a dataset record. @@ -46,9 +46,21 @@ boundaries. The p-code field is unique only in combination with reference_period_start, since p-codes may be reused in different versions of geographical metadata. -When bringing in data from sub-categories, HDX HAPI uses p-code matching -from [`hdx-python-country`](https://hdx-python-country.readthedocs.io/en/latest/). -Thie can involve .... TODO +HDX HAPI utilises p-code handling provided by the +[`hdx-python-country`](https://hdx-python-country.readthedocs.io/en/latest/) +library. When bringing in data from sub-categories, p-codes are used if +present. Sometimes, these p-codes may not strictly follow the standard with +variations such as shorter or longer country or admin unit codes. +The `hdx-python-country` library has a p-code length matching algorithm to +ensure correct admin units are determined. + +This algorithm works as follows: First, a p-code is verified against the +expected p-code format. Then, a length-matching check is performed, +ensuring that for each segment (country code or admin unit code), +the length matches the predefined format for the country. For example, +a 3-letter country code could be converted to a 2-letter one or vice-versa. +Digits can be prefixed by 0's to increase the length, or have 0's dropped +from the prefix. ## Location @@ -152,10 +164,14 @@ The currency table is populated using the WFP VAM Data Bridges API. [`Food Prices`](food_security_and_nutrition.md#food-prices) Markets are defined as the physical locations where buyers and sellers -come together to trade goods and services. This table is populated from the -food prices data, which contains location names down to admin 2 as well as -a latitude and longitude, however is not p-coded. -We therefore p-code .... TODO +come together to trade goods and services. +This table is populated from the food prices data, which contains location +names down to admin 2 as well as +a latitude and longitude, but is not p-coded. Consequently, admin 1 and 2 +names must be matched to p-codes using the algorithm provided by the +[`hdx-python-country`](https://hdx-python-country.readthedocs.io/en/latest/) +library which includes phonetic name matching and +manual overrides. ### WFP Commodity diff --git a/docs/data_usage_guides/subcategory_details/humanitarian_needs_details.yaml b/docs/data_usage_guides/subcategory_details/humanitarian_needs_details.yaml index bd2f65cb..84ecac1d 100644 --- a/docs/data_usage_guides/subcategory_details/humanitarian_needs_details.yaml +++ b/docs/data_usage_guides/subcategory_details/humanitarian_needs_details.yaml @@ -1,7 +1,8 @@ - Attribute: Data provider Details: OCHA HPC - Attribute: Dataset - Details: Coming soon to HDX + Details: > + [Humanitarian Needs Overview tag](https://data.humdata.org/dataset/?vocab_Topics=humanitarian+needs+overview-hno) - Attribute: Time series Details: Past datasets are available in the HPC API, but are sparse prior to 2023 - Attribute: Update frequency From c3ffaa6f1d77532ff9e67e27d81c7a7d8cb9586f Mon Sep 17 00:00:00 2001 From: Monica Turner Date: Wed, 5 Jun 2024 13:51:50 +0200 Subject: [PATCH 7/8] HDXDSYS-731 fixes some bugs --- .../enum_parameters/risk_class_parameters.yaml | 4 ++-- docs/data_usage_guides/metadata.md | 14 +++++++------- 2 files changed, 9 insertions(+), 9 deletions(-) diff --git a/docs/data_usage_guides/enum_parameters/risk_class_parameters.yaml b/docs/data_usage_guides/enum_parameters/risk_class_parameters.yaml index d6a50388..78baaf87 100644 --- a/docs/data_usage_guides/enum_parameters/risk_class_parameters.yaml +++ b/docs/data_usage_guides/enum_parameters/risk_class_parameters.yaml @@ -1,6 +1,6 @@ - risk class: 1 description: very low - risk_score min: 0.0 + risk score min: 0.0 risk score max: 1.9 - risk class: 2 description: low @@ -9,7 +9,7 @@ - risk class: 3 description: medium risk score min: 3.5 - risk score_max: 4.9 + risk score max: 4.9 - risk class: 4 description: high risk score min: 5.0 diff --git a/docs/data_usage_guides/metadata.md b/docs/data_usage_guides/metadata.md index f335c011..926b93f3 100644 --- a/docs/data_usage_guides/metadata.md +++ b/docs/data_usage_guides/metadata.md @@ -6,7 +6,7 @@ All of the data in HDX HAPI comes from publicly-available datasets on HDX, and the HDX metadata can be used refer back to the original data to see what HAPI has simplified or omitted. The two primary HDX metadata tables are -[`dataset`](dataset) and [`resource`](resource), and they +[`dataset`](metadata.md#dataset) and [`resource`](metadata.md#resource), and they contain a small subset of the metadata available through the CKAN API or HDX library. @@ -17,11 +17,11 @@ back to a dataset record. ### Dataset -**Used in:** [`Resource`](resource), all sub-category tables +**Used in:** [`Resource`](metadata.md#resource), all sub-category tables This table contains the HDX-specific metadata associated with all datasets used to populate the HAPI sub-category tables. Every dataset has at least -one child [resource](resource). +one child [resource](metadata.md#resource). ### Resource @@ -29,7 +29,7 @@ one child [resource](resource). This table contains the HDX-specific metadata associated with all resources used to populate the HAPI sub-category tables. Every resource has one -parent [dataset](dataset). +parent [dataset](metadata.md#dataset). ## Geographical Metadata @@ -64,7 +64,7 @@ from the prefix. ## Location -**Used in:** [`Admin 1`](admin1), +**Used in:** [`Admin 1`](metadata.md#admin1), [`Refugees & Persons of Concern`](affected_people.md#refugees), [`Funding`](coordination_and_context.md#funding), [`National Risk`](coordination_and_context.md#national-risk) @@ -74,7 +74,7 @@ Country or country-like entities in HDX HAPI, from the CODs ## Admin 1 -**Used in:** [`Admin 2](admin2), +**Used in:** [`Admin 2](metadata.md#admin2), [`Poverty Rate`](population_and_socio-economy.md#poverty-rate) Admin 1 level names and p-codes in HDX HAPI, from the CODs @@ -116,7 +116,7 @@ methodology: ### Org Type **Used in:** -[`Org`](org), +[`Org`](metadata.md#org), [`3W - Who is Doing What Where`](coordination_and_context.md#operational-presence) The table is populated using From 97c3c69b932f75171fd1c880250e214367f2948e Mon Sep 17 00:00:00 2001 From: Monica Turner Date: Wed, 5 Jun 2024 16:33:39 +0200 Subject: [PATCH 8/8] HDXDSYS-731 delete unused file --- docs/subcategory_details.md | 260 ------------------------------------ 1 file changed, 260 deletions(-) delete mode 100644 docs/subcategory_details.md diff --git a/docs/subcategory_details.md b/docs/subcategory_details.md deleted file mode 100644 index c0b15109..00000000 --- a/docs/subcategory_details.md +++ /dev/null @@ -1,260 +0,0 @@ -# Data subcateogry details and usage notes - ---- - -This page contains detailed information about the data in HAPI, including usage notes on each subcategory of data. - -## Category: Affected People - -### Sub-category: Humanitarian Needs - -This data represents the shared understanding of OCHA Humanitarian Country Teams of people's widespread emergency needs during crises. It includes the estimated number of people who need assistance, often referred to as People in Need (PIN). This PIN data is derived from the [Joint Intersectoral Analysis Framework (JIAF)](https://www.jiaf.info/), which consolidates and analyses evidence to inform and direct strategic response planning. - -The JIAF data is available for all Humanitarian Response Plan (HRP) countries on HDX. However as the raw data comes from multiple providers (OCHA offices), and is not standardised at this time, HDX HAPI obtains the numbers from the [HPC Tools API](https://api.hpc.tools/docs/v1/). Datasets from the HPC API will soon be available on HDX. - -#### Details - - -||| -|:----|:----| -|Time series|Past datasets are available in the HPC API, but are sparse prior to 2023.| -|Update frequency|Annually| -|Data provider|OCHA HPC| -|Datasets|| -|Related endpoints and schemas|Sector, Disabled Marker, Gender, Population Group, Population Status| - -#### Usage Notes - -- The PIN should *not* be summed across sectors, as the same people can be counted across multiple sectors. For the number of people affected across all sectors, please use the PIN value where sector=intersectoral. -- Methodology in Yemen leads to negative population values in some admin 2 level areas. Where negative values appear they have been omitted from the API.https://www.jiaf.info/ - -### Sub-category: Refugees & Persons of Concern - -This dataset, compiled by the UNHCR, provides annual age- and gender-disaggregated statistics on refugees and others of concern, categorised by their country of origin and country of asylum. The data are sourced primarily from governments hosting these populations, UNHCR's own registration data, and occasionally data published by non-governmental organisations. - -#### Details -||| -|:----|:----| -|Time series|Yes, annual dating back to 2001| -|Update frequency|Annually| -|Data provider|UNHCR| -|Dataset|[Data on forcibly displaced populations and stateless persons (Global)](https://data.humdata.org/dataset/unhcr-population-data-for-world)| -|Related endpoints and schemas|Gender, Population Group| - -#### Usage Notes - -- The source data is not p-coded and only includes vague location descriptions. Since these are not consistently mappable to admin subdivisions, we aggregate to the national tables and enumslevel. -- The original data source contains several population groups such as IDPs and asylum seekers, but we only consider refugees (REF) and others of concern (OOC) -- Note that an “all” value in the` `gender` (age_range) column indicates a sum over all genders (age ranges) - -## Category: Coordination & Context - -### Sub-category: Operation Presence (3W - Who is Doing What Where) - -The [Who does What Where (3W)](https://3w.unocha.org/) is a core humanitarian coordination dataset that contains the geographic and sectoral spread of humanitarian activities and partners. It is critical to know where humanitarian organisations are working and what they are doing in order to ensure collaboration and efficient resource allocation, avoid duplication of efforts, identify gaps, and plan for future humanitarian response. - -#### Details - -||| -|:----|:----| -|Time series|Not available for this subcategory. While older resources are available on HDX, HDX HAPI currently only supports the latest resource.| -|Update frequency|Irregular, depending on the country. Common update frequencies include quarterly and annually. Please check individual resources for more information| -|Data provider|OCHA country and regional offices| -|Datasets|See the [data grids](https://data.humdata.org/dashboards/overview-of-data-grids?)| -|Related endpoints and schemas|Org, Org Type, Sector| - -#### Usage Notes - -- This data comes from multiple providers (OCHA offices), in many different formats and levels of detail -- For consistency and interoperability, we aggregate to an [operational presence](https://humanitarian.atlassian.net/wiki/spaces/imtoolbox/pages/214499412/Who+does+What+Where+3W) level (3W:OP, per org, sector, and admin2), even if the original 3W data is more detailed (e.g. the source lists individual activities) - -### Sub-category: Funding - -FTS publishes data on humanitarian funding flows as reported by donors and recipient organisations. It presents all humanitarian funding to a country and funding that is reported or that can be specifically mapped against funding requirements stated in Humanitarian Response Plans. - -#### Details - -||| -|:----|:----| -|Time series|Data contains funding for several years, but the timepoints are not regular| -|Update frequency|Annually| -|Data provider|OCHA FTS| -|Dataset|[OCHA FTS - Requirements and Funding Data series](https://data.humdata.org/dataset/?dataseries_name=OCHA+FTS+-+Requirements+and+Funding+Data)| -|Related endpoints and schemas|None| - -#### Usage Notes - -- The present version of the API currently captures only funding associated with an appeal. Funding data without associated appeals will be added in a future version. - -### Sub-category: Conflict Events - -[ACLED](https://acleddata.com/) collects real-time data on the locations, dates, actors, fatalities, and types of all reported political violence and protest events around the world. - -#### Details - -||| -|:----|:----| -|Time series|Yes, monthly| -|Update frequency|Weekly| -|Data provider|ACLED| -|Dataset|[ACLED Conflict Events Data series](https://data.humdata.org/dataset/?dataseries_name=ACLED+-+Conflict+Events)| -|Related endpoints and schemas|Event Type| - -#### Usage Notes - -- The API uses ACLED’s public aggregated data -- The data for political violence events, civilian targeting events, and demonstrations are in separate resource on HDX, but are combined into a single endpoint in the API - -### Sub-category: National Risk - -The [INFORM Risk Index](https://drmkc.jrc.ec.europa.eu/inform-index/INFORM-Risk) is a global, open-source risk assessment for humanitarian crises and disasters. It can support decisions about prevention, preparedness and response. For more information on the methodology, see [here](https://drmkc.jrc.ec.europa.eu/inform-index/INFORM-Risk/Methodology). - -#### Details - -||| -|:----|:----| -|Time series|Not available for this subcategory| -|Update frequency|Annually| -|Data provider|INFORM| -|Dataset|[INFORM Risk Index](https://data.humdata.org/dataset/inform-risk-index-2021)| -|Related endpoints and schemas|RiskClass| - -## Category: Food Security & Nutrition - -### Sub-category: Food Security - -The [IPC Acute Food Insecurity (IPC AFI) classification](https://www.ipcinfo.org/ipcinfo-website/ipc-overview-and-classification-system/ipc-acute-food-insecurity-classification/en/) provides strategically relevant information to decision makers that focuses on short-term objectives to prevent, mitigate or decrease severe food insecurity. - -#### Details - -||| -|:----|:----| -|Time series|Yes, with projections| -|Update frequency|Annually| -|Data provider|Food Security and Nutrition Working Group, West and Central Africa| -|Dataset|[West & Central Africa Food Security Data - Cadre Harmonise (CH) and Integrated Food Security Phase Classification (IPC) data](https://data.humdata.org/dataset/cadre-harmonise)| -|Related endpoints and schemas|IPCPhase, IPCType| - -#### Usage Notes - -- The Beta release only contains data from the [Cadre Harmonisé](https://www.cadreharmonise.org/en_GB), as it is p-coded. In a future release we will p-code and expand coverage to other IPC datasets. -- The reference period refers to the time frame that the projection covers, not when the projection was made -= The IPC fraction is computed in the HDX HAPI API pipeline, buy dividing the population by the the total population (ipc_phase=all). -- The total population (ipc_phase=all) is not necessarily equal to the sum of the populations in phases 1-5. - -### Sub-category: Food Prices - -The World Food Programme Price Database covers foods such as maize, rice, beans, fish, and sugar for 98 countries and some 3000 markets. It is updated weekly but primarily contains data with a monthly update frequency. - -#### Details - -||| -|:----|:----| -|Time series|Yes, primarily monthly| -|Update frequency|Weekly| -|Data provider|The World Food Programme| -|Dataset|[WFP Food Prices data series](https://data.humdata.org/dataset/?dataseries_name=WFP+-+Food+Prices)| -|Related endpoints and schemas|Commodity, Currency, Market| - -#### Usage Notes - -- The source data is not p-coded, however we have used the admin 1 and 2 names to p-code most markets. See the Markets section for more details. - -## Category: Population & Socio-economy - -### Sub-category: Baseline Population - -This data comprises population statistics sourced from the [common operational datasets](https://cod.unocha.org/) (CODs), typical disaggregated by age and/or gender, and reaching administrative levels 1 or 2. The primary sources include the UNFPA and OCHA country offices, utilised for coordination purposes, although disparities may arise when compared to alternative sources. - -#### Details - -||| -|:----|:----| -|Time series|No| -|Update frequency|Annually| -|Data provider|UNFPA, OCHA country offices| -|Dataset|[COD - subnational population statistics data series](https://data.humdata.org/dataset/?dataseries_name=COD+-+Subnational+Population+Statistics)| -|Related endpoints and schemas|Gender| - -### Sub-category: Poverty Rate - -The global [Oxford Multidimensional Poverty Index](https://ophi.org.uk/global-mpi) (MPI) measures multidimensional poverty in over 100 developing countries, using internationally comparable datasets. -The MPI assesses poverty through three main dimensions: health, education, and living standards, each of which is represented by specific indicators.Please see the [OPHI methodological note](https://ophi.org.uk/publications/MN-54) for more details. - -#### Details - -||| -|:----|:----| -|Time series|Annual, with some combined years| -|Update frequency|Annually| -|Data provider|Oxford Poverty and Human Development Initiative (OPHI)| -|Dataset|[Oxford Poverty and Human Development Initiative - Global Multidimensional Poverty Index data series](https://data.humdata.org/dataset/?dataseries_name=Oxford+Poverty+and+Human+Development+Initiative+-+Global+Multidimensional+Poverty+Index)| -|Related endpoints and schemas|None| - - -#### Usage Notes - -The source data is very detailed. We’ve chosen the following subset of indicators, detailed below: - -|Indicator|format|Description| -|:----|:----|:----| -|mpi|fraction|The multidimensional poverty index. Derived as a product of the headcount ratio and intensity of deprivation.| -|headcount_ratio|%|The percentage of people deprived in 33% or more indicators| -|intensity_of_deprivation|%|The average proportion of indicators in which people are deprived| -|vulnerable_to_poverty|%|The percentage of people deprived in 20-33% of indicators| -|in_severe_poverty|%|The percentage of people deprived in 50% or more indicators| - -## Metadata - -### Sectors - -- The list of sectors is available through the sector endpoint -- This table is populated using the Global Coordination Groups, with the following additional entries: - - cash - - humanitarian assistance - - multi-sector - - intersectoral -The sector name strings in the 3W data are normalised and then aligned to the sector table, using the “sector_map” section of this configuration file if needed: https://github.com/OCHA-DAP/hapi-pipelines/blob/main/src/hapi/pipelines/configs/core.yaml. In the absence of a direct match, phonetic matching is used for strings > 5 characters. - -### Org - -- The organisation table is populated from the 3W data -- Organisation name and acronym strings are normalised. If an acronym isn’t available, the first 32 characters of the name are used. -- This [organisation mapping](https://docs.google.com/spreadsheets/d/e/2PACX-1vSfBWvSu3fKA743VvHtgf-pIGkYH7zhy-NP7DZgEV9_a6YU7vtCeWhbLM56aUL1iIfrfv5UBvvjVt7B/pub?gid=1040329566&single=true&output=csv) is used for common alternative names -- Organisations without a sector are not included -- Organisations can have an associated organisation type. If available, the organisation type code is taken directly from the 3W data, otherwise the name string is normalised and matched to the org type names. In the absence of a direct match, phonetic matching is used for strings > 5 characters. If no match is found, the organisation is skipped. - -### OrgType - -- The table is populated using [OCHA Digital Services organization types list](https://data.humdata.org/dataset/organization-types-beta), with the addition of: - - Civil Society - - Observer - - Development Programme - - Local NGO -Organisation types all have an associated name and code - -### IPC Code - -The IPC classification includes 5 different phases of increasing severity, described in detail on page 53 of [the IPC technical manual version 3.1](https://www.ipcinfo.org/fileadmin/user_upload/ipcinfo/manual/IPC_Technical_Manual_3_Final.pdf), and summarised below: - -|Phase|Name|Description| -|:----|:----|:----| -|1|None/Minimal|Households are able to meet essential food and non-food needs without engaging in atypical and unsustainable strategies to access food and income.| -|2|Stressed|Households have minimally adequate food consumption but are unable to afford some essential non-food expenditures without engaging in stress-coping strategies.| -|3|Crisis|Households either have food consumption gaps that are reflected by high or above-usual acute malnutrition, or are marginally able to meet minimum food needs but only by depleting essential livelihood assets or through crisis-coping strategies.| -|4|Emergency|Households either have large food consumption gaps which are reflected in very high acute malnutrition and excess mortality, or are able to mitigate large food consumption gaps but only by employing emergency livelihood strategies and asset liquidation.| -|5|Catastrophe/Famine|Households have an extreme lack of food and/or other basic needs even after full employment of coping strategies. Starvation, death, destitution and extremely critical acute malnutrition levels are evident. (For Famine Classification, an area needs to have extreme critical levels of acute malnutrition and mortality.)| -|3|In Need of Action|Sum of population in phases 3, 4, and 5. The population in Phase 3+ does not necessarily reflect the full population in need of urgent action. This is because some households may be in Phase 2 or even 1 but only because of receipt of assistance, and thus, they may be in need of continued action.| -|all|Total population|Total population| - -The above table also includes Phase 3+, to highlight the population in need of action, and total population, used to compute fractions. - -### IPC Type - -The IPC provides different projections to aid in planning and response efforts, outlined in the following table: - -|IPC type|Definition| -|:----|:----| -|current|Food insecurity that is occurring in the current analysis period.| -|first projection|Projected food insecurity occurring in the period immediately following the current analysis period.| -|second projection|Projected food insecurity occurring in the period immediately following the first projection period.| \ No newline at end of file