ensemble-table.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "4f32094f-61ee-4424-847c-8f47dd10d251",
   "metadata": {},
   "source": [
    "# Region and ensemble member mean table #24\n",
    "\n",
    "Make a table by averaging values nationally and in each NCRA region for each RCM and second by computing the 10, 50, and 90 percentiles from the ensemble of RCM averaged values.\n",
    "The table would have national and each NCRA region the 10,50 and 90 percentile change in the hazard metric from GWL1.2 for GWL 1.5, 2, 3\n",
    "\n",
    "|   |rcm1\t|...\t|rcm13|\tens p10th\t|ens p50th\t|ens p90th|\n",
    "|---|-------|-------|-----|-------------|-----------|---------|\n",
    "|NCRA region 0|\t\t\t\t\t\t\n",
    "|...|\t\t\t\t\t\t\n",
    "|NCRA region 9|\n",
    "|AUS |\n",
    "\n",
    "Start with mean change in FFDI from GWL1.2.\n",
    "Data is at /g/data/ia39/ncra/fire/ffdi for each RCM (20 years of data in each zarr collection).\n",
    "Compute the average change in FFDI for each RCM and provide that in figures and tables."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7afa4bf7-0321-4d60-a203-b6a56fdbf79d",
   "metadata": {},
   "source": [
    "Now, looking at the hazards we have computed, you can apply code/plotting/table to the hazard metrics for the heat, drought, extratropical storms, and fire hazards (i.e. 3 threshold changes files).   Use the change fields from GWL1.2.  Some hazards will have multiple indices – heat – EHT and Tmax above some threshold …"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3fdce7b4-d81c-4a94-b774-f28f18047b01",
   "metadata": {},
   "outputs": [],
   "source": [
    "# times in this file are using 28 cores, 126GB mem"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "efc86a71-f604-41c8-a263-534ceafee872",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/g/data/mn51/users/gt3409/plotting_maps\n"
     ]
    }
   ],
   "source": [
    "cd /g/data/mn51/users/gt3409/plotting_maps/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "1785c6bd-2da9-4d7b-a751-caf1b4e046e4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# import needed packages\n",
    "from acs_area_statistics import acs_regional_stats, get_regions\n",
    "import xarray as xr\n",
    "import geopandas as gpd\n",
    "from glob import glob\n",
    "import pandas as pd\n",
    "import regionmask\n",
    "PATH = \"/g/data/ia39/aus-ref-clim-data-nci/shapefiles/data\"\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "\n",
    "import dask"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1664b938-3bef-4984-be96-9eb026f51748",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "b3cf7bf6-136a-4fb7-8b29-5f0f28891340",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<regionmask.Regions 'ncra_regions-australia'>\n",
       "overlap:  True\n",
       "\n",
       "Regions:\n",
       "0  WANorth                          Western Australia North\n",
       "1  WASouth                          Western Australia South\n",
       "2      NSW New South Wales and Australian Capital Territory\n",
       "3      VIC                                         Victoria\n",
       "4       SA                                  South Australia\n",
       "5      TAS                                         Tasmania\n",
       "6       NT                               Northern Territory\n",
       "7 QLDNorth                                 Queensland North\n",
       "8 QLDSouth                                 Queensland South\n",
       "9      AUS                                        Australia\n",
       "\n",
       "[10 regions]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# NCRA regions from acs_area_statistics code\n",
    "# these are the names of your regions\n",
    "regions = get_regions([\n",
    "                          \"ncra_regions\", \"australia\"\n",
    "                      ])\n",
    "regions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "e12644f7-a04a-47da-927c-ad63b51a9068",
   "metadata": {},
   "outputs": [],
   "source": [
    "# get the individual models for GWL30-GWL12 anomalies only\n",
    "var = \"FFDI\"\n",
    "gwl = \"GWL1.5\"\n",
    "filelist = glob(f\"/g/data/ia39/ncra/fire/change_ffdi/*{gwl}*\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "93f09450-bfae-4048-a405-9d4adf19d413",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['/g/data/ia39/ncra/fire/change_ffdi/AGCD-05i_BOM_ACCESS-CM2_ssp370_r4i1p1f1_BARPA-R_v1-r1_day_GWL1.5_change_ffdi.zarr',\n",
       " '/g/data/ia39/ncra/fire/change_ffdi/AGCD-05i_BOM_CMCC-ESM2_ssp370_r1i1p1f1_BARPA-R_v1-r1_day_GWL1.5_change_ffdi.zarr',\n",
       " '/g/data/ia39/ncra/fire/change_ffdi/AGCD-05i_BOM_NorESM2-MM_ssp370_r1i1p1f1_BARPA-R_v1-r1_day_GWL1.5_change_ffdi.zarr',\n",
       " '/g/data/ia39/ncra/fire/change_ffdi/AGCD-05i_BOM_CESM2_ssp370_r11i1p1f1_BARPA-R_v1-r1_day_GWL1.5_change_ffdi.zarr',\n",
       " '/g/data/ia39/ncra/fire/change_ffdi/AGCD-05i_CSIRO_CMCC-ESM2_ssp370_r1i1p1f1_CCAM-v2203-SN_v1-r1_day_GWL1.5_change_ffdi.zarr',\n",
       " '/g/data/ia39/ncra/fire/change_ffdi/AGCD-05i_BOM_MPI-ESM1-2-HR_ssp370_r1i1p1f1_BARPA-R_v1-r1_day_GWL1.5_change_ffdi.zarr',\n",
       " '/g/data/ia39/ncra/fire/change_ffdi/AGCD-05i_CSIRO_CESM2_ssp370_r11i1p1f1_CCAM-v2203-SN_v1-r1_day_GWL1.5_change_ffdi.zarr',\n",
       " '/g/data/ia39/ncra/fire/change_ffdi/AGCD-05i_BOM_EC-Earth3_ssp370_r1i1p1f1_BARPA-R_v1-r1_day_GWL1.5_change_ffdi.zarr',\n",
       " '/g/data/ia39/ncra/fire/change_ffdi/AGCD-05i_CSIRO_ACCESS-ESM1-5_ssp370_r6i1p1f1_CCAM-v2203-SN_v1-r1_day_GWL1.5_change_ffdi.zarr',\n",
       " '/g/data/ia39/ncra/fire/change_ffdi/AGCD-05i_CSIRO_EC-Earth3_ssp370_r1i1p1f1_CCAM-v2203-SN_v1-r1_day_GWL1.5_change_ffdi.zarr',\n",
       " '/g/data/ia39/ncra/fire/change_ffdi/AGCD-05i_CSIRO_ACCESS-CM2_ssp370_r4i1p1f1_CCAM-v2203-SN_v1-r1_day_GWL1.5_change_ffdi.zarr',\n",
       " '/g/data/ia39/ncra/fire/change_ffdi/AGCD-05i_CSIRO_CNRM-ESM2-1_ssp370_r1i1p1f2_CCAM-v2203-SN_v1-r1_day_GWL1.5_change_ffdi.zarr']"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "filelist"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "b12464e4-489c-46e0-8367-fbbc150f42ea",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 5.31 s, sys: 3.26 s, total: 8.56 s\n",
      "Wall time: 29.8 s\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div><svg style=\"position: absolute; width: 0; height: 0; overflow: hidden\">\n",
       "<defs>\n",
       "<symbol id=\"icon-database\" viewBox=\"0 0 32 32\">\n",
       "<path d=\"M16 0c-8.837 0-16 2.239-16 5v4c0 2.761 7.163 5 16 5s16-2.239 16-5v-4c0-2.761-7.163-5-16-5z\"></path>\n",
       "<path d=\"M16 17c-8.837 0-16-2.239-16-5v6c0 2.761 7.163 5 16 5s16-2.239 16-5v-6c0 2.761-7.163 5-16 5z\"></path>\n",
       "<path d=\"M16 26c-8.837 0-16-2.239-16-5v6c0 2.761 7.163 5 16 5s16-2.239 16-5v-6c0 2.761-7.163 5-16 5z\"></path>\n",
       "</symbol>\n",
       "<symbol id=\"icon-file-text2\" viewBox=\"0 0 32 32\">\n",
       "<path d=\"M28.681 7.159c-0.694-0.947-1.662-2.053-2.724-3.116s-2.169-2.030-3.116-2.724c-1.612-1.182-2.393-1.319-2.841-1.319h-15.5c-1.378 0-2.5 1.121-2.5 2.5v27c0 1.378 1.122 2.5 2.5 2.5h23c1.378 0 2.5-1.122 2.5-2.5v-19.5c0-0.448-0.137-1.23-1.319-2.841zM24.543 5.457c0.959 0.959 1.712 1.825 2.268 2.543h-4.811v-4.811c0.718 0.556 1.584 1.309 2.543 2.268zM28 29.5c0 0.271-0.229 0.5-0.5 0.5h-23c-0.271 0-0.5-0.229-0.5-0.5v-27c0-0.271 0.229-0.5 0.5-0.5 0 0 15.499-0 15.5 0v7c0 0.552 0.448 1 1 1h7v19.5z\"></path>\n",
       "<path d=\"M23 26h-14c-0.552 0-1-0.448-1-1s0.448-1 1-1h14c0.552 0 1 0.448 1 1s-0.448 1-1 1z\"></path>\n",
       "<path d=\"M23 22h-14c-0.552 0-1-0.448-1-1s0.448-1 1-1h14c0.552 0 1 0.448 1 1s-0.448 1-1 1z\"></path>\n",
       "<path d=\"M23 18h-14c-0.552 0-1-0.448-1-1s0.448-1 1-1h14c0.552 0 1 0.448 1 1s-0.448 1-1 1z\"></path>\n",
       "</symbol>\n",
       "</defs>\n",
       "</svg>\n",
       "<style>/* CSS stylesheet for displaying xarray objects in jupyterlab.\n",
       " *\n",
       " */\n",
       "\n",
       ":root {\n",
       "  --xr-font-color0: var(--jp-content-font-color0, rgba(0, 0, 0, 1));\n",
       "  --xr-font-color2: var(--jp-content-font-color2, rgba(0, 0, 0, 0.54));\n",
       "  --xr-font-color3: var(--jp-content-font-color3, rgba(0, 0, 0, 0.38));\n",
       "  --xr-border-color: var(--jp-border-color2, #e0e0e0);\n",
       "  --xr-disabled-color: var(--jp-layout-color3, #bdbdbd);\n",
       "  --xr-background-color: var(--jp-layout-color0, white);\n",
       "  --xr-background-color-row-even: var(--jp-layout-color1, white);\n",
       "  --xr-background-color-row-odd: var(--jp-layout-color2, #eeeeee);\n",
       "}\n",
       "\n",
       "html[theme=dark],\n",
       "body[data-theme=dark],\n",
       "body.vscode-dark {\n",
       "  --xr-font-color0: rgba(255, 255, 255, 1);\n",
       "  --xr-font-color2: rgba(255, 255, 255, 0.54);\n",
       "  --xr-font-color3: rgba(255, 255, 255, 0.38);\n",
       "  --xr-border-color: #1F1F1F;\n",
       "  --xr-disabled-color: #515151;\n",
       "  --xr-background-color: #111111;\n",
       "  --xr-background-color-row-even: #111111;\n",
       "  --xr-background-color-row-odd: #313131;\n",
       "}\n",
       "\n",
       ".xr-wrap {\n",
       "  display: block !important;\n",
       "  min-width: 300px;\n",
       "  max-width: 700px;\n",
       "}\n",
       "\n",
       ".xr-text-repr-fallback {\n",
       "  /* fallback to plain text repr when CSS is not injected (untrusted notebook) */\n",
       "  display: none;\n",
       "}\n",
       "\n",
       ".xr-header {\n",
       "  padding-top: 6px;\n",
       "  padding-bottom: 6px;\n",
       "  margin-bottom: 4px;\n",
       "  border-bottom: solid 1px var(--xr-border-color);\n",
       "}\n",
       "\n",
       ".xr-header > div,\n",
       ".xr-header > ul {\n",
       "  display: inline;\n",
       "  margin-top: 0;\n",
       "  margin-bottom: 0;\n",
       "}\n",
       "\n",
       ".xr-obj-type,\n",
       ".xr-array-name {\n",
       "  margin-left: 2px;\n",
       "  margin-right: 10px;\n",
       "}\n",
       "\n",
       ".xr-obj-type {\n",
       "  color: var(--xr-font-color2);\n",
       "}\n",
       "\n",
       ".xr-sections {\n",
       "  padding-left: 0 !important;\n",
       "  display: grid;\n",
       "  grid-template-columns: 150px auto auto 1fr 20px 20px;\n",
       "}\n",
       "\n",
       ".xr-section-item {\n",
       "  display: contents;\n",
       "}\n",
       "\n",
       ".xr-section-item input {\n",
       "  display: none;\n",
       "}\n",
       "\n",
       ".xr-section-item input + label {\n",
       "  color: var(--xr-disabled-color);\n",
       "}\n",
       "\n",
       ".xr-section-item input:enabled + label {\n",
       "  cursor: pointer;\n",
       "  color: var(--xr-font-color2);\n",
       "}\n",
       "\n",
       ".xr-section-item input:enabled + label:hover {\n",
       "  color: var(--xr-font-color0);\n",
       "}\n",
       "\n",
       ".xr-section-summary {\n",
       "  grid-column: 1;\n",
       "  color: var(--xr-font-color2);\n",
       "  font-weight: 500;\n",
       "}\n",
       "\n",
       ".xr-section-summary > span {\n",
       "  display: inline-block;\n",
       "  padding-left: 0.5em;\n",
       "}\n",
       "\n",
       ".xr-section-summary-in:disabled + label {\n",
       "  color: var(--xr-font-color2);\n",
       "}\n",
       "\n",
       ".xr-section-summary-in + label:before {\n",
       "  display: inline-block;\n",
       "  content: '►';\n",
       "  font-size: 11px;\n",
       "  width: 15px;\n",
       "  text-align: center;\n",
       "}\n",
       "\n",
       ".xr-section-summary-in:disabled + label:before {\n",
       "  color: var(--xr-disabled-color);\n",
       "}\n",
       "\n",
       ".xr-section-summary-in:checked + label:before {\n",
       "  content: '▼';\n",
       "}\n",
       "\n",
       ".xr-section-summary-in:checked + label > span {\n",
       "  display: none;\n",
       "}\n",
       "\n",
       ".xr-section-summary,\n",
       ".xr-section-inline-details {\n",
       "  padding-top: 4px;\n",
       "  padding-bottom: 4px;\n",
       "}\n",
       "\n",
       ".xr-section-inline-details {\n",
       "  grid-column: 2 / -1;\n",
       "}\n",
       "\n",
       ".xr-section-details {\n",
       "  display: none;\n",
       "  grid-column: 1 / -1;\n",
       "  margin-bottom: 5px;\n",
       "}\n",
       "\n",
       ".xr-section-summary-in:checked ~ .xr-section-details {\n",
       "  display: contents;\n",
       "}\n",
       "\n",
       ".xr-array-wrap {\n",
       "  grid-column: 1 / -1;\n",
       "  display: grid;\n",
       "  grid-template-columns: 20px auto;\n",
       "}\n",
       "\n",
       ".xr-array-wrap > label {\n",
       "  grid-column: 1;\n",
       "  vertical-align: top;\n",
       "}\n",
       "\n",
       ".xr-preview {\n",
       "  color: var(--xr-font-color3);\n",
       "}\n",
       "\n",
       ".xr-array-preview,\n",
       ".xr-array-data {\n",
       "  padding: 0 5px !important;\n",
       "  grid-column: 2;\n",
       "}\n",
       "\n",
       ".xr-array-data,\n",
       ".xr-array-in:checked ~ .xr-array-preview {\n",
       "  display: none;\n",
       "}\n",
       "\n",
       ".xr-array-in:checked ~ .xr-array-data,\n",
       ".xr-array-preview {\n",
       "  display: inline-block;\n",
       "}\n",
       "\n",
       ".xr-dim-list {\n",
       "  display: inline-block !important;\n",
       "  list-style: none;\n",
       "  padding: 0 !important;\n",
       "  margin: 0;\n",
       "}\n",
       "\n",
       ".xr-dim-list li {\n",
       "  display: inline-block;\n",
       "  padding: 0;\n",
       "  margin: 0;\n",
       "}\n",
       "\n",
       ".xr-dim-list:before {\n",
       "  content: '(';\n",
       "}\n",
       "\n",
       ".xr-dim-list:after {\n",
       "  content: ')';\n",
       "}\n",
       "\n",
       ".xr-dim-list li:not(:last-child):after {\n",
       "  content: ',';\n",
       "  padding-right: 5px;\n",
       "}\n",
       "\n",
       ".xr-has-index {\n",
       "  font-weight: bold;\n",
       "}\n",
       "\n",
       ".xr-var-list,\n",
       ".xr-var-item {\n",
       "  display: contents;\n",
       "}\n",
       "\n",
       ".xr-var-item > div,\n",
       ".xr-var-item label,\n",
       ".xr-var-item > .xr-var-name span {\n",
       "  background-color: var(--xr-background-color-row-even);\n",
       "  margin-bottom: 0;\n",
       "}\n",
       "\n",
       ".xr-var-item > .xr-var-name:hover span {\n",
       "  padding-right: 5px;\n",
       "}\n",
       "\n",
       ".xr-var-list > li:nth-child(odd) > div,\n",
       ".xr-var-list > li:nth-child(odd) > label,\n",
       ".xr-var-list > li:nth-child(odd) > .xr-var-name span {\n",
       "  background-color: var(--xr-background-color-row-odd);\n",
       "}\n",
       "\n",
       ".xr-var-name {\n",
       "  grid-column: 1;\n",
       "}\n",
       "\n",
       ".xr-var-dims {\n",
       "  grid-column: 2;\n",
       "}\n",
       "\n",
       ".xr-var-dtype {\n",
       "  grid-column: 3;\n",
       "  text-align: right;\n",
       "  color: var(--xr-font-color2);\n",
       "}\n",
       "\n",
       ".xr-var-preview {\n",
       "  grid-column: 4;\n",
       "}\n",
       "\n",
       ".xr-index-preview {\n",
       "  grid-column: 2 / 5;\n",
       "  color: var(--xr-font-color2);\n",
       "}\n",
       "\n",
       ".xr-var-name,\n",
       ".xr-var-dims,\n",
       ".xr-var-dtype,\n",
       ".xr-preview,\n",
       ".xr-attrs dt {\n",
       "  white-space: nowrap;\n",
       "  overflow: hidden;\n",
       "  text-overflow: ellipsis;\n",
       "  padding-right: 10px;\n",
       "}\n",
       "\n",
       ".xr-var-name:hover,\n",
       ".xr-var-dims:hover,\n",
       ".xr-var-dtype:hover,\n",
       ".xr-attrs dt:hover {\n",
       "  overflow: visible;\n",
       "  width: auto;\n",
       "  z-index: 1;\n",
       "}\n",
       "\n",
       ".xr-var-attrs,\n",
       ".xr-var-data,\n",
       ".xr-index-data {\n",
       "  display: none;\n",
       "  background-color: var(--xr-background-color) !important;\n",
       "  padding-bottom: 5px !important;\n",
       "}\n",
       "\n",
       ".xr-var-attrs-in:checked ~ .xr-var-attrs,\n",
       ".xr-var-data-in:checked ~ .xr-var-data,\n",
       ".xr-index-data-in:checked ~ .xr-index-data {\n",
       "  display: block;\n",
       "}\n",
       "\n",
       ".xr-var-data > table {\n",
       "  float: right;\n",
       "}\n",
       "\n",
       ".xr-var-name span,\n",
       ".xr-var-data,\n",
       ".xr-index-name div,\n",
       ".xr-index-data,\n",
       ".xr-attrs {\n",
       "  padding-left: 25px !important;\n",
       "}\n",
       "\n",
       ".xr-attrs,\n",
       ".xr-var-attrs,\n",
       ".xr-var-data,\n",
       ".xr-index-data {\n",
       "  grid-column: 1 / -1;\n",
       "}\n",
       "\n",
       "dl.xr-attrs {\n",
       "  padding: 0;\n",
       "  margin: 0;\n",
       "  display: grid;\n",
       "  grid-template-columns: 125px auto;\n",
       "}\n",
       "\n",
       ".xr-attrs dt,\n",
       ".xr-attrs dd {\n",
       "  padding: 0;\n",
       "  margin: 0;\n",
       "  float: left;\n",
       "  padding-right: 10px;\n",
       "  width: auto;\n",
       "}\n",
       "\n",
       ".xr-attrs dt {\n",
       "  font-weight: normal;\n",
       "  grid-column: 1;\n",
       "}\n",
       "\n",
       ".xr-attrs dt:hover span {\n",
       "  display: inline-block;\n",
       "  background: var(--xr-background-color);\n",
       "  padding-right: 10px;\n",
       "}\n",
       "\n",
       ".xr-attrs dd {\n",
       "  grid-column: 2;\n",
       "  white-space: pre-wrap;\n",
       "  word-break: break-all;\n",
       "}\n",
       "\n",
       ".xr-icon-database,\n",
       ".xr-icon-file-text2,\n",
       ".xr-no-icon {\n",
       "  display: inline-block;\n",
       "  vertical-align: middle;\n",
       "  width: 1em;\n",
       "  height: 1.5em !important;\n",
       "  stroke-width: 0;\n",
       "  stroke: currentColor;\n",
       "  fill: currentColor;\n",
       "}\n",
       "</style><pre class='xr-text-repr-fallback'>&lt;xarray.Dataset&gt; Size: 36GB\n",
       "Dimensions:  (lat: 691, lon: 886, time: 7300)\n",
       "Coordinates:\n",
       "  * lat      (lat) float64 6kB -44.5 -44.45 -44.4 -44.35 ... -10.1 -10.05 -10.0\n",
       "  * lon      (lon) float64 7kB 112.0 112.0 112.1 112.2 ... 156.2 156.2 156.2\n",
       "  * time     (time) object 58kB 2022-01-01 12:00:00 ... 2041-12-31 12:00:00\n",
       "Data variables:\n",
       "    FFDI     (lat, lon, time) float64 36GB dask.array&lt;chunksize=(33, 43, 7300), meta=np.ndarray&gt;</pre><div class='xr-wrap' style='display:none'><div class='xr-header'><div class='xr-obj-type'>xarray.Dataset</div></div><ul class='xr-sections'><li class='xr-section-item'><input id='section-a51fd970-9d9f-4845-a3b1-1bfa1e7bded4' class='xr-section-summary-in' type='checkbox' disabled ><label for='section-a51fd970-9d9f-4845-a3b1-1bfa1e7bded4' class='xr-section-summary'  title='Expand/collapse section'>Dimensions:</label><div class='xr-section-inline-details'><ul class='xr-dim-list'><li><span class='xr-has-index'>lat</span>: 691</li><li><span class='xr-has-index'>lon</span>: 886</li><li><span class='xr-has-index'>time</span>: 7300</li></ul></div><div class='xr-section-details'></div></li><li class='xr-section-item'><input id='section-f73b5d44-9c5f-4f6c-ab58-3aeac196fd52' class='xr-section-summary-in' type='checkbox'  checked><label for='section-f73b5d44-9c5f-4f6c-ab58-3aeac196fd52' class='xr-section-summary' >Coordinates: <span>(3)</span></label><div class='xr-section-inline-details'></div><div class='xr-section-details'><ul class='xr-var-list'><li class='xr-var-item'><div class='xr-var-name'><span class='xr-has-index'>lat</span></div><div class='xr-var-dims'>(lat)</div><div class='xr-var-dtype'>float64</div><div class='xr-var-preview xr-preview'>-44.5 -44.45 -44.4 ... -10.05 -10.0</div><input id='attrs-4e4871a4-c982-4447-bf42-5bdee44c3d2d' class='xr-var-attrs-in' type='checkbox' disabled><label for='attrs-4e4871a4-c982-4447-bf42-5bdee44c3d2d' title='Show/Hide attributes'><svg class='icon xr-icon-file-text2'><use xlink:href='#icon-file-text2'></use></svg></label><input id='data-877537da-9c72-4820-94a5-de3685646fa0' class='xr-var-data-in' type='checkbox'><label for='data-877537da-9c72-4820-94a5-de3685646fa0' title='Show/Hide data repr'><svg class='icon xr-icon-database'><use xlink:href='#icon-database'></use></svg></label><div class='xr-var-attrs'><dl class='xr-attrs'></dl></div><div class='xr-var-data'><pre>array([-44.5 , -44.45, -44.4 , ..., -10.1 , -10.05, -10.  ])</pre></div></li><li class='xr-var-item'><div class='xr-var-name'><span class='xr-has-index'>lon</span></div><div class='xr-var-dims'>(lon)</div><div class='xr-var-dtype'>float64</div><div class='xr-var-preview xr-preview'>112.0 112.0 112.1 ... 156.2 156.2</div><input id='attrs-1a55f1f3-a7ea-4d81-a606-3c001be7d668' class='xr-var-attrs-in' type='checkbox' disabled><label for='attrs-1a55f1f3-a7ea-4d81-a606-3c001be7d668' title='Show/Hide attributes'><svg class='icon xr-icon-file-text2'><use xlink:href='#icon-file-text2'></use></svg></label><input id='data-37058cb4-87e5-45ee-8538-c13712986435' class='xr-var-data-in' type='checkbox'><label for='data-37058cb4-87e5-45ee-8538-c13712986435' title='Show/Hide data repr'><svg class='icon xr-icon-database'><use xlink:href='#icon-database'></use></svg></label><div class='xr-var-attrs'><dl class='xr-attrs'></dl></div><div class='xr-var-data'><pre>array([112.  , 112.05, 112.1 , ..., 156.15, 156.2 , 156.25])</pre></div></li><li class='xr-var-item'><div class='xr-var-name'><span class='xr-has-index'>time</span></div><div class='xr-var-dims'>(time)</div><div class='xr-var-dtype'>object</div><div class='xr-var-preview xr-preview'>2022-01-01 12:00:00 ... 2041-12-...</div><input id='attrs-b63ea626-0988-4a74-8bf9-54e399384923' class='xr-var-attrs-in' type='checkbox' disabled><label for='attrs-b63ea626-0988-4a74-8bf9-54e399384923' title='Show/Hide attributes'><svg class='icon xr-icon-file-text2'><use xlink:href='#icon-file-text2'></use></svg></label><input id='data-81d96d2d-5cc6-4bd1-ab0a-f04d51bdb8c6' class='xr-var-data-in' type='checkbox'><label for='data-81d96d2d-5cc6-4bd1-ab0a-f04d51bdb8c6' title='Show/Hide data repr'><svg class='icon xr-icon-database'><use xlink:href='#icon-database'></use></svg></label><div class='xr-var-attrs'><dl class='xr-attrs'></dl></div><div class='xr-var-data'><pre>array([cftime.DatetimeNoLeap(2022, 1, 1, 12, 0, 0, 0, has_year_zero=True),\n",
       "       cftime.DatetimeNoLeap(2022, 1, 2, 12, 0, 0, 0, has_year_zero=True),\n",
       "       cftime.DatetimeNoLeap(2022, 1, 3, 12, 0, 0, 0, has_year_zero=True), ...,\n",
       "       cftime.DatetimeNoLeap(2041, 12, 29, 12, 0, 0, 0, has_year_zero=True),\n",
       "       cftime.DatetimeNoLeap(2041, 12, 30, 12, 0, 0, 0, has_year_zero=True),\n",
       "       cftime.DatetimeNoLeap(2041, 12, 31, 12, 0, 0, 0, has_year_zero=True)],\n",
       "      dtype=object)</pre></div></li></ul></div></li><li class='xr-section-item'><input id='section-006e346b-fe1f-4772-84f2-f0fd814e3e4b' class='xr-section-summary-in' type='checkbox'  checked><label for='section-006e346b-fe1f-4772-84f2-f0fd814e3e4b' class='xr-section-summary' >Data variables: <span>(1)</span></label><div class='xr-section-inline-details'></div><div class='xr-section-details'><ul class='xr-var-list'><li class='xr-var-item'><div class='xr-var-name'><span>FFDI</span></div><div class='xr-var-dims'>(lat, lon, time)</div><div class='xr-var-dtype'>float64</div><div class='xr-var-preview xr-preview'>dask.array&lt;chunksize=(33, 43, 7300), meta=np.ndarray&gt;</div><input id='attrs-f20a3233-c57e-4b8c-b348-2d6e915f2cde' class='xr-var-attrs-in' type='checkbox' disabled><label for='attrs-f20a3233-c57e-4b8c-b348-2d6e915f2cde' title='Show/Hide attributes'><svg class='icon xr-icon-file-text2'><use xlink:href='#icon-file-text2'></use></svg></label><input id='data-85000370-03e7-4932-bacd-9d4fef8ae1b2' class='xr-var-data-in' type='checkbox'><label for='data-85000370-03e7-4932-bacd-9d4fef8ae1b2' title='Show/Hide data repr'><svg class='icon xr-icon-database'><use xlink:href='#icon-database'></use></svg></label><div class='xr-var-attrs'><dl class='xr-attrs'></dl></div><div class='xr-var-data'><table>\n",
       "    <tr>\n",
       "        <td>\n",
       "            <table style=\"border-collapse: collapse;\">\n",
       "                <thead>\n",
       "                    <tr>\n",
       "                        <td> </td>\n",
       "                        <th> Array </th>\n",
       "                        <th> Chunk </th>\n",
       "                    </tr>\n",
       "                </thead>\n",
       "                <tbody>\n",
       "                    \n",
       "                    <tr>\n",
       "                        <th> Bytes </th>\n",
       "                        <td> 33.30 GiB </td>\n",
       "                        <td> 79.03 MiB </td>\n",
       "                    </tr>\n",
       "                    \n",
       "                    <tr>\n",
       "                        <th> Shape </th>\n",
       "                        <td> (691, 886, 7300) </td>\n",
       "                        <td> (33, 43, 7300) </td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <th> Dask graph </th>\n",
       "                        <td colspan=\"2\"> 441 chunks in 2 graph layers </td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <th> Data type </th>\n",
       "                        <td colspan=\"2\"> float64 numpy.ndarray </td>\n",
       "                    </tr>\n",
       "                </tbody>\n",
       "            </table>\n",
       "        </td>\n",
       "        <td>\n",
       "        <svg width=\"202\" height=\"112\" style=\"stroke:rgb(0,0,0);stroke-width:1\" >\n",
       "\n",
       "  <!-- Horizontal lines -->\n",
       "  <line x1=\"10\" y1=\"0\" x2=\"32\" y2=\"22\" style=\"stroke-width:2\" />\n",
       "  <line x1=\"10\" y1=\"1\" x2=\"32\" y2=\"24\" />\n",
       "  <line x1=\"10\" y1=\"3\" x2=\"32\" y2=\"26\" />\n",
       "  <line x1=\"10\" y1=\"5\" x2=\"32\" y2=\"28\" />\n",
       "  <line x1=\"10\" y1=\"7\" x2=\"32\" y2=\"30\" />\n",
       "  <line x1=\"10\" y1=\"9\" x2=\"32\" y2=\"32\" />\n",
       "  <line x1=\"10\" y1=\"11\" x2=\"32\" y2=\"34\" />\n",
       "  <line x1=\"10\" y1=\"13\" x2=\"32\" y2=\"35\" />\n",
       "  <line x1=\"10\" y1=\"15\" x2=\"32\" y2=\"37\" />\n",
       "  <line x1=\"10\" y1=\"17\" x2=\"32\" y2=\"39\" />\n",
       "  <line x1=\"10\" y1=\"21\" x2=\"32\" y2=\"43\" />\n",
       "  <line x1=\"10\" y1=\"23\" x2=\"32\" y2=\"45\" />\n",
       "  <line x1=\"10\" y1=\"25\" x2=\"32\" y2=\"47\" />\n",
       "  <line x1=\"10\" y1=\"26\" x2=\"32\" y2=\"49\" />\n",
       "  <line x1=\"10\" y1=\"28\" x2=\"32\" y2=\"51\" />\n",
       "  <line x1=\"10\" y1=\"30\" x2=\"32\" y2=\"53\" />\n",
       "  <line x1=\"10\" y1=\"32\" x2=\"32\" y2=\"55\" />\n",
       "  <line x1=\"10\" y1=\"34\" x2=\"32\" y2=\"57\" />\n",
       "  <line x1=\"10\" y1=\"36\" x2=\"32\" y2=\"59\" />\n",
       "  <line x1=\"10\" y1=\"39\" x2=\"32\" y2=\"62\" style=\"stroke-width:2\" />\n",
       "\n",
       "  <!-- Vertical lines -->\n",
       "  <line x1=\"10\" y1=\"0\" x2=\"10\" y2=\"39\" style=\"stroke-width:2\" />\n",
       "  <line x1=\"11\" y1=\"1\" x2=\"11\" y2=\"40\" />\n",
       "  <line x1=\"12\" y1=\"2\" x2=\"12\" y2=\"41\" />\n",
       "  <line x1=\"13\" y1=\"3\" x2=\"13\" y2=\"42\" />\n",
       "  <line x1=\"14\" y1=\"4\" x2=\"14\" y2=\"43\" />\n",
       "  <line x1=\"15\" y1=\"5\" x2=\"15\" y2=\"45\" />\n",
       "  <line x1=\"16\" y1=\"6\" x2=\"16\" y2=\"46\" />\n",
       "  <line x1=\"17\" y1=\"7\" x2=\"17\" y2=\"47\" />\n",
       "  <line x1=\"18\" y1=\"8\" x2=\"18\" y2=\"48\" />\n",
       "  <line x1=\"19\" y1=\"9\" x2=\"19\" y2=\"49\" />\n",
       "  <line x1=\"21\" y1=\"11\" x2=\"21\" y2=\"51\" />\n",
       "  <line x1=\"22\" y1=\"12\" x2=\"22\" y2=\"52\" />\n",
       "  <line x1=\"23\" y1=\"13\" x2=\"23\" y2=\"53\" />\n",
       "  <line x1=\"25\" y1=\"15\" x2=\"25\" y2=\"54\" />\n",
       "  <line x1=\"26\" y1=\"16\" x2=\"26\" y2=\"55\" />\n",
       "  <line x1=\"27\" y1=\"17\" x2=\"27\" y2=\"56\" />\n",
       "  <line x1=\"28\" y1=\"18\" x2=\"28\" y2=\"57\" />\n",
       "  <line x1=\"29\" y1=\"19\" x2=\"29\" y2=\"59\" />\n",
       "  <line x1=\"30\" y1=\"20\" x2=\"30\" y2=\"60\" />\n",
       "  <line x1=\"32\" y1=\"22\" x2=\"32\" y2=\"62\" style=\"stroke-width:2\" />\n",
       "\n",
       "  <!-- Colored Rectangle -->\n",
       "  <polygon points=\"10.0,0.0 32.523762084563465,22.523762084563465 32.523762084563465,62.1644071824761 10.0,39.64064509791263\" style=\"fill:#8B4903A0;stroke-width:0\"/>\n",
       "\n",
       "  <!-- Horizontal lines -->\n",
       "  <line x1=\"10\" y1=\"0\" x2=\"130\" y2=\"0\" style=\"stroke-width:2\" />\n",
       "  <line x1=\"11\" y1=\"1\" x2=\"131\" y2=\"1\" />\n",
       "  <line x1=\"12\" y1=\"2\" x2=\"132\" y2=\"2\" />\n",
       "  <line x1=\"13\" y1=\"3\" x2=\"133\" y2=\"3\" />\n",
       "  <line x1=\"14\" y1=\"4\" x2=\"134\" y2=\"4\" />\n",
       "  <line x1=\"15\" y1=\"5\" x2=\"135\" y2=\"5\" />\n",
       "  <line x1=\"16\" y1=\"6\" x2=\"136\" y2=\"6\" />\n",
       "  <line x1=\"17\" y1=\"7\" x2=\"137\" y2=\"7\" />\n",
       "  <line x1=\"18\" y1=\"8\" x2=\"138\" y2=\"8\" />\n",
       "  <line x1=\"19\" y1=\"9\" x2=\"139\" y2=\"9\" />\n",
       "  <line x1=\"21\" y1=\"11\" x2=\"141\" y2=\"11\" />\n",
       "  <line x1=\"22\" y1=\"12\" x2=\"142\" y2=\"12\" />\n",
       "  <line x1=\"23\" y1=\"13\" x2=\"143\" y2=\"13\" />\n",
       "  <line x1=\"25\" y1=\"15\" x2=\"145\" y2=\"15\" />\n",
       "  <line x1=\"26\" y1=\"16\" x2=\"146\" y2=\"16\" />\n",
       "  <line x1=\"27\" y1=\"17\" x2=\"147\" y2=\"17\" />\n",
       "  <line x1=\"28\" y1=\"18\" x2=\"148\" y2=\"18\" />\n",
       "  <line x1=\"29\" y1=\"19\" x2=\"149\" y2=\"19\" />\n",
       "  <line x1=\"30\" y1=\"20\" x2=\"150\" y2=\"20\" />\n",
       "  <line x1=\"32\" y1=\"22\" x2=\"152\" y2=\"22\" style=\"stroke-width:2\" />\n",
       "\n",
       "  <!-- Vertical lines -->\n",
       "  <line x1=\"10\" y1=\"0\" x2=\"32\" y2=\"22\" style=\"stroke-width:2\" />\n",
       "  <line x1=\"130\" y1=\"0\" x2=\"152\" y2=\"22\" style=\"stroke-width:2\" />\n",
       "\n",
       "  <!-- Colored Rectangle -->\n",
       "  <polygon points=\"10.0,0.0 130.0,0.0 152.52376208456346,22.523762084563465 32.523762084563465,22.523762084563465\" style=\"fill:#8B4903A0;stroke-width:0\"/>\n",
       "\n",
       "  <!-- Horizontal lines -->\n",
       "  <line x1=\"32\" y1=\"22\" x2=\"152\" y2=\"22\" style=\"stroke-width:2\" />\n",
       "  <line x1=\"32\" y1=\"24\" x2=\"152\" y2=\"24\" />\n",
       "  <line x1=\"32\" y1=\"26\" x2=\"152\" y2=\"26\" />\n",
       "  <line x1=\"32\" y1=\"28\" x2=\"152\" y2=\"28\" />\n",
       "  <line x1=\"32\" y1=\"30\" x2=\"152\" y2=\"30\" />\n",
       "  <line x1=\"32\" y1=\"32\" x2=\"152\" y2=\"32\" />\n",
       "  <line x1=\"32\" y1=\"34\" x2=\"152\" y2=\"34\" />\n",
       "  <line x1=\"32\" y1=\"35\" x2=\"152\" y2=\"35\" />\n",
       "  <line x1=\"32\" y1=\"37\" x2=\"152\" y2=\"37\" />\n",
       "  <line x1=\"32\" y1=\"39\" x2=\"152\" y2=\"39\" />\n",
       "  <line x1=\"32\" y1=\"43\" x2=\"152\" y2=\"43\" />\n",
       "  <line x1=\"32\" y1=\"45\" x2=\"152\" y2=\"45\" />\n",
       "  <line x1=\"32\" y1=\"47\" x2=\"152\" y2=\"47\" />\n",
       "  <line x1=\"32\" y1=\"49\" x2=\"152\" y2=\"49\" />\n",
       "  <line x1=\"32\" y1=\"51\" x2=\"152\" y2=\"51\" />\n",
       "  <line x1=\"32\" y1=\"53\" x2=\"152\" y2=\"53\" />\n",
       "  <line x1=\"32\" y1=\"55\" x2=\"152\" y2=\"55\" />\n",
       "  <line x1=\"32\" y1=\"57\" x2=\"152\" y2=\"57\" />\n",
       "  <line x1=\"32\" y1=\"59\" x2=\"152\" y2=\"59\" />\n",
       "  <line x1=\"32\" y1=\"62\" x2=\"152\" y2=\"62\" style=\"stroke-width:2\" />\n",
       "\n",
       "  <!-- Vertical lines -->\n",
       "  <line x1=\"32\" y1=\"22\" x2=\"32\" y2=\"62\" style=\"stroke-width:2\" />\n",
       "  <line x1=\"152\" y1=\"22\" x2=\"152\" y2=\"62\" style=\"stroke-width:2\" />\n",
       "\n",
       "  <!-- Colored Rectangle -->\n",
       "  <polygon points=\"32.523762084563465,22.523762084563465 152.52376208456346,22.523762084563465 152.52376208456346,62.1644071824761 32.523762084563465,62.1644071824761\" style=\"fill:#8B4903A0;stroke-width:0\"/>\n",
       "\n",
       "  <!-- Text -->\n",
       "  <text x=\"92.523762\" y=\"82.164407\" font-size=\"1.0rem\" font-weight=\"100\" text-anchor=\"middle\" >7300</text>\n",
       "  <text x=\"172.523762\" y=\"42.344085\" font-size=\"1.0rem\" font-weight=\"100\" text-anchor=\"middle\" transform=\"rotate(-90,172.523762,42.344085)\">886</text>\n",
       "  <text x=\"11.261881\" y=\"70.902526\" font-size=\"1.0rem\" font-weight=\"100\" text-anchor=\"middle\" transform=\"rotate(45,11.261881,70.902526)\">691</text>\n",
       "</svg>\n",
       "        </td>\n",
       "    </tr>\n",
       "</table></div></li></ul></div></li><li class='xr-section-item'><input id='section-25cac200-5557-41c7-a84f-99a14b2a9daa' class='xr-section-summary-in' type='checkbox'  ><label for='section-25cac200-5557-41c7-a84f-99a14b2a9daa' class='xr-section-summary' >Indexes: <span>(3)</span></label><div class='xr-section-inline-details'></div><div class='xr-section-details'><ul class='xr-var-list'><li class='xr-var-item'><div class='xr-index-name'><div>lat</div></div><div class='xr-index-preview'>PandasIndex</div><div></div><input id='index-12e23b71-ce4a-43c6-a0e4-880a2f901061' class='xr-index-data-in' type='checkbox'/><label for='index-12e23b71-ce4a-43c6-a0e4-880a2f901061' title='Show/Hide index repr'><svg class='icon xr-icon-database'><use xlink:href='#icon-database'></use></svg></label><div class='xr-index-data'><pre>PandasIndex(Index([ -44.5, -44.45,  -44.4, -44.35,  -44.3, -44.25,  -44.2, -44.15,  -44.1,\n",
       "       -44.05,\n",
       "       ...\n",
       "       -10.45,  -10.4, -10.35,  -10.3, -10.25,  -10.2, -10.15,  -10.1, -10.05,\n",
       "        -10.0],\n",
       "      dtype=&#x27;float64&#x27;, name=&#x27;lat&#x27;, length=691))</pre></div></li><li class='xr-var-item'><div class='xr-index-name'><div>lon</div></div><div class='xr-index-preview'>PandasIndex</div><div></div><input id='index-37082dc2-543b-4b62-9ac4-5f53b3acc46e' class='xr-index-data-in' type='checkbox'/><label for='index-37082dc2-543b-4b62-9ac4-5f53b3acc46e' title='Show/Hide index repr'><svg class='icon xr-icon-database'><use xlink:href='#icon-database'></use></svg></label><div class='xr-index-data'><pre>PandasIndex(Index([ 112.0, 112.05,  112.1, 112.15,  112.2, 112.25,  112.3, 112.35,  112.4,\n",
       "       112.45,\n",
       "       ...\n",
       "        155.8, 155.85,  155.9, 155.95,  156.0, 156.05,  156.1, 156.15,  156.2,\n",
       "       156.25],\n",
       "      dtype=&#x27;float64&#x27;, name=&#x27;lon&#x27;, length=886))</pre></div></li><li class='xr-var-item'><div class='xr-index-name'><div>time</div></div><div class='xr-index-preview'>PandasIndex</div><div></div><input id='index-001f51b1-5a20-4d10-b66c-0e4d2aa42e16' class='xr-index-data-in' type='checkbox'/><label for='index-001f51b1-5a20-4d10-b66c-0e4d2aa42e16' title='Show/Hide index repr'><svg class='icon xr-icon-database'><use xlink:href='#icon-database'></use></svg></label><div class='xr-index-data'><pre>PandasIndex(CFTimeIndex([2022-01-01 12:00:00, 2022-01-02 12:00:00, 2022-01-03 12:00:00,\n",
       "             2022-01-04 12:00:00, 2022-01-05 12:00:00, 2022-01-06 12:00:00,\n",
       "             2022-01-07 12:00:00, 2022-01-08 12:00:00, 2022-01-09 12:00:00,\n",
       "             2022-01-10 12:00:00,\n",
       "             ...\n",
       "             2041-12-22 12:00:00, 2041-12-23 12:00:00, 2041-12-24 12:00:00,\n",
       "             2041-12-25 12:00:00, 2041-12-26 12:00:00, 2041-12-27 12:00:00,\n",
       "             2041-12-28 12:00:00, 2041-12-29 12:00:00, 2041-12-30 12:00:00,\n",
       "             2041-12-31 12:00:00],\n",
       "            dtype=&#x27;object&#x27;, length=7300, calendar=&#x27;noleap&#x27;, freq=&#x27;D&#x27;))</pre></div></li></ul></div></li><li class='xr-section-item'><input id='section-aee7bbec-e50a-4b87-adfc-436b1b2d3354' class='xr-section-summary-in' type='checkbox' disabled ><label for='section-aee7bbec-e50a-4b87-adfc-436b1b2d3354' class='xr-section-summary'  title='Expand/collapse section'>Attributes: <span>(0)</span></label><div class='xr-section-inline-details'></div><div class='xr-section-details'><dl class='xr-attrs'></dl></div></li></ul></div></div>"
      ],
      "text/plain": [
       "<xarray.Dataset> Size: 36GB\n",
       "Dimensions:  (lat: 691, lon: 886, time: 7300)\n",
       "Coordinates:\n",
       "  * lat      (lat) float64 6kB -44.5 -44.45 -44.4 -44.35 ... -10.1 -10.05 -10.0\n",
       "  * lon      (lon) float64 7kB 112.0 112.0 112.1 112.2 ... 156.2 156.2 156.2\n",
       "  * time     (time) object 58kB 2022-01-01 12:00:00 ... 2041-12-31 12:00:00\n",
       "Data variables:\n",
       "    FFDI     (lat, lon, time) float64 36GB dask.array<chunksize=(33, 43, 7300), meta=np.ndarray>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "ds = xr.open_zarr(filelist[1], use_cftime=True)\n",
    "ds"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "dc1436af-5604-4511-b84f-ed2c746a8c7d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 17.8 s, sys: 1.83 s, total: 19.6 s\n",
      "Wall time: 19.6 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "# calculate weighted mask this is a very slow part of the code and\n",
    "# can be reused for any datasets using the same regions and the same lat lon\n",
    "mask_frac = regions.mask_3D_frac_approx(ds[var])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "ec9e7b93-ce43-4233-8462-93755e06948c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "BOM-BARPA-R-ACCESS-CM2-r4i1p1f1\n",
      "BOM-BARPA-R-CMCC-ESM2-r1i1p1f1\n",
      "BOM-BARPA-R-NorESM2-MM-r1i1p1f1\n",
      "BOM-BARPA-R-CESM2-r11i1p1f1\n",
      "CSIRO-CCAM-v2203-SN-CMCC-ESM2-r1i1p1f1\n",
      "BOM-BARPA-R-MPI-ESM1-2-HR-r1i1p1f1\n",
      "CSIRO-CCAM-v2203-SN-CESM2-r11i1p1f1\n",
      "BOM-BARPA-R-EC-Earth3-r1i1p1f1\n",
      "CSIRO-CCAM-v2203-SN-ACCESS-ESM1-5-r6i1p1f1\n",
      "CSIRO-CCAM-v2203-SN-EC-Earth3-r1i1p1f1\n",
      "CSIRO-CCAM-v2203-SN-ACCESS-CM2-r4i1p1f1\n",
      "CSIRO-CCAM-v2203-SN-CNRM-ESM2-1-r1i1p1f2\n",
      "merging...\n",
      "calculating percentiles...\n",
      "compute...\n",
      "convert to dataframe...\n",
      "CPU times: user 3h 57min 49s, sys: 1h 11min 6s, total: 5h 8min 55s\n",
      "Wall time: 18min 1s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "# calculate the area averages for each region\n",
    "dict_means = {}\n",
    "for file in filelist:\n",
    "    ensemble_member = \"-\".join([file.split(\"/\")[-1].split(\"_\")[i] for i in (1,5,2,4)])\n",
    "    print(ensemble_member)\n",
    "    ds = xr.open_zarr(file, use_cftime = True,)\n",
    "    ds = ds.expand_dims(dim= {\"ensemble_member\":[ensemble_member]}, axis=0)\n",
    "    df = acs_regional_stats(ds=ds,\n",
    "                                var=var, \n",
    "                                mask=mask_frac, \n",
    "                                dims = (\"time\", \"lat\", \"lon\",), \n",
    "                                how = [\"mean\"])\n",
    "    # Round to two decimal places (eg for temperature anomalies)\n",
    "    df[f\"{var}_mean\"] = df[f\"{var}_mean\"].round(2)\n",
    "\n",
    "    dict_means.update({ensemble_member:df})\n",
    "\n",
    "# dict_means = dask.compute(dict_means, traverse=False)\n",
    "# merge each ensemble member into one xarray dataset\n",
    "print(\"merging...\")\n",
    "ds_means = xr.merge([dict_means[key] for key in dict_means.keys()])\n",
    "\n",
    "# calculate the ensemble median per region\n",
    "print(\"calculating percentiles...\")\n",
    "ens_p10 = ds_means.quantile(0.1, dim=\"ensemble_member\").expand_dims(dim= {\"ensemble_member\":[\"ensemble_p10\"]}, axis=0).drop_vars(\"quantile\")\n",
    "ens_p50 = ds_means.quantile(0.5, dim=\"ensemble_member\").expand_dims(dim= {\"ensemble_member\":[\"ensemble_p50\"]}, axis=0).drop_vars(\"quantile\")\n",
    "ens_p90 = ds_means.quantile(0.9, dim=\"ensemble_member\").expand_dims(dim= {\"ensemble_member\":[\"ensemble_p90\"]}, axis=0).drop_vars(\"quantile\")\n",
    "\n",
    "# then add it to the xr dataset\n",
    "print(\"compute...\")\n",
    "ds_merged_stats = xr.merge([ds_means, ens_p10, ens_p50, ens_p90]).compute()\n",
    "print(\"convert to dataframe...\")\n",
    "df_means = ds_merged_stats.to_dataframe()[f\"{var}_mean\"].unstack().T\n",
    "\n",
    "df_means[\"abbrevs\"] = regions.abbrevs\n",
    "df_means[\"names\"] = regions.names\n",
    "df_means = df_means[[\"abbrevs\", \"names\"] + list(df_means.keys()[0:-2])]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "81d2bbad-1afe-4a8e-b7cd-bc5e7cca324c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>ensemble_member</th>\n",
       "      <th>abbrevs</th>\n",
       "      <th>names</th>\n",
       "      <th>BOM-BARPA-R-ACCESS-CM2-r4i1p1f1</th>\n",
       "      <th>BOM-BARPA-R-CESM2-r11i1p1f1</th>\n",
       "      <th>BOM-BARPA-R-CMCC-ESM2-r1i1p1f1</th>\n",
       "      <th>BOM-BARPA-R-EC-Earth3-r1i1p1f1</th>\n",
       "      <th>BOM-BARPA-R-MPI-ESM1-2-HR-r1i1p1f1</th>\n",
       "      <th>BOM-BARPA-R-NorESM2-MM-r1i1p1f1</th>\n",
       "      <th>CSIRO-CCAM-v2203-SN-ACCESS-CM2-r4i1p1f1</th>\n",
       "      <th>CSIRO-CCAM-v2203-SN-ACCESS-ESM1-5-r6i1p1f1</th>\n",
       "      <th>CSIRO-CCAM-v2203-SN-CESM2-r11i1p1f1</th>\n",
       "      <th>CSIRO-CCAM-v2203-SN-CMCC-ESM2-r1i1p1f1</th>\n",
       "      <th>CSIRO-CCAM-v2203-SN-CNRM-ESM2-1-r1i1p1f2</th>\n",
       "      <th>CSIRO-CCAM-v2203-SN-EC-Earth3-r1i1p1f1</th>\n",
       "      <th>ensemble_p10</th>\n",
       "      <th>ensemble_p50</th>\n",
       "      <th>ensemble_p90</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>region</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>WANorth</td>\n",
       "      <td>Western Australia North</td>\n",
       "      <td>0.81</td>\n",
       "      <td>-0.53</td>\n",
       "      <td>2.07</td>\n",
       "      <td>-0.55</td>\n",
       "      <td>1.33</td>\n",
       "      <td>2.55</td>\n",
       "      <td>0.61</td>\n",
       "      <td>2.11</td>\n",
       "      <td>-0.51</td>\n",
       "      <td>3.32</td>\n",
       "      <td>-1.35</td>\n",
       "      <td>-0.54</td>\n",
       "      <td>-0.549</td>\n",
       "      <td>0.710</td>\n",
       "      <td>2.506</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>WASouth</td>\n",
       "      <td>Western Australia South</td>\n",
       "      <td>0.70</td>\n",
       "      <td>-0.25</td>\n",
       "      <td>1.83</td>\n",
       "      <td>-0.66</td>\n",
       "      <td>0.55</td>\n",
       "      <td>1.84</td>\n",
       "      <td>0.83</td>\n",
       "      <td>1.33</td>\n",
       "      <td>-0.17</td>\n",
       "      <td>2.80</td>\n",
       "      <td>-1.04</td>\n",
       "      <td>0.16</td>\n",
       "      <td>-0.619</td>\n",
       "      <td>0.625</td>\n",
       "      <td>1.839</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>NSW</td>\n",
       "      <td>New South Wales and Australian Capital Territory</td>\n",
       "      <td>-0.17</td>\n",
       "      <td>-0.21</td>\n",
       "      <td>1.09</td>\n",
       "      <td>-0.33</td>\n",
       "      <td>3.05</td>\n",
       "      <td>1.42</td>\n",
       "      <td>-0.10</td>\n",
       "      <td>1.89</td>\n",
       "      <td>-0.08</td>\n",
       "      <td>1.71</td>\n",
       "      <td>0.20</td>\n",
       "      <td>0.27</td>\n",
       "      <td>-0.206</td>\n",
       "      <td>0.235</td>\n",
       "      <td>1.872</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>VIC</td>\n",
       "      <td>Victoria</td>\n",
       "      <td>0.17</td>\n",
       "      <td>0.28</td>\n",
       "      <td>0.53</td>\n",
       "      <td>-0.34</td>\n",
       "      <td>1.55</td>\n",
       "      <td>0.97</td>\n",
       "      <td>-0.15</td>\n",
       "      <td>0.86</td>\n",
       "      <td>0.08</td>\n",
       "      <td>0.66</td>\n",
       "      <td>0.49</td>\n",
       "      <td>-0.06</td>\n",
       "      <td>-0.141</td>\n",
       "      <td>0.385</td>\n",
       "      <td>0.959</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>SA</td>\n",
       "      <td>South Australia</td>\n",
       "      <td>0.20</td>\n",
       "      <td>0.19</td>\n",
       "      <td>1.91</td>\n",
       "      <td>-0.54</td>\n",
       "      <td>2.25</td>\n",
       "      <td>1.65</td>\n",
       "      <td>0.45</td>\n",
       "      <td>2.09</td>\n",
       "      <td>0.02</td>\n",
       "      <td>3.39</td>\n",
       "      <td>-0.09</td>\n",
       "      <td>0.07</td>\n",
       "      <td>-0.079</td>\n",
       "      <td>0.325</td>\n",
       "      <td>2.234</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>TAS</td>\n",
       "      <td>Tasmania</td>\n",
       "      <td>0.10</td>\n",
       "      <td>0.10</td>\n",
       "      <td>0.05</td>\n",
       "      <td>-0.02</td>\n",
       "      <td>0.13</td>\n",
       "      <td>0.19</td>\n",
       "      <td>0.04</td>\n",
       "      <td>0.15</td>\n",
       "      <td>0.03</td>\n",
       "      <td>-0.01</td>\n",
       "      <td>0.13</td>\n",
       "      <td>0.04</td>\n",
       "      <td>-0.006</td>\n",
       "      <td>0.075</td>\n",
       "      <td>0.148</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>NT</td>\n",
       "      <td>Northern Territory</td>\n",
       "      <td>0.28</td>\n",
       "      <td>-0.50</td>\n",
       "      <td>1.85</td>\n",
       "      <td>-0.65</td>\n",
       "      <td>2.24</td>\n",
       "      <td>2.00</td>\n",
       "      <td>0.47</td>\n",
       "      <td>2.44</td>\n",
       "      <td>-0.52</td>\n",
       "      <td>2.80</td>\n",
       "      <td>-0.68</td>\n",
       "      <td>-0.18</td>\n",
       "      <td>-0.637</td>\n",
       "      <td>0.375</td>\n",
       "      <td>2.420</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>QLDNorth</td>\n",
       "      <td>Queensland North</td>\n",
       "      <td>0.10</td>\n",
       "      <td>0.27</td>\n",
       "      <td>1.14</td>\n",
       "      <td>-0.49</td>\n",
       "      <td>1.98</td>\n",
       "      <td>1.42</td>\n",
       "      <td>-0.04</td>\n",
       "      <td>1.72</td>\n",
       "      <td>-0.03</td>\n",
       "      <td>1.58</td>\n",
       "      <td>0.24</td>\n",
       "      <td>0.05</td>\n",
       "      <td>-0.039</td>\n",
       "      <td>0.255</td>\n",
       "      <td>1.706</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>QLDSouth</td>\n",
       "      <td>Queensland South</td>\n",
       "      <td>-0.03</td>\n",
       "      <td>0.13</td>\n",
       "      <td>1.62</td>\n",
       "      <td>-0.32</td>\n",
       "      <td>2.70</td>\n",
       "      <td>1.74</td>\n",
       "      <td>-0.00</td>\n",
       "      <td>3.02</td>\n",
       "      <td>0.06</td>\n",
       "      <td>2.43</td>\n",
       "      <td>0.05</td>\n",
       "      <td>-0.14</td>\n",
       "      <td>-0.129</td>\n",
       "      <td>0.095</td>\n",
       "      <td>2.673</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>AUS</td>\n",
       "      <td>Australia</td>\n",
       "      <td>0.31</td>\n",
       "      <td>-0.15</td>\n",
       "      <td>1.65</td>\n",
       "      <td>-0.52</td>\n",
       "      <td>1.87</td>\n",
       "      <td>1.80</td>\n",
       "      <td>0.36</td>\n",
       "      <td>2.01</td>\n",
       "      <td>-0.19</td>\n",
       "      <td>2.58</td>\n",
       "      <td>-0.45</td>\n",
       "      <td>-0.05</td>\n",
       "      <td>-0.424</td>\n",
       "      <td>0.335</td>\n",
       "      <td>1.996</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "ensemble_member   abbrevs                                             names  \\\n",
       "region                                                                        \n",
       "0                 WANorth                           Western Australia North   \n",
       "1                 WASouth                           Western Australia South   \n",
       "2                     NSW  New South Wales and Australian Capital Territory   \n",
       "3                     VIC                                          Victoria   \n",
       "4                      SA                                   South Australia   \n",
       "5                     TAS                                          Tasmania   \n",
       "6                      NT                                Northern Territory   \n",
       "7                QLDNorth                                  Queensland North   \n",
       "8                QLDSouth                                  Queensland South   \n",
       "9                     AUS                                         Australia   \n",
       "\n",
       "ensemble_member  BOM-BARPA-R-ACCESS-CM2-r4i1p1f1  BOM-BARPA-R-CESM2-r11i1p1f1  \\\n",
       "region                                                                          \n",
       "0                                           0.81                        -0.53   \n",
       "1                                           0.70                        -0.25   \n",
       "2                                          -0.17                        -0.21   \n",
       "3                                           0.17                         0.28   \n",
       "4                                           0.20                         0.19   \n",
       "5                                           0.10                         0.10   \n",
       "6                                           0.28                        -0.50   \n",
       "7                                           0.10                         0.27   \n",
       "8                                          -0.03                         0.13   \n",
       "9                                           0.31                        -0.15   \n",
       "\n",
       "ensemble_member  BOM-BARPA-R-CMCC-ESM2-r1i1p1f1  \\\n",
       "region                                            \n",
       "0                                          2.07   \n",
       "1                                          1.83   \n",
       "2                                          1.09   \n",
       "3                                          0.53   \n",
       "4                                          1.91   \n",
       "5                                          0.05   \n",
       "6                                          1.85   \n",
       "7                                          1.14   \n",
       "8                                          1.62   \n",
       "9                                          1.65   \n",
       "\n",
       "ensemble_member  BOM-BARPA-R-EC-Earth3-r1i1p1f1  \\\n",
       "region                                            \n",
       "0                                         -0.55   \n",
       "1                                         -0.66   \n",
       "2                                         -0.33   \n",
       "3                                         -0.34   \n",
       "4                                         -0.54   \n",
       "5                                         -0.02   \n",
       "6                                         -0.65   \n",
       "7                                         -0.49   \n",
       "8                                         -0.32   \n",
       "9                                         -0.52   \n",
       "\n",
       "ensemble_member  BOM-BARPA-R-MPI-ESM1-2-HR-r1i1p1f1  \\\n",
       "region                                                \n",
       "0                                              1.33   \n",
       "1                                              0.55   \n",
       "2                                              3.05   \n",
       "3                                              1.55   \n",
       "4                                              2.25   \n",
       "5                                              0.13   \n",
       "6                                              2.24   \n",
       "7                                              1.98   \n",
       "8                                              2.70   \n",
       "9                                              1.87   \n",
       "\n",
       "ensemble_member  BOM-BARPA-R-NorESM2-MM-r1i1p1f1  \\\n",
       "region                                             \n",
       "0                                           2.55   \n",
       "1                                           1.84   \n",
       "2                                           1.42   \n",
       "3                                           0.97   \n",
       "4                                           1.65   \n",
       "5                                           0.19   \n",
       "6                                           2.00   \n",
       "7                                           1.42   \n",
       "8                                           1.74   \n",
       "9                                           1.80   \n",
       "\n",
       "ensemble_member  CSIRO-CCAM-v2203-SN-ACCESS-CM2-r4i1p1f1  \\\n",
       "region                                                     \n",
       "0                                                   0.61   \n",
       "1                                                   0.83   \n",
       "2                                                  -0.10   \n",
       "3                                                  -0.15   \n",
       "4                                                   0.45   \n",
       "5                                                   0.04   \n",
       "6                                                   0.47   \n",
       "7                                                  -0.04   \n",
       "8                                                  -0.00   \n",
       "9                                                   0.36   \n",
       "\n",
       "ensemble_member  CSIRO-CCAM-v2203-SN-ACCESS-ESM1-5-r6i1p1f1  \\\n",
       "region                                                        \n",
       "0                                                      2.11   \n",
       "1                                                      1.33   \n",
       "2                                                      1.89   \n",
       "3                                                      0.86   \n",
       "4                                                      2.09   \n",
       "5                                                      0.15   \n",
       "6                                                      2.44   \n",
       "7                                                      1.72   \n",
       "8                                                      3.02   \n",
       "9                                                      2.01   \n",
       "\n",
       "ensemble_member  CSIRO-CCAM-v2203-SN-CESM2-r11i1p1f1  \\\n",
       "region                                                 \n",
       "0                                              -0.51   \n",
       "1                                              -0.17   \n",
       "2                                              -0.08   \n",
       "3                                               0.08   \n",
       "4                                               0.02   \n",
       "5                                               0.03   \n",
       "6                                              -0.52   \n",
       "7                                              -0.03   \n",
       "8                                               0.06   \n",
       "9                                              -0.19   \n",
       "\n",
       "ensemble_member  CSIRO-CCAM-v2203-SN-CMCC-ESM2-r1i1p1f1  \\\n",
       "region                                                    \n",
       "0                                                  3.32   \n",
       "1                                                  2.80   \n",
       "2                                                  1.71   \n",
       "3                                                  0.66   \n",
       "4                                                  3.39   \n",
       "5                                                 -0.01   \n",
       "6                                                  2.80   \n",
       "7                                                  1.58   \n",
       "8                                                  2.43   \n",
       "9                                                  2.58   \n",
       "\n",
       "ensemble_member  CSIRO-CCAM-v2203-SN-CNRM-ESM2-1-r1i1p1f2  \\\n",
       "region                                                      \n",
       "0                                                   -1.35   \n",
       "1                                                   -1.04   \n",
       "2                                                    0.20   \n",
       "3                                                    0.49   \n",
       "4                                                   -0.09   \n",
       "5                                                    0.13   \n",
       "6                                                   -0.68   \n",
       "7                                                    0.24   \n",
       "8                                                    0.05   \n",
       "9                                                   -0.45   \n",
       "\n",
       "ensemble_member  CSIRO-CCAM-v2203-SN-EC-Earth3-r1i1p1f1  ensemble_p10  \\\n",
       "region                                                                  \n",
       "0                                                 -0.54        -0.549   \n",
       "1                                                  0.16        -0.619   \n",
       "2                                                  0.27        -0.206   \n",
       "3                                                 -0.06        -0.141   \n",
       "4                                                  0.07        -0.079   \n",
       "5                                                  0.04        -0.006   \n",
       "6                                                 -0.18        -0.637   \n",
       "7                                                  0.05        -0.039   \n",
       "8                                                 -0.14        -0.129   \n",
       "9                                                 -0.05        -0.424   \n",
       "\n",
       "ensemble_member  ensemble_p50  ensemble_p90  \n",
       "region                                       \n",
       "0                       0.710         2.506  \n",
       "1                       0.625         1.839  \n",
       "2                       0.235         1.872  \n",
       "3                       0.385         0.959  \n",
       "4                       0.325         2.234  \n",
       "5                       0.075         0.148  \n",
       "6                       0.375         2.420  \n",
       "7                       0.255         1.706  \n",
       "8                       0.095         2.673  \n",
       "9                       0.335         1.996  "
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_means"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "74034b59-3699-45a7-a7e9-185f65d58761",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_means.to_csv(f\"/g/data/mn51/users/gt3409/plotting_maps/ffdi_ncra-aus-regions_MME_ssp370_v1-r1_{gwl}-GWL1.2.csv\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:analysis3-24.04] *",
   "language": "python",
   "name": "conda-env-analysis3-24.04-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}