EnvriHub Marine EOV Broker SPARQL.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "b7c72676",
   "metadata": {},
   "source": [
    "# Marine EOV Broker\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "e9adae3a",
   "metadata": {},
   "outputs": [],
   "source": [
    "from marine_eov_broker import MarineRiBroker\n",
    "import logging\n",
    "import matplotlib.pyplot as plt\n",
    "import xarray as xr\n",
    "import pandas as pd\n",
    "\n",
    "logger = logging.getLogger()\n",
    "logger.setLevel(logging.INFO)\n",
    "# logger.setLevel(logging.DEBUG)\n",
    "\n",
    "# print(MarineRiBroker.ERDDAP_OUTPUT_FORMATS)\n",
    "# print(MarineRiBroker.EOV_LIST)\n",
    "# import sys\n",
    "# sys.prefix"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d84b2c9e",
   "metadata": {},
   "source": [
    "## Start the broker\n",
    "\n",
    "It will take some time (though it still needs improvements on performances). This is because the broker will :\n",
    "* load vocabularies upon startup\n",
    "* load erddap datasets metadata from all erddap servers\n",
    "\n",
    "\n",
    "**Question :**\n",
    "Do we want to work with all datasets on Erddap servers ? Or do we want to build a fixed list for them ?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "fab5fc3f",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Querying vocabulary server for EOV : EV_OXY\n",
      "INFO:root:Querying vocabulary server for EOV : EV_SEATEMP\n",
      "INFO:root:Querying vocabulary server for EOV : EV_SALIN\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 188 ms, sys: 93.8 ms, total: 281 ms\n",
      "Wall time: 5.48 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "broker = MarineRiBroker.MarineBroker({\"https://www.ifremer.fr/erddap\": [\"ArgoFloats-synthetic-BGC\"]})\n",
    "\n",
    "#broker = MarineRiBroker.MarineBroker({\n",
    "#         \"https://www.ifremer.fr/erddap\": [\"ArgoFloats\", \"ArgoFloats-synthetic-BGC\", \n",
    "#                                           \"SDC_BAL_CLIM_TS_V2_m\", \"SDC_BAL_CLIM_TS_V2_s\",\n",
    "#                                           \"SDC_GLO_AGG_V2\", \n",
    "#                                           \"SDC_GLO_CLIM_TS_V2_1\", \"SDC_GLO_CLIM_TS_V2_2\",\n",
    "#                                           \"SDC_BLS_CLIM_TS_V2_m\", \"SDC_BLS_CLIM_TS_V2_s\",\n",
    "#                                           \"SDC_MED_CLIM_TS_V2_m_pre_post_emt\",\n",
    "#                                           \"SDC_MED_CLIM_TS_V2_m_whole_period\",\n",
    "#                                           \"SDC_MED_CLIM_TS_V2_s_decades\",\n",
    "#                                           \"SDC_MED_CLIM_TS_V2_s_pre_post_emt\",\n",
    "#                                           \"SDC_MED_CLIM_TS_V2_s_whole_period\",\n",
    "#                                           \"SDC_NAT_CLIM_TS_V2_050_m\", \"SDC_NAT_CLIM_TS_V2_050_s\"]\n",
    "#     })"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "240149d1",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Create a request to the broker :\n",
    "The user must provide the EOVs, min/max date/lat/lon.\n",
    "In the case of Argo, the user must provide a SPARQL query returning the platforms WMO."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "3a51e770",
   "metadata": {},
   "outputs": [],
   "source": [
    "#eovs_request = [\"EV_SALIN\", \"EV_OXY\", \"EV_SEATEMP\"]\n",
    "eovs_request = [\"EV_OXY\"]\n",
    "\n",
    "start_date = \"2022-01-16\"\n",
    "end_date = \"2022-01-17\"\n",
    "# North-east Atlantic Ocean\n",
    "min_lon = -40\n",
    "min_lat = 35\n",
    "max_lon = 2\n",
    "max_lat = 62\n",
    "\n",
    "# logger.setLevel(logging.DEBUG)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "025348d8",
   "metadata": {},
   "source": [
    "## Query the SPARQL endpoint\n",
    "Once you defined the eov variables you want, you can query the SPARQL endpoint of your choice that will do a selection over the metadata you wish.\n",
    "In this example, we are going to use `platform_number` as the link between the SPARQL endpoint and ERDAPP (the variable name in the SPARQL query has to be the same than the ERDAPP variable). You then specify the ERDAPP server on which the data will be fetched (here `ArgoFloats-synthetic-BGC`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "fb80b0b6",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 2.53 s, sys: 469 ms, total: 3 s\n",
      "Wall time: 9.79 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "response = broker.submit_sparql_query(\"\"\"\n",
    "prefix geo: <https://www.w3.org/2003/01/geo/wgs84_pos#>\n",
    "prefix owl: <http://www.w3.org/2002/07/owl#>\n",
    "prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> \n",
    "prefix ssn: <http://www.w3.org/ns/ssn/> \n",
    "prefix xml: <http://www.w3.org/XML/1998/namespace> \n",
    "prefix xsd: <http://www.w3.org/2001/XMLSchema#> \n",
    "prefix argo: <https://co.ifremer.fr/co/argo-linked-data/doc/argo-floats.ttl#> \n",
    "prefix foaf: <http://xmlns.com/foaf/0.1/> \n",
    "prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> \n",
    "prefix sosa: <http://www.w3.org/ns/sosa/>\n",
    "prefix nerc: <http://vocab.nerc.ac.uk/collection/>\n",
    "prefix dct: <http://purl.org/dc/terms/>\n",
    "prefix prov: <https://www.w3.org/TR/prov-o/>\n",
    "prefix dcat: <http://www.w3.org/ns/dcat#>\n",
    "prefix xsd: <http://www.w3.org/2001/XMLSchema#>\n",
    "\n",
    "SELECT ?platform_number ?lat ?lon WHERE { \n",
    "?float a dcat:Dataset;\n",
    "       dct:identifier ?platform_number;\n",
    "       argo:cycle ?cycle;\n",
    "       sosa:hosts ?sensor.\n",
    "?sensor sosa:observes <http://vocab.nerc.ac.uk/collection/R03/current/DOXY>.\n",
    "?cycle argo:startDate ?date;\n",
    "       geo:latitude ?lat;\n",
    "       geo:longitude ?lon.\n",
    "\n",
    "FILTER(?lat >= 35)\n",
    "FILTER(?lat <= 62)\n",
    "FILTER(?lon >= -40)\n",
    "FILTER(?lon <= 2)\n",
    "FILTER(?date >= \"2022-01-16\"^^xsd:date)\n",
    "FILTER(?date <= \"2022-01-17\"^^xsd:date)\n",
    "}\n",
    "\"\"\", broker.DEFAULT_SPARQL_ENDPOINTS[\"Argo\"], \n",
    "     eovs_request, \n",
    "     start_date,\n",
    "     end_date,\n",
    "     min_lon,\n",
    "     min_lat,\n",
    "     max_lon,\n",
    "     max_lat,\n",
    "     \"nc\",\n",
    "     \"platform_number\",\n",
    "     \"ArgoFloats-synthetic-BGC\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c874442c",
   "metadata": {},
   "source": [
    "## Results\n",
    "\n",
    "Once we have the SPARQL results containing the list of values that will be processed to ERDAPP, we can send the requests and compile them into a single Pandas frame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "b81eb9e6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>time</th>\n",
       "      <th>latitude</th>\n",
       "      <th>longitude</th>\n",
       "      <th>doxy</th>\n",
       "      <th>mlpl_doxy</th>\n",
       "      <th>pres</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>row</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2022-01-16 12:05:19</td>\n",
       "      <td>50.36314</td>\n",
       "      <td>-31.49648</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.607000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2022-01-16 12:05:19</td>\n",
       "      <td>50.36314</td>\n",
       "      <td>-31.49648</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.900000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2022-01-16 12:05:19</td>\n",
       "      <td>50.36314</td>\n",
       "      <td>-31.49648</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4.310000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2022-01-16 12:05:19</td>\n",
       "      <td>50.36314</td>\n",
       "      <td>-31.49648</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>5.440000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2022-01-16 12:05:19</td>\n",
       "      <td>50.36314</td>\n",
       "      <td>-31.49648</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>6.010000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>505</th>\n",
       "      <td>2022-01-16 04:15:45</td>\n",
       "      <td>42.60000</td>\n",
       "      <td>-16.77500</td>\n",
       "      <td>246.669815</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1749.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>506</th>\n",
       "      <td>2022-01-16 04:15:45</td>\n",
       "      <td>42.60000</td>\n",
       "      <td>-16.77500</td>\n",
       "      <td>246.785339</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1799.099976</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>507</th>\n",
       "      <td>2022-01-16 04:15:45</td>\n",
       "      <td>42.60000</td>\n",
       "      <td>-16.77500</td>\n",
       "      <td>246.422516</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1848.400024</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>508</th>\n",
       "      <td>2022-01-16 04:15:45</td>\n",
       "      <td>42.60000</td>\n",
       "      <td>-16.77500</td>\n",
       "      <td>246.775803</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1898.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>509</th>\n",
       "      <td>2022-01-16 04:15:45</td>\n",
       "      <td>42.60000</td>\n",
       "      <td>-16.77500</td>\n",
       "      <td>239.572922</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1949.800049</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5852 rows × 6 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                   time  latitude  longitude        doxy  mlpl_doxy  \\\n",
       "row                                                                   \n",
       "0   2022-01-16 12:05:19  50.36314  -31.49648         NaN        NaN   \n",
       "1   2022-01-16 12:05:19  50.36314  -31.49648         NaN        NaN   \n",
       "2   2022-01-16 12:05:19  50.36314  -31.49648         NaN        NaN   \n",
       "3   2022-01-16 12:05:19  50.36314  -31.49648         NaN        NaN   \n",
       "4   2022-01-16 12:05:19  50.36314  -31.49648         NaN        NaN   \n",
       "..                  ...       ...        ...         ...        ...   \n",
       "505 2022-01-16 04:15:45  42.60000  -16.77500  246.669815        NaN   \n",
       "506 2022-01-16 04:15:45  42.60000  -16.77500  246.785339        NaN   \n",
       "507 2022-01-16 04:15:45  42.60000  -16.77500  246.422516        NaN   \n",
       "508 2022-01-16 04:15:45  42.60000  -16.77500  246.775803        NaN   \n",
       "509 2022-01-16 04:15:45  42.60000  -16.77500  239.572922        NaN   \n",
       "\n",
       "            pres  \n",
       "row               \n",
       "0       1.607000  \n",
       "1       1.900000  \n",
       "2       4.310000  \n",
       "3       5.440000  \n",
       "4       6.010000  \n",
       "..           ...  \n",
       "505  1749.500000  \n",
       "506  1799.099976  \n",
       "507  1848.400024  \n",
       "508  1898.500000  \n",
       "509  1949.800049  \n",
       "\n",
       "[5852 rows x 6 columns]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "response.compile_results()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b5456212",
   "metadata": {},
   "source": [
    "![SPARQL combined with ERDDAP](images/SPARQL_broker.JPG)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}