demo_client.ipynb

{
 "metadata": {
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6-final"
  },
  "orig_nbformat": 2,
  "kernelspec": {
   "name": "python3",
   "display_name": "Python 3 (ipykernel)",
   "language": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2,
 "cells": [
  {
   "source": [
    "# Tiled Python Client Demonstration\n",
    "\n",
    "Demonstrate examples of a Python client accessing a tiled data server (running on `localhost`).  The server provides two databroker catalogs (`bdp2022` and `20idb_usaxs`) and some selected data files from APS beam lines in a nested directory known to the tiled data server as `directory`.\n",
    "\n",
    "Show two types of Python client:\n",
    "\n",
    "- The Python [`requests`](https://requests.readthedocs.io/) package\n",
    "  (instead of [`urllib.request`](https://docs.python.org/3.11/library/urllib.request.html)\n",
    "  from the Python Standard Library).\n",
    "  Python programmers with at least intermediate experience may have already\n",
    "  used `requests` (for reasons\n",
    "  best [summarized](https://stackoverflow.com/questions/2018026) by others.)\n",
    "  to *scrape* information from a web page.\n",
    "- TODO: The [`tiled.client`](https://blueskyproject.io/tiled/reference/python-client.html)\n",
    "  Python library from the Bluesky Framework.  The `tiled.client` makes it easier to work\n",
    "  directly with data structures such as numpy arrays, pandas DataFrames, and xarray tables.\n",
    "\n",
    "For each type of client, show some specific queries and responses.\n",
    "\n",
    "* [x] Find all runs in a catalog between these two ISO8601 dates.\n",
    "* [x] Find run(s) which match given metadata.\n",
    "* [x] Get overall metadata from given run.\n",
    "* [x] What are the data streams in this run?\n",
    "* [x] What is the metadata for this stream?\n",
    "* [x] Get the data from the data stream named primary (the canonical main data)."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "source": [
    "## Structure of Bluesky's databroker data\n",
    "\n",
    "Bluesky data is structured in databroker into three distinct levels which will be described below.\n",
    "\n",
    "- databroker *catalog* : a set of bluesky *runs*\n",
    "- bluesky *run* : data (& metadata) for a single run, including zero or more *streams*\n",
    "- run *stream* : a set of related data acquisition events (including metadata)"
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "source": [
    "### databroker *catalog*\n",
    "\n",
    "The Bluesky `databroker` stores data from a Bluesky instrument in a [MongoDB *collection*](https://www.mongodb.com/docs/manual/core/databases-and-collections/) which is called a *catalog*.  A single MongoDB server may host several databroker catalogs.  Tiled will access a catalog with a configuration such as this template called `45id_instrument`:\n",
    "\n",
    "```\n",
    "  - path: 45id_instrument\n",
    "    tree: databroker.mongo_normalized:Tree.from_uri\n",
    "    args:\n",
    "      uri: mongodb://DB_SERVER.xray.aps.anl.gov:27017/45id_instrument-bluesky\n",
    "```\n",
    "\n",
    "The *catalog* contains data & metadata from all the measurement *run*s.\n",
    "\n",
    "Note: The MongoDB documentation also describes:\n",
    "\n",
    "> In MongoDB, *databases* hold one or more collections of documents.\n",
    "\n",
    "To avoid further name confusion, this will be the only time we speak of a MongoDB *database*."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "source": [
    "### bluesky *run*\n",
    "\n",
    "A bluesky [*run*](https://blueskyproject.io/bluesky/documents.html#overview-of-a-run) contains all the data acquired for a single measurement sequence (such as a step scan).  It is stored by databroker as a set of JSON documents.\n",
    "\n",
    "The [documents](https://blueskyproject.io/event-model/external.html#the-documents) are structured according to the [Bluesky Event Model](https://blueskyproject.io/event-model), briefly summarized here.\n",
    "\n",
    "The `start` document contains metadata keys describing the run as it begins, such as the assigned [`uid`](https://docs.python.org/3/library/uuid.html#uuid.uuid4) (used by databroker to quickly locate the run) and the `time` (number of seconds since 1970-01-01 00:00:00 UTC).  Additional metadata keys may be supplied by the acquisition sequence code (the *plan*), the instrument, and/or the user.\n",
    "\n",
    "The `stop` document contains metadata keys describing how the run ended, including any data streams that were recorded.\n",
    "\n",
    "The tiled server adds an additional `summary` metadata key to the run that includes content from both `start` and `end`, including a `datetime` key that is a text version of the date & time when the run started.\n",
    "\n",
    "There are a couple other document types, used for [external file resources](https://blueskyproject.io/bluesky/hardware.html#external-asset-writing-interface) such as area detector images."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "source": [
    "### bluesky data *stream*\n",
    "\n",
    "During a bluesky run, data is acquired into streams, each with its own name.  It is common for the main data to be acquired.  A stream will have one or more descriptor documents and zero or more `event` documents.  The `descriptor` documents, combined by the tiled server into the stream's metadata, describe the signals measured in the stream.  The `event` documents contain each of the time-stamped data acquisition events in the stream.  The tiled server gathers these events together and provides data arrays organized by the signal name."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "source": [
    "### metadata\n",
    "\n",
    "As described above, there are two types of metadata for each run, the *run* metadata and the *stream* metadata.  See the two sections immediately preceding for details."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "source": [
    "# `requests` Client\n",
    "\n",
    "We'll use the `requests` package, to search the tiled server's API using tiled's `http://` interface by assembling a URI string.  Since the URI is a string consisting of several parts, we'll build that string up from its parts in each example below.\n",
    "\n",
    "The tiled server response is [JSON](https://www.json.org/) formatted, which is readable as a Python dictionary.  We'll let Python handle and report any Exceptions that might occur.  Here, we just import the `requests` package.\n",
    "\n",
    "The calls will look like:\n",
    "\n",
    "```py\n",
    "r = requests.get(uri).json()\n",
    "```\n",
    "\n",
    "where `r` is a Python dictionary of the results of the search request."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests"
   ]
  },
  {
   "source": [
    "As a convenience, make a function that converts a string representation of the date and time in ISO-8601 format into the Linux EPOCH floating-point representation needed for tiled's API.  This makes a long function call much shorter."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import datetime\n",
    "\n",
    "def iso_to_ts(isotime):\n",
    "    return datetime.datetime.fromisoformat(isotime).timestamp()"
   ]
  },
  {
   "source": [
    "We'll search the BDP project's databroker catalog, known to the tiled server (running on workstation `localhost` on port `8000`) by the text name `bdp2022`."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "server = \"localhost\"\n",
    "port = 8000\n",
    "catalog = \"bdp2022\""
   ]
  },
  {
   "source": [
    "### Find runs within range of dates\n",
    "\n",
    "Define the ends of the time span for the search query:"
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Find all runs in a catalog between these two ISO8601 dates.\n",
    "start_time = \"2022-05-01\"\n",
    "end_time = \"2022-11-01\"\n",
    "tz = \"US/Central\""
   ]
  },
  {
   "source": [
    "Using the `requests` package, ask the tiled server for all runs in the catalog that match the time range.\n",
    "\n",
    "Here, we build up the URI suffix in parts to expose how the search query is constructed.  The response is a Python dictionary.  We won't print the entire dictionary here since it likely contains a lot of information, perhaps too much to show in full."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "uri='http://localhost:8000/api/v1/node/search/bdp2022?page[limit]=0&filter[time_range][condition][since]=1651381200.0&filter[time_range][condition][until]=1667278800.0&filter[time_range][condition][timezone]=US/Central&sort=time'\n"
     ]
    }
   ],
   "source": [
    "uri =(\n",
    "    f\"http://{server}:{port}\"  # standard prefix\n",
    "    \"/api/v1/node/search\"  # API command\n",
    "    f\"/{catalog}\"  # catalog\n",
    "    \"?\"  # begin any command options\n",
    "    \"page[limit]=0\"  # 0: all matching\n",
    "    \"&\"  # separator between any additional options\n",
    "    f\"filter[time_range][condition][since]={iso_to_ts(start_time)}\"\n",
    "    f\"&filter[time_range][condition][until]={iso_to_ts(end_time)}\"\n",
    "    f\"&filter[time_range][condition][timezone]={tz}\"\n",
    "    \"&sort=time\"\n",
    ")\n",
    "print(f\"{uri=}\")\n",
    "r = requests.get(uri).json()"
   ]
  },
  {
   "source": [
    "Summarize the results (in object `r`):"
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def print_results_summary(r):\n",
    "    \"\"\"We'll use this a few times.\"\"\"\n",
    "    xref = dict(First=0, Last=-1)\n",
    "    for k, v in dict(First=0, Last=-1).items():\n",
    "        md = r[\"data\"][v][\"attributes\"][\"metadata\"]\n",
    "        # md keys: start  stop  summary\n",
    "        # summary key is composed by tiled server\n",
    "        plan_name = md[\"summary\"][\"plan_name\"]\n",
    "        scan_id = md[\"summary\"][\"scan_id\"]\n",
    "        started = md[\"summary\"][\"datetime\"]\n",
    "        print(f\"{k:5s} run: {started=} {scan_id=} {plan_name=}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "Search of catalog='bdp2022' has 397 runs.\nFirst run: started='2022-05-03T08:37:21.510276' scan_id=1596 plan_name='take_image'\nLast  run: started='2022-09-08T13:54:25.178280' scan_id=1960 plan_name='push_images'\n"
     ]
    }
   ],
   "source": [
    "print(f'Search of {catalog=} has {len(r[\"data\"])} runs.')\n",
    "print_results_summary(r)"
   ]
  },
  {
   "source": [
    "### Find runs matching a given plan name\n",
    "\n",
    "Find run(s) which match some given metadata.  In this search, let's find all the runs that match a given `plan_name`.  Let's use the `take_image` plan from the previous results."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "Search for plan_name='take_image'\nuri='http://localhost:8000/api/v1/node/search/bdp2022?page[limit]=0&filter[eq][condition][key]=plan_name&filter[eq][condition][value]=\"take_image\"&sort=time'\n"
     ]
    }
   ],
   "source": [
    "plan_name = \"take_image\"\n",
    "print(f\"Search for {plan_name=}\")\n",
    "\n",
    "uri = (\n",
    "    f\"http://{server}:{port}\"\n",
    "    \"/api/v1/node/search\"\n",
    "    f\"/{catalog}\"\n",
    "    \"?page[limit]=0\"  # 0: all matching\n",
    "    \"&filter[eq][condition][key]=plan_name\"\n",
    "    f'&filter[eq][condition][value]=\"{plan_name}\"'\n",
    "    \"&sort=time\"\n",
    ")\n",
    "print(f\"{uri=}\")\n",
    "r = requests.get(uri).json()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "Search of catalog='bdp2022' has 1029 runs.\nFirst run: started='2022-03-22T16:48:41.130881' scan_id=1 plan_name='take_image'\nLast  run: started='2022-08-30T15:06:37.662096' scan_id=1959 plan_name='take_image'\n"
     ]
    }
   ],
   "source": [
    "print(f'Search of {catalog=} has {len(r[\"data\"])} runs.')\n",
    "print_results_summary(r)"
   ]
  },
  {
   "source": [
    "## Find runs with given text\n",
    "\n",
    "Find run(s) that include some text in the metadata.  Here, let's find all the runs that have the text `M9 demo`."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "uri='http://localhost:8000/api/v1/node/search/bdp2022?page[limit]=0&filter[fulltext][condition][text]=M9 demo&filter[fulltext][condition][case_sensitive]=true&sort=time'\n",
      "Search of catalog='bdp2022' has 375 runs.\n",
      "First run: started='2022-07-15T23:14:54.974411' scan_id=1 plan_name='push_images'\n",
      "Last  run: started='2022-11-23T11:17:32.495794' scan_id=2035 plan_name='m9_push_images'\n"
     ]
    }
   ],
   "source": [
    "search_text = \"M9 demo\"\n",
    "case_sensitive = True\n",
    "uri = (\n",
    "    f\"http://{server}:{port}\"\n",
    "    \"/api/v1/node/search\"\n",
    "    f\"/{catalog}\"\n",
    "    \"?page[limit]=0\"  # 0: all matching\n",
    "    f\"&filter[fulltext][condition][text]={search_text}\"\n",
    "    f\"&filter[fulltext][condition][case_sensitive]={str(case_sensitive).lower()}\"\n",
    "    \"&sort=time\"\n",
    ")\n",
    "\n",
    "print(f\"{uri=}\")\n",
    "r = requests.get(uri).json()\n",
    "\n",
    "print(f'Search of {catalog=} has {len(r[\"data\"])} runs.')\n",
    "print_results_summary(r)"
   ]
  },
  {
   "source": [
    "### Show a run's metadata\n",
    "\n",
    "Let's show the various metadata available from a Bluesky *run*.  We'll use the last run from the previous search.\n",
    "\n",
    "```\n",
    "http://localhost:8000/api/v1/node/metadata/bdp2022/a1233634-1259-438f-b9f0-f77c26f48f54\n",
    "```"
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "run = r[\"data\"][-1]  # most recent run from previous results"
   ]
  },
  {
   "source": [
    "The `run` object is a dictionary.  The interesting keys are:\n",
    "\n",
    "key | content\n",
    ":--- | :---\n",
    "`id` | `uid` universal identifier of this `run` (used by the database)\n",
    "`attributes` | contents of this `run`\n",
    "\n",
    "The `attributes` contents are a dictionary with these interesting keys (there are other keys, as well):\n",
    "\n",
    "key | content\n",
    ":--- | :---\n",
    "`metadata` | metadata dictionary of this `run`\n",
    "\n",
    "The `metadata` dictionary has these keys:\n",
    "\n",
    "key | content\n",
    ":--- | :---\n",
    "`start` | Metadata created as the run started (includes user-supplied, scan-specific, facility-specific, and bluesky metadata).  The `start` dictionary keys will vary between runs and catalogs.  Only a few are expected, including: `uid`, `time`, & `versions`.\n",
    "`stop` | Metadata about how the run ended (exit status and reason if problem, stream names, end time stamp)\n",
    "`summary` | tiled server provides this additional high-level summary with ISO8601 start date and run duration\n",
    "\n",
    "Note: the run's data streams are obtained by a different query, using the run's `uid`.  Keep track of the `uid` for that reason.\n",
    "\n",
    "To show the structure of this dictionary, we just access Python to show the object's value."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "{'start': {'uid': 'a1233634-1259-438f-b9f0-f77c26f48f54',\n",
       "  'time': 1669223852.495794,\n",
       "  'versions': {'apstools': '1.6.9.dev89+gbf712d9',\n",
       "   'bluesky': '1.10.0',\n",
       "   'bluesky_queueserver': '0.0.18',\n",
       "   'databroker': '1.2.5',\n",
       "   'epics': '3.5.0',\n",
       "   'h5py': '3.7.0',\n",
       "   'matplotlib': '3.6.2',\n",
       "   'numpy': '1.23.4',\n",
       "   'ophyd': '1.7.0',\n",
       "   'pyRestTable': '2020.0.6',\n",
       "   'spec2nexus': '2021.2.4'},\n",
       "  'databroker_catalog': 'bdp2022',\n",
       "  'login_id': 'bdp@mona3.xray.aps.anl.gov',\n",
       "  'beamline_id': 'BDP',\n",
       "  'instrument_name': 'APS-U Beamline Data Pipelines project in 2022',\n",
       "  'proposal_id': 'bdp2022',\n",
       "  'milestone': 'BDP M9 demo',\n",
       "  'pid': 61262,\n",
       "  'scan_id': 2035,\n",
       "  'plan_type': 'generator',\n",
       "  'plan_name': 'm9_push_images',\n",
       "  'purpose': 'publish image frames via PVaccess',\n",
       "  'num_images': 12000,\n",
       "  'frame_rate': 1000.0,\n",
       "  'datetime': '2022-11-23 11:17:32.480612',\n",
       "  'client': '/clhome/BDP/DM/workflows/example-06/qserver_client.py',\n",
       "  'session': 'M9 demo'},\n",
       " 'stop': {'run_start': 'a1233634-1259-438f-b9f0-f77c26f48f54',\n",
       "  'time': 1669223872.8862715,\n",
       "  'uid': 'b7b51924-624f-405f-be3a-557f7fa56432',\n",
       "  'exit_status': 'success',\n",
       "  'reason': '',\n",
       "  'num_events': {'primary': 1}},\n",
       " 'summary': {'uid': 'a1233634-1259-438f-b9f0-f77c26f48f54',\n",
       "  'scan_id': 2035,\n",
       "  'timestamp': 1669223852.495794,\n",
       "  'datetime': '2022-11-23T11:17:32.495794',\n",
       "  'plan_name': 'm9_push_images',\n",
       "  'stream_names': ['primary'],\n",
       "  'duration': 20.390477418899536}}"
      ]
     },
     "metadata": {},
     "execution_count": 13
    }
   ],
   "source": [
    "run[\"attributes\"][\"metadata\"]"
   ]
  },
  {
   "source": [
    "### What data streams are available with this run?\n",
    "\n",
    "Use the last `take_image` run.  The stream names are in the `stop` metadata, where the number of data events is shown for each stream.\n",
    "\n",
    "```json\n",
    "                \"stop\": {\n",
    "                    \"exit_status\": \"success\",\n",
    "                    \"num_events\": {\n",
    "                        \"primary\": 1\n",
    "                    },\n",
    "                    \"reason\": \"\",\n",
    "                    \"run_start\": \"a4edf4b3-8a12-4724-b817-fd45958488da\",\n",
    "                    \"time\": 1661889999.67191,\n",
    "                    \"uid\": \"0cb0e643-877f-41b3-96b2-7adf75bef657\"\n",
    "                },\n",
    "```"
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "uri='http://localhost:8000/api/v1/node/search/bdp2022?page[limit]=0&filter[eq][condition][key]=plan_name&filter[eq][condition][value]=\"take_image\"&sort=time'\n",
      "Run uid='a4edf4b3-8a12-4724-b817-fd45958488da' has 1 streams: streams=['primary']\n"
     ]
    }
   ],
   "source": [
    "uri = (\n",
    "    f\"http://{server}:{port}\"\n",
    "    \"/api/v1/node/search\"\n",
    "    f\"/{catalog}\"\n",
    "    \"?page[limit]=0\"  # 0: all matching\n",
    "    \"&filter[eq][condition][key]=plan_name\"\n",
    "    '&filter[eq][condition][value]=\"take_image\"'\n",
    "    \"&sort=time\"\n",
    ")\n",
    "print(f\"{uri=}\")\n",
    "r = requests.get(uri).json()\n",
    "\n",
    "stop_md = r[\"data\"][-1][\"attributes\"][\"metadata\"][\"stop\"]\n",
    "streams = list(stop_md[\"num_events\"].keys())\n",
    "uid = stop_md[\"run_start\"]\n",
    "print(f'Run {uid=} has {len(streams)} streams: {streams=}')"
   ]
  },
  {
   "source": [
    "### What is the metadata for the `primary` stream of this run?\n",
    "\n",
    "The stream metadata has information about each of the signals acquired in the stream.  PV names, units, limits, data type and shape, ... and the timestamps when each of these nuggets was received."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "uri='http://localhost:8000/api/v1/node/metadata/bdp2022/a4edf4b3-8a12-4724-b817-fd45958488da/primary'\ncatalog='bdp2022' uid='a4edf4b3-8a12-4724-b817-fd45958488da' metadata for stream_name='primary':\n"
     ]
    },
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "{'descriptors': [{'run_start': 'a4edf4b3-8a12-4724-b817-fd45958488da',\n",
       "   'time': 1661889999.655602,\n",
       "   'data_keys': {'adsimdet_image': {'shape': [1, 1024, 1024],\n",
       "     'source': 'PV:bdpSimExample:',\n",
       "     'dtype': 'array',\n",
       "     'external': 'FILESTORE:',\n",
       "     'object_name': 'adsimdet'}},\n",
       "   'uid': 'f78f5d29-36dc-49e2-9e9a-8eee6120873b',\n",
       "   'configuration': {'adsimdet': {'data': {'adsimdet_cam_acquire_period': 0.251,\n",
       "      'adsimdet_cam_acquire_time': 0.25,\n",
       "      'adsimdet_cam_image_mode': 0,\n",
       "      'adsimdet_cam_manufacturer': 'Simulated detector',\n",
       "      'adsimdet_cam_model': 'Basic simulator',\n",
       "      'adsimdet_cam_num_exposures': 1,\n",
       "      'adsimdet_cam_num_images': 1,\n",
       "      'adsimdet_cam_trigger_mode': 0},\n",
       "     'timestamps': {'adsimdet_cam_acquire_period': 1661889997.531498,\n",
       "      'adsimdet_cam_acquire_time': 1661889997.520722,\n",
       "      'adsimdet_cam_image_mode': 1661889997.5001307,\n",
       "      'adsimdet_cam_manufacturer': 1661889153.029876,\n",
       "      'adsimdet_cam_model': 1661889153.029901,\n",
       "      'adsimdet_cam_num_exposures': 1661889413.3754888,\n",
       "      'adsimdet_cam_num_images': 1661889997.5106463,\n",
       "      'adsimdet_cam_trigger_mode': 1661889177.3299406},\n",
       "     'data_keys': {'adsimdet_cam_acquire_period': {'source': 'PV:bdpSimExample:cam1:AcquirePeriod_RBV',\n",
       "       'dtype': 'number',\n",
       "       'shape': [],\n",
       "       'units': '',\n",
       "       'lower_ctrl_limit': 0.0,\n",
       "       'upper_ctrl_limit': 0.0,\n",
       "       'precision': 3},\n",
       "      'adsimdet_cam_acquire_time': {'source': 'PV:bdpSimExample:cam1:AcquireTime_RBV',\n",
       "       'dtype': 'number',\n",
       "       'shape': [],\n",
       "       'units': '',\n",
       "       'lower_ctrl_limit': 0.0,\n",
       "       'upper_ctrl_limit': 0.0,\n",
       "       'precision': 3},\n",
       "      'adsimdet_cam_image_mode': {'source': 'PV:bdpSimExample:cam1:ImageMode_RBV',\n",
       "       'dtype': 'integer',\n",
       "       'shape': [],\n",
       "       'units': None,\n",
       "       'lower_ctrl_limit': None,\n",
       "       'upper_ctrl_limit': None,\n",
       "       'enum_strs': ['Single', 'Multiple', 'Continuous']},\n",
       "      'adsimdet_cam_manufacturer': {'source': 'PV:bdpSimExample:cam1:Manufacturer_RBV',\n",
       "       'dtype': 'string',\n",
       "       'shape': [],\n",
       "       'units': None,\n",
       "       'lower_ctrl_limit': None,\n",
       "       'upper_ctrl_limit': None},\n",
       "      'adsimdet_cam_model': {'source': 'PV:bdpSimExample:cam1:Model_RBV',\n",
       "       'dtype': 'string',\n",
       "       'shape': [],\n",
       "       'units': None,\n",
       "       'lower_ctrl_limit': None,\n",
       "       'upper_ctrl_limit': None},\n",
       "      'adsimdet_cam_num_exposures': {'source': 'PV:bdpSimExample:cam1:NumExposures_RBV',\n",
       "       'dtype': 'integer',\n",
       "       'shape': [],\n",
       "       'units': '',\n",
       "       'lower_ctrl_limit': 0,\n",
       "       'upper_ctrl_limit': 0},\n",
       "      'adsimdet_cam_num_images': {'source': 'PV:bdpSimExample:cam1:NumImages_RBV',\n",
       "       'dtype': 'integer',\n",
       "       'shape': [],\n",
       "       'units': '',\n",
       "       'lower_ctrl_limit': 0,\n",
       "       'upper_ctrl_limit': 0},\n",
       "      'adsimdet_cam_trigger_mode': {'source': 'PV:bdpSimExample:cam1:TriggerMode_RBV',\n",
       "       'dtype': 'integer',\n",
       "       'shape': [],\n",
       "       'units': None,\n",
       "       'lower_ctrl_limit': None,\n",
       "       'upper_ctrl_limit': None,\n",
       "       'enum_strs': ['Internal', 'External']}}}},\n",
       "   'name': 'primary',\n",
       "   'hints': {'adsimdet': {'fields': []}},\n",
       "   'object_keys': {'adsimdet': ['adsimdet_image']}}],\n",
       " 'stream_name': 'primary'}"
      ]
     },
     "metadata": {},
     "execution_count": 15
    }
   ],
   "source": [
    "stream_name = \"primary\"\n",
    "uri = (\n",
    "    f\"http://{server}:{port}\"\n",
    "    \"/api/v1/node/metadata\"\n",
    "    f\"/{catalog}\"\n",
    "    f\"/{uid}\"\n",
    "    f\"/{stream_name}\"\n",
    ")\n",
    "print(f\"{uri=}\")\n",
    "r = requests.get(uri).json()\n",
    "print(f\"{catalog=} {uid=} metadata for {stream_name=}:\")\n",
    "r['data']['attributes']['metadata']"
   ]
  },
  {
   "source": [
    "### Is there any information about external files?\n",
    "\n",
    "For data that is deemed too *large* for storage in MongoDB documents, such as area detector image frames which may be several MB in size and a milllion pixels, that content is stored in files.  In such cases, the RunEngine records references to the file, its path, and the file format and frame information within the file.\n",
    "\n",
    "Here is an example from our `take_image()` plan which records a single image frame to an HDF5 file.  The `'spec': 'AD_HDF5'` is a reference to a [handler](https://github.com/bluesky/area-detector-handlers/blob/c3db30dd1bee8c9cc26b369c66963f1bd6d36802/area_detector_handlers/handlers.py#L162-L183) which expects the image frame to be in this HDF5 file at hard-coded address `/entry/data/data`."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "uri='http://localhost:8000/api/v1/documents/bdp2022/43044b6e-f6ba-48cb-a975-90d236dcbaaa?fill=false'\n"
     ]
    },
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "{'resource': {'spec': 'AD_HDF5',\n",
       "  'root': '/',\n",
       "  'resource_path': 'clhome/BDP/voyager/adsimdet/2022/08/30/d585f272-dd9b-4ac0-b521_000.h5',\n",
       "  'resource_kwargs': {'frame_per_point': 1},\n",
       "  'path_semantics': 'posix',\n",
       "  'uid': '506944a6-7632-4db8-9448-82b258211ed4',\n",
       "  'run_start': '43044b6e-f6ba-48cb-a975-90d236dcbaaa'},\n",
       " 'datum_page': {'resource': '506944a6-7632-4db8-9448-82b258211ed4',\n",
       "  'datum_id': ['506944a6-7632-4db8-9448-82b258211ed4/0'],\n",
       "  'datum_kwargs': {'point_number': [0]}}}"
      ]
     },
     "metadata": {},
     "execution_count": 43
    }
   ],
   "source": [
    "uid = \"43044b6e-f6ba-48cb-a975-90d236dcbaaa\"\n",
    "uri = (\n",
    "    f\"http://{server}:{port}\"\n",
    "    \"/api/v1/documents\"\n",
    "    f\"/{catalog}\"\n",
    "    f\"/{uid}\"\n",
    "    \"?fill=false\"\n",
    ")\n",
    "print(f\"{uri=}\")\n",
    "\n",
    "# special handling here to overcome a problem with this API request\n",
    "import json\n",
    "r = requests.get(uri)  # note no json() call here, it's a problem in the server's response\n",
    "r = f\"[{r.text.strip()}]\".replace(\"\\n\", \",\")\n",
    "r = json.loads(r)\n",
    "\n",
    "{k:doc for k, doc in r if k in (\"datum_page\", \"resource\")}"
   ]
  },
  {
   "source": [
    "### Get the data from the data stream named primary (the canonical main data).\n",
    "\n",
    "To get the data, we need to change the type of search using `/api/v1/node/full` (so far, we have been using the default search for metadata: `/api/v1/node/search`) and specify the format of the result.  One format is `json`.  Let's pick some data from the M6 demo (with `M6-gallery` matching text) that has a very long list (1-D) of floating point numbers."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "uri='http://localhost:8000/api/v1/node/search/bdp2022?page[limit]=0&filter[fulltext][condition][text]=M6-gallery&filter[fulltext][condition][case_sensitive]=true&sort=time'\n"
     ]
    }
   ],
   "source": [
    "search_text = \"M6-gallery\"\n",
    "case_sensitive = True\n",
    "uri = (\n",
    "    f\"http://{server}:{port}\"\n",
    "    \"/api/v1/node/search\"\n",
    "    f\"/{catalog}\"\n",
    "    \"?page[limit]=0\"  # 0: all matching\n",
    "    f\"&filter[fulltext][condition][text]={search_text}\"\n",
    "    f\"&filter[fulltext][condition][case_sensitive]={str(case_sensitive).lower()}\"\n",
    "    \"&sort=time\"\n",
    ")\n",
    "print(f\"{uri=}\")\n",
    "r = requests.get(uri).json()"
   ]
  },
  {
   "source": [
    "Next, pick the last run (`r[\"data\"][-1]`):"
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "uri='http://localhost:8000/api/v1/array/full/bdp2022/ae762f9c-4933-4aa4-a720-147f4aaab6fd/primary/data/adpvadet_pva1_execution_time?format=json'\n"
     ]
    }
   ],
   "source": [
    "uid = r[\"data\"][-1][\"attributes\"][\"metadata\"][\"start\"][\"uid\"]\n",
    "data_format = \"json\"\n",
    "stream_name = \"primary\"\n",
    "data_name = \"adpvadet_pva1_execution_time\"\n",
    "\n",
    "uri = (\n",
    "    f\"http://{server}:{port}\"\n",
    "    \"/api/v1/array/full\"\n",
    "    f\"/{catalog}\"\n",
    "    f\"/{uid}\"\n",
    "    f\"/{stream_name}\"\n",
    "    \"/data\"\n",
    "    f\"/{data_name}\"\n",
    "    f\"?format={data_format}\"\n",
    ")\n",
    "print(f\"{uri=}\")\n",
    "arr = requests.get(uri).json()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "len(arr)=10261 min(arr)=0.213429 max(arr)=2.024461\n"
     ]
    }
   ],
   "source": [
    "print(f\"{len(arr)=} {min(arr)=} {max(arr)=}\")"
   ]
  },
  {
   "source": [
    "### Get data from a file served by *tiled*.\n",
    "\n",
    "This file is in the catalog named (in our tiled server) as `directory`.  It happens to be a NeXus (HDF5) data file from the [NeXus documentation](https://manual.nexusformat.org/examples/python/index.html#simple-example-plot).\n",
    "\n",
    "First find the signal names.  Since this is not a `BlueskyRun`, the results dictionary is a bit different.  Instead of a `data` key (as above), the information we seek is in the `contents` key."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "uri='http://localhost:8000/api/v1/node/full/directory/hdf5/writer_1_3.h5/Scan/data?format=json'\n"
     ]
    },
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "{'counts': {'contents': {},\n",
       "  'metadata': {'axes': 'two_theta', 'signal': '1', 'units': 'counts'}},\n",
       " 'two_theta': {'contents': {}, 'metadata': {'units': 'degrees'}}}"
      ]
     },
     "metadata": {},
     "execution_count": 20
    }
   ],
   "source": [
    "uri = \"http://localhost:8000/api/v1/node/full/directory/hdf5/writer_1_3.h5/Scan/data?format=json\"\n",
    "print(f\"{uri=}\")\n",
    "r = requests.get(uri).json()\n",
    "r[\"contents\"]"
   ]
  },
  {
   "source": [
    "Get the `counts` array."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "uri='http://localhost:8000/api/v1/array/full/directory/hdf5/writer_1_3.h5/Scan/data/counts?format=application/json&slice=0:31'\n"
     ]
    }
   ],
   "source": [
    "uri = \"http://localhost:8000/api/v1/array/full/directory/hdf5/writer_1_3.h5/Scan/data/counts?format=application/json&slice=0:31\"\n",
    "print(f\"{uri=}\")\n",
    "counts = requests.get(uri).json()"
   ]
  },
  {
   "source": [
    "Get the `two_theta` array."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "uri='http://localhost:8000/api/v1/array/full/directory/hdf5/writer_1_3.h5/Scan/data/two_theta?format=application/json&slice=0:31'\n"
     ]
    }
   ],
   "source": [
    "uri = \"http://localhost:8000/api/v1/array/full/directory/hdf5/writer_1_3.h5/Scan/data/two_theta?format=application/json&slice=0:31\"\n",
    "print(f\"{uri=}\")\n",
    "tth = requests.get(uri).json()"
   ]
  },
  {
   "source": [
    "Print the data in two columns."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "source": [
    "print(\"two_theta  counts\")\n",
    "for x, y in zip(tth, counts):\n",
    "    print(f\"{x}  {y}\")"
   ],
   "cell_type": "code",
   "metadata": {},
   "execution_count": 23,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "two_theta  counts\n17.92608  1037\n17.92591  1318\n17.92575  1704\n17.92558  2857\n17.92541  4516\n17.92525  9998\n17.92508  23819\n17.92491  31662\n17.92475  40458\n17.92458  49087\n17.92441  56514\n17.92425  63499\n17.92408  66802\n17.92391  66863\n17.92375  66599\n17.92358  66206\n17.92341  65747\n17.92325  65250\n17.92308  64129\n17.92291  63044\n17.92275  60796\n17.92258  56795\n17.92241  51550\n17.92225  43710\n17.92208  29315\n17.92191  19782\n17.92175  12992\n17.92158  6622\n17.92141  4198\n17.92125  2248\n17.92108  1321\n"
     ]
    }
   ]
  },
  {
   "source": [
    "## `tiled.client` Client"
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tiled.client import from_uri\n",
    "from tiled.client.cache import Cache\n",
    "import tiled.queries\n",
    "from tiled.utils import tree"
   ]
  },
  {
   "source": [
    "TODO\n",
    "\n",
    "Until then, look at the [Python code file](./pyapi_client.py), used for development."
   ],
   "cell_type": "markdown",
   "metadata": {}
  }
 ]
}