Improvements to forwarding

[timcowlishaw]

TODO: flesh this ticket out more

• Decouple generation of readings JSON from rendering the current state of the device model, so we can render each reading as it comes it directly (rather than rendering the latest state of the device).

[timcowlishaw]

Ok, I think as part of this i'd like to make a few small changes to the way we represent things, which should make the responses clearer and more consistent (and make the code for generating responses much simpler and more robust, and therefore more easily reused and adapted to things like forwarding):

The idea is we'd have fixed representations of all of our data types which we use everywhere (so a device on a user looks the same as a device on the index page, as a device on the show page, ditto for sensors, measurements and users) - with one specific exception which i'll get to in a bit.

The device json would look like the following (which also implicitly provides examples of a user, a sensor, a measurement and a reading on the nested objects):

{
  "id": 16080,
  "uuid": "0974aad9-07b1-4c8f-b095-eb3f2373066c",
  "name": "MINKE_demo #",
  "description": "",
  "state": "has_published",
  "system_tags": [
    "indoor",
    "offline",
    "test_device"
  ],
  "user_tags": [
    "Research",
    "Experimental",
    "Lab"
  ],
  "last_reading_at": "2023-05-11T09:01:17Z",
  "created_at": "2023-02-21T17:27:58Z",
  "updated_at": "2024-06-27T17:03:24Z",
  "notify": {
    "stopped_publishing": false,
    "low_battery": false
  },
  "device_token": null,
  "mac_address": null,
  "postprocessing": null,
  "location": {
    "exposure": "indoor",
    "elevation": null,
    "latitude": 41.39685,
    "longitude": 2.19435,
    "geohash": "sp3e9bjfhh",
    "city": "Barcelona",
    "country_code": "ES",
    "country": "Spain"
  },
  "data_policy": null,
  "hardware": {
    "name": "SCK 2.1 Sea Water",
    "type": "SCK",
    "version": "2.1",
    "slug": "sck:2,1:seawater",
    "last_status_message": null
  },
  "owner": {
    "id": 6556,
    "uuid": "76affe1f-a1b8-4a47-bc0b-07bb8f9c700f",
    "role": "admin",
    "username": "team",
    "url": "http://pral2a.com",
    "profile_picture": "https://api.smartcitizen.me/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBalFDIiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--9d0516494e820f91076abc46728daab4d7c2ede3/1djs79t.earth.jpg",
    "location": {
      "city": "Barcelona",
      "country": "Spain",
      "country_code": "ES"
    },
    "email": null,
    "legacy_api_key": null,
    "created_at":"2023-02-21T17:27:58Z",
    "updated_at":,"2023-02-21T17:27:58Z"
  },
  "components": [
      {
        "key": "atlas_do",
        "latest_value": 7.3,
        "prev_value": 7.3,
        "last_reading_at": "2023-05-11T09:01:17Z",
        "sensor": {
          "id": 48,
          "parent_id": 47,
          "name": "AS  EZO Dissolved Oxygen",
          "description": "Atlas Scientific EZO™ Dissolved Oxygen",
          "unit": "mg/L",
          "created_at": "2017-08-04T12:05:30Z",
          "updated_at": "2017-08-04T12:05:30Z",
          "uuid": "dd7baf73-705f-4ac0-a119-c75fedd3ddcf",
          "default_key": "atlas_do",
          "datasheet": null,
          "unit_definition": null,
          "measurement": {
            "id": 21,
            "name": "Dissolved Oxygen",
            "description": "Absolute measure of the concentration of oxygen that is dissolved in Water",
            "unit": null,
            "uuid": "86e4fe06-64fd-49af-9534-5d5cf8763ec6",
            "definition": null
          },
          "tags": []
        },
        "readings": [
          {
            "timestamp": "2023-05-11T09:01:17Z",
            "value": 7.3
          }
        ]
      },
  ],
  "unauthorized_fields": [
    "device_token",
    "mac_address",
    "data_policy",
    "hardware.last_status_message",
    "owner.email",
    "owner.legacy_api_key"
  ]
}

The key changes are as follows:

• Device stays mostly the same, except we no long leave out the location when displayed on user pages (not really worth doing for privacy if you can just go to the device page to see it)
• The user, when displayed nested as the owner of a device, does not list the other devices.
• In turn, the device, when displayed in a list of the devices owned by a user, would not include either its owner or the list of components. (The components could also be optional, for instance, on the device index page).
• The data property disappears, and is replaced by a list of components (much more descriptive of what's actually happening there). Each one has a sensor property as well as the existing last_reading_at, latest_value, previous_value (formerly on the sensor itself), and the component key which i think is only missing currently as an oversight.
• Sensors and measurements stay the same, apart from the fact that the sensor consistently returns a parent_id instead of the ancestryproperty (we use a mixture currently in different places: its exactly the same value, except expressed as an int instead of a string)
• In forwarding, each component can optionally have a list of readings, each of which has a timestamp and a value.
• The user object is updated to remove the redundant user_id property (a dupe of id) and the profile_picture2 which is a bug, and to add the created_at time (which seems to be a bit of an omission.
• Any fields which are not shown as the user is not authorized are present with the value null, and the root element of the response contains an unauthorized_fields list which lists these keys, so the user knows they can re-request, passing credentials, to access these.
• The location ip field (Which, bizarrely, is hard-coded to return null, always) disappears.

We use these representations everywhere we represent a thing as JSON, apart from on the world map, where, due to the volume of devices, it's important to be as concise as possible. The `world_map' representation therefore becomes something like this, which contains only the info needed to render the map:

{
  "id": 16080,
  "name": "MINKE_demo #",
  "description": "",
  "state": "has_published",
  "system_tags": [
    "indoor",
    "offline",
    "test_device"
  ],
  "user_tags": [
    "Research",
    "Experimental",
    "Lab"
  ],
  "last_reading_at": "2023-05-11T09:01:17Z",
  "location": {
    "exposure": "indoor",
    "elevation": null,
    "latitude": 41.39685,
    "longitude": 2.19435,
    "geohash": "sp3e9bjfhh",
    "city": "Barcelona",
    "country_code": "ES",
    "country": "Spain"
  },
  "hardware": {
    "name": "SCK 2.1 Sea Water",
    "type": "SCK",
    "version": "2.1",
    "slug": "sck:2,1:seawater",
  },
  "owner": {
    "id": 6556,
    "username": "team",
    "url": "http://pral2a.com",
  },
}

This will be the same for authorized and unauthorized users, as there's no privileged information in there.

[timcowlishaw]

@oscgonfer what do you think? there's a couple of small user-facing changes there but i think overall the responses would be much clearer.