engine, api: protype new check-in API

[bassosimone]

We discussed with @FedericoCeratto improving the check-in API. We are going to create a prototype new API in the private namespace (/api/_/check-in) as an experiment to support developing the probe alongside it.

We determined we need a new check-in API for the following reasons:

1. we want to support richer input inside the check-in API;
2. we want to serve OONI Run work directly through the check-in API to cut round trips;
3. we want to reduce OONI Probe code complexity.

The new check-in API will now return the following response:

{
  "conf": {
    "features": {
      "webconnectivity_0.5": true
    },  
    "test_helpers": {}
  },
  "probe_asn": "AS30722",
  "probe_cc": "IT",
  "probe_network_name": "Vodafone Italia S.p.A.",
  "nettests": [
    {   
      "report_id": "20230329T092643Z_webconnectivity_IT_30722_n1_cOJE8Hl0l7RcJqcJ",
      "targets": [
        {
          "attributes": {
            "category_code": "GRP",
            "country_code": "XX"
          },
          "options": {},
          "input": "https://www.facebook.com/"
        }
      ],
      "test_name": "web_connectivity"
    }
  ],
  "utc_time": "2023-03-29T09:26:43Z",
  "v": 1
}

The two main changes between this new experimental data format and the previous format are:

1. that we input options specific to each target (to support richer input);
2. that nettests is now a list of nettests to run as opposed to being just a dictionary (because the dictionary based format is possibly too rigid and binding, while the list seems more flexible in light of future, unforeseen changes).

The data format with which we describe each nettest is the same we want to have for a OONI Run v2 version that supports running experiments along with richer input. In Go the new data format looks like:

type CheckInV2NettestTarget struct {
  Attributes map[string]any // or json.RawMessage
  Options map[string]any // or json.RawMessage
  Input string
}

type CheckInV2NettestDescriptor struct {
  ReportID string
  Targets []CheckInV2NettestTarget
  TestName string
}

The reason why we're using such a data format for nettests is that it describes a superset of the operations one could run from the command line using miniooni (the map[string]any type used for options allows us to potentially include any type we could include for JSON, while using json.RawMessage directly allows to use any data format used by JSON by it seems we need additional refactoring to use such a type directly).

In addition to changing the data format, we would also need to (1) ensure we generate a ReportID for each possible experiment and (2) include proper input for any experiment requiring it. The former change ensures we can always use the check-in API to obtain a report ID and we never need to call the "open report" API again. The latter change ensures that we do not need to vendor experiment specific input inside of experiments.

While at it, it may also be a good place to clarify what the experiment input is: URL or string? It seems it's a URL in most cases, and we need to continue the conversation on this topic and collect more data.

This issue is part of ooni/ooni.org#1295 and ooni/ooni.org#1291.

The following list summarizes the changes we need to implement:

• check whether we should enforce input being a URL for experiments [answer: no];
• expose /api/_/check-in with the desired data format;
• include a ReportID for each possible experiment;
• make sure we include default input for each experiment needing it.

[bassosimone]

As explained in ooni/2023-05-richer-input@7871d3b, the https://github.com/ooni/2023-05-richer-input contains a prototype that explores the richer input domain and redesigns ooniprobe around richer input. Notably, that repository sketches out what the check-in v2 with richer input response should look like.

[jbonisteel]

There is an updated plan for implementing richer input here: ooni/ooni.org#1295 (comment)

Assuming we move forward with this plan, this issue is likely out of scope for DRL. Once we agree on the plan, we will decide what to do with this issue.

[bassosimone]

@jbonisteel today we need to discuss the fate of this issue!

[bassosimone]

So, our updated plan for richer input does not need to modify the check-in API at this time. Therefore, we can close this issue as out of scope. We may want to change the check-in API later if we hit some bottleneck. In such a case, we will create a new issue describing the scope of work.