Expand OONI’s testing model to support richer testing input

[agrabeli]

We aim to expand OONI’s testing model to support richer testing input and, by extension, enable all OONI Probe app users worldwide to more easily run novel experiments, automatically process these measurements and present findings on OONI Explorer as open data. This involves making improvements to the communication layer between the measurement coordination infrastructure and probes to improve our ability to provide a richer set of configuration parameters to network experiments that goes beyond just providing them with URLs to test.

Specifically, this involves making all the components of the OONI ecosystem (the backend API, OONI Probe clients, the data processing pipeline, the database model, and OONI Explorer) aware that the input for a measurement is now an object containing a URL, as well as other configuration parameters (for example, the backend would aggregate measurements based on a richer definition of input to score HTTP/2 measurements differently than HTTP/3 measurements).

To this end, this may entail:

• Changes to the backend API to serve richer testing input to probes;
• Changes to OONI Probe’s API to pass the rich input object to experiments;
• Changes to the data processing pipeline to use the richer input model to group and score measurements;
• Changes to the database model to use the richer definition of input for the indexing of measurements;
• Adding support to OONI Explorer for querying measurements with richer input and displaying the full input (instead of just measured URLs).

This work is a requirement for expanding our censorship measurement capabilities, and will enable us to ship a new methodology for measuring throttling. For example, if we suspect that access to https://twitter.com is being throttled in Russia, we could serve richer input (including a boolean flag) to OONI Probe clients in Russia to perform additional measurements that are useful for detecting throttling. In such a case, for example, the probe would fetch the whole page (as opposed to fetching a small part of the webpage) and would collect and submit download speed measurements to OONI’s backend.

[agrabeli]

I have set a tentative timeline on zenhub which probably needs to be adjusted.

[agrabeli]

I'm closing this ticket because it already exists: #1291