feat: minipipeline to analyze measurements #1393
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
We're introducing failure modes that do not exist hence it seems this is not the correct way of moving forward.
I'm doing this mainly to explore whether we could have more robust webconnectivity v0.5 analysis code
Because I am dropping the requests again, we break again the tests with the redirects. I could possibly fix it by putting requests back again but I am not super happy about doing this because that would cause the DSL to do some strange work and I'd honestly rather not do this.
what remains to be done now is to make sure we make green all the tests that are currently skipped we also need to account for differences between the two
then next step is to sort out this mess :-)
(I am thankful there's a ~comprehensive test suite.)
this happens because LTE sucessfully handshakes with the wrong address
bassosimone
changed the title
Murphy-OrangeMud
pushed a commit
to Murphy-OrangeMud/probe-cli
that referenced
this pull request
Feb 13, 2024
This diff introduces the `minipipeline` package and command. The general idea here is to analyze experiments results using an architecture that matches https://github.com/ooni/data as closely as possible. The advantage of using this architecture is that we can neatly divide the data analysis process into three phases: 1. ingestion converts an OONI measurement into observations, which are flat data structures; 2. analysis converts observations into an analysis structure, which is significantly simpler than observations; 3. scoring is the process (implemented by each experiment) to turn the analysis into top-level keys. That is: ```mermaid stateDiagram-v2 state "{ingestion}" as ingestion MEASUREMENT --> ingestion ingestion --> OBSERVATIONS state "{analysis}" as analysis OBSERVATIONS --> analysis analysis --> ANALYSIS state "{scoring}" as scoring ANALYSIS --> scoring scoring --> TOP_LEVEL_KEYS ``` This diff in particular tackles points 1 and 2. We'll implement 3 for Web Connectivity LTE in a separate diff. The diff itself commits several observations and analysis files along with the measurements from which they originate, which we all collectively use to write regression tests for the `minipipeline` package. The motivation for implementing these changes is that I need a more clearly defined model to address the differences between Web Connectivity LTE and v0.4 (see also ooni#1392). Because there is already an architectural model for implementing these changes in ooni/data, it seems logical to borrow liberally from ooni/data. Incidentally, this change has the benefit of making it easier, in the future, to align ooni/probe-cli and ooni/data. The reference issue is ooni/probe#2634. Implementing this change seems now a precondition for implementing extra changes that would address such an issue for good, by changing LTE's scoring. Unlike the ooni/data pipeline, this pipeline is minimal (hence the name): it only aims to process measurements collected by a recent version of ooniprobe, rather than all measurements from the beginning of time. While there, disable signal integration tests because of ooni/probe#2636.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This diff introduces the
minipipelinepackage and command. The general idea here is to analyze experiments results using an architecture that matches https://github.com/ooni/data as closely as possible.The advantage of using this architecture is that we can neatly divide the data analysis process into three phases:
That is:
stateDiagram-v2 state "{ingestion}" as ingestion MEASUREMENT --> ingestion ingestion --> OBSERVATIONS state "{analysis}" as analysis OBSERVATIONS --> analysis analysis --> ANALYSIS state "{scoring}" as scoring ANALYSIS --> scoring scoring --> TOP_LEVEL_KEYSThis diff in particular tackles points 1 and 2. We'll implement 3 for Web Connectivity LTE in a separate diff.
The diff itself commits several observations and analysis files along with the measurements from which they originate, which we all collectively use to write regression tests for the
minipipelinepackage.The motivation for implementing these changes is that I need a more clearly defined model to address the differences between Web Connectivity LTE and v0.4 (see also #1392). Because there is already an architectural model for implementing these changes in ooni/data, it seems logical to borrow liberally from ooni/data.
Incidentally, this change has the benefit of making it easier, in the future, to align ooni/probe-cli and ooni/data.
The reference issue is ooni/probe#2634. Implementing this change seems now a precondition for implementing extra changes that would address such an issue for good, by changing LTE's scoring.
Unlike the ooni/data pipeline, this pipeline is minimal (hence the name): it only aims to process measurements collected by a recent version of ooniprobe, rather than all measurements from the beginning of time.
While there, disable signal integration tests because of ooni/probe#2636.