Create/ modify Azure event for more accurate OKR Metrics Dashboard

[jsutantio]

As a product manager and user of the OKR Metrics Dashboard, I want the dashboard to convey the most accurate information so that I can be confident in what is being conveyed and make decisions off of it.

Description/Use Case

Currently the OKR Metrics Dashboard conveys the volume of results designated to be routed (based on the qty of test performed instances) to receivers of ReportStream. The Azure app event that contains this information is ITEM_ROUTED; however, this does not take into account actions to occur after this event is logged (e.g., filtering out conditions, filtering out pos/ negs, resends, items that get stuck in poison queue or last mile) – which ultimately leads to an inaccurate representation of the quantity of results that get sent to the receiver.
The request is to create or modify an event to contain the metadata necessary for the OKR Metrics Dashboard, but can more accurately capture the delivered/ sent status of the result to the receiver.

Additional context of the issue:

On Aug 16, 2024, the OKR Metrics Dashboard displayed that 18272 test performed observations were routed to IL PHD. Although this is accurate in terms of how many messages were passed through the "route" step, this is not closely accurate in conveying what was received by the receiver. What happened was that on "8/12/2024, it was 108 messages sent to IL without PID-3-1. [And engagement engineer] requested Simple Report to ask [them] to resubmit messages that missing PID-3-1. At 8/17/2023, it was 100 messages sent to IL without PID-3-1. I corrected those messages and resubmitted to their production folder and verified that they all there... [in addition, there were] Negative results. IL only wants Positive results to route to them. Therefore, those messages have been filtered out by conditionFilter."
This means that using the ITEM_ROUTED is insufficient for capturing the volume of results received by the STLT, as it does not take into about receiver's filter settings and any resends.

Risks/Impacts/Considerations

Evaluate how this may impact current and future observability/ monitoring for the engagement team/ RS admins.
Ensure that the following metadata continues to be captured:

• child reportID and/ or tracking ID
• observation containing the tests performed (code system, code [typically LOINC], display name)
• info that enables identification and filtering of AOEs (typically the LOINC display name (mentioned above) works
• date/time
• sender
• receiver
• topic
• It would be nice to have all the metadata that is currently provided within the ITEM_ROUTED event

Dev Notes

• Consider leveraging the last logged event in the pipeline (send)
• Because this impacts how the OKR Metrics Dashboard will import data, please denote the exact time that changes in prod will go into effect. This will help with switching the data import settings.
• Reports in send step may contain thousands of items so a single event wont be able to hold all the condition data. One option is to create a "ItemSentEvent" for each ITEM in the report.
  • Another option is the ItemSentEvent can contain the trackingId and SubmittedReportId which Jessica S can use to link to an event that contains the condition info

Acceptance Criteria

• Create/ modify azure event, that captures the sent/ delivery state of the report/ result, to contain the necessary info for the OKR Metrics Dashboard
• Notify @jsutantio exact time when changes go into effect in prod

[Andrey-Glazkv]

Hey team! Please add your planning poker estimate with Zenhub @adegolier @arnejduranovic @brick-green @david-navapbc @jack-h-wang @jalbinson @JFisk42 @mkalish @thetaurean