Implement draft-ietf-ppm-dap-07
#1669
Assignees
Comments
tgeoghegan
added a commit
that referenced
this issue
Aug 4, 2023
The bulk of the changes here deal with the change to the representation of `Prio3Histogram`. Since `prio` 0.14.x implements VDAF-06, taking this change will break compatibility with DAP-04. Part of #1669
Merged
tgeoghegan
added a commit
that referenced
this issue
Aug 4, 2023
The bulk of the changes here deal with the change to the representation of `Prio3Histogram`. Since `prio` 0.14.x implements VDAF-06, taking this change will break compatibility with DAP-04. Part of #1669
tgeoghegan
added a commit
that referenced
this issue
Aug 7, 2023
We are starting our DAP-05 implementation (#1669) and so must branch for DAP-04.
tgeoghegan
added a commit
that referenced
this issue
Aug 7, 2023
We are starting our DAP-05 implementation (#1669) and so must branch for DAP-04.
tgeoghegan
added a commit
that referenced
this issue
Aug 8, 2023
The bulk of the changes here deal with the change to the representation of `Prio3Histogram`. Since `prio` 0.14.x implements VDAF-06, taking this change will break compatibility with DAP-04. Part of #1669
tgeoghegan
added a commit
that referenced
this issue
Aug 14, 2023
The bulk of the changes here deal with the change to the representation of `Prio3Histogram`. Since `prio` 0.14.x implements VDAF-06, taking this change will break compatibility with DAP-04. Part of #1669
tgeoghegan
added a commit
that referenced
this issue
Aug 14, 2023
We are starting our DAP-05 implementation (#1669) and so must branch for DAP-04.
tgeoghegan
added a commit
that referenced
this issue
Aug 15, 2023
We are starting our DAP-05 implementation (#1669) and so must branch for DAP-04.
tgeoghegan
added a commit
that referenced
this issue
Aug 24, 2023
This commit takes changes to various messages so that we handle exactly one leader and exactly one helper, as specified in [DAP-05][1]. These changes are voluminous but mostly mechanical. This commit also adds variants for `Poplar1` to a couple of enums and some macros, because later changes will need it. [1]: https://datatracker.ietf.org/doc/draft-ietf-ppm-dap/05/ Part of #1669
tgeoghegan
added a commit
that referenced
this issue
Aug 24, 2023
This commit takes changes to various messages so that we handle exactly one leader and exactly one helper, as specified in [DAP-05][1]. These changes are voluminous but mostly mechanical. This commit also adds variants for `Poplar1` to a couple of enums and some macros, because later changes will need it. [1]: https://datatracker.ietf.org/doc/draft-ietf-ppm-dap/05/ Part of #1669
tgeoghegan
pushed a commit
that referenced
this issue
Aug 27, 2023
This change implements the DAP-05 ping-pong topology in which aggregators take turns preprocessing prepare shares into prepare messages. While this topology first appeared in DAP-05, this implementation follows the changes in [1], which should appear in DAP-06. This change depends on the implementation of the VDAF ping-pong topology added to crate `prio` in [2], which in turn conforms to the specification added after VDAF-06 and further tweaked in [3] (we expect this to be published soon as VDAF-07). This commit makes some changes to what intermediate values are stored by aggregators. In the case where an aggregator is continuing, it will have computed a prepare state, a prepare message for the current round and a prepare share for the next round. The existing implementation would store all three objects in the database, significantly increasing the per-report storage requirements. In particular, this makes things worse for the Helper, which previously never needed to store a prepare share because the Leader always took responsibility for combining prepare shares. To mitigate this, we instead have aggregators store a `prio::ping_pong::topology::Transition`, which will contain a prepare state and a prepare message (both of which are generally much smaller than prepare shares), from which the next prepare state and importantly prepare share can be recomputed. The main benefit of this change is to reduce how many round trips between aggregators are needed to prepare reports. Quite a few tests used Prio3 but depended on having the leader or helper in the `Waiting` state after running aggregation initialization. Accordingly, those tests are changed to run Poplar1, which now takes 2 rounds. [1]: ietf-wg-ppm/draft-ietf-ppm-dap#494 [2]: divviup/libprio-rs#683 [3]: cfrg/draft-irtf-cfrg-vdaf#281 Part of #1669
Merged
tgeoghegan
added a commit
that referenced
this issue
Aug 29, 2023
DAP-05 changes a few error types. `unrecongizedMessage -> invalidMessage` `queryMismatch` is gone (now we use `invalidMessage` in that case) `roundMismatch -> stepMismatch` We still use the word "round" in several places where "step" would be more appropriate given DAP-05 text. Renaming those variables, etc., will arrive in a later commit to avoid adding unnecessary noise here. Part of #1669
|
DAP-06 is out with a more clear and implementable ping-pong spec, so we'll jump ahead to implementing that. |
tgeoghegan
changed the title
draft-ietf-ppm-dap-05draft-ietf-ppm-dap-06
tgeoghegan
pushed a commit
that referenced
this issue
Aug 31, 2023
This change implements the DAP-05 ping-pong topology in which aggregators take turns preprocessing prepare shares into prepare messages. While this topology first appeared in DAP-05, this implementation follows the changes in [1], which should appear in DAP-06. This change depends on the implementation of the VDAF ping-pong topology added to crate `prio` in [2], which in turn conforms to the specification added after VDAF-06 and further tweaked in [3] (we expect this to be published soon as VDAF-07). This commit makes some changes to what intermediate values are stored by aggregators. In the case where an aggregator is continuing, it will have computed a prepare state, a prepare message for the current round and a prepare share for the next round. The existing implementation would store all three objects in the database, significantly increasing the per-report storage requirements. In particular, this makes things worse for the Helper, which previously never needed to store a prepare share because the Leader always took responsibility for combining prepare shares. To mitigate this, we instead have aggregators store a `prio::ping_pong::topology::Transition`, which will contain a prepare state and a prepare message (both of which are generally much smaller than prepare shares), from which the next prepare state and importantly prepare share can be recomputed. The main benefit of this change is to reduce how many round trips between aggregators are needed to prepare reports. Quite a few tests used Prio3 but depended on having the leader or helper in the `Waiting` state after running aggregation initialization. Accordingly, those tests are changed to run Poplar1, which now takes 2 rounds. [1]: ietf-wg-ppm/draft-ietf-ppm-dap#494 [2]: divviup/libprio-rs#683 [3]: cfrg/draft-irtf-cfrg-vdaf#281 Part of #1669
tgeoghegan
pushed a commit
that referenced
this issue
Sep 13, 2023
This change implements the DAP-05 ping-pong topology in which aggregators take turns preprocessing prepare shares into prepare messages. While this topology first appeared in DAP-05, this implementation follows DAP-06. This change depends on the implementation of the VDAF ping-pong topology added to crate `prio` in [1], which in turn conforms to the specification in VDAF-07. In the ping-pong topology, each DAP-layer step of aggregation now spans two VDAF rounds. An aggregator will use the prepare message it gets from its peer to advance by one VDAF round, and then can use the prepare share it just computed along with the peer's prepare share to advance by another. This incurs some changes to what intermediate values are stored by aggregators. In the case where a leader is continuing/waiting, it will have computed a prepare state, a prepare message for the current round and a prepare share for the next round. The naive implementation would store all three objects in the database, significantly increasing the per-report storage use. To mitigate this, the leader stores a `prio::topology::ping_pong::PingPongTransition`, which will contain a prepare state and a prepare message (both of which are generally much smaller than prepare shares), from which the next prepare state and importantly prepare share can be recomputed. On the helper side, there's no way around storing the prepare share: we store the most recently computed `PrepareResp` so that we can handle aggregation jobs idempotently. But to avoid storing prepare messages twice, the continuing/waiting helper stores just a prepare state and a `PingPongMessage`. The main benefit of this change is to reduce how many round trips between aggregators are needed to prepare reports. Quite a few tests used Prio3 but depended on having the leader or helper in the `Waiting` state after running aggregation initialization. Accordingly, those tests are changed to run Poplar1, which now takes 2 rounds. [1]: divviup/libprio-rs#683 Co-authored-by: Tim Geoghegan <[email protected]> Part of #1669
tgeoghegan
pushed a commit
that referenced
this issue
Sep 13, 2023
This change implements the DAP-05 ping-pong topology in which aggregators take turns preprocessing prepare shares into prepare messages. While this topology first appeared in DAP-05, this implementation follows DAP-06. This change depends on the implementation of the VDAF ping-pong topology added to crate `prio` in [1], which in turn conforms to the specification in VDAF-07. In the ping-pong topology, each DAP-layer step of aggregation now spans two VDAF rounds. An aggregator will use the prepare message it gets from its peer to advance by one VDAF round, and then can use the prepare share it just computed along with the peer's prepare share to advance by another. This incurs some changes to what intermediate values are stored by aggregators. In the case where a leader is continuing/waiting, it will have computed a prepare state, a prepare message for the current round and a prepare share for the next round. The naive implementation would store all three objects in the database, significantly increasing the per-report storage use. To mitigate this, the leader stores a `prio::topology::ping_pong::PingPongTransition`, which will contain a prepare state and a prepare message (both of which are generally much smaller than prepare shares), from which the next prepare state and importantly prepare share can be recomputed. On the helper side, there's no way around storing the prepare share: we store the most recently computed `PrepareResp` so that we can handle aggregation jobs idempotently. But to avoid storing prepare messages twice, the continuing/waiting helper stores just a prepare state and a `PingPongMessage`. The main benefit of this change is to reduce how many round trips between aggregators are needed to prepare reports. Quite a few tests used Prio3 but depended on having the leader or helper in the `Waiting` state after running aggregation initialization. Accordingly, those tests are changed to run Poplar1, which now takes 2 rounds. [1]: divviup/libprio-rs#683 Co-authored-by: Tim Geoghegan <[email protected]> Part of #1669
tgeoghegan
added a commit
that referenced
this issue
Sep 13, 2023
This change implements the DAP-05 ping-pong topology in which aggregators take turns preprocessing prepare shares into prepare messages. While this topology first appeared in DAP-05, this implementation follows DAP-06. This change depends on the implementation of the VDAF ping-pong topology added to crate `prio` in [1], which in turn conforms to the specification in VDAF-07. In the ping-pong topology, each DAP-layer step of aggregation now spans two VDAF rounds. An aggregator will use the prepare message it gets from its peer to advance by one VDAF round, and then can use the prepare share it just computed along with the peer's prepare share to advance by another. This incurs some changes to what intermediate values are stored by aggregators. In the case where a leader is continuing/waiting, it will have computed a prepare state, a prepare message for the current round and a prepare share for the next round. The naive implementation would store all three objects in the database, significantly increasing the per-report storage use. To mitigate this, the leader stores a `prio::topology::ping_pong::PingPongTransition`, which will contain a prepare state and a prepare message (both of which are generally much smaller than prepare shares), from which the next prepare state and importantly prepare share can be recomputed. On the helper side, there's no way around storing the prepare share: we store the most recently computed `PrepareResp` so that we can handle aggregation jobs idempotently. But to avoid storing prepare messages twice, the continuing/waiting helper stores just a prepare state and a `PingPongMessage`. The main benefit of this change is to reduce how many round trips between aggregators are needed to prepare reports. Quite a few tests used Prio3 but depended on having the leader or helper in the `Waiting` state after running aggregation initialization. Accordingly, those tests are changed to run Poplar1, which now takes 2 rounds. [1]: divviup/libprio-rs#683 Co-authored-by: Tim Geoghegan <[email protected]> Part of #1669 Co-authored-by: Brandon Pitman <[email protected]>
tgeoghegan
added a commit
that referenced
this issue
Sep 13, 2023
DAP-05 changes a few error types. `unrecongizedMessage -> invalidMessage` `queryMismatch` is gone (now we use `invalidMessage` in that case) `roundMismatch -> stepMismatch` We still use the word "round" in several places where "step" would be more appropriate given DAP-05 text. Renaming those variables, etc., will arrive in a later commit to avoid adding unnecessary noise here. Part of #1669
tgeoghegan
added a commit
that referenced
this issue
Sep 13, 2023
When DAP-05 introduced the ping-pong topology, it also started talking about aggregation protocol "steps" instead of "rounds". This commit corrects the word usage across Janus to line up with the specification. Relevant to #1669
tgeoghegan
added a commit
that referenced
this issue
Sep 14, 2023
DAP-05 changes a few error types. `unrecongizedMessage -> invalidMessage` `queryMismatch` is gone (now we use `invalidMessage` in that case) `roundMismatch -> stepMismatch` We still use the word "round" in several places where "step" would be more appropriate given DAP-05 text. Renaming those variables, etc., will arrive in a later commit to avoid adding unnecessary noise here. Part of #1669
tgeoghegan
added a commit
that referenced
this issue
Sep 14, 2023
When DAP-05 introduced the ping-pong topology, it also started talking about aggregation protocol "steps" instead of "rounds". This commit corrects the word usage across Janus to line up with the specification. Relevant to #1669
tgeoghegan
added a commit
that referenced
this issue
Sep 14, 2023
tgeoghegan
added a commit
that referenced
this issue
Sep 14, 2023
tgeoghegan
added a commit
that referenced
this issue
Sep 14, 2023
When DAP-05 introduced the ping-pong topology, it also started talking about aggregation protocol "steps" instead of "rounds". This commit corrects the word usage across Janus to line up with the specification. Relevant to #1669
|
The wrong commit got tagged as DAP-06, leaving it with the wrong HPKE domain separation strings and an outdated reference to DAF. So we published |
tgeoghegan
changed the title
draft-ietf-ppm-dap-06draft-ietf-ppm-dap-07
tgeoghegan
added a commit
that referenced
this issue
Sep 14, 2023
tgeoghegan
added a commit
that referenced
this issue
Sep 15, 2023
* Update domain separation strings for DAP-07 * Add aggregation parameter to `AggregateShareAad` * Update README.md for DAP-07 Relevant to #1669
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
DAP-05 is out and Janus should implement it. We need:
prio0.14.0 #1673)re-enable Daphne test once they have a DAP-06 implementationprotocolto aggregator api config/capabilities #1743)Draft diff
The text was updated successfully, but these errors were encountered: