Skip to content

Commit

Permalink
Squashed commit of the following:
Browse files Browse the repository at this point in the history 
commit dc117c586a01d6dbef41eafcaf3d0be2e37f2d12
Merge: 32514d1c36 b8eb8920ae
Author: Carlos Baez <[email protected]>
Date:   Thu Nov 14 11:36:36 2024 +0100

    Merge tag 'testnet-v1.37.3' into testnet4-wit-v1.37.3

    Sui testnet v1.37.3 release

commit b8eb8920aeca592d0cf15fbddc7f64ed4aad7202
Author: Andrew Schran <[email protected]>
Date:   Tue Nov 12 15:34:20 2024 -0500

    Revert "Add support for TLS connections with self-signed cert on validator gRPC interface" (#20232) (#20234)

    Reverts MystenLabs/sui#19796 due to suspicion it's causing performance
    issues

    ---

    ## Release notes

    Check each box that your changes affect. If none of the boxes relate to
    your changes, release notes aren't required.

    For each box you select, include information after the relevant heading
    that describes the impact of your changes that a user might notice and
    any actions they must take to implement updates.

    - [ ] Protocol:
    - [ ] Nodes (Validators and Full nodes):
    - [ ] Indexer:
    - [ ] JSON-RPC:
    - [ ] GraphQL:
    - [ ] CLI:
    - [ ] Rust SDK:
    - [ ] REST API:

commit e205b302bd5c5742d21fbc18811b705980714c81
Author: Eugene Boguslavsky <[email protected]>
Date:   Tue Nov 12 16:56:05 2024 +0000

    Sui v1.37.3 Version Bump

commit f77489d1cf177b6bacda47c3c3ec6f120d1fed45
Author: Emma Zhong <[email protected]>
Date:   Mon Nov 11 19:22:47 2024 -0800

    [cherrypick][jsonrpc] fix estimated rewards during safe mode (#20182) (#20187)

    ## Description

    Currently if the exchange rate for an epoch is not found, we would
    assign it the default exchange rate while we should've used the exchange
    rate from the previous existing epoch instead. This PR fixes that.

    ## Test plan

    Added some unit tests.

    ---

    ## Release notes

    Check each box that your changes affect. If none of the boxes relate to
    your changes, release notes aren't required.

    For each box you select, include information after the relevant heading
    that describes the impact of your changes that a user might notice and
    any actions they must take to implement updates.

    - [ ] Protocol:
    - [ ] Nodes (Validators and Full nodes):
    - [ ] Indexer:
    - [x] JSON-RPC: fixed reward estimation in `getStakes` API so that we
    don't overestimate stake rewards of stakes activated during safe mode
    epochs.
    - [ ] GraphQL:
    - [ ] CLI:
    - [ ] Rust SDK:
    - [ ] REST API:

    ## Description

    Describe the changes or additions included in this PR.

    ## Test plan

    How did you test the new or updated feature?

    ---

    ## Release notes

    Check each box that your changes affect. If none of the boxes relate to
    your changes, release notes aren't required.

    For each box you select, include information after the relevant heading
    that describes the impact of your changes that a user might notice and
    any actions they must take to implement updates.

    - [ ] Protocol:
    - [ ] Nodes (Validators and Full nodes):
    - [ ] Indexer:
    - [ ] JSON-RPC:
    - [ ] GraphQL:
    - [ ] CLI:
    - [ ] Rust SDK:
    - [ ] REST API:

commit 102280ae7591ec1cf187a433624443b8798e5605
Author: Eugene Boguslavsky <[email protected]>
Date:   Thu Nov 7 19:25:00 2024 +0000

    Sui v1.37.2 Version Bump

commit b15a076619641002381f4be001f28be5dce7a3e9
Author: wlmyng <[email protected]>
Date:   Tue Nov 5 14:18:50 2024 -0800

    [graphql][cherry-pick] need to add object_status=0 to use partial indexes (#20176) (#20181)

    ## Description

    cherry pick

    ## Test plan

    How did you test the new or updated feature?

    ---

    ## Release notes

    Check each box that your changes affect. If none of the boxes relate to
    your changes, release notes aren't required.

    For each box you select, include information after the relevant heading
    that describes the impact of your changes that a user might notice and
    any actions they must take to implement updates.

    - [ ] Protocol:
    - [ ] Nodes (Validators and Full nodes):
    - [ ] Indexer:
    - [ ] JSON-RPC:
    - [ ] GraphQL:
    - [ ] CLI:
    - [ ] Rust SDK:
    - [ ] REST API:

commit 7839b9501066108cb2322ba9039120a41781a1b0
Author: John Martin <[email protected]>
Date:   Mon Nov 4 14:39:56 2024 -0800

    add client-request-method to CORS allowed headers (#20160) (#20168)

    ## Description

    adds `client-request-method` to allowed cors methods

    ## Test plan

    ---

    ## Release notes

    Check each box that your changes affect. If none of the boxes relate to
    your changes, release notes aren't required.

    For each box you select, include information after the relevant heading
    that describes the impact of your changes that a user might notice and
    any actions they must take to implement updates.

    - [ ] Protocol:
    - [ ] Nodes (Validators and Full nodes):
    - [ ] Indexer:
    - [ ] JSON-RPC:
    - [ ] GraphQL:
    - [ ] CLI:
    - [ ] Rust SDK:
    - [ ] REST API:

    ## Description

    Describe the changes or additions included in this PR.

    ## Test plan

    How did you test the new or updated feature?

    ---

    ## Release notes

    Check each box that your changes affect. If none of the boxes relate to
    your changes, release notes aren't required.

    For each box you select, include information after the relevant heading
    that describes the impact of your changes that a user might notice and
    any actions they must take to implement updates.

    - [ ] Protocol:
    - [ ] Nodes (Validators and Full nodes):
    - [ ] Indexer:
    - [ ] JSON-RPC:
    - [ ] GraphQL:
    - [ ] CLI:
    - [ ] Rust SDK:
    - [ ] REST API:

commit ecffa454e86b994b79dfc6fb9032d02ee95a0697
Author: John Martin <[email protected]>
Date:   Mon Nov 4 13:13:22 2024 -0800

    fix: remove high cardinality address label from safe_client_latency (… (#20169)

    …#20165)

    ## Description

    safe_client_latency is the highest cardinality metric across our
    observability system due to the `address` label + being a histogram:

    ```
    root@sui-node-mysten-rpc-0:/sui# curl -s localhost:9184/metrics | grep safe_client_latency_bucket | wc -l
    8512
    ```

    ---

    ## Release notes

    Check each box that your changes affect. If none of the boxes relate to
    your changes, release notes aren't required.

    For each box you select, include information after the relevant heading
    that describes the impact of your changes that a user might notice and
    any actions they must take to implement updates.

    - [ ] Protocol:
    - [ ] Nodes (Validators and Full nodes):
    - [ ] Indexer:
    - [ ] JSON-RPC:
    - [ ] GraphQL:
    - [ ] CLI:
    - [ ] Rust SDK:
    - [ ] REST API:

    ## Description

    Describe the changes or additions included in this PR.

    ## Test plan

    How did you test the new or updated feature?

    ---

    ## Release notes

    Check each box that your changes affect. If none of the boxes relate to
    your changes, release notes aren't required.

    For each box you select, include information after the relevant heading
    that describes the impact of your changes that a user might notice and
    any actions they must take to implement updates.

    - [ ] Protocol:
    - [ ] Nodes (Validators and Full nodes):
    - [ ] Indexer:
    - [ ] JSON-RPC:
    - [ ] GraphQL:
    - [ ] CLI:
    - [ ] Rust SDK:
    - [ ] REST API:

commit 1493483238672e1d5c22dbff15eb670b1d57883a
Author: Eugene Boguslavsky <[email protected]>
Date:   Mon Nov 4 20:00:02 2024 +0000

    Sui v1.37.1 Version Bump

commit 6646b75dd68808b7bedb679c41706b12ae79a5a9
Author: Eugene Boguslavsky <[email protected]>
Date:   Thu Oct 31 18:46:36 2024 -0700

    Sui `v1.37.0` Framework Bytecode Snapshot (#20131)

    ## Description
    Sui `v1.37.0` Framework Bytecode Snapshot

    ## Test plan
    `cargo run --bin sui-framework-snapshot`

commit fa5d48fcb3e06a11677dc14e90f12cac59e8f2fc
Author: Ashok Menon <[email protected]>
Date:   Fri Nov 1 01:03:44 2024 +0000

    chore(indexer-alt): alphabetize Cargo.toml

commit e116dfa6547e96608bee6767a483e080bb0a1e99
Author: Ashok Menon <[email protected]>
Date:   Fri Nov 1 00:39:20 2024 +0000

    indexer-alt: watermark timestamps and other logging improvements (#20120)

    ## Description

    Track the timestamp associated with a watermark, include it in tracing
    messages, and introduce gauges to track the timestamps of the latest
    checkpoint gathered and written to the DB.

    This change also introduces a couple of other timestamp/watermark
    related tracing improvements:

    - Measure watermarks gathered and in DB for the sequential pipeline
    (previously just tracked the watermark after they were written to the
    DB). These will usually be the exact same, but may differ if the
    pipeline needs to retry a write.
    - Gets rid of the ordering and equality impls for `CommitterWatermark`
    -- these are no longer required.
    - Standardise the ordering of watermark traces between the sequential
    and concurrent pipelines, so that they are easier to compare with each
    other (they now share a common prefix).
    - Standardised tracing messages during committer/watermark teardown: Now
    in each exit edge, we log the reason for the exit, and then a message
    with the final watermark.

    ## Test plan

    Run `sum_obj_types` and `wal_obj_types` with a consistent range of
    `3600` and note that the difference in their timestamps is roughly an
    hour (to begin with, it's about 70 minutes):

    ```
    sui$ cargo run -p sui-indexer-alt --release --                                   \
      --database-url 'postgres://postgres:postgrespw@localhost:5432/sui_indexer_alt' \
      indexer --remote-store-url 'https://checkpoints.mainnet.sui.io'                \
      --consistent-range 3600 --pipeline wal_obj_types --pipeline sum_obj_types`
         Running `/Users/ashokmenon/sui/idx-poc/target/release/sui-indexer-alt --database-url 'postgres://postgres:postgrespw@localhost:5432/sui_indexer_alt' indexer --remote-store-url 'https://checkpoints.mainnet.sui.io' --consistent-range 3600 --pipeline wal_obj_types --pipeline sum_obj_types`
    2024-10-31T15:16:26.867221Z  INFO sui_indexer_alt::db: Running migrations ...
    2024-10-31T15:16:26.889145Z  INFO sui_indexer_alt::db: Migrations complete.
    2024-10-31T15:16:27.024406Z  INFO sui_indexer_alt: Skipping pipeline ev_emit_mod
    2024-10-31T15:16:27.024413Z  INFO sui_indexer_alt: Skipping pipeline ev_struct_inst
    2024-10-31T15:16:27.024414Z  INFO sui_indexer_alt: Skipping pipeline kv_checkpoints
    2024-10-31T15:16:27.024414Z  INFO sui_indexer_alt: Skipping pipeline kv_objects
    2024-10-31T15:16:27.024415Z  INFO sui_indexer_alt: Skipping pipeline kv_transactions
    2024-10-31T15:16:27.024416Z  INFO sui_indexer_alt: Skipping pipeline obj_versions
    2024-10-31T15:16:27.024416Z  INFO sui_indexer_alt: Skipping pipeline tx_affected_objects
    2024-10-31T15:16:27.024417Z  INFO sui_indexer_alt: Skipping pipeline tx_balance_changes
    2024-10-31T15:16:27.024417Z  INFO sui_indexer_alt: Skipping pipeline wal_coin_balances
    2024-10-31T15:16:27.033550Z  INFO sui_indexer_alt: Skipping pipeline sum_coin_balances
    2024-10-31T15:16:27.033607Z  INFO sui_indexer_alt::pipeline::processor: Starting processor pipeline="wal_obj_types"
    2024-10-31T15:16:27.033856Z  INFO sui_indexer_alt::pipeline::concurrent::collector: Starting collector pipeline="wal_obj_types"
    2024-10-31T15:16:27.033869Z  INFO sui_indexer_alt::pipeline::concurrent::committer: Starting committer pipeline="wal_obj_types"
    2024-10-31T15:16:27.033983Z  INFO sui_indexer_alt: Skipping pipeline sum_packages
    2024-10-31T15:16:27.034011Z  INFO sui_indexer_alt::pipeline::processor: Starting processor pipeline="sum_obj_types"
    2024-10-31T15:16:27.034183Z  INFO sui_indexer_alt::pipeline::concurrent::watermark: Starting watermark pipeline="wal_obj_types" watermark=CommitterWatermark { pipeline: "wal_obj_types", epoch_hi_inclusive: 0, checkpoint_hi_inclusive: 0, tx_hi: 0, timestamp_ms_hi_inclusive: 0 }
    2024-10-31T15:16:27.034260Z  INFO sui_indexer_alt::pipeline::sequential::committer: Starting committer pipeline="sum_obj_types" watermark=CommitterWatermark { pipeline: "sum_obj_types", epoch_hi_inclusive: 0, checkpoint_hi_inclusive: 0, tx_hi: 0, timestamp_ms_hi_inclusive: 0 }
    2024-10-31T15:16:27.034305Z  INFO sui_indexer_alt: Ingestion range first_checkpoint=0 last_checkpoint=None
    2024-10-31T15:16:27.034339Z  INFO sui_indexer_alt::metrics: Starting metrics service on 0.0.0.0:9184
    2024-10-31T15:16:27.034351Z  INFO sui_indexer_alt::ingestion::regulator: Starting ingestion regulator
    2024-10-31T15:16:27.034355Z  INFO sui_indexer_alt::ingestion::broadcaster: Starting ingestion broadcaster
    2024-10-31T15:16:27.541400Z  INFO sui_indexer_alt::pipeline::concurrent::watermark: Watermark pipeline="wal_obj_types" epoch=0 checkpoint=212 transaction=213 timestamp=2023-04-13 13:28:25.836 UTC updated=true elapsed_ms=5.486208
    2024-10-31T15:16:28.043533Z  INFO sui_indexer_alt::pipeline::concurrent::watermark: Watermark pipeline="wal_obj_types" epoch=0 checkpoint=515 transaction=516 timestamp=2023-04-13 13:36:17.240 UTC updated=true elapsed_ms=5.512666
    2024-10-31T15:16:28.538421Z  INFO sui_indexer_alt::pipeline::concurrent::watermark: Watermark pipeline="wal_obj_types" epoch=0 checkpoint=1017 transaction=1018 timestamp=2023-04-13 13:48:04.584 UTC updated=true elapsed_ms=3.468209
    2024-10-31T15:16:29.041788Z  INFO sui_indexer_alt::pipeline::concurrent::watermark: Watermark pipeline="wal_obj_types" epoch=0 checkpoint=1422 transaction=1423 timestamp=2023-04-13 13:57:11.577 UTC updated=true elapsed_ms=7.184292
    2024-10-31T15:16:29.540675Z  INFO sui_indexer_alt::pipeline::concurrent::watermark: Watermark pipeline="wal_obj_types" epoch=0 checkpoint=1824 transaction=1825 timestamp=2023-04-13 14:05:59.154 UTC updated=true elapsed_ms=5.7144580000000005
    2024-10-31T15:16:30.040987Z  INFO sui_indexer_alt::pipeline::concurrent::watermark: Watermark pipeline="wal_obj_types" epoch=0 checkpoint=2226 transaction=2227 timestamp=2023-04-13 14:14:54.721 UTC updated=true elapsed_ms=5.955416
    2024-10-31T15:16:30.539394Z  INFO sui_indexer_alt::pipeline::concurrent::watermark: Watermark pipeline="wal_obj_types" epoch=0 checkpoint=2630 transaction=2631 timestamp=2023-04-13 14:23:21.636 UTC updated=true elapsed_ms=5.0003340000000005
    2024-10-31T15:16:31.035944Z  INFO sui_indexer_alt::pipeline::concurrent::watermark: Watermark pipeline="wal_obj_types" epoch=0 checkpoint=3142 transaction=3143 timestamp=2023-04-13 14:34:00.192 UTC updated=true elapsed_ms=1.368042
    2024-10-31T15:16:31.532346Z  INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=71 transaction=72 timestamp=2023-04-13 13:24:36.662 UTC
    2024-10-31T15:16:31.535237Z  INFO sui_indexer_alt::pipeline::concurrent::watermark: Watermark pipeline="wal_obj_types" epoch=0 checkpoint=3646 transaction=3647 timestamp=2023-04-13 14:45:19.227 UTC updated=true elapsed_ms=0.7080409999999999
    2024-10-31T15:16:31.556459Z  INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=103 transaction=104 timestamp=2023-04-13 13:25:35.491 UTC
    2024-10-31T15:16:31.605247Z  INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=151 transaction=152 timestamp=2023-04-13 13:27:01.087 UTC
    2024-10-31T15:16:31.658676Z  INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=203 transaction=204 timestamp=2023-04-13 13:28:14.431 UTC
    2024-10-31T15:16:31.709093Z  INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=251 transaction=252 timestamp=2023-04-13 13:29:41.350 UTC
    2024-10-31T15:16:31.774101Z  INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=304 transaction=305 timestamp=2023-04-13 13:31:05.962 UTC
    2024-10-31T15:16:31.823847Z  INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=351 transaction=352 timestamp=2023-04-13 13:32:06.748 UTC
    2024-10-31T15:16:31.875613Z  INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=404 transaction=405 timestamp=2023-04-13 13:33:26.122 UTC
    2024-10-31T15:16:31.923673Z  INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=452 transaction=453 timestamp=2023-04-13 13:34:41.672 UTC
    2024-10-31T15:16:31.973523Z  INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=502 transaction=503 timestamp=2023-04-13 13:35:54.245 UTC
    2024-10-31T15:16:32.020968Z  INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=553 transaction=554 timestamp=2023-04-13 13:37:12.723 UTC
    2024-10-31T15:16:32.034336Z  INFO sui_indexer_alt::pipeline::concurrent::watermark: Watermark pipeline="wal_obj_types" epoch=0 checkpoint=4147 transaction=4148 timestamp=2023-04-13 14:56:30.470 UTC updated=true elapsed_ms=0.478916
    2024-10-31T15:16:32.081115Z  INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=603 transaction=604 timestamp=2023-04-13 13:38:26.226 UTC
    2024-10-31T15:16:32.143228Z  INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=657 transaction=658 timestamp=2023-04-13 13:39:32.583 UTC
    2024-10-31T15:16:32.201323Z  INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=710 transaction=711 timestamp=2023-04-13 13:40:44.461 UTC
    2024-10-31T15:16:32.239715Z  INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=751 transaction=752 timestamp=2023-04-13 13:41:40.955 UTC
    2024-10-31T15:16:32.306828Z  INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=803 transaction=804 timestamp=2023-04-13 13:43:00.433 UTC
    2024-10-31T15:16:32.351995Z  INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=854 transaction=855 timestamp=2023-04-13 13:44:21.487 UTC
    2024-10-31T15:16:32.408044Z  INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=903 transaction=904 timestamp=2023-04-13 13:45:33.468 UTC
    2024-10-31T15:16:32.478623Z  INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=963 transaction=964 timestamp=2023-04-13 13:46:48.531 UTC
    2024-10-31T15:16:32.517822Z  INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=1004 transaction=1005 timestamp=2023-04-13 13:47:43.997 UTC
    2024-10-31T15:16:32.540701Z  INFO sui_indexer_alt::pipeline::concurrent::watermark: Watermark pipeline="wal_obj_types" epoch=0 checkpoint=4550 transaction=4551 timestamp=2023-04-13 15:05:30.690 UTC updated=true elapsed_ms=5.776375
    2024-10-31T15:16:32.574600Z  INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=1052 transaction=1053 timestamp=2023-04-13 13:48:55.253 UTC
    2024-10-31T15:16:32.625726Z  INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=1102 transaction=1103 timestamp=2023-04-13 13:49:57.288 UTC
    2024-10-31T15:16:32.679135Z  INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=1159 transaction=1160 timestamp=2023-04-13 13:51:06.819 UTC
    2024-10-31T15:16:32.729858Z  INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=1203 transaction=1204 timestamp=2023-04-13 13:52:11.201 UTC
    2024-10-31T15:16:32.777044Z  INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=1257 transaction=1258 timestamp=2023-04-13 13:53:26.750 UTC
    2024-10-31T15:16:32.819603Z  INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=1301 transaction=1302 timestamp=2023-04-13 13:54:17.123 UTC
    2024-10-31T15:16:32.870028Z  INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=1351 transaction=1352 timestamp=2023-04-13 13:55:24.124 UTC
    2024-10-31T15:16:32.923059Z  INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=1402 transaction=1403 timestamp=2023-04-13 13:56:49.078 UTC
    2024-10-31T15:16:32.969159Z  INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=1451 transaction=1452 timestamp=2023-04-13 13:57:56.351 UTC
    2024-10-31T15:16:33.034831Z  INFO sui_indexer_alt::pipeline::concurrent::watermark: Watermark pipeline="wal_obj_types" epoch=0 checkpoint=5055 transaction=5056 timestamp=2023-04-13 15:16:38.038 UTC updated=true elapsed_ms=1.0442500000000001
    2024-10-31T15:16:33.036266Z  INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=1501 transaction=1502 timestamp=2023-04-13 13:58:56.227 UTC
    ^C2024-10-31T15:16:33.091189Z  INFO sui_indexer_alt::pipeline::concurrent::collector: Shutdown received, stopping collector pipeline="wal_obj_types"
    2024-10-31T15:16:33.091195Z  INFO sui_indexer_alt::pipeline::concurrent::watermark: Shutdown received pipeline="wal_obj_types"
    2024-10-31T15:16:33.091201Z  INFO sui_indexer_alt::ingestion::regulator: Shutdown received, stopping regulator
    2024-10-31T15:16:33.091213Z  INFO sui_indexer_alt::metrics: Shutdown received, stopping metrics service
    2024-10-31T15:16:33.091225Z  INFO sui_indexer_alt::pipeline::concurrent::watermark: Stopping watermark task pipeline="wal_obj_types" watermark=CommitterWatermark { pipeline: "wal_obj_types", epoch_hi_inclusive: 0, checkpoint_hi_inclusive: 5055, tx_hi: 5056, timestamp_ms_hi_inclusive: 1681398998038 }
    2024-10-31T15:16:33.091410Z  INFO sui_indexer_alt::pipeline::concurrent::committer: Batches done, stopping committer pipeline="wal_obj_types"
    2024-10-31T15:16:33.091721Z  INFO sui_indexer_alt::pipeline::sequential::committer: Shutdown received pipeline="sum_obj_types"
    2024-10-31T15:16:33.091725Z  INFO sui_indexer_alt::pipeline::sequential::committer: Stopping committer pipeline="sum_obj_types" watermark=CommitterWatermark { pipeline: "sum_obj_types", epoch_hi_inclusive: 0, checkpoint_hi_inclusive: 1546, tx_hi: 1547, timestamp_ms_hi_inclusive: 1681394391225 }
    2024-10-31T15:16:33.093829Z  INFO sui_indexer_alt::pipeline::processor: Shutdown received, stopping processor pipeline="wal_obj_types"
    2024-10-31T15:16:33.094014Z  INFO sui_indexer_alt::pipeline::processor: Shutdown received, stopping processor pipeline="sum_obj_types"
    2024-10-31T15:16:33.094406Z  INFO sui_indexer_alt::ingestion::broadcaster: Shutdown received, stopping ingestion broadcaster
    2024-10-31T15:16:33.094513Z  INFO sui_indexer_alt: Indexing pipeline gracefully shut down
    ```

    ## Stack

    - #20089
    - #20114
    - #20116
    - #20117
    - #20119

    ---

    ## Release notes

    Check each box that your changes affect. If none of the boxes relate to
    your changes, release notes aren't required.

    For each box you select, include information after the relevant heading
    that describes the impact of your changes that a user might notice and
    any actions they must take to implement updates.

    - [ ] Protocol:
    - [ ] Nodes (Validators and Full nodes):
    - [ ] Indexer:
    - [ ] JSON-RPC:
    - [ ] GraphQL:
    - [ ] CLI:
    - [ ] Rust SDK:
    - [ ] REST API:

commit 4b727cbbfd89db7ebfd09b21e05faa58ac2d8c96
Author: Ashok Menon <[email protected]>
Date:   Fri Nov 1 00:37:19 2024 +0000

    indexer-alt: allow sequential pipeline immediate reset (#20119)

    ## Description

    If the sequential pipeline committer can guarantee that it could process
    more checkpoints by looking at its pending buffer, we now reset the
    polling interval immediately, so it does not wait to issue the next
    write. This mimics a similar behaviour in the concurrent pipeline.

    I made this change after noticing how the pipeline behaves when
    ingestion is stuck retrying a checkpoint, on my local machine. Usually
    when running locally, performance is limited by checkpoint download
    rate, but in a sequential pipeline, if a checkpoint failed to download,
    it is possible for many checkpoints to end up processed and pending.

    With the previous implementation, once ingestion had recovered (the
    checkpoint is fetched), the pending buffer kept growing because it was
    only able to land `MAX_BATCH_CHECKPOINTS / commit_interval`, so if
    checkpoints were getting added faster than that, it would never recover.

    With this change, the pipeline recovers almost instantly, and I expect
    that in GCP where bandwidth is not the rate limiting factor, this should
    improve throughput during backfill, and synthetic benchmarks.

    ## Test plan

    Run the indexer with a large ingestion buffer and concurrency, wait for
    ingestion to fail to fetch a checkpoint, and then notice how the
    situation recovers (instead of getting worse until the pipeline
    eventually complains that it has too many pending checkpoints):

    ```
    sui$ cargo run -p sui-indexer-alt --release --                                   \
      --database-url "postgres://postgres:postgrespw@localhost:5432/sui_indexer_alt" \
      indexer --remote-store-url https://checkpoints.mainnet.sui.io                  \
      --last-checkpoint 1200000 --pipeline sum_packages                              \
      --checkpoint-buffer-size 50000 --ingest-concurrency 20000
    ```

    ## Stack

    - #20089
    - #20114
    - #20116
    - #20117

    ---

    ## Release notes

    Check each box that your changes affect. If none of the boxes relate to
    your changes, release notes aren't required.

    For each box you select, include information after the relevant heading
    that describes the impact of your changes that a user might notice and
    any actions they must take to implement updates.

    - [ ] Protocol:
    - [ ] Nodes (Validators and Full nodes):
    - [ ] Indexer:
    - [ ] JSON-RPC:
    - [ ] GraphQL:
    - [ ] CLI:
    - [ ] Rust SDK:
    - [ ] REST API:

commit b65f34c6a00801620e11469575523895a8e074ac
Author: Ashok Menon <[email protected]>
Date:   Fri Nov 1 00:21:38 2024 +0000

    indexer-alt: wal_coin_balances pipeline (#20117)

    ## Description

    `wal_coin_balances` is to `sum_coin_balances` what `wal_obj_types` is to
    `sum_obj_types`.

    ## Test plan

    Run the indexer, and correlate the live object set calculated from the
    write-ahead log against the one that's already in the summary table:

    ```
    sui$ cargo run -p sui-indexer-alt --release --                                   \
      --database-url "postgres://postgres:postgrespw@localhost:5432/sui_indexer_alt" \
      indexer --remote-store-url https://checkpoints.mainnet.sui.io/                 \
      --last-checkpoint 5000
    ```

    ```
    sui_indexer_alt=# SELECT COUNT(*) FROM sum_coin_balances;
     count
    -------
       178
    (1 row)

    sui_indexer_alt=# SELECT
        COUNT(*)
    FROM (
        SELECT DISTINCT ON (object_id)
            *
        FROM
            wal_coin_balances
        ORDER BY
            object_id,
            object_version DESC
    ) o
    WHERE
        o.owner_id IS NOT NULL;
     count
    -------
       178
    (1 row)
    ```

    ## Stack

    - #20089
    - #20114
    - #20116

    ---

    ## Release notes

    Check each box that your changes affect. If none of the boxes relate to
    your changes, release notes aren't required.

    For each box you select, include information after the relevant heading
    that describes the impact of your changes that a user might notice and
    any actions they must take to implement updates.

    - [ ] Protocol:
    - [ ] Nodes (Validators and Full nodes):
    - [ ] Indexer:
    - [ ] JSON-RPC:
    - [ ] GraphQL:
    - [ ] CLI:
    - [ ] Rust SDK:
    - [ ] REST API:

commit 71f1faa1a8557a178e0bf89f20829a8ea503f156
Author: Ashok Menon <[email protected]>
Date:   Fri Nov 1 00:18:46 2024 +0000

    indexer-alt: wal_obj_types pipeline (#20116)

    ## Description

    Adds the concurrent pipeline that writes the write-ahead log for
    `sum_obj_types`. It re-uses the `process` implementation from
    `sum_obj_types` and then writes it into an append-only table.

    Note that today:

    - The pipelines are completely independent, which means the WAL pipeline
    redoes the processing work of the summary pipeline (this is presumably
    not an issue because the repeated work is not particularly heavy).
    - This change does not include the pruning necessary to keep this
    table's size in check (in practice it should only be a couple of gigs in
    size). This will come in a follow-up PR.

    ## Test plan

    Run the indexer and cross check the live-object set calculated from the
    write-ahead log with the summary:

    ```
    sui$ cargo run -p sui-indexer-alt --release --                                   \
      --database-url "postgres://postgres:postgrespw@localhost:5432/sui_indexer_alt" \
      indexer --remote-store-url https://checkpoints.mainnet.sui.io                  \
      --last-checkpoint 5000
    ```

    ```
    sui_indexer_alt=# SELECT COUNT(*) FROM sum_obj_types;
     count
    -------
       592
    (1 row)

                                                      ^
    sui_indexer_alt=# SELECT
        COUNT(*)
    FROM (
        SELECT DISTINCT ON (object_id)
            *
        FROM
            wal_obj_types
        ORDER BY
            object_id,
            object_version DESC
    ) o
    WHERE
        o.owner_kind IS NOT NULL;
     count
    -------
       592
    (1 row)
    ```

    ## Stack

    - #20089
    - #20114

    ---

    ## Release notes

    Check each box that your changes affect. If none of the boxes relate to
    your changes, release notes aren't required.

    For each box you select, include information after the relevant heading
    that describes the impact of your changes that a user might notice and
    any actions they must take to implement updates.

    - [ ] Protocol:
    - [ ] Nodes (Validators and Full nodes):
    - [ ] Indexer:
    - [ ] JSON-RPC:
    - [ ] GraphQL:
    - [ ] CLI:
    - [ ] Rust SDK:
    - [ ] REST API:

commit 7de02417c27844c8a086866b1279b4736ab862ec
Author: Ashok Menon <[email protected]>
Date:   Thu Oct 31 23:22:42 2024 +0000

    indexer-alt: obj_versions pipeline (#20114)

    ## Description

    Pipeline to fill `obj_versions` table -- this is the table that serves
    "parent version" (dynamic field) queries.

    ## Test plan

    ```
    sui$ cargo run -p sui-indexer-alt --release --                                   \
      --database-url "postgres://postgres:postgrespw@localhost:5432/sui_indexer_alt" \
      indexer --remote-url "https://checkpoints.mainnet.sui.io"                      \
      --last-checkpoint 5000
    ```

    ## Stack

    - #20089

    ---

    ## Release notes

    Check each box that your changes affect. If none of the boxes relate to
    your changes, release notes aren't required.

    For each box you select, include information after the relevant heading
    that describes the impact of your changes that a user might notice and
    any actions they must take to implement updates.

    - [ ] Protocol:
    - [ ] Nodes (Validators and Full nodes):
    - [ ] Indexer:
    - [ ] JSON-RPC:
    - [ ] GraphQL:
    - [ ] CLI:
    - [ ] Rust SDK:
    - [ ] REST API:

commit 53eaed5e300a67f0d01cf68119a145410925f9de
Author: Ashok Menon <[email protected]>
Date:   Wed Oct 30 00:06:07 2024 +0000

    indexer-alt: delays for sequential pipelines

    ## Description

    Add the ability to include an optional delay to sequential pipelines.
    This can be used to implement behaviour similar to `objects_snapshot` in
    the new architecture.

    Sequential pipelines honour this configuration in their committer tasks:

    - They process checkpoints at the same rate as other pipelines
    - But hold back data in the committer's "pending" map until the
      committer witnesses a checkpoint that is at least `checkpoint_lag`
      checkpoints ahead.

    This also requires slightly complicating the set-up and tear-down logic:

    - When setting up, we may need to wait until the pending buffer has been
      primed enough to set a valid commit upperbound.
    - If the pipeline runs dry before that point, we need to add an extra
      check to close down the committer.
    - When tearing down the committer after its channel has closed, we need
      to check whether batching would have picked up the first pending
      checkpoint (whether it is a valid "next checkpoint" and it's not held
      up because of a lag parameter).

    ## Test plan

    Run the indexer with various parameterisations for the consistent range,
    and note that the watermark for the summary pipelines lag behind the
    watermarks for other pipelines.

    ```
    sui$ cargo run -p sui-indexer-alt --release --                                   \
      --database-url "postgres://postgres:postgrespw@localhost:5432/sui_indexer_alt" \
      indexer --remote-store-url https://checkpoints.mainnet.sui.io                  \
      --pipeline kv_objects                                                          \
      --pipeline sum_obj_types --pipeline sum_coin_balances                          \
      --last-checkpoint 5000 --consistent-range 1000
    [...]
    2024-10-30T00:24:45.148259Z  INFO sui_indexer_alt::ingestion::regulator: Checkpoints done, stopping regulator
    2024-10-30T00:24:45.161682Z  INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=3605 transaction=3606
    2024-10-30T00:24:45.212394Z  INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_coin_balances" epoch=0 checkpoint=2703 transaction=2704
    2024-10-30T00:24:45.222986Z  INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=3656 transaction=3657
    2024-10-30T00:24:45.274318Z  INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=3702 transaction=3703
    2024-10-30T00:24:45.340714Z  INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=3753 transaction=3754
    2024-10-30T00:24:45.355934Z  INFO sui_indexer_alt::pipeline::concurrent::watermark: Watermark pipeline="kv_objects" elapsed_ms=0.949083 updated=true epoch=0 checkpoint=4760 transaction=4761
    2024-10-30T00:24:45.401262Z  INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=3801 transaction=3802
    2024-10-30T00:24:45.463134Z  INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=3852 transaction=3853
    2024-10-30T00:24:45.544933Z  INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=3904 transaction=3905
    2024-10-30T00:24:45.609068Z  INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=3952 transaction=3953
    2024-10-30T00:24:45.668560Z  INFO sui_indexer_alt::ingestion::broadcaster: Checkpoints done, stopping ingestion broadcaster
    2024-10-30T00:24:45.668593Z  INFO sui_indexer_alt::pipeline::processor: Checkpoints done, stopping processor pipeline="kv_objects"
    2024-10-30T00:24:45.668605Z  INFO sui_indexer_alt::pipeline::processor: Checkpoints done, stopping processor pipeline="sum_obj_types"
    2024-10-30T00:24:45.668605Z  INFO sui_indexer_alt::pipeline::processor: Checkpoints done, stopping processor pipeline="sum_coin_balances"
    2024-10-30T00:24:45.674305Z  INFO sui_indexer_alt::pipeline::sequential::committer: Processor closed channel, pending rows empty, stopping committer pipeline="sum_obj_types" watermark=CommitterWatermark { pipeline: "sum_obj_types", epoch_hi_inclusive: 0, checkpoint_hi_inclusive: 4000, tx_hi: 4001 }
    2024-10-30T00:24:45.713856Z  INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_coin_balances" epoch=0 checkpoint=3003 transaction=3004
    2024-10-30T00:24:45.863837Z  INFO sui_indexer_alt::pipeline::concurrent::watermark: Watermark pipeline="kv_objects" elapsed_ms=1.83825 updated=true epoch=0 checkpoint=4960 transaction=4961
    2024-10-30T00:24:46.120459Z  INFO sui_indexer_alt::pipeline::concurrent::collector: Processor closed channel, pending rows empty, stopping collector pipeline="kv_objects"
    2024-10-30T00:24:46.131407Z  INFO sui_indexer_alt::pipeline::concurrent::committer: Batches done, stopping committer pipeline="kv_objects"
    2024-10-30T00:24:46.214267Z  INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_coin_balances" epoch=0 checkpoint=3303 transaction=3304
    2024-10-30T00:24:46.356941Z  INFO sui_indexer_alt::pipeline::concurrent::watermark: Watermark pipeline="kv_objects" elapsed_ms=0.68 updated=true epoch=0 checkpoint=5000 transaction=5001
    2024-10-30T00:24:46.356970Z  INFO sui_indexer_alt::pipeline::concurrent::watermark: Committer closed channel, stopping watermark task pipeline="kv_objects" watermark=CommitterWatermark { pipeline: "kv_objects", epoch_hi_inclusive: 0, checkpoint_hi_inclusive: 5000, tx_hi: 5001 }
    2024-10-30T00:24:46.714498Z  INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_coin_balances" epoch=0 checkpoint=3603 transaction=3604
    2024-10-30T00:24:47.214758Z  INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_coin_balances" epoch=0 checkpoint=3903 transaction=3904
    2024-10-30T00:24:47.715841Z  INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_coin_balances" epoch=0 checkpoint=4000 transaction=4001
    2024-10-30T00:24:47.715917Z  INFO sui_indexer_alt::pipeline::sequential::committer: Processor closed channel, pending rows empty, stopping committer pipeline="sum_coin_balances" watermark=CommitterWatermark { pipeline: "sum_coin_balances", epoch_hi_inclusive: 0, checkpoint_hi_inclusive: 4000, tx_hi: 4001 }
    2024-10-30T00:24:47.716485Z  INFO sui_indexer_alt: Indexing pipeline gracefully shut down
    2024-10-30T00:24:47.716647Z  INFO sui_indexer_alt::metrics: Shutdown received, stopping metrics service
    ```

commit d57326492b62ec8ba7ff8826ad473ec1b5008f11
Author: Ashok Menon <[email protected]>
Date:   Wed Oct 30 00:01:45 2024 +0000

    fix(indexer-alt): remove double logging on pipeline skip

    ## Description

    We were logging the pipeline being skipped twice.

    ## Test plan

    Check logs while the indexer starts up:

    ```
    sui$ cargo run -p sui-indexer-alt --release --                                   \
      --database-url "postgres://postgres:postgrespw@localhost:5432/sui_indexer_alt" \
      indexer --remote-store-url "https://checkpoints.mainnet.sui.io"                \
      --last-checkpoint 5000 --pipeline sum_obj_types
    ```

commit ef0d78c638e3ef0d0a3b28787b618ea56a140284
Author: Ge Gao <[email protected]>
Date:   Thu Oct 31 19:30:51 2024 -0400

    add sui-field-count for sui-indexer-alt (#20126)

    ## Description

    title, so that we can replace hard-coded field count in pipelines.
    a crate did very similar things but it's archived, so prob better to
    have our own impl, also b/c the functionality is relatively simple
    https://github.com/discosultan/field-count

    ## Test plan

    added tests in models/

    ---

    ## Release notes

    Check each box that your changes affect. If none of the boxes relate to
    your changes, release notes aren't required.

    For each box you select, include information after the relevant heading
    that describes the impact of your changes that a user might notice and
    any actions they must take to implement updates.

    - [ ] Protocol:
    - [ ] Nodes (Validators and Full nodes):
    - [ ] Indexer:
    - [ ] JSON-RPC:
    - [ ] GraphQL:
    - [ ] CLI:
    - [ ] Rust SDK:
    - [ ] REST API:

commit c92dd1b4f16074ced8c1fe5c56a10e675f68bbe6
Author: phoenix <[email protected]>
Date:   Thu Oct 31 18:28:34 2024 -0400

    [kv store] bigtable. Add support for querying checkpoint by digest (#20122)

    ## Description

    Adds support for querying checkpoint by digest

    ---

    ## Release notes

    Check each box that your changes affect. If none of the boxes relate to
    your changes, release notes aren't required.

    For each box you select, include information after the relevant heading
    that describes the impact of your changes that a user might notice and
    any actions they must take to implement updates.

    - [ ] Protocol:
    - [ ] Nodes (Validators and Full nodes):
    - [ ] Indexer:
    - [ ] JSON-RPC:
    - [ ] GraphQL:
    - [ ] CLI:
    - [ ] Rust SDK:
    - [ ] REST API:

commit 74ace7b6166f10bddfd714568e0ddc20db2ca75d
Author: JasonRUAN <[email protected]>
Date:   Fri Nov 1 06:25:44 2024 +0800

    [CLI] fix `sui keytool import` --alias command and default alias does not work (#20111)

    ## Description

    fix `sui keytool import` --alias command and default alias does not work

    Before Fix：
    <img width="1295" alt="image"
    src="https://github.com/user-attachments/assets/4cb40d92-e266-4b6e-9c39-c05ac52d7a9d">

    After Fix：

    - **--alias**
    <img width="1301" alt="image"
    src="https://github.com/user-attachments/assets/b5c711e8-2a4d-4861-9965-30cde260e93f">

    - **default alias**
    <img width="1295" alt="image"
    src="https://github.com/user-attachments/assets/4513db65-0e10-4179-b5f0-de188871cf78">

    ## Test plan

    How did you test the new or updated feature?

    ---

    ## Release notes

    Check each box that your changes affect. If none of the boxes relate to
    your changes, release notes aren't required.

    For each box you select, include information after the relevant heading
    that describes the impact of your changes that a user might notice and
    any actions they must take to implement updates.

    - [ ] Protocol:
    - [ ] Nodes (Validators and Full nodes):
    - [ ] Indexer:
    - [ ] JSON-RPC:
    - [ ] GraphQL:
    - [x] CLI: `sui keytool import --alias` is now fixed and correctly
    records the provided alias.
    - [ ] Rust SDK:
    - [ ] REST API:

commit e41a759769fdea08f660160a23e945bbad80b337
Author: sui-merge-bot[bot] <114704316+sui-merge-bot[bot]@users.noreply.github.com>
Date:   Thu Oct 31 23:50:37 2024 +0200

    Version Packages (#20127)

    This PR was opened by the [Changesets
    release](https://github.com/changesets/action) GitHub action. When
    you're ready to do a release, you can merge this and publish to npm
    yourself or [setup this action to publish
    automatically](https://github.com/changesets/action#with-publishing). If
    you're not ready to do a release yet, that's fine, whenever you add more
    changesets to main, this PR will be updated.

    # Releases
    ## @mysten/[email protected]

    ### Patch Changes

    -   02c9e46: Fix exports on the bundled package

    Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

commit 02c9e4691b429b4a0ec85b71eace37b72515d055
Author: Manolis Liolios <[email protected]>
Date:   Thu Oct 31 23:38:15 2024 +0200

    Fix build, re-publish @mysten/kms (#20125)

    ## Description

    Build will always treat `src` as the base for outputs when building.

    ## Test plan

    How did you test the new or updated feature?

    ---

    ## Release notes

    Check each box that your changes affect. If none of the boxes relate to
    your changes, release notes aren't required.

    For each box you select, include information after the relevant heading
    that describes the impact of your changes that a user might notice and
    any actions they must take to implement updates.

    - [ ] Protocol:
    - [ ] Nodes (Validators and Full nodes):
    - [ ] Indexer:
    - [ ] JSON-RPC:
    - [ ] GraphQL:
    - [ ] CLI:
    - [ ] Rust SDK:
    - [ ] REST API:

commit 16f2683a99e7b8824aa5879d3f20ffef936a7546
Author: William Smith <[email protected]>
Date:   Thu Oct 31 17:27:16 2024 -0400

    [sui-benchmark] Add expected failure payloads (#19831)

    ## Description

    Introduce a new payload type into the benchmark tool -
    `ExpectedFailurePayload`.

    This payload type is configured with some expected failure type
    (currently only one implemented - user signature failure - but others
    can be added with minimal scaffolding) such that
    `payload.make_transaction()` generates a transaction that will fail in
    this manner. Note that the failures that this payload type is concerned
    with are failures to execute the transaction itself, rather than during
    execution by the MoveVM. In other words, it is expected that the failure
    mode will not consume gas or produce effects.

    Note that for this reason, its failure mode is inverted. It will be
    tallied for metrics purposes as an `error` if it succeeds and tallied as
    an `expected_error` (which is a success) if it fails.

    Also note that transaction responses for this type are handled by
    producing a `NextOp::Retry` (with some additional logging/metrics) since
    it is functionally equivalent to a retryable error.

    ## Test plan

    Ran the following:
    ```
    SIM_STRESS_TEST_QPS=200 SIM_STRESS_TEST_WORKERS=20 RUST_LOG=debug cargo simtest --nocapture test_simulated_load_expected_failure_traffic_control
    ```

    Observed the following in logs
    ```
    2022-01-03T02:05:47.734507Z  INFO node{id=1 name="client"}: simtest::test: crates/sui-benchmark/tests/simtest.rs:1088: end of test BenchmarkStats { duration: 50.000002242s, num_error_txes: 0, num_expected_error_txes: 8049, num_success_txes: 348, num_success_cmds: 1977, total_gas_used: 3215776400, latency_ms: HistogramWrapper { histogram: Histogram { auto_resize: false, highest_trackable_value: 120000, lowest_discernible_value: 1, significant_value_digits: 3, bucket_count: 7, sub_bucket_count: 2048, sub_bucket_half_count: 1024, sub_bucket_half_count_magnitude: 10, sub_bucket_mask: 2047, leading_zero_count_base: 53, unit_magnitude: 0, unit_magnitude_mask: 0, max_value: 14359, min_non_zero_value: 72, total_count: 348, counts: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 2, 8, 12, 9, 20, 25, 22, 19
    ...
    ```

    [Also ran in
    PTN](https://metrics.sui.io/d/adl51ctsvmkg0a/traffic-control-dos-protection-dashboard?from=2024-10-31T19:00:50.548Z&to=2024-10-31T20:20:03.673Z&timezone=browser&var-network=private-testnet&var-host=$__all&var-fullnode=$__all)

    ---

    ## Release notes

    Check each box that your changes affect. If none of the boxes relate to
    your changes, release notes aren't required.

    For each box you select, include information after the relevant heading
    that describes the impact of your changes that a user might notice and
    any actions they must take to implement updates.

    - [ ] Protocol:
    - [ ] Nodes (Validators and Full nodes):
    - [ ] Indexer:
    - [ ] JSON-RPC:
    - [ ] GraphQL:
    - [ ] CLI:
    - [ ] Rust SDK:
    - [ ] REST API:

commit d95918cea0d8fa0f1a4007273f477e3c12b61302
Author: phoenix <[email protected]>
Date:   Thu Oct 31 17:18:17 2024 -0400

    [data ingestion] add support for bigtable in main workflow binary (#20018)

    ## Description

    adding BigTable KV as a variant to main binary that manages internal
    workflows

    ---

    ## Release notes

    Check each box that your changes affect. If none of the boxes relate to
    your changes, release notes aren't required.

    For each box you select, include information after the relevant heading
    that describes the impact of your changes that a user might notice and
    any actions they must take to implement updates.

    - [ ] Protocol:
    - [ ] Nodes (Validators and Full nodes):
    - [ ] Indexer:
    - [ ] JSON-RPC:
    - [ ] GraphQL:
    - [ ] CLI:
    - [ ] Rust SDK:
    - [ ] REST API:

commit c33374e735257b2a460a2f882e062619f8958bc5
Author: wlmyng <[email protected]>
Date:   Thu Oct 31 12:51:47 2024 -0700

    [mvr] indexer schema subset and reduced persist tasks (#20100)

    ## Description

    Describe the changes or additions included in this PR.

    ## Test plan

    How did you test the new or updated feature?

    ---

    ## Release notes

    Check each box that your changes affect. If none of the boxes relate to
    your changes, release notes aren't required.

    For each box you select, include information after the relevant heading
    that describes the impact of your changes that a user might notice and
    any actions they must take to implement updates.

    - [ ] Protocol:
    - [ ] Nodes (Validators and Full nodes):
    - [ ] Indexer:
    - [ ] JSON-RPC:
    - [ ] GraphQL:
    - [ ] CLI:
    - [ ] Rust SDK:
    - [ ] REST API:

commit 7c517e7e5ce7193d1f51ef7f20a31b56f0e66797
Author: Xun Li <[email protected]>
Date:   Thu Oct 31 12:49:58 2024 -0700

    Add system_invariant_violation macro (#20105)

    ## Description

    Introduce a system invariant violation macro.
    We can then set up alerts whenever the counter value changes.
    Used it in one place as an example.

    ## Test plan

    How did you test the new or updated feature?

    ---

    ## Release notes

    Check each box that your changes affect. If none of the boxes relate to
    your changes, release notes aren't required.

    For each box you select, include information after the relevant heading
    that describes the impact of your changes that a user might notice and
    any actions they must take to implement updates.

    - [ ] Protocol:
    - [ ] Nodes (Validators and Full nodes):
    - [ ] Indexer:
    - [ ] JSON-RPC:
    - [ ] GraphQL:
    - [ ] CLI:
    - [ ] Rust SDK:
    - [ ] REST API:

commit b248f35879801b3bae9b76c4d204f36540d816f8
Author: Andrew Schran <[email protected]>
Date:   Thu Oct 31 14:16:48 2024 -0400

    Add support for TLS connections with self-signed cert on validator gRPC interface (#19796)

    Client side use of TLS will be enabled-by-default in a future PR.

    ---

    ## Release notes

    Check each box that your changes affect. If none of the boxes relate to
    your changes, release notes aren't required.

    For each box you select, include information after the relevant heading
    that describes the impact of your changes that a user might notice and
    any actions they must take to implement updates.

    - [ ] Protocol:
    - [x] Nodes (Validators and Full nodes): Adds support for TLS on
    validator gRPC interface.
    - [ ] Indexer:
    - [ ] JSON-RPC:
    - [ ] GraphQL:
    - [ ] CLI:
    - [ ] Rust SDK:
    - [ ] REST API:

commit e06610823b985bdc01c40e23c9a9a2f8d5b571d1
Author: sui-merge-bot[bot] <114704316+sui-merge-bot[bot]@users.noreply.github.com>
Date:   Thu Oct 31 19:58:58 2024 +0200

    Version Packages (#20076)

    This PR was opened by the [Changesets
    release](https://github.com/changesets/action) GitHub action. When
    you're ready to do a release, you can merge this and publish to npm
    yourself or [setup this action to publish
    automatically](https://github.com/changesets/action#with-publishing). If
    you're not ready to do a release yet, that's fine, whenever you add more
    changesets to main, this PR will be updated.

    # Releases
    ## @mysten/[email protected]

    ### Minor Changes

    - c24814b: Adds a custom header; 'Client-Request-Method' which will
    contain the method name used in
        each outgoing jsonrpc request

    ## @mysten/[email protected]

    ### Patch Changes

    -   Updated dependencies [c24814b]
        -   @mysten/[email protected]
        -   @mysten/[email protected]

    ## @mysten/[email protected]

    ### Patch Changes

    -   Updated dependencies [c24814b]
        -   @mysten/[email protected]
        -   @mysten/[email protected]
        -   @mysten/[email protected]

    ## @mysten/[email protected]

    ### Patch Changes

    -   Updated dependencies [c24814b]
        -   @mysten/[email protected]

    ## @mysten/[email protected]

    ### Patch Changes

    -   Updated dependencies [c24814b]
        -   @mysten/[email protected]

    ## @mysten/[email protected]

    ### Patch Changes

    -   Updated dependencies [c24814b]
        -   @mysten/[email protected]
        -   @mysten/[email protected]

    ## @mysten/[email protected]

    ### Patch Changes

    -   Updated dependencies [c24814b]
        -   @mysten/[email protected]

    ## @mysten/[email protected]

    ### Patch Changes

    -   4166d71: Fix doc comment on `getKiosk` command
    -   Updated dependencies [c24814b]
        -   @mysten/[email protected]

    ## @mysten/[email protected]

    ### Patch Changes

    - b3f3925: Introduces @mysten/kms which initially exposes a Sui AWS KMS
    signer
    -   Updated dependencies [c24814b]
        -   @mysten/[email protected]

    ## @mysten/[email protected]

    ### Patch Changes

    -   Updated dependencies [c24814b]
        -   @mysten/[email protected]

    ## @mysten/[email protected]

    ### Patch Changes

    -   Updated dependencies [c24814b]
        -   @mysten/[email protected]

    ## @mysten/[email protected]

    ### Patch Changes

    -   Updated dependencies [c24814b]
        -   @mysten/[email protected]

    ## @mysten/[email protected]

    ### Patch Changes

    -   Updated dependencies [c24814b]
        -   @mysten/[email protected]
        -   @mysten/[email protected]

    Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

commit 093954a09bce1942a9f9773b845f59bf5124efbf
Author: ronny-mysten <[email protected]>
Date:   Thu Oct 31 11:35:27 2024 -0600

    [docs] Rpc update (#20094)

    ## Description

    Updates the RPC render to include component schemas. Also updates a few
    comments.

    ## Test plan

    How did you test the new or updated feature?

    ---

    ## Release notes

    Check each box that your changes affect. If none of the boxes relate to
    your changes, release notes aren't required.

    For each box you select, include information after the relevant heading
    that describes the impact of your changes that a user might notice and
    any actions they must take to implement updates.

    - [ ] Protocol:
    - [ ] Nodes (Validators and Full nodes):
    - [ ] Indexer:
    - [ ] JSON-RPC:
    - [ ] GraphQL:
    - [ ] CLI:
    - [ ] Rust SDK:
    - [ ] REST API:

commit fa66617b7dfa7096fd64a2dc1b002d154223be4a
Author: wlmyng <[email protected]>
Date:   Thu Oct 31 10:18:57 2024 -0700

    [mvr] renaming to sui-mvr-indexer and Dockerfile (#20099)

    ## Description

    Describe the changes or additions included in this PR.

    ## Test plan

    How did you test the new or updated feature?

    ---

    ## Release notes

    Check each box that your changes affect. If none of the boxes relate to
    your changes, release notes aren't required.

    For each box you select, include information after the relevant heading
    that describes the impact of your changes that a user might notice and
    any actions they must take to implement updates.

    - [ ] Protocol:
    - [ ] Nodes (Validators and Full nodes):
    - [ ] Indexer:
    - [ ] JSON-RPC:
    - [ ] GraphQL:
    - [ ] CLI:
    - [ ] Rust SDK:
    - [ ] REST API:

commit 434206f14a0a21bc62ea08480b4a5c4d36d6cc25
Author: Todd Nowacki <[email protected]>
Date:   Thu Oct 31 10:18:46 2024 -0700

    [move-compiler] Re-organize warning filters (#20103)

    ## Description

    - Re-organize warning filters. It should make it a bit cleaner to
    refactor diagnostic reporting

    ## Test plan

    - ran tests

    ---

    ## Release notes

    Check each box that your changes affect. If none of the boxes relate to
    your changes, release notes aren't required.

    For each box you select, include information after the relevant heading
    that describes the impact of your changes that a user might notice and
    any actions they must take to implement updates.

    - [ ] Protocol:
    - [ ] Nodes (Validators and Full nodes):
    - [ ] Indexer:
    - [ ] JSON-RPC:
    - [ ] GraphQL:
    - [ ] CLI:
    - [ ] Rust SDK:
    - [ ] REST API:

commit 0d16b6f8b7c4ff79f70400e8a94d7486b2b2c3fe
Author: wlmyng <[email protected]>
Date:   Thu Oct 31 09:36:37 2024 -0700

    [mvr] new crate for custom mvr-indexer, copied from sui-indexer crate (#20098)

    ## Description

    Simple copy-paste of the sui-indexer crate for the custom mvr indexer

    ## Test plan

    How did you test the new or updated feature?

    ---

    ## Release notes

    Check each box that your changes affect. If none of the boxes relate to
    your changes, release notes aren't required.

    For each box you select, include information after the relevant heading
    that describes the impact of your changes that a user might notice and
    any actions they must take to implement updates.

    - [ ] Protocol:
    - [ ] Nodes (Validators and Full nodes):
    - [ ] Indexer:
    - [ ] JSON-RPC:
    - [ ] GraphQL:
    - [ ] CLI:
    - [ ] Rust SDK:
    - [ ] REST API:

commit b3f39253ff0153b69b42bc298d1ecdbb0524e276
Author: John Naulty Jr. <[email protected]>
Date:   Thu Oct 31 09:21:29 2024 -0700

    AWS KMS TS Signer (#16139)

    ## Description

    Describe the changes or additions included in this PR.

    ## Test Plan

    How did you test the new or updated feature?

    ---
    If your changes are not user-facing and do not break anything, you can
    skip the following section. Otherwise, please briefly describe what has
    changed under the Release Notes section.

    ### Type of Change (Check all that apply)

    - [ ] protocol change
    - [ ] user-visible impact
    - [ ] breaking change for a client SDKs
    - [ ] breaking change for FNs (FN binary must upgrade)
    - [ ] breaking change for validators or node operators (must upgrade
    binaries)
    - [ ] breaking change for on-chain data layout
    - [ ] necessitate either a data wipe or data migration

    ### Release notes

    ---------

    Co-authored-by: Manolis Liolios <[email protected]>

commit f78c7b006c28c4c1afd3f7d42e21e53970902f5d
Author: Brandon Williams <[email protected]>
Date:   Wed Oct 30 19:50:45 2024 -0500

    rest: revert client change to requested protobuf and request bcs instead

    Revert the client change to request protobuf as a response type, and
    instead request bcs, because the support for protobuf hasn't rolled out
    to other networks yet.

commit 108ad4c8ea53c68bd90d0b3fcdc88fbc75640c61
Author: Brandon Williams <[email protected]>
Date:   Wed Oct 30 12:50:43 2024 -0500

    rest: support protobuf format for committee endpoints

commit 718a244bca584382faedde56f8247c6b26225b07
Author: Brandon Williams <[email protected]>
Date:   Wed Oct 30 11:03:43 2024 -0500

    rest: support protobuf format for resolve, simulate, and execute transaction endpoints

commit 2987f5967ecffa8ddbec2b8ebf9ee2eec09dad60
Author: Brandon Williams <[email protected]>
Date:   Wed Oct 30 08:55:51 2024 -0500

    rest: support protobuf format for GetTransaction and ListTransactions endpoints

commit 672ecbd2a58c7514aa24d139c442316a03d7f89e
Author: Brandon Williams <[email protected]>
Date:   Tue Oct 29 21:30:42 2024 -0500

    rest: support protobuf format for GetObject endpoints

commit c8472c057523ff88040ae688f1410d34164b0ebc
Author: Brandon Williams <[email protected]>
Date:   Tue Oct 29 19:00:39 2024 -0500

    rest: remove ListFullCheckpoint api

    Remove the ListFullCheckpoint api as its a bit too heavy-weight and
    instead we'll look to introduce a streaming api in the future.

commit 39f1ae21fd5467e71522729519720d2add9468dc
Author: Xun Li <[email protected]>
Date:   Wed Oct 30 22:25:50 2024 -0700

    Clean up local execution (#20086)

    ## Description

    Describe the changes or additions included in this PR.

    ## Test plan

    How did you test the new or updated feature?

    ---

    ## Release notes

    Check each box that your changes affect. If none of the boxes relate to
    your changes, release notes aren't required.

    For each box you select, include information after the relevant heading
    that describes the impact of your changes that a user might notice and
    any actions they must take to implement updates.

    - [ ] Protocol:
    - [ ] Nodes (Validators and Full nodes):
    - [ ] Indexer:
    - [ ] JSON-RPC:
    - [ ] GraphQL:
    - [ ] CLI:
    - [ ] Rust SDK:
    - [ ] REST API:

commit 65b02e203ad897f438f87d0e5cfe46c56a1d4247
Author: phoenix <[email protected]>
Date:   Wed Oct 30 21:26:54 2024 -0400

    [pruner] enable periodic compaction of individual SST files by default (#20101)

    ## Description
    enables periodic compaction of individual SST files by default

    ---

    ## Release notes

    Check each box that your changes affect. If none of the boxes relate to
    your changes, release notes aren't required.

    For each box you select, include information after the relevant heading
    that describes the impact of your changes that a user might notice and
    any actions they must take to implement updates.

    - [ ] Protocol:
    - [ ] Nodes (Validators and Full nodes):
    - [ ] Indexer:
    - [ ] JSON-RPC:
    - [ ] GraphQL:
    - [ ] CLI:
    - [ ] Rust SDK:
    - [ ] REST API:

commit f2670718f694edd59e8a062bc932d4efd96ba765
Author: Tim Zakian <[email protected]>
Date:   Wed Oct 30 16:49:01 2024 -0700

    [execution] Update to deps-only mode  (#20102)

    ## Description

    Disallow new modules from being added to packages in deps-only mode.

    ## Test plan

    Added new tests to check for both previous and new behavior.

    ---

    ## Release notes

    Check each box that your changes affect. If none of the boxes relate to
    your changes, release notes aren't required.

    For each box you select, include information after the relevant heading
    that describes the impact of your changes that a user might notice and
    any actions they must take to implement updates.

    - [X] Protocol: Updated restrictions on deps-only packages to not allow
    adding new modules to the package.
    - [ ] Nodes (Validators and Full nodes):
    - [ ] Indexer:
    - [ ] JSON-RPC:
    - [ ] GraphQL:
    - [ ] CLI:
    - [ ] Rust SDK:
    - [ ] REST API:

commit 7e6319fa33b7021a94ff7dcc07e479a4bae92b55
Author: Adam Welc <[email protected]>
Date:   Wed Oct 30 15:51:37 2024 -0700

    [trace-view] Added support for native functions and global locations (#20080)

    ## Description

    Adds support for native functions where the `CloseFrame` event happens
    right after `OpenFrame` event and both need to be skipped to skip over
    the native function. Also added support for handling global locations -
    they should be parsed properly but largely ignored in the trace as they
    cannot directly affect values of local variables

    ## Test plan

    All old and new tests must pass

commit 21639a51847f1a04b6b221ac77df3875c0430be7
Author: Xun Li <[email protected]>
Date:   Wed Oct 30 14:30:59 2024 -0700

    [RPC] Fix coin metadata api inefficiency (#19794)

    ## Description

    This PR fixes a minor inefficiency in the coin metadata api.
    It does not need to fetch the transaction, as it only needs the effects.
    This likely avoids a remote KV fetch.

    ## Test plan

    CI

    ---

    ## Release notes

    Check each box that your changes affect. If none of the boxes relate to
    your changes, release notes aren't required.

    For each box you select, include information after the relevant heading
    that describes the impact of your changes that a user might notice and
    any actions they must take to implement updates.

    - [ ] Protocol:
    - [ ] Nodes (Validators and Full nodes):
    - [ ] Indexer:
    - [ ] JSON-RPC:
    - [ ] GraphQL:
    - [ ] CLI:
    - [ ] Rust SDK:
    - [ ] REST API:

comm…
  • Loading branch information
@cmevton
cmevton committed Nov 14, 2024
1 parent 32514d1 commit 4125e40
Show file tree
Hide file tree
Showing 38 changed files with 1,348 additions and 1,161 deletions.
507 changes: 259 additions & 248 deletions Cargo.lock
View file

Large diffs are not rendered by default.

16 changes: 7 additions & 9 deletions Cargo.toml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -209,7 +209,7 @@ members = [

[workspace.package]
# This version string will be inherited by sui-core, sui-faucet, sui-node, sui-tools, sui-sdk, sui-move-build, and sui crates.
version = "1.37.1"
version = "1.37.3"

[profile.release]
# debug = 1 means line charts only, which is minimum needed for good stack traces
Expand Down Expand Up @@ -262,7 +262,6 @@ async-graphql = "=7.0.1"
async-graphql-axum = "=7.0.1"
async-graphql-value = "=7.0.1"
async-recursion = "1.0.4"
async-stream = "0.3.6"
async-trait = "0.1.61"
atomic_float = "0.1"
aws-config = "0.56"
Expand Down Expand Up @@ -328,10 +327,9 @@ cynic-codegen = "= 3.7.3"
dashmap = "5.5.3"
# datatest-stable = "0.1.2"
datatest-stable = { git = "https://github.com/nextest-rs/datatest-stable.git", rev = "72db7f6d1bbe36a5407e96b9488a581f763e106f" }
derivative = "2.2.0"
derive-syn-parse = "0.1.5"
derive_builder = "0.12.0"
derive_more = "0.99.17"
derive_more = "1.0.0"
diesel = "2.2"
diesel_migrations = "2.2"
diesel-async = "0.5"
Expand Down Expand Up @@ -583,17 +581,17 @@ move-abstract-interpreter = { path = "external-crates/move/crates/move-abstract-
move-abstract-stack = { path = "external-crates/move/crates/move-abstract-stack" }
move-analyzer = { path = "external-crates/move/crates/move-analyzer" }

mamoru-sniffer = { git = "https://github.com/Mamoru-Foundation/mamoru-core", rev = "d0c8237f85ccedc286ffd12ec260abc384146383" }
mamoru-sui-types = { git = "https://github.com/Mamoru-Foundation/mamoru-core", rev = "d0c8237f85ccedc286ffd12ec260abc384146383" }
#mamoru-sniffer = { path = "../mamoru-core/mamoru-sniffer" }
#mamoru-sui-types = { path = "../mamoru-core/blockchain-types/mamoru-sui-types" }

fastcrypto = { git = "https://github.com/MystenLabs/fastcrypto", rev = "2f502fd8570fe4e9cff36eea5bbd6fef22002898" }
fastcrypto-tbls = { git = "https://github.com/MystenLabs/fastcrypto", rev = "2f502fd8570fe4e9cff36eea5bbd6fef22002898" }
fastcrypto-zkp = { git = "https://github.com/MystenLabs/fastcrypto", rev = "2f502fd8570fe4e9cff36eea5bbd6fef22002898", package = "fastcrypto-zkp" }
fastcrypto-vdf = { git = "https://github.com/MystenLabs/fastcrypto", rev = "2f502fd8570fe4e9cff36eea5bbd6fef22002898", features = [
"experimental",
] }
mamoru-sniffer = { git = "https://github.com/Mamoru-Foundation/mamoru-core", rev = "d0c8237f85ccedc286ffd12ec260abc384146383" }
mamoru-sui-types = { git = "https://github.com/Mamoru-Foundation/mamoru-core", rev = "d0c8237f85ccedc286ffd12ec260abc384146383" }
#mamoru-sniffer = { path = "../mamoru-core/mamoru-sniffer" }
#mamoru-sui-types = { path = "../mamoru-core/blockchain-types/mamoru-sui-types" }

passkey-types = { version = "0.2.0" }
passkey-client = { version = "0.2.0" }
passkey-authenticator = { version = "0.2.0" }
Expand Down
28 changes: 5 additions & 23 deletions consensus/core/src/network/tonic_network.rs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,6 @@ use mysten_network::{
Multiaddr,
};
use parking_lot::RwLock;
use sui_tls::AllowPublicKeys;
use tokio::{
pin,
task::JoinSet,
Expand All @@ -45,6 +44,7 @@ use super::{
consensus_service_client::ConsensusServiceClient,
consensus_service_server::ConsensusService,
},
tonic_tls::create_rustls_client_config,
BlockStream, NetworkClient, NetworkManager, NetworkService,
};
use crate::{
Expand All @@ -54,7 +54,7 @@ use crate::{
error::{ConsensusError, ConsensusResult},
network::{
tonic_gen::consensus_service_server::ConsensusServiceServer,
tonic_tls::certificate_server_name,
tonic_tls::create_rustls_server_config,
},
CommitIndex, Round,
};
Expand Down Expand Up @@ -381,16 +381,7 @@ impl ChannelPool {
let address = format!("https://{address}");
let config = &self.context.parameters.tonic;
let buffer_size = config.connection_buffer_size;
let client_tls_config = sui_tls::create_rustls_client_config(
self.context
.committee
.authority(peer)
.network_key
.clone()
.into_inner(),
certificate_server_name(&self.context),
Some(network_keypair.private_key().into_inner()),
);
let client_tls_config = create_rustls_client_config(&self.context, network_keypair, peer);
let endpoint = tonic_rustls::Channel::from_shared(address.clone())
.unwrap()
.connect_timeout(timeout)
Expand Down Expand Up @@ -737,17 +728,8 @@ impl<S: NetworkService> NetworkManager<S> for TonicManager {
Arc::new(builder)
};

let tls_server_config = sui_tls::create_rustls_server_config(
self.network_keypair.clone().private_key().into_inner(),
certificate_server_name(&self.context),
AllowPublicKeys::new(
self.context
.committee
.authorities()
.map(|(_i, a)| a.network_key.clone().into_inner())
.collect(),
),
);
let tls_server_config =
create_rustls_server_config(&self.context, self.network_keypair.clone());
let tls_acceptor = TlsAcceptor::from(Arc::new(tls_server_config));

// Create listener to incoming connections.
Expand Down
58 changes: 57 additions & 1 deletion consensus/core/src/network/tonic_tls.rs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,63 @@
// SPDX-License-Identifier: Apache-2.0

use crate::context::Context;
use consensus_config::{AuthorityIndex, NetworkKeyPair};
use sui_tls::AllowPublicKeys;
use tokio_rustls::rustls::{ClientConfig, ServerConfig};

pub(crate) fn certificate_server_name(context: &Context) -> String {
pub(crate) fn create_rustls_server_config(
context: &Context,
network_keypair: NetworkKeyPair,
) -> ServerConfig {
let allower = AllowPublicKeys::new(
context
.committee
.authorities()
.map(|(_i, a)| a.network_key.clone().into_inner())
.collect(),
);
let verifier = sui_tls::ClientCertVerifier::new(allower, certificate_server_name(context));
// TODO: refactor to use key bytes
let self_signed_cert = sui_tls::SelfSignedCertificate::new(
network_keypair.private_key().into_inner(),
&certificate_server_name(context),
);
let tls_cert = self_signed_cert.rustls_certificate();
let tls_private_key = self_signed_cert.rustls_private_key();
let mut tls_config = verifier
.rustls_server_config(vec![tls_cert], tls_private_key)
.unwrap_or_else(|e| panic!("Failed to create TLS server config: {:?}", e));
tls_config.alpn_protocols = vec![b"h2".to_vec()];
tls_config
}

pub(crate) fn create_rustls_client_config(
context: &Context,
network_keypair: NetworkKeyPair,
target: AuthorityIndex,
) -> ClientConfig {
let target_public_key = context
.committee
.authority(target)
.network_key
.clone()
.into_inner();
let self_signed_cert = sui_tls::SelfSignedCertificate::new(
network_keypair.private_key().into_inner(),
&certificate_server_name(context),
);
let tls_cert = self_signed_cert.rustls_certificate();
let tls_private_key = self_signed_cert.rustls_private_key();
let mut tls_config =
sui_tls::ServerCertVerifier::new(target_public_key, certificate_server_name(context))
.rustls_client_config(vec![tls_cert], tls_private_key)
.unwrap_or_else(|e| panic!("Failed to create TLS client config: {:?}", e));
// ServerCertVerifier sets alpn for completeness, but alpn cannot be predefined when
// using HttpsConnector from hyper-rustls, as in TonicManager.
tls_config.alpn_protocols = vec![];
tls_config
}

fn certificate_server_name(context: &Context) -> String {
format!("consensus_epoch_{}", context.committee.epoch())
}
3 changes: 0 additions & 3 deletions crates/mysten-network/Cargo.toml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,6 @@ publish = false

[dependencies]
anemo.workspace = true
async-stream.workspace = true
bcs.workspace = true
bytes.workspace = true
eyre.workspace = true
Expand All @@ -19,10 +18,8 @@ multiaddr.workspace = true
serde.workspace = true
once_cell.workspace = true
snap.workspace = true
hyper-rustls.workspace = true
hyper-util.workspace = true
tokio = { workspace = true, features = ["sync", "rt", "macros"] }
tokio-rustls.workspace = true
tokio-stream.workspace = true
tonic.workspace = true
tonic-health.workspace = true
Expand Down
87 changes: 19 additions & 68 deletions crates/mysten-network/src/client.rs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -21,67 +21,53 @@ use std::{
vec,
};
use tokio::task::JoinHandle;
use tokio_rustls::rustls::ClientConfig;
use tonic::transport::{Channel, Endpoint, Uri};
use tower::Service;
use tracing::{info, trace};

pub async fn connect(address: &Multiaddr, tls_config: Option<ClientConfig>) -> Result<Channel> {
let channel = endpoint_from_multiaddr(address, tls_config)?
.connect()
.await?;
pub async fn connect(address: &Multiaddr) -> Result<Channel> {
let channel = endpoint_from_multiaddr(address)?.connect().await?;
Ok(channel)
}

pub fn connect_lazy(address: &Multiaddr, tls_config: Option<ClientConfig>) -> Result<Channel> {
let channel = endpoint_from_multiaddr(address, tls_config)?.connect_lazy();
pub fn connect_lazy(address: &Multiaddr) -> Result<Channel> {
let channel = endpoint_from_multiaddr(address)?.connect_lazy();
Ok(channel)
}

pub(crate) async fn connect_with_config(
address: &Multiaddr,
tls_config: Option<ClientConfig>,
config: &Config,
) -> Result<Channel> {
let channel = endpoint_from_multiaddr(address, tls_config)?
pub(crate) async fn connect_with_config(address: &Multiaddr, config: &Config) -> Result<Channel> {
let channel = endpoint_from_multiaddr(address)?
.apply_config(config)
.connect()
.await?;
Ok(channel)
}

pub(crate) fn connect_lazy_with_config(
address: &Multiaddr,
tls_config: Option<ClientConfig>,
config: &Config,
) -> Result<Channel> {
let channel = endpoint_from_multiaddr(address, tls_config)?
pub(crate) fn connect_lazy_with_config(address: &Multiaddr, config: &Config) -> Result<Channel> {
let channel = endpoint_from_multiaddr(address)?
.apply_config(config)
.connect_lazy();
Ok(channel)
}

fn endpoint_from_multiaddr(
addr: &Multiaddr,
tls_config: Option<ClientConfig>,
) -> Result<MyEndpoint> {
fn endpoint_from_multiaddr(addr: &Multiaddr) -> Result<MyEndpoint> {
let mut iter = addr.iter();

let channel = match iter.next().ok_or_else(|| eyre!("address is empty"))? {
Protocol::Dns(_) => {
let (dns_name, tcp_port, http_or_https) = parse_dns(addr)?;
let uri = format!("{http_or_https}://{dns_name}:{tcp_port}");
MyEndpoint::try_from_uri(uri, tls_config)?
MyEndpoint::try_from_uri(uri)?
}
Protocol::Ip4(_) => {
let (socket_addr, http_or_https) = parse_ip4(addr)?;
let uri = format!("{http_or_https}://{socket_addr}");
MyEndpoint::try_from_uri(uri, tls_config)?
MyEndpoint::try_from_uri(uri)?
}
Protocol::Ip6(_) => {
let (socket_addr, http_or_https) = parse_ip6(addr)?;
let uri = format!("{http_or_https}://{socket_addr}");
MyEndpoint::try_from_uri(uri, tls_config)?
MyEndpoint::try_from_uri(uri)?
}
unsupported => return Err(eyre!("unsupported protocol {unsupported}")),
};
Expand All @@ -91,25 +77,21 @@ fn endpoint_from_multiaddr(

struct MyEndpoint {
endpoint: Endpoint,
tls_config: Option<ClientConfig>,
}

static DISABLE_CACHING_RESOLVER: OnceCell<bool> = OnceCell::new();

impl MyEndpoint {
fn new(endpoint: Endpoint, tls_config: Option<ClientConfig>) -> Self {
Self {
endpoint,
tls_config,
}
fn new(endpoint: Endpoint) -> Self {
Self { endpoint }
}

fn try_from_uri(uri: String, tls_config: Option<ClientConfig>) -> Result<Self> {
fn try_from_uri(uri: String) -> Result<Self> {
let uri: Uri = uri
.parse()
.with_context(|| format!("unable to create Uri from '{uri}'"))?;
let endpoint = Endpoint::from(uri);
Ok(Self::new(endpoint, tls_config))
Ok(Self::new(endpoint))
}

fn apply_config(mut self, config: &Config) -> Self {
Expand All @@ -125,51 +107,20 @@ impl MyEndpoint {
});

if disable_caching_resolver {
if let Some(tls_config) = self.tls_config {
self.endpoint.connect_with_connector_lazy(
hyper_rustls::HttpsConnectorBuilder::new()
.with_tls_config(tls_config)
.https_only()
.enable_http2()
.build(),
)
} else {
self.endpoint.connect_lazy()
}
self.endpoint.connect_lazy()
} else {
let mut http = HttpConnector::new_with_resolver(CachingResolver::new());
http.enforce_http(false);
http.set_nodelay(true);
http.set_keepalive(None);
http.set_connect_timeout(None);

if let Some(tls_config) = self.tls_config {
let https = hyper_rustls::HttpsConnectorBuilder::new()
.with_tls_config(tls_config)
.https_only()
.enable_http1()
.wrap_connector(http);
self.endpoint.connect_with_connector_lazy(https)
} else {
self.endpoint.connect_with_connector_lazy(http)
}
self.endpoint.connect_with_connector_lazy(http)
}
}

async fn connect(self) -> Result<Channel> {
if let Some(tls_config) = self.tls_config {
let https_connector = hyper_rustls::HttpsConnectorBuilder::new()
.with_tls_config(tls_config)
.https_only()
.enable_http2()
.build();
self.endpoint
.connect_with_connector(https_connector)
.await
.map_err(Into::into)
} else {
self.endpoint.connect().await.map_err(Into::into)
}
self.endpoint.connect().await.map_err(Into::into)
}
}

Expand Down
19 changes: 5 additions & 14 deletions crates/mysten-network/src/config.rs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,6 @@ use crate::{
use eyre::Result;
use serde::{Deserialize, Serialize};
use std::time::Duration;
use tokio_rustls::rustls::ClientConfig;
use tonic::transport::Channel;

#[derive(Debug, Default, Deserialize, Serialize)]
Expand Down Expand Up @@ -91,19 +90,11 @@ impl Config {
ServerBuilder::from_config(self, metrics_provider)
}

pub async fn connect(
&self,
addr: &Multiaddr,
tls_config: Option<ClientConfig>,
) -> Result<Channel> {
connect_with_config(addr, tls_config, self).await
pub async fn connect(&self, addr: &Multiaddr) -> Result<Channel> {
connect_with_config(addr, self).await
}

pub fn connect_lazy(
&self,
addr: &Multiaddr,
tls_config: Option<ClientConfig>,
) -> Result<Channel> {
connect_lazy_with_config(addr, tls_config, self)
pub fn connect_lazy(&self, addr: &Multiaddr) -> Result<Channel> {
connect_lazy_with_config(addr, self)
}
}
}
Loading

0 comments on commit 4125e40

Please sign in to comment.