[mvr] Introduce mvr-mode on production indexer #20271
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…t set, and mvr-mode toggles so only subset of tables are written to
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
3 Skipped Deployments
|
wlmyng
temporarily deployed
to
sui-typescript-aws-kms-test-env
GitHub Actions
Inactive
wlmyng
temporarily deployed
to
sui-typescript-aws-kms-test-env
GitHub Actions
Inactive
gegaowp
reviewed
Nov 14, 2024
wlmyng
temporarily deployed
to
sui-typescript-aws-kms-test-env
GitHub Actions
Inactive
wlmyng
temporarily deployed
to
sui-typescript-aws-kms-test-env
GitHub Actions
Inactive
gegaowp
approved these changes
Nov 15, 2024
|
|
||
| // Check that objects_history is being correctly pruned. At epoch 3, we should only have data | ||
| // between 2 and 3 inclusive. | ||
| loop { |
wlmyng
temporarily deployed
to
sui-typescript-aws-kms-test-env
GitHub Actions
Inactive
wlmyng
temporarily deployed
to
sui-typescript-aws-kms-test-env
GitHub Actions
Inactive
wlmyng
added a commit
that referenced
this pull request
Nov 15, 2024
## Description introduce mvr-mode flag. If true, enables pruning objects_history, and toggles committer to write only a subset of tables to db. Compatible with the production indexer schema. ## Test plan How did you test the new or updated feature? --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API:
wlmyng
added a commit
that referenced
this pull request
Nov 15, 2024
## Description introduce mvr-mode flag. If true, enables pruning objects_history, and toggles committer to write only a subset of tables to db. Compatible with the production indexer schema. ## Test plan How did you test the new or updated feature? --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API:
wlmyng
added a commit
that referenced
this pull request
Nov 15, 2024
## Description introduce mvr-mode flag. If true, enables pruning objects_history, and toggles committer to write only a subset of tables to db. Compatible with the production indexer schema. ## Test plan How did you test the new or updated feature? --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API: ## Description Describe the changes or additions included in this PR. ## Test plan How did you test the new or updated feature? --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API:
wlmyng
added a commit
that referenced
this pull request
Nov 15, 2024
## Description introduce mvr-mode flag. If true, enables pruning objects_history, and toggles committer to write only a subset of tables to db. Compatible with the production indexer schema. ## Test plan How did you test the new or updated feature? --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API: ## Description Describe the changes or additions included in this PR. ## Test plan How did you test the new or updated feature? --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API:
carlosbmamoru
pushed a commit
to Mamoru-Foundation/sui
that referenced
this pull request
Nov 19, 2024
commit cd214a3d7e8e9e47ccf31ab8f5315e8c114313be Author: wlmyng <[email protected]> Date: Fri Nov 15 13:21:24 2024 -0800 [mvr] Introduce mvr-mode on production indexer (#20271) (#20288) ## Description introduce mvr-mode flag. If true, enables pruning objects_history, and toggles committer to write only a subset of tables to db. Compatible with the production indexer schema. ## Test plan How did you test the new or updated feature? --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API: ## Description Describe the changes or additions included in this PR. ## Test plan How did you test the new or updated feature? --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API: commit ab6863576baf7ff74f0de95ab5abef9040fc8fcd Author: Eugene Boguslavsky <[email protected]> Date: Thu Nov 14 18:41:02 2024 -0800 Sui `v1.38.0` Framework Bytecode Snapshot (#20277) ## Description Sui `v1.38.0` Framework Bytecode Snapshot ## Test plan 👀 commit 5d70ec6f5d4b7ff611649432aeb8c538efb63e80 Author: Sadhan Sood <[email protected]> Date: Thu Nov 14 17:08:27 2024 -0800 feat(analytics): Add package filter to object handler (#20272) ## Description This PR adds a package filter for object handler in analytics for being able to generate an object table specifically for walrus in testnet. ## Test Invoking with `--package-id-filter 0x9f992cc2430a1f442ca7a5ca7638169f5d5c00e0ebc3977a65e9ac6e497fe5ef --starting-checkpoint-seq-num 120132840` to filter for walrus specific objects. Here is the walrus package object: https://testnet.suivision.xyz/package/0x9f992cc2430a1f442ca7a5ca7638169f5d5c00e0ebc3977a65e9ac6e497fe5ef commit b36024d32ce349ba9465b38bd6706b8b2068ee67 Author: stefan-mysten <[email protected]> Date: Thu Nov 14 16:27:12 2024 -0800 [CLI] Add back the warning about version mismatch (#20273) ## Description #20259 removed the warning for the SDK, so this PR adds it back for the CLI. ## Test plan How did you test the new or updated feature? --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API: commit 81f75843464ce1140040403df0a0fa0852f87386 Author: Adam Welc <[email protected]> Date: Thu Nov 14 13:01:50 2024 -0800 [move-compiler] Added function location info to the source map (#20261) ## Description This PR adds additional field to the source map describing the location of the entire function. This is needed to enable better handling of macros in the trace viewer (we need to know which instructions in a given function are outside of a function body to identify those that belong to inlined macro functions). ## Test plan All existing tests must pass commit 8d90acd980b4490a7f5488cee2aad465abe8e663 Author: Tim Zakian <[email protected]> Date: Thu Nov 14 11:53:41 2024 -0800 [json-rpc] Add enums in `getNormalizedMoveModule` (#20231) ## Description Adds enums to the `NormalizedMoveModule` for json-rpc. This fixes #20205 ## Test plan Tested manually on the repro provided in the issue to make sure enums are included in the output. --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [x] JSON-RPC: Fixes an issue where enum types were not included in the output of `getNormalizedMoveModule` and `getNormalizedMoveModuleForPackage` - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API: commit 53dd080a841710c91f492de17f7312a29662235f Author: William Smith <[email protected]> Date: Thu Nov 14 14:03:22 2024 -0500 [TrafficControl] Add salt to TrafficSketch (#20245) ## Description False positives stem from IP collisions during traffic tallying. Because validators tend to see closely correlated transaction execution traffic patterns, it is possible that a false positive resulting in an erroneously blocked IP could be correlated across many validators at the same time. If this happens on a quorum of validators, this would result in an IP being effectively denied service until the TTL expires. Luckily, we can exploit this property to lower the effective false positive rate (as observed by the client) by removing this correlation. This PR adds a salt key to the `SketchKey`. The value is generated in memory at startup. ## Test plan Existing tests --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API: commit 97838b20eb28b5abd12d275f5924c70c97b86e2c Author: sui-merge-bot[bot] <114704316+sui-merge-bot[bot]@users.noreply.github.com> Date: Thu Nov 14 10:25:13 2024 -0800 Version Packages (#20269) This PR was opened by the [Changesets release](https://github.com/changesets/action) GitHub action. When you're ready to do a release, you can merge this and publish to npm yourself or [setup this action to publish automatically](https://github.com/changesets/action#with-publishing). If you're not ready to do a release yet, that's fine, whenever you add more changesets to main, this PR will be updated. # Releases ## @mysten/[email protected] ### Patch Changes - @mysten/[email protected] ## @mysten/[email protected] ### Patch Changes - Updated dependencies [1765780] - @mysten/[email protected] ## @mysten/[email protected] ### Patch Changes - 1765780: fix zksend link network detection Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> commit 176578020ba15ff321d42b487f87af0a650bece3 Author: Pavlos Chrysochoidis <[email protected]> Date: Thu Nov 14 18:10:02 2024 +0000 zksend sdk fix network detection (#20268) ## Description Describe the changes or additions included in this PR. ## Test plan How did you test the new or updated feature? --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API: commit 28e184a694636fc3b5a589c219f47b549250d6c1 Author: xiaodi <[email protected]> Date: Fri Nov 15 00:24:24 2024 +0800 Update regulated.mdx (#20267) ## Description Replaced "Regulated Coin and Deny List" with "Loyalty Tokens" in documentation. ## Test plan How did you test the new or updated feature? --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API: commit 0bd58999ae2be185c6677ebc4a44b3e5dcd15296 Author: sui-merge-bot[bot] <114704316+sui-merge-bot[bot]@users.noreply.github.com> Date: Thu Nov 14 09:18:08 2024 -0500 Version Packages (#20265) This PR was opened by the [Changesets release](https://github.com/changesets/action) GitHub action. When you're ready to do a release, you can merge this and publish to npm yourself or [setup this action to publish automatically](https://github.com/changesets/action#with-publishing). If you're not ready to do a release yet, that's fine, whenever you add more changesets to main, this PR will be updated. # Releases ## @mysten/[email protected] ### Patch Changes - 4a42691: NS token pools added Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> commit 4a426918ef263103020aca25cf05809e4531ec51 Author: Tony Lee <[email protected]> Date: Thu Nov 14 08:26:36 2024 -0500 NS Deepbook Pools (#20256) commit d5d51aed177432e62950385336b79f8395c940f9 Author: stefan-mysten <[email protected]> Date: Wed Nov 13 17:17:13 2024 -0800 [RustSDK] Less noise when rpc version and local CLI version mismatch (#20259) ## Description This PR removes the SDK tracing warning on mismatch local + RPC API versions. ## Test plan How did you test the new or updated feature? --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API: commit c85e7c07c0713c1edf7cdb59eab46372ef29fe24 Author: Tim Zakian <[email protected]> Date: Wed Nov 13 17:16:49 2024 -0800 [docs] Update Sui CLI install instructions to include the tracing feature (#20255) ## Description Update docs for installing the Sui CLI to include `--features tracing` so that they work with the trace viewer/debugger. ## Test plan 👀 commit 9dfd26a723e5efdb5816d6c2492cb0bb27b2f2c5 Author: Eugene Boguslavsky <[email protected]> Date: Wed Nov 13 15:33:46 2024 -0800 Use tracing feature for sui builds (#20260) ## Description Use tracing feature for sui builds ## Test plan 👀 commit 047de09d84bf90b633fd183b0cecef38e9387317 Author: Xun Li <[email protected]> Date: Wed Nov 13 14:30:23 2024 -0800 [indexer-alt] Add processor lag metrics (#20235) ## Description Describe the changes or additions included in this PR. ## Test plan How did you test the new or updated feature? --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API: commit c03132d75e80b569c2fd8447539ac0b7bddc8ecc Author: Andrew Schran <[email protected]> Date: Wed Nov 13 16:51:15 2024 -0500 Add support for TLS connections with self-signed cert on validator gRPC interface (#20238) Client side use of TLS will be enabled-by-default in a future PR. This has corrected TLS upgrade behavior compared to older PR #19796. --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [x] Nodes (Validators and Full nodes): Adds support for TLS on validator gRPC interface. - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API: commit 313bf8b43a1b8b511cb9b012a23d5803444b3ea5 Author: Tim Zakian <[email protected]> Date: Wed Nov 13 13:16:01 2024 -0800 [move] Add Move 2024 (non-beta) edition and make it the default edition (#20242) ## Description Add Move `2024` edition and make it the default edition. ## Test plan Updated existing tests. --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [X] CLI: Added the Move `2024` edition, and set it as the default Move edition both when creating a new Move package and when migrating from an existing package. Existing `2024.beta` and `2024.alpha` editions remain supported. - [ ] Rust SDK: - [ ] REST API: commit 8f21cd702b8ec9f4796f317aa9ec6bfd56330f0e Author: phoenix <[email protected]> Date: Wed Nov 13 15:26:23 2024 -0500 [data ingestion] bigtable: set env credentials parameter (#20240) ## Description Pass filename with credential parameters to the ingestion binary --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API: commit 3c444d5dada018e5b10a929efa6f17e3445367d5 Author: Adam Welc <[email protected]> Date: Wed Nov 13 11:04:35 2024 -0800 [move-ide] Fixed auto-completion on the dot character (#20175) ## Description This PR fixes a certain problem with auto-completion on the `.` character. Previously, once a part of the access chain after the `.` was parsed, auto-completion information for the `.` character location was no longer generated. This is particularly problematic if someone expects auto-completion in the middle of the function code, for example: ```move public struct SomeStruct has drop { some_field: u64, } public fun foo(_param: SomeStruct) {} public fun bar(): SomeStruct { let val = SomeStruct{some_field: 42}; val. SomeStruct {some_field: 42} } ``` It is expected that auto-completion would work correctly after `val.` but previously it did not because this expression was actually parsed as `val.SomeStruct`. This PR fixes this:  Previously, auto-completion for the `.` location would also not work if what follows the `.` is a function call (whether the function being called was resolved correctly or not). It now works correctly in both cases:   ## Test plan All new and (updated) all tests must pass commit 8d647c0f00809bba411079ce765fdb4fbecef00c Author: sui-merge-bot[bot] <114704316+sui-merge-bot[bot]@users.noreply.github.com> Date: Wed Nov 13 18:49:45 2024 +0000 Version Packages (#20252) This PR was opened by the [Changesets release](https://github.com/changesets/action) GitHub action. When you're ready to do a release, you can merge this and publish to npm yourself or [setup this action to publish automatically](https://github.com/changesets/action#with-publishing). If you're not ready to do a release yet, that's fine, whenever you add more changesets to main, this PR will be updated. # Releases ## @mysten/[email protected] ### Minor Changes - 60bdb62: Update graphql query and use stashed host for claim api - 364a93a: Remove decoding spaces to `+` in zksend wallet url hashes - 364a93a: Add claimedBy property that is set when claimedAssets are loaded ## @mysten/[email protected] ### Patch Changes - @mysten/[email protected] ## @mysten/[email protected] ### Patch Changes - Updated dependencies [60bdb62] - Updated dependencies [364a93a] - Updated dependencies [364a93a] - @mysten/[email protected] Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> commit 8117d8393f25fa85390094bca61952ab24c9b595 Author: Tim Zakian <[email protected]> Date: Wed Nov 13 10:38:18 2024 -0800 [move/tracing] Panic unit tests if tracing without being compiled with tracing support (#20253) ## Description Make sure that if the `--trace-execution` flag is passed that the binary was compiled with the `tracing` feature flag set. ## Test plan CI --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API: commit 60bdb62627a821bb54b7c7415a63193b7395a05d Author: hayes-mysten <[email protected]> Date: Wed Nov 13 10:08:56 2024 -0800 [zksend] Update garphql query, and use getstashed.com as claim api host (#20237) ## Description fixes #20222 Describe the changes or additions included in this PR. ## Test plan How did you test the new or updated feature? --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API: commit 364a93a1b10be70aabe0fb8937b3c1af610d1eba Author: hayes-mysten <[email protected]> Date: Wed Nov 13 10:00:16 2024 -0800 [zksend] add claimedBy property when loading assets of claimed link (#20244) ## Description Describe the changes or additions included in this PR. ## Test plan How did you test the new or updated feature? --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API: commit 43d7c3dcfcdf062cad98b480f0c0a62e218026e9 Author: Xun Li <[email protected]> Date: Wed Nov 13 09:14:22 2024 -0800 [indexer-alt] Skip commit if batch is empty (#20124) ## Description If we did not collect anything to the batch, we should not commit to the DB. We should also avoid reporting any metrics. ## Test plan CI --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API: --------- Co-authored-by: Ashok Menon <[email protected]> commit f04e4cddfdde35a5d5cba9c7fd7c0f9482d4193b Author: sui-merge-bot[bot] <114704316+sui-merge-bot[bot]@users.noreply.github.com> Date: Wed Nov 13 11:07:19 2024 -0500 Version Packages (#20239) This PR was opened by the [Changesets release](https://github.com/changesets/action) GitHub action. When you're ready to do a release, you can merge this and publish to npm yourself or [setup this action to publish automatically](https://github.com/changesets/action#with-publishing). If you're not ready to do a release yet, that's fine, whenever you add more changesets to main, this PR will be updated. # Releases ## @mysten/[email protected] ### Patch Changes - Updated dependencies [d5a23d7] - @mysten/[email protected] - @mysten/[email protected] ## @mysten/[email protected] ### Patch Changes - Updated dependencies [d5a23d7] - @mysten/[email protected] - @mysten/[email protected] - @mysten/[email protected] ## @mysten/[email protected] ### Patch Changes - Updated dependencies [d5a23d7] - @mysten/[email protected] ## @mysten/[email protected] ### Patch Changes - a43ae92: NS Token - Updated dependencies [d5a23d7] - @mysten/[email protected] ## @mysten/[email protected] ### Patch Changes - Updated dependencies [d5a23d7] - @mysten/[email protected] - @mysten/[email protected] ## @mysten/[email protected] ### Patch Changes - Updated dependencies [d5a23d7] - @mysten/[email protected] ## @mysten/[email protected] ### Patch Changes - Updated dependencies [d5a23d7] - @mysten/[email protected] ## @mysten/[email protected] ### Patch Changes - Updated dependencies [d5a23d7] - @mysten/[email protected] ## @mysten/[email protected] ### Patch Changes - Updated dependencies [d5a23d7] - @mysten/[email protected] ## @mysten/[email protected] ### Patch Changes - d5a23d7: Add tx.object.option for creatnig object options in transaction builder ## @mysten/[email protected] ### Patch Changes - Updated dependencies [d5a23d7] - @mysten/[email protected] ## @mysten/[email protected] ### Patch Changes - Updated dependencies [d5a23d7] - @mysten/[email protected] ## @mysten/[email protected] ### Patch Changes - Updated dependencies [d5a23d7] - @mysten/[email protected] - @mysten/[email protected] Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> commit a43ae921386d1e8406f81d73d2c317ad92bb211e Author: Tony Lee <[email protected]> Date: Wed Nov 13 10:38:16 2024 -0500 NS token in deepbook sdk (#20251) ## Description NS token in deepbook sdk ## Test plan How did you test the new or updated feature? Testnet and local ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API: commit 91dec7c02d776f3e68a5a1e5b86851c7d289c206 Author: Ashok Menon <[email protected]> Date: Wed Nov 13 14:00:52 2024 +0000 indexer-alt: remove watermarks.epoch_lo (#20218) ## Description This field turns out not to be used by the new pruner implementation, because it is entirely based on checkpoints. ## Test plan CI ## Stack - #20149 - #20150 - #20166 - #20216 - #20217 --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API: commit c5f275ea9fab4a099e998a7b4f65926fe1ae8e8e Author: Ashok Menon <[email protected]> Date: Mon Nov 11 02:43:06 2024 +0000 indexer-alt: pruner task ## Description Add the task that actually deletes data, based on the reader low watermark. ## Test plan Run the indexer and note the following: - Metrics related to deleted rows by the pruner (from `localhost:9184/metrics`) - The contents of the `watermarks` table. ``` sui$ cargo run -p sui-indexer-alt --release -- \ --database-url "postgres://postgres:postgrespw@localhost:5432/sui_indexer_alt" \ indexer --remote-store-url https://checkpoints.mainnet.sui.io \ --last-checkpoint 10000 \ --consistent-range 100 --consistent-pruning-interval 10 \ --pipeline sum_obj_types --pipeline wal_obj_types ``` Also tested running the indexer for an extended period of time (1M checkpoints over roughly half an hour in local testing), and noted how the pruner behaves. When configured as it would be in production (roughly one hour of consistent range, and a 5 minute pruning interval and a 2 minute pruning delay): - Many rows accumulated during backfill -- by the end of the 1M checkpoints, the pruner had only pruned up to between checkpoint 500K and checkpoint 700K depending on the pipeline. This should not be an issue under normal operation where the indexer will run for long enough for pruning to stabilise at the tip of the network (and it would be recommended practice to start from formal snapshot and therefore only need to run pruning from that point forward). - Because the reader watermark task and the pruner task use the same interval, it can take up to two ticks of that interval for the pruner to act on a change to its upperbound -- again, it should be okay, as the pruner's interval should be at least an order of magnitude smaller than its retention period. commit 3a2e332c68cfe9491eb56d09f087ff8cc559e6e4 Author: Ashok Menon <[email protected]> Date: Mon Nov 11 02:37:06 2024 +0000 fix(indexer-alt): overly chatty loud watermark updates ## Description The loud watermark update logic always bumped the previous update watermark by the same interval. When the indexer was running well ahead of the loud watermark update rate, this would cause many updates to be issued successively. This change makes it so that the next update is always `LOUD_WATERMARK_UPDATE_INTERVAL` away from the last loud update. ## Test plan Run the indexer. Previously updates -- particularly from summary tables that gathered changes up and write them out in big batches -- would come in bursts, and this behaviour is no longer apparent after the change: ``` sui$ cargo run -p sui-indexer --release -- \ --database-url "postgres://postgres:postgrespw@localhost:5432/sui_indexer_alt" \ indexer --remote-store-url https://checkpoints.mainnet.sui.io \ --last-checkpoint 1000000 --consistent-range 3600 ``` commit 6d31be68d0ef82f90aa0447e13fd4c82eb40f732 Author: Ashok Menon <[email protected]> Date: Sun Nov 10 17:19:53 2024 +0000 indexer-alt: reader watermark task ## Description The reader watermark task is one half of the pruning implementation. It applies only to concurrent pipelines, and tracks the lowest checkpoint sequence number the reader is guaranteed to get data for, and the timestamp (from the DB), that this threshold was set at. This task does not do any actual pruning, but it sets the limit that the pruner can go to, and a time at which it can start pruning to that point. A later PR will introduce the actual pruning task. Unlike the existing pruner implementation, this system does not support pruning by epoch, only by checkpoint. This was for two reasons: - Pruning no longer works by dropping partitions, so we can keep the implementation simple by only supporting the form of pruning we need (by checkpoint sequence number). - Pruning is part of the indexing framework, rather than a specific part of our indexer implementation, so pruning by epoch would require that the indexing framework kept track of a mapping from epoch to checkpoint sequence number. Along with the reader watermark, the following changes were made: - The existing watermark task was renamed to the `committer_watermark` to distinguish it. - The interface between pipelines and the indexing framework has been simplified to return just one `JoinHandle`. This was because clippy complained that returning 6 handles was too complex, and additionally, because the reader watermark task is not connected to the rest of the tasks for a pipeline, it needs a more complicated tear down logic, where firstly the committer tasks are wound down, and then the pruner is explicitly cancelled, before its tasks are awaited as well. The change also sets up pruning configs for write-ahead log tables, so that we can test reader watermark behaviour. ## Test plan Run the indexer with just one of the consistent read summary and write-ahead logs, and then query the watermarks table: ``` cargo run -p sui-indexer-alt --release -- \ --database-url "postgres://postgres:postgrespw@localhost:5432/sui_indexer_alt" \ indexer --remote-store-url https://checkpoints.mainnet.sui.io \ --last-checkpoint 10000 \ --consistent-range 100 --consistent-pruning-interval 10 \ --pipeline sum_obj_types --pipeline wal_obj_types ``` ``` sui_indexer_alt=# SELECT * FROM watermarks; pipeline | epoch_hi_inclusive | checkpoint_hi_inclusive | tx_hi | timestamp_ms_hi_inclusive | epoch_lo | reader_lo | pruner_timestamp | pruner_hi ---------------+--------------------+-------------------------+-------+---------------------------+----------+-----------+----------------------------+----------- sum_obj_types | 1 | 9900 | 9902 | 1681405380769 | 0 | 0 | 1970-01-01 00:00:00 | 0 wal_obj_types | 1 | 10000 | 10002 | 1681405504878 | 0 | 6321 | 2024-11-10 17:33:11.459593 | 0 (2 rows) ``` Note that `reader_lo` is not guaranteed to be as advanced as it could be, because this task checks in relatively infrequently, and the indexer as a whole might shut down efore it has a chance to catch up. However, on restart, the indexer should fast-forward the watermark (i.e. if the steps above are repeated, `reader_lo` will be updated): ``` sui_indexer_alt=# SELECT * FROM watermarks; pipeline | epoch_hi_inclusive | checkpoint_hi_inclusive | tx_hi | timestamp_ms_hi_inclusive | epoch_lo | reader_lo | pruner_timestamp | pruner_hi ---------------+--------------------+-------------------------+-------+---------------------------+----------+-----------+----------------------------+----------- sum_obj_types | 1 | 9900 | 9902 | 1681405380769 | 0 | 0 | 1970-01-01 00:00:00 | 0 wal_obj_types | 1 | 10000 | 10002 | 1681405504878 | 0 | 9801 | 2024-11-10 17:33:49.913129 | 0 (2 rows) ``` commit 81474ef2fb2bc2e1ae2a39bbf599e2493504ab1d Author: Ashok Menon <[email protected]> Date: Sun Nov 10 17:15:47 2024 +0000 indexer-alt: watermarks.pruner_timestamp is a TIMESTAMP ## Description Change `watermark.pruner_timestamp_ms BIGINT` to `watermark.pruner_timestamp TIMESTAMP`. This is because unlike the other timestamps in this table, which are read from checkpoint data, this timestamp is read from the database, and so it is easiest to represent it using the database's native type for timestamps. ## Test plan CI commit 1d568438c5ca3806946d93900e1561b31d06daa3 Author: Ashok Menon <[email protected]> Date: Wed Nov 13 13:54:12 2024 +0000 chore(indexer-alt): rename tx_calls_fun -> tx_calls (#20166) ## Description I partially simplified the naming of `tx_calls_fun` to `tx_calls` but did not do a complete job -- this change completes it. ## Test plan CI ## Stack - #20118 - #20132 - #20147 - #20148 - #20149 - #20150 --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API: commit 85c0f810e907cabd503a28c2e134e8af282351e8 Author: Ashok Menon <[email protected]> Date: Wed Nov 13 13:53:31 2024 +0000 indexer-alt: epochs pipelines (#20150) ## Description Adds two tables and pipelines: `kv_epoch_starts` and `kv_epoch_ends`, to index epoch information. This pipeline is different from epoch indexing in `sui-indexer` in a number of ways: - It is an append-only pipeline. The columns that are written at the start and end of the epoch are split into two separate tables that can be written to concurrently (by separate pipelines). - The first row of `kv_epoch_starts` is written by the bootstrap process which seeds the `kv_genesis` table as well, this avoids having to condition on whether the checkpoint being processed is the genesis checkpoint in the main pipeline. - Instead of indexing the number of transactions in the epoch, it tracks the transaction high watermark -- readers will need to read the records to calculate the number of total transactions (this avoids having to read the last epoch's total transactions in the write path). - We index the `SuiSystemState` object as BCS, rather than the summary structure. - We explicitly record whether the epoch advancement at the end of the epoch triggered safe mode (the system state object also tracks whether the epoch was started in safe mode). - Fields related to information that came from `SystemEpochInfoEvent` have all been consolidated in `kv_epoch_ends`, and they are all optional, in case of safe mode. It's worth elaborating on the last bullet point, because this is quite a subtle, but large change: - Today, `total_stake` and `storage_fund_balance` are written at the start of an epoch based on the fields of the `SystemEpochInfoEvent` emitted from the previous epoch, and are `NOT NULL`. - The remaining fields were nullable, but only because they would be written to later, once the epoch was over. This was awkward to work with in a couple of ways: - It meant that for the genesis epoch, we needed to make some numbers up (all zeroes) because we did not have an event to read from. - We had to do something similar if we hit safe mode. - When indexing the start and end of epochs separately, it meant that we had to duplicate work (finding the system epoch info event). By making the fields nullable, and consolidating them in `kv_epoch_ends`, we can simplify the pipelines: - `kv_epoch_starts` and the bootstrapping logic can work purely based on the system state object. - `kv_epoch_ends` can work purely based on the `SystemEpochInfoEvent`, and can leave fields `NULL` if we are in safe mode. In the case of `kv_epoch_starts` we could also have cut down fields to just `epoch`, `cp_lo` and `system_state`. I chose not to do this because the system state is actually quite a large object, and it is beneficial to avoid having to deserialize to answer simpler queries. ## Test plan Ran the indexer on the first 1M checkpoints, and correlated the resulting info in the respective tables from the data that the current indexer produced: ``` sui_indexer_alt=# SELECT epoch, protocol_version, cp_lo, TO_TIMESTAMP(start_timestamp_ms / 1000), reference_gas_price FROM kv_epoch_starts; epoch | protocol_version | cp_lo | to_timestamp | reference_gas_price -------+------------------+--------+------------------------+--------------------- 0 | 4 | 0 | 2023-04-12 18:00:00+01 | 1000 1 | 4 | 9770 | 2023-04-13 18:00:02+01 | 1000 2 | 4 | 85169 | 2023-04-14 18:00:04+01 | 1000 3 | 4 | 161192 | 2023-04-15 18:00:08+01 | 1000 4 | 4 | 237074 | 2023-04-16 18:00:11+01 | 1000 5 | 4 | 314160 | 2023-04-17 18:00:15+01 | 1000 6 | 4 | 391107 | 2023-04-18 18:00:18+01 | 1000 7 | 4 | 467716 | 2023-04-19 18:00:21+01 | 1000 8 | 4 | 544978 | 2023-04-20 18:00:26+01 | 1000 9 | 5 | 621933 | 2023-04-21 18:00:28+01 | 1000 10 | 6 | 699410 | 2023-04-22 18:00:31+01 | 1000 11 | 6 | 777074 | 2023-04-23 18:00:34+01 | 1000 12 | 6 | 855530 | 2023-04-24 18:00:36+01 | 1000 13 | 6 | 933559 | 2023-04-25 18:00:39+01 | 1000 (14 rows) sui_indexer_alt=# SELECT epoch, cp_hi, tx_hi, TO_TIMESTAMP(end_timestamp_ms / 1000), safe_mode, storage_fund_balance, storage_fund_reinvestment, storage_charge, storage_rebate, stake_subsidy_amount, total_gas_fees, total_stake_rewards_distributed, leftover_storage_fund_inflow FROM kv_epoch_ends ORDER BY epoch ASC; epoch | cp_hi | tx_hi | to_timestamp | safe_mode | storage_fund_balance | storage_fund_reinvestment | storage_charge | storage_rebate | stake_subsidy_amount | total_gas_fees | total_stake_rewards_distributed | leftover_storage_fund_inflow -------+--------+--------+------------------------+-----------+----------------------+---------------------------+----------------+----------------+----------------------+----------------+---------------------------------+------------------------------ 0 | 9770 | 9771 | 2023-04-13 18:00:02+01 | f | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 1 | 85169 | 85174 | 2023-04-14 18:00:04+01 | f | 2973880 | 0 | 3952000 | 978120 | 0 | 102000000 | 102000000 | 0 2 | 161192 | 161199 | 2023-04-15 18:00:08+01 | f | 717398960 | 0 | 715403200 | 978120 | 0 | 1000000 | 1000000 | 0 3 | 237074 | 237084 | 2023-04-16 18:00:11+01 | f | 733657184 | 0 | 1430198400 | 1413940176 | 0 | 2000000 | 2000000 | 0 4 | 314160 | 314171 | 2023-04-17 18:00:15+01 | f | 733657184 | 0 | 0 | 0 | 0 | 0 | 0 | 0 5 | 391107 | 391119 | 2023-04-18 18:00:18+01 | f | 733657184 | 0 | 0 | 0 | 0 | 0 | 0 | 0 6 | 467716 | 467730 | 2023-04-19 18:00:21+01 | f | 735633184 | 0 | 1976000 | 0 | 0 | 1000000 | 1000000 | 0 7 | 544978 | 544994 | 2023-04-20 18:00:26+01 | f | 729859616 | 0 | 702475600 | 708249168 | 0 | 1000000 | 1000000 | 0 8 | 621933 | 621950 | 2023-04-21 18:00:28+01 | f | 729859616 | 0 | 0 | 0 | 0 | 0 | 0 | 0 9 | 699410 | 699428 | 2023-04-22 18:00:31+01 | f | 729859616 | 0 | 0 | 0 | 0 | 0 | 0 | 0 10 | 777074 | 777093 | 2023-04-23 18:00:34+01 | f | 729859616 | 0 | 0 | 0 | 0 | 0 | 0 | 0 11 | 855530 | 855550 | 2023-04-24 18:00:36+01 | f | 729859616 | 0 | 0 | 0 | 0 | 0 | 0 | 0 12 | 933559 | 933586 | 2023-04-25 18:00:39+01 | f | 735866656 | 0 | 13832000 | 7824960 | 0 | 6000000 | 6000000 | 0 (13 rows) ``` ## Stack - #20118 - #20132 - #20147 - #20148 - #20149 --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API: commit e96fca283cf8f00fd09291fdbdcd401c2063b71b Author: Ashok Menon <[email protected]> Date: Sat Nov 2 02:05:09 2024 +0000 indexer-alt: kv_feature_flags pipeline ## Description Like `kv_protocol_configs`, but for feature flags. ## Test plan Run the indexer over the first 700K checkpoints -- this will include two protocol config upgrades, then check the database for feature flags. ``` sui$ cargo run -p sui-indexer-alt --release -- \ --database-url "postgres://postgres:postgrespw@localhost:5432/sui_indexer_alt" \ indexer --remote-store-url https://checkpoints.mainnet.sui.io \ --last-checkpoint 700000 ``` ``` sui_indexer_alt=# SELECT * FROM kv_feature_flags ORDER BY protocol_version DESC, flag_name; protocol_version | flag_name | flag_value ------------------+-------------------------------------------------------------+------------ 6 | accept_zklogin_in_multisig | f 6 | advance_epoch_start_time_in_safe_mode | t 6 | advance_to_highest_supported_protocol_version | f 6 | allow_receiving_object_id | f 6 | authority_capabilities_v2 | f [...] ``` commit 1aab0cf2b23b6d682446eae86937542f9f698e3c Author: Ashok Menon <[email protected]> Date: Sat Nov 2 01:40:35 2024 +0000 indexer-alt: kv_protocol_configs pipeline ## Description Add a pipeline for indexing protocol configs. ## Test plan Index the first ~1M checkpoints, which should pick up the first protocol version upgrade (from 4 to 5 for epoch 10) and then query `kv_protocol_configs` results after that: ``` sui$ cargo run -p sui-indexer-alt -- \ --database-url postgres://postgres:postgrespw@localhost:5432/sui_indexer_alt \ --remote-store-url https://checkpoints.mainnet.sui.io \ --last-checkpoint 500000 ``` ``` sui_indexer_alt=# SELECT * FROM kv_protocol_configs; ``` commit 97932677f340404a8843ac8fb46ae56ea0b7501c Author: Ashok Menon <[email protected]> Date: Sat Nov 2 01:18:58 2024 +0000 refactor(indexer-alt): configurable processor ## Description Change the pipeline/handler/processor interface to accept a value representing the handler/processor. This allows the processor to include configuration. This will be used by upcoming pipelines for processing protocol configs and feature flags. ## Test plan CI commit ae6275f6957ac936caffe04e82b7762ba192c4f8 Author: Anastasios Kichidis <[email protected]> Date: Wed Nov 13 12:38:22 2024 +0000 [Consensus] resubmit GC'ed transactions (#19572) ## Description Transactions of GC'ed blocks should be retried to remain compliant with the so far system assumptions. Refactored the consensus side to signal back the submitter about the status of the block that the submitted transaction has been included to. Practically two possible outcomes: 1. Sequenced (when the block has been committed and consequently the tx as well) 2. GarbageCollected (when the tx's block has not been committed and passed gc_round) also took the opportunity for a small refactoring around the `SubmitToConsensus` trait and the `ConsensusClient` as intentions started diverging. ## Test plan CI/PT --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API: commit bb0288b9b8de0307690a434eaf62a930cf1280e8 Author: benr-ml <[email protected]> Date: Wed Nov 13 14:35:19 2024 +0200 Enable G1Uncompressed in testnet (#20226) ## Description Describe the changes or additions included in this PR. ## Test plan How did you test the new or updated feature? --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API: commit 6ea6bc9b112bf34a3dcfb136430726924b03e0d0 Author: Mark Logan <[email protected]> Date: Tue Nov 12 23:08:08 2024 -0800 Fixes for overload protection (#20248) - Only check overload queues for mutable shared inputs - Increase max tx queue age to 500ms ## Description Describe the changes or additions included in this PR. ## Test plan How did you test the new or updated feature? --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API: commit 297913c0adcb677eb369d520e9de7c80696fbde3 Author: Mark Logan <[email protected]> Date: Tue Nov 12 21:21:06 2024 -0800 Fix bug with object_by_id inserts. (#20236) The previous version of this code was based on a misunderstanding of the Moka Cache API. It assumed that: let entry = cache.entry(k).or_insert_with(|| v); entry.is_fresh() could be used to detect whether there was a race with cache.insert(k, v) In fact this is not the case! Both threads must use `or_insert_with` in order to handle the race correctly. The bug is instantly reproducible with the following code: use moka::sync::Cache; use std::sync::{Arc, Mutex}; fn monotonic_update(cache: Arc<Cache<u64, Arc<Mutex<u64>>>>, key: u64, value: u64) { let entry = cache .entry(key) .or_insert_with(|| Arc::new(Mutex::new(value))); if !entry.is_fresh() { let mut entry = entry.value().lock().unwrap(); // only update if the new value is greater than the current value if value > *entry { *entry = value; } } } fn blind_write(cache: Arc<Cache<u64, Arc<Mutex<u64>>>>, key: u64, value: u64) { cache.insert(key, Arc::new(Mutex::new(value))); } fn main() { for _ in 0..1000 { let cache = Arc::new(Cache::new(1000)); let cache1 = cache.clone(); let cache2 = cache.clone(); let handle1 = std::thread::spawn(move || { monotonic_update(cache1, 1, 1); }); let handle2 = std::thread::spawn(move || { // Correct way to update the value // monotonic_update(cache2, 1, 2); // Incorrect way to update the value blind_write(cache2, 1, 2); }); handle1.join().unwrap(); handle2.join().unwrap(); let entry = cache.get(&1).unwrap(); let value = entry.lock().unwrap(); assert_eq!(*value, 2); } } ## Description Describe the changes or additions included in this PR. ## Test plan How did you test the new or updated feature? --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API: commit ec6d01642ad68b1dea024a3f34d5966359a3617e Author: Todd Nowacki <[email protected]> Date: Tue Nov 12 15:51:12 2024 -0800 [move-compiler] Improve interactions with Sui mode and interface files (#20196) ## Description - Sui mode used only named addresses for type comparison. This is not sufficient for interface files that might be using only the numerical address. - Switched `is` to use only the numerical address. We might want to evaluate differently if we start using it more outside of Sui mode ## Test plan - Ran tests --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API: commit 7b61e1ff70f4b2fbfd67085db2f795bcaa291105 Author: Avrora <[email protected]> Date: Wed Nov 13 01:42:06 2024 +0200 Docs: Fix typo (#20225) Fix typo ## Description Describe the changes or additions included in this PR. ## Test plan How did you test the new or updated feature? --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API: --------- Co-authored-by: ronny-mysten <[email protected]> commit f716589e9b7d966cea77dbe8145ed892fe4fd0cd Author: ronny-mysten <[email protected]> Date: Tue Nov 12 16:41:39 2024 -0700 [docs] Removed orphaned page (#20228) ## Description Removed orphaned page and updates the link that pointed to it. ## Test plan How did you test the new or updated feature? --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API: commit d5a23d78a9b30ec1c5ae5ac3cafdbe2817d68df6 Author: hayes-mysten <[email protected]> Date: Tue Nov 12 12:49:03 2024 -0800 [ts-sdk] Add tx.object.option for creatnig object options in transact… (#20194) …ion builder ## Description Describe the changes or additions included in this PR. ## Test plan How did you test the new or updated feature? --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API: commit de2c7de6f6b5d643560c0ca343b9009c96e7fefd Author: benr-ml <[email protected]> Date: Tue Nov 12 21:44:54 2024 +0200 Update fastcrypto, including moving some structs between dkg->dkg_v1 (#20224) ## Description Describe the changes or additions included in this PR. ## Test plan How did you test the new or updated feature? --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API: commit 44fe9eb64fed3a10ad5da8d3ad0d542b2f06b281 Author: Andrew Schran <[email protected]> Date: Tue Nov 12 14:44:34 2024 -0500 Revert "Add support for TLS connections with self-signed cert on validator gRPC interface" (#20232) Reverts MystenLabs/sui#19796 due to suspicion it's causing performance issues commit 73dbd005bb0f828a64eb4e811d4f8e2b3c80bdfe Author: sui-merge-bot[bot] <114704316+sui-merge-bot[bot]@users.noreply.github.com> Date: Tue Nov 12 13:38:32 2024 -0500 Version Packages (#20229) This PR was opened by the [Changesets release](https://github.com/changesets/action) GitHub action. When you're ready to do a release, you can merge this and publish to npm yourself or [setup this action to publish automatically](https://github.com/changesets/action#with-publishing). If you're not ready to do a release yet, that's fine, whenever you add more changesets to main, this PR will be updated. # Releases ## @mysten/[email protected] ### Patch Changes - a24d43d: getOrders and decodeOrderId support Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> commit c4afb26067e9a4e9eea75a706032f1764c4c3ded Author: Xun Li <[email protected]> Date: Tue Nov 12 10:17:08 2024 -0800 [indexer-alt] Add checkpoint ingestion lag metrics (#20191) ## Description This PR adds two metrics, one that tracks the ingestion lag on the latest fetched checkpoint, one that tracks histogram. These two combined could give us the full picture on the lag. ## Test plan CI. --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API: commit 4ba4b29a0ca9c473e89a1ba5b01caf71c31c5c21 Author: phoenix <[email protected]> Date: Tue Nov 12 12:47:34 2024 -0500 [kv store] add timeout to bigtable ingestion + backfill mode (#20193) ## Description adds timeouts to the Bigtable ingestion pipeline and introduces a backfill mode that skips most progress updates to DynamoDB during backfilling --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API: commit a24d43d709e1c05f98b092d50c2ac87b5c172489 Author: Tony Lee <[email protected]> Date: Tue Nov 12 12:38:38 2024 -0500 Deepbook SDK Examples (#20227) ## Description Getting balance for balance manager, getting live orders for a balance manager, and price/quantity map for a balance manager. Balance: ``` Manager: MANAGER_1 { coinType: '0x0000000000000000000000000000000000000000000000000000000000000002::sui::SUI', balance: 374527.5686 } { coinType: '0xdba34672e30cb065b1f93e3ab55318768fd6fef66c15942c9f7cb846e2f900e7::usdc::USDC', balance: 1492902.476423 } ``` Live Orders: ``` Manager: MANAGER_1 SUI_USDC [ '170141183460526324404595434778137174870', '170141183460526490425292098164101718871', '170141183460526490425292098164101718872', '56723756473400944927347700', '56502395544516430307955699', '55819866013789176898163698', '56723756473400944927347696' ] DEEP_SUI [ '413576002151014891304428578578', '421323634661972902983148578577', '170141183888249226801011805690925112362', '414498339354700368885228578576' ] ``` Price/Quantity Map: ``` Manager: MANAGER_1 SUI_USDC bid orders: [ [ 3.046, 6.6 ], [ 3.035, 21418.1 ], [ 3.027, 26427.4 ], [ 2.995, 46747.6 ] ] SUI_USDC ask orders: [ [ 3.049, 6.6 ], [ 3.065, 21207.2 ], [ 3.073, 26034.1 ], [ 3.096, 45223.3 ] ] ``` ## Test plan How did you test the new or updated feature? Locally tested using mainnet ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading …
carlosbmamoru
pushed a commit
to Mamoru-Foundation/sui
that referenced
this pull request
Nov 19, 2024
commit cd214a3d7e8e9e47ccf31ab8f5315e8c114313be Author: wlmyng <[email protected]> Date: Fri Nov 15 13:21:24 2024 -0800 [mvr] Introduce mvr-mode on production indexer (#20271) (#20288) ## Description introduce mvr-mode flag. If true, enables pruning objects_history, and toggles committer to write only a subset of tables to db. Compatible with the production indexer schema. ## Test plan How did you test the new or updated feature? --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API: ## Description Describe the changes or additions included in this PR. ## Test plan How did you test the new or updated feature? --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API: commit ab6863576baf7ff74f0de95ab5abef9040fc8fcd Author: Eugene Boguslavsky <[email protected]> Date: Thu Nov 14 18:41:02 2024 -0800 Sui `v1.38.0` Framework Bytecode Snapshot (#20277) ## Description Sui `v1.38.0` Framework Bytecode Snapshot ## Test plan 👀 commit 5d70ec6f5d4b7ff611649432aeb8c538efb63e80 Author: Sadhan Sood <[email protected]> Date: Thu Nov 14 17:08:27 2024 -0800 feat(analytics): Add package filter to object handler (#20272) ## Description This PR adds a package filter for object handler in analytics for being able to generate an object table specifically for walrus in testnet. ## Test Invoking with `--package-id-filter 0x9f992cc2430a1f442ca7a5ca7638169f5d5c00e0ebc3977a65e9ac6e497fe5ef --starting-checkpoint-seq-num 120132840` to filter for walrus specific objects. Here is the walrus package object: https://testnet.suivision.xyz/package/0x9f992cc2430a1f442ca7a5ca7638169f5d5c00e0ebc3977a65e9ac6e497fe5ef commit b36024d32ce349ba9465b38bd6706b8b2068ee67 Author: stefan-mysten <[email protected]> Date: Thu Nov 14 16:27:12 2024 -0800 [CLI] Add back the warning about version mismatch (#20273) ## Description #20259 removed the warning for the SDK, so this PR adds it back for the CLI. ## Test plan How did you test the new or updated feature? --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API: commit 81f75843464ce1140040403df0a0fa0852f87386 Author: Adam Welc <[email protected]> Date: Thu Nov 14 13:01:50 2024 -0800 [move-compiler] Added function location info to the source map (#20261) ## Description This PR adds additional field to the source map describing the location of the entire function. This is needed to enable better handling of macros in the trace viewer (we need to know which instructions in a given function are outside of a function body to identify those that belong to inlined macro functions). ## Test plan All existing tests must pass commit 8d90acd980b4490a7f5488cee2aad465abe8e663 Author: Tim Zakian <[email protected]> Date: Thu Nov 14 11:53:41 2024 -0800 [json-rpc] Add enums in `getNormalizedMoveModule` (#20231) ## Description Adds enums to the `NormalizedMoveModule` for json-rpc. This fixes #20205 ## Test plan Tested manually on the repro provided in the issue to make sure enums are included in the output. --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [x] JSON-RPC: Fixes an issue where enum types were not included in the output of `getNormalizedMoveModule` and `getNormalizedMoveModuleForPackage` - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API: commit 53dd080a841710c91f492de17f7312a29662235f Author: William Smith <[email protected]> Date: Thu Nov 14 14:03:22 2024 -0500 [TrafficControl] Add salt to TrafficSketch (#20245) ## Description False positives stem from IP collisions during traffic tallying. Because validators tend to see closely correlated transaction execution traffic patterns, it is possible that a false positive resulting in an erroneously blocked IP could be correlated across many validators at the same time. If this happens on a quorum of validators, this would result in an IP being effectively denied service until the TTL expires. Luckily, we can exploit this property to lower the effective false positive rate (as observed by the client) by removing this correlation. This PR adds a salt key to the `SketchKey`. The value is generated in memory at startup. ## Test plan Existing tests --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API: commit 97838b20eb28b5abd12d275f5924c70c97b86e2c Author: sui-merge-bot[bot] <114704316+sui-merge-bot[bot]@users.noreply.github.com> Date: Thu Nov 14 10:25:13 2024 -0800 Version Packages (#20269) This PR was opened by the [Changesets release](https://github.com/changesets/action) GitHub action. When you're ready to do a release, you can merge this and publish to npm yourself or [setup this action to publish automatically](https://github.com/changesets/action#with-publishing). If you're not ready to do a release yet, that's fine, whenever you add more changesets to main, this PR will be updated. # Releases ## @mysten/[email protected] ### Patch Changes - @mysten/[email protected] ## @mysten/[email protected] ### Patch Changes - Updated dependencies [1765780] - @mysten/[email protected] ## @mysten/[email protected] ### Patch Changes - 1765780: fix zksend link network detection Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> commit 176578020ba15ff321d42b487f87af0a650bece3 Author: Pavlos Chrysochoidis <[email protected]> Date: Thu Nov 14 18:10:02 2024 +0000 zksend sdk fix network detection (#20268) ## Description Describe the changes or additions included in this PR. ## Test plan How did you test the new or updated feature? --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API: commit 28e184a694636fc3b5a589c219f47b549250d6c1 Author: xiaodi <[email protected]> Date: Fri Nov 15 00:24:24 2024 +0800 Update regulated.mdx (#20267) ## Description Replaced "Regulated Coin and Deny List" with "Loyalty Tokens" in documentation. ## Test plan How did you test the new or updated feature? --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API: commit 0bd58999ae2be185c6677ebc4a44b3e5dcd15296 Author: sui-merge-bot[bot] <114704316+sui-merge-bot[bot]@users.noreply.github.com> Date: Thu Nov 14 09:18:08 2024 -0500 Version Packages (#20265) This PR was opened by the [Changesets release](https://github.com/changesets/action) GitHub action. When you're ready to do a release, you can merge this and publish to npm yourself or [setup this action to publish automatically](https://github.com/changesets/action#with-publishing). If you're not ready to do a release yet, that's fine, whenever you add more changesets to main, this PR will be updated. # Releases ## @mysten/[email protected] ### Patch Changes - 4a42691: NS token pools added Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> commit 4a426918ef263103020aca25cf05809e4531ec51 Author: Tony Lee <[email protected]> Date: Thu Nov 14 08:26:36 2024 -0500 NS Deepbook Pools (#20256) commit d5d51aed177432e62950385336b79f8395c940f9 Author: stefan-mysten <[email protected]> Date: Wed Nov 13 17:17:13 2024 -0800 [RustSDK] Less noise when rpc version and local CLI version mismatch (#20259) ## Description This PR removes the SDK tracing warning on mismatch local + RPC API versions. ## Test plan How did you test the new or updated feature? --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API: commit c85e7c07c0713c1edf7cdb59eab46372ef29fe24 Author: Tim Zakian <[email protected]> Date: Wed Nov 13 17:16:49 2024 -0800 [docs] Update Sui CLI install instructions to include the tracing feature (#20255) ## Description Update docs for installing the Sui CLI to include `--features tracing` so that they work with the trace viewer/debugger. ## Test plan 👀 commit 9dfd26a723e5efdb5816d6c2492cb0bb27b2f2c5 Author: Eugene Boguslavsky <[email protected]> Date: Wed Nov 13 15:33:46 2024 -0800 Use tracing feature for sui builds (#20260) ## Description Use tracing feature for sui builds ## Test plan 👀 commit 047de09d84bf90b633fd183b0cecef38e9387317 Author: Xun Li <[email protected]> Date: Wed Nov 13 14:30:23 2024 -0800 [indexer-alt] Add processor lag metrics (#20235) ## Description Describe the changes or additions included in this PR. ## Test plan How did you test the new or updated feature? --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API: commit c03132d75e80b569c2fd8447539ac0b7bddc8ecc Author: Andrew Schran <[email protected]> Date: Wed Nov 13 16:51:15 2024 -0500 Add support for TLS connections with self-signed cert on validator gRPC interface (#20238) Client side use of TLS will be enabled-by-default in a future PR. This has corrected TLS upgrade behavior compared to older PR #19796. --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [x] Nodes (Validators and Full nodes): Adds support for TLS on validator gRPC interface. - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API: commit 313bf8b43a1b8b511cb9b012a23d5803444b3ea5 Author: Tim Zakian <[email protected]> Date: Wed Nov 13 13:16:01 2024 -0800 [move] Add Move 2024 (non-beta) edition and make it the default edition (#20242) ## Description Add Move `2024` edition and make it the default edition. ## Test plan Updated existing tests. --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [X] CLI: Added the Move `2024` edition, and set it as the default Move edition both when creating a new Move package and when migrating from an existing package. Existing `2024.beta` and `2024.alpha` editions remain supported. - [ ] Rust SDK: - [ ] REST API: commit 8f21cd702b8ec9f4796f317aa9ec6bfd56330f0e Author: phoenix <[email protected]> Date: Wed Nov 13 15:26:23 2024 -0500 [data ingestion] bigtable: set env credentials parameter (#20240) ## Description Pass filename with credential parameters to the ingestion binary --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API: commit 3c444d5dada018e5b10a929efa6f17e3445367d5 Author: Adam Welc <[email protected]> Date: Wed Nov 13 11:04:35 2024 -0800 [move-ide] Fixed auto-completion on the dot character (#20175) ## Description This PR fixes a certain problem with auto-completion on the `.` character. Previously, once a part of the access chain after the `.` was parsed, auto-completion information for the `.` character location was no longer generated. This is particularly problematic if someone expects auto-completion in the middle of the function code, for example: ```move public struct SomeStruct has drop { some_field: u64, } public fun foo(_param: SomeStruct) {} public fun bar(): SomeStruct { let val = SomeStruct{some_field: 42}; val. SomeStruct {some_field: 42} } ``` It is expected that auto-completion would work correctly after `val.` but previously it did not because this expression was actually parsed as `val.SomeStruct`. This PR fixes this:  Previously, auto-completion for the `.` location would also not work if what follows the `.` is a function call (whether the function being called was resolved correctly or not). It now works correctly in both cases:   ## Test plan All new and (updated) all tests must pass commit 8d647c0f00809bba411079ce765fdb4fbecef00c Author: sui-merge-bot[bot] <114704316+sui-merge-bot[bot]@users.noreply.github.com> Date: Wed Nov 13 18:49:45 2024 +0000 Version Packages (#20252) This PR was opened by the [Changesets release](https://github.com/changesets/action) GitHub action. When you're ready to do a release, you can merge this and publish to npm yourself or [setup this action to publish automatically](https://github.com/changesets/action#with-publishing). If you're not ready to do a release yet, that's fine, whenever you add more changesets to main, this PR will be updated. # Releases ## @mysten/[email protected] ### Minor Changes - 60bdb62: Update graphql query and use stashed host for claim api - 364a93a: Remove decoding spaces to `+` in zksend wallet url hashes - 364a93a: Add claimedBy property that is set when claimedAssets are loaded ## @mysten/[email protected] ### Patch Changes - @mysten/[email protected] ## @mysten/[email protected] ### Patch Changes - Updated dependencies [60bdb62] - Updated dependencies [364a93a] - Updated dependencies [364a93a] - @mysten/[email protected] Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> commit 8117d8393f25fa85390094bca61952ab24c9b595 Author: Tim Zakian <[email protected]> Date: Wed Nov 13 10:38:18 2024 -0800 [move/tracing] Panic unit tests if tracing without being compiled with tracing support (#20253) ## Description Make sure that if the `--trace-execution` flag is passed that the binary was compiled with the `tracing` feature flag set. ## Test plan CI --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API: commit 60bdb62627a821bb54b7c7415a63193b7395a05d Author: hayes-mysten <[email protected]> Date: Wed Nov 13 10:08:56 2024 -0800 [zksend] Update garphql query, and use getstashed.com as claim api host (#20237) ## Description fixes #20222 Describe the changes or additions included in this PR. ## Test plan How did you test the new or updated feature? --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API: commit 364a93a1b10be70aabe0fb8937b3c1af610d1eba Author: hayes-mysten <[email protected]> Date: Wed Nov 13 10:00:16 2024 -0800 [zksend] add claimedBy property when loading assets of claimed link (#20244) ## Description Describe the changes or additions included in this PR. ## Test plan How did you test the new or updated feature? --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API: commit 43d7c3dcfcdf062cad98b480f0c0a62e218026e9 Author: Xun Li <[email protected]> Date: Wed Nov 13 09:14:22 2024 -0800 [indexer-alt] Skip commit if batch is empty (#20124) ## Description If we did not collect anything to the batch, we should not commit to the DB. We should also avoid reporting any metrics. ## Test plan CI --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API: --------- Co-authored-by: Ashok Menon <[email protected]> commit f04e4cddfdde35a5d5cba9c7fd7c0f9482d4193b Author: sui-merge-bot[bot] <114704316+sui-merge-bot[bot]@users.noreply.github.com> Date: Wed Nov 13 11:07:19 2024 -0500 Version Packages (#20239) This PR was opened by the [Changesets release](https://github.com/changesets/action) GitHub action. When you're ready to do a release, you can merge this and publish to npm yourself or [setup this action to publish automatically](https://github.com/changesets/action#with-publishing). If you're not ready to do a release yet, that's fine, whenever you add more changesets to main, this PR will be updated. # Releases ## @mysten/[email protected] ### Patch Changes - Updated dependencies [d5a23d7] - @mysten/[email protected] - @mysten/[email protected] ## @mysten/[email protected] ### Patch Changes - Updated dependencies [d5a23d7] - @mysten/[email protected] - @mysten/[email protected] - @mysten/[email protected] ## @mysten/[email protected] ### Patch Changes - Updated dependencies [d5a23d7] - @mysten/[email protected] ## @mysten/[email protected] ### Patch Changes - a43ae92: NS Token - Updated dependencies [d5a23d7] - @mysten/[email protected] ## @mysten/[email protected] ### Patch Changes - Updated dependencies [d5a23d7] - @mysten/[email protected] - @mysten/[email protected] ## @mysten/[email protected] ### Patch Changes - Updated dependencies [d5a23d7] - @mysten/[email protected] ## @mysten/[email protected] ### Patch Changes - Updated dependencies [d5a23d7] - @mysten/[email protected] ## @mysten/[email protected] ### Patch Changes - Updated dependencies [d5a23d7] - @mysten/[email protected] ## @mysten/[email protected] ### Patch Changes - Updated dependencies [d5a23d7] - @mysten/[email protected] ## @mysten/[email protected] ### Patch Changes - d5a23d7: Add tx.object.option for creatnig object options in transaction builder ## @mysten/[email protected] ### Patch Changes - Updated dependencies [d5a23d7] - @mysten/[email protected] ## @mysten/[email protected] ### Patch Changes - Updated dependencies [d5a23d7] - @mysten/[email protected] ## @mysten/[email protected] ### Patch Changes - Updated dependencies [d5a23d7] - @mysten/[email protected] - @mysten/[email protected] Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> commit a43ae921386d1e8406f81d73d2c317ad92bb211e Author: Tony Lee <[email protected]> Date: Wed Nov 13 10:38:16 2024 -0500 NS token in deepbook sdk (#20251) ## Description NS token in deepbook sdk ## Test plan How did you test the new or updated feature? Testnet and local ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API: commit 91dec7c02d776f3e68a5a1e5b86851c7d289c206 Author: Ashok Menon <[email protected]> Date: Wed Nov 13 14:00:52 2024 +0000 indexer-alt: remove watermarks.epoch_lo (#20218) ## Description This field turns out not to be used by the new pruner implementation, because it is entirely based on checkpoints. ## Test plan CI ## Stack - #20149 - #20150 - #20166 - #20216 - #20217 --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API: commit c5f275ea9fab4a099e998a7b4f65926fe1ae8e8e Author: Ashok Menon <[email protected]> Date: Mon Nov 11 02:43:06 2024 +0000 indexer-alt: pruner task ## Description Add the task that actually deletes data, based on the reader low watermark. ## Test plan Run the indexer and note the following: - Metrics related to deleted rows by the pruner (from `localhost:9184/metrics`) - The contents of the `watermarks` table. ``` sui$ cargo run -p sui-indexer-alt --release -- \ --database-url "postgres://postgres:postgrespw@localhost:5432/sui_indexer_alt" \ indexer --remote-store-url https://checkpoints.mainnet.sui.io \ --last-checkpoint 10000 \ --consistent-range 100 --consistent-pruning-interval 10 \ --pipeline sum_obj_types --pipeline wal_obj_types ``` Also tested running the indexer for an extended period of time (1M checkpoints over roughly half an hour in local testing), and noted how the pruner behaves. When configured as it would be in production (roughly one hour of consistent range, and a 5 minute pruning interval and a 2 minute pruning delay): - Many rows accumulated during backfill -- by the end of the 1M checkpoints, the pruner had only pruned up to between checkpoint 500K and checkpoint 700K depending on the pipeline. This should not be an issue under normal operation where the indexer will run for long enough for pruning to stabilise at the tip of the network (and it would be recommended practice to start from formal snapshot and therefore only need to run pruning from that point forward). - Because the reader watermark task and the pruner task use the same interval, it can take up to two ticks of that interval for the pruner to act on a change to its upperbound -- again, it should be okay, as the pruner's interval should be at least an order of magnitude smaller than its retention period. commit 3a2e332c68cfe9491eb56d09f087ff8cc559e6e4 Author: Ashok Menon <[email protected]> Date: Mon Nov 11 02:37:06 2024 +0000 fix(indexer-alt): overly chatty loud watermark updates ## Description The loud watermark update logic always bumped the previous update watermark by the same interval. When the indexer was running well ahead of the loud watermark update rate, this would cause many updates to be issued successively. This change makes it so that the next update is always `LOUD_WATERMARK_UPDATE_INTERVAL` away from the last loud update. ## Test plan Run the indexer. Previously updates -- particularly from summary tables that gathered changes up and write them out in big batches -- would come in bursts, and this behaviour is no longer apparent after the change: ``` sui$ cargo run -p sui-indexer --release -- \ --database-url "postgres://postgres:postgrespw@localhost:5432/sui_indexer_alt" \ indexer --remote-store-url https://checkpoints.mainnet.sui.io \ --last-checkpoint 1000000 --consistent-range 3600 ``` commit 6d31be68d0ef82f90aa0447e13fd4c82eb40f732 Author: Ashok Menon <[email protected]> Date: Sun Nov 10 17:19:53 2024 +0000 indexer-alt: reader watermark task ## Description The reader watermark task is one half of the pruning implementation. It applies only to concurrent pipelines, and tracks the lowest checkpoint sequence number the reader is guaranteed to get data for, and the timestamp (from the DB), that this threshold was set at. This task does not do any actual pruning, but it sets the limit that the pruner can go to, and a time at which it can start pruning to that point. A later PR will introduce the actual pruning task. Unlike the existing pruner implementation, this system does not support pruning by epoch, only by checkpoint. This was for two reasons: - Pruning no longer works by dropping partitions, so we can keep the implementation simple by only supporting the form of pruning we need (by checkpoint sequence number). - Pruning is part of the indexing framework, rather than a specific part of our indexer implementation, so pruning by epoch would require that the indexing framework kept track of a mapping from epoch to checkpoint sequence number. Along with the reader watermark, the following changes were made: - The existing watermark task was renamed to the `committer_watermark` to distinguish it. - The interface between pipelines and the indexing framework has been simplified to return just one `JoinHandle`. This was because clippy complained that returning 6 handles was too complex, and additionally, because the reader watermark task is not connected to the rest of the tasks for a pipeline, it needs a more complicated tear down logic, where firstly the committer tasks are wound down, and then the pruner is explicitly cancelled, before its tasks are awaited as well. The change also sets up pruning configs for write-ahead log tables, so that we can test reader watermark behaviour. ## Test plan Run the indexer with just one of the consistent read summary and write-ahead logs, and then query the watermarks table: ``` cargo run -p sui-indexer-alt --release -- \ --database-url "postgres://postgres:postgrespw@localhost:5432/sui_indexer_alt" \ indexer --remote-store-url https://checkpoints.mainnet.sui.io \ --last-checkpoint 10000 \ --consistent-range 100 --consistent-pruning-interval 10 \ --pipeline sum_obj_types --pipeline wal_obj_types ``` ``` sui_indexer_alt=# SELECT * FROM watermarks; pipeline | epoch_hi_inclusive | checkpoint_hi_inclusive | tx_hi | timestamp_ms_hi_inclusive | epoch_lo | reader_lo | pruner_timestamp | pruner_hi ---------------+--------------------+-------------------------+-------+---------------------------+----------+-----------+----------------------------+----------- sum_obj_types | 1 | 9900 | 9902 | 1681405380769 | 0 | 0 | 1970-01-01 00:00:00 | 0 wal_obj_types | 1 | 10000 | 10002 | 1681405504878 | 0 | 6321 | 2024-11-10 17:33:11.459593 | 0 (2 rows) ``` Note that `reader_lo` is not guaranteed to be as advanced as it could be, because this task checks in relatively infrequently, and the indexer as a whole might shut down efore it has a chance to catch up. However, on restart, the indexer should fast-forward the watermark (i.e. if the steps above are repeated, `reader_lo` will be updated): ``` sui_indexer_alt=# SELECT * FROM watermarks; pipeline | epoch_hi_inclusive | checkpoint_hi_inclusive | tx_hi | timestamp_ms_hi_inclusive | epoch_lo | reader_lo | pruner_timestamp | pruner_hi ---------------+--------------------+-------------------------+-------+---------------------------+----------+-----------+----------------------------+----------- sum_obj_types | 1 | 9900 | 9902 | 1681405380769 | 0 | 0 | 1970-01-01 00:00:00 | 0 wal_obj_types | 1 | 10000 | 10002 | 1681405504878 | 0 | 9801 | 2024-11-10 17:33:49.913129 | 0 (2 rows) ``` commit 81474ef2fb2bc2e1ae2a39bbf599e2493504ab1d Author: Ashok Menon <[email protected]> Date: Sun Nov 10 17:15:47 2024 +0000 indexer-alt: watermarks.pruner_timestamp is a TIMESTAMP ## Description Change `watermark.pruner_timestamp_ms BIGINT` to `watermark.pruner_timestamp TIMESTAMP`. This is because unlike the other timestamps in this table, which are read from checkpoint data, this timestamp is read from the database, and so it is easiest to represent it using the database's native type for timestamps. ## Test plan CI commit 1d568438c5ca3806946d93900e1561b31d06daa3 Author: Ashok Menon <[email protected]> Date: Wed Nov 13 13:54:12 2024 +0000 chore(indexer-alt): rename tx_calls_fun -> tx_calls (#20166) ## Description I partially simplified the naming of `tx_calls_fun` to `tx_calls` but did not do a complete job -- this change completes it. ## Test plan CI ## Stack - #20118 - #20132 - #20147 - #20148 - #20149 - #20150 --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API: commit 85c0f810e907cabd503a28c2e134e8af282351e8 Author: Ashok Menon <[email protected]> Date: Wed Nov 13 13:53:31 2024 +0000 indexer-alt: epochs pipelines (#20150) ## Description Adds two tables and pipelines: `kv_epoch_starts` and `kv_epoch_ends`, to index epoch information. This pipeline is different from epoch indexing in `sui-indexer` in a number of ways: - It is an append-only pipeline. The columns that are written at the start and end of the epoch are split into two separate tables that can be written to concurrently (by separate pipelines). - The first row of `kv_epoch_starts` is written by the bootstrap process which seeds the `kv_genesis` table as well, this avoids having to condition on whether the checkpoint being processed is the genesis checkpoint in the main pipeline. - Instead of indexing the number of transactions in the epoch, it tracks the transaction high watermark -- readers will need to read the records to calculate the number of total transactions (this avoids having to read the last epoch's total transactions in the write path). - We index the `SuiSystemState` object as BCS, rather than the summary structure. - We explicitly record whether the epoch advancement at the end of the epoch triggered safe mode (the system state object also tracks whether the epoch was started in safe mode). - Fields related to information that came from `SystemEpochInfoEvent` have all been consolidated in `kv_epoch_ends`, and they are all optional, in case of safe mode. It's worth elaborating on the last bullet point, because this is quite a subtle, but large change: - Today, `total_stake` and `storage_fund_balance` are written at the start of an epoch based on the fields of the `SystemEpochInfoEvent` emitted from the previous epoch, and are `NOT NULL`. - The remaining fields were nullable, but only because they would be written to later, once the epoch was over. This was awkward to work with in a couple of ways: - It meant that for the genesis epoch, we needed to make some numbers up (all zeroes) because we did not have an event to read from. - We had to do something similar if we hit safe mode. - When indexing the start and end of epochs separately, it meant that we had to duplicate work (finding the system epoch info event). By making the fields nullable, and consolidating them in `kv_epoch_ends`, we can simplify the pipelines: - `kv_epoch_starts` and the bootstrapping logic can work purely based on the system state object. - `kv_epoch_ends` can work purely based on the `SystemEpochInfoEvent`, and can leave fields `NULL` if we are in safe mode. In the case of `kv_epoch_starts` we could also have cut down fields to just `epoch`, `cp_lo` and `system_state`. I chose not to do this because the system state is actually quite a large object, and it is beneficial to avoid having to deserialize to answer simpler queries. ## Test plan Ran the indexer on the first 1M checkpoints, and correlated the resulting info in the respective tables from the data that the current indexer produced: ``` sui_indexer_alt=# SELECT epoch, protocol_version, cp_lo, TO_TIMESTAMP(start_timestamp_ms / 1000), reference_gas_price FROM kv_epoch_starts; epoch | protocol_version | cp_lo | to_timestamp | reference_gas_price -------+------------------+--------+------------------------+--------------------- 0 | 4 | 0 | 2023-04-12 18:00:00+01 | 1000 1 | 4 | 9770 | 2023-04-13 18:00:02+01 | 1000 2 | 4 | 85169 | 2023-04-14 18:00:04+01 | 1000 3 | 4 | 161192 | 2023-04-15 18:00:08+01 | 1000 4 | 4 | 237074 | 2023-04-16 18:00:11+01 | 1000 5 | 4 | 314160 | 2023-04-17 18:00:15+01 | 1000 6 | 4 | 391107 | 2023-04-18 18:00:18+01 | 1000 7 | 4 | 467716 | 2023-04-19 18:00:21+01 | 1000 8 | 4 | 544978 | 2023-04-20 18:00:26+01 | 1000 9 | 5 | 621933 | 2023-04-21 18:00:28+01 | 1000 10 | 6 | 699410 | 2023-04-22 18:00:31+01 | 1000 11 | 6 | 777074 | 2023-04-23 18:00:34+01 | 1000 12 | 6 | 855530 | 2023-04-24 18:00:36+01 | 1000 13 | 6 | 933559 | 2023-04-25 18:00:39+01 | 1000 (14 rows) sui_indexer_alt=# SELECT epoch, cp_hi, tx_hi, TO_TIMESTAMP(end_timestamp_ms / 1000), safe_mode, storage_fund_balance, storage_fund_reinvestment, storage_charge, storage_rebate, stake_subsidy_amount, total_gas_fees, total_stake_rewards_distributed, leftover_storage_fund_inflow FROM kv_epoch_ends ORDER BY epoch ASC; epoch | cp_hi | tx_hi | to_timestamp | safe_mode | storage_fund_balance | storage_fund_reinvestment | storage_charge | storage_rebate | stake_subsidy_amount | total_gas_fees | total_stake_rewards_distributed | leftover_storage_fund_inflow -------+--------+--------+------------------------+-----------+----------------------+---------------------------+----------------+----------------+----------------------+----------------+---------------------------------+------------------------------ 0 | 9770 | 9771 | 2023-04-13 18:00:02+01 | f | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 1 | 85169 | 85174 | 2023-04-14 18:00:04+01 | f | 2973880 | 0 | 3952000 | 978120 | 0 | 102000000 | 102000000 | 0 2 | 161192 | 161199 | 2023-04-15 18:00:08+01 | f | 717398960 | 0 | 715403200 | 978120 | 0 | 1000000 | 1000000 | 0 3 | 237074 | 237084 | 2023-04-16 18:00:11+01 | f | 733657184 | 0 | 1430198400 | 1413940176 | 0 | 2000000 | 2000000 | 0 4 | 314160 | 314171 | 2023-04-17 18:00:15+01 | f | 733657184 | 0 | 0 | 0 | 0 | 0 | 0 | 0 5 | 391107 | 391119 | 2023-04-18 18:00:18+01 | f | 733657184 | 0 | 0 | 0 | 0 | 0 | 0 | 0 6 | 467716 | 467730 | 2023-04-19 18:00:21+01 | f | 735633184 | 0 | 1976000 | 0 | 0 | 1000000 | 1000000 | 0 7 | 544978 | 544994 | 2023-04-20 18:00:26+01 | f | 729859616 | 0 | 702475600 | 708249168 | 0 | 1000000 | 1000000 | 0 8 | 621933 | 621950 | 2023-04-21 18:00:28+01 | f | 729859616 | 0 | 0 | 0 | 0 | 0 | 0 | 0 9 | 699410 | 699428 | 2023-04-22 18:00:31+01 | f | 729859616 | 0 | 0 | 0 | 0 | 0 | 0 | 0 10 | 777074 | 777093 | 2023-04-23 18:00:34+01 | f | 729859616 | 0 | 0 | 0 | 0 | 0 | 0 | 0 11 | 855530 | 855550 | 2023-04-24 18:00:36+01 | f | 729859616 | 0 | 0 | 0 | 0 | 0 | 0 | 0 12 | 933559 | 933586 | 2023-04-25 18:00:39+01 | f | 735866656 | 0 | 13832000 | 7824960 | 0 | 6000000 | 6000000 | 0 (13 rows) ``` ## Stack - #20118 - #20132 - #20147 - #20148 - #20149 --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API: commit e96fca283cf8f00fd09291fdbdcd401c2063b71b Author: Ashok Menon <[email protected]> Date: Sat Nov 2 02:05:09 2024 +0000 indexer-alt: kv_feature_flags pipeline ## Description Like `kv_protocol_configs`, but for feature flags. ## Test plan Run the indexer over the first 700K checkpoints -- this will include two protocol config upgrades, then check the database for feature flags. ``` sui$ cargo run -p sui-indexer-alt --release -- \ --database-url "postgres://postgres:postgrespw@localhost:5432/sui_indexer_alt" \ indexer --remote-store-url https://checkpoints.mainnet.sui.io \ --last-checkpoint 700000 ``` ``` sui_indexer_alt=# SELECT * FROM kv_feature_flags ORDER BY protocol_version DESC, flag_name; protocol_version | flag_name | flag_value ------------------+-------------------------------------------------------------+------------ 6 | accept_zklogin_in_multisig | f 6 | advance_epoch_start_time_in_safe_mode | t 6 | advance_to_highest_supported_protocol_version | f 6 | allow_receiving_object_id | f 6 | authority_capabilities_v2 | f [...] ``` commit 1aab0cf2b23b6d682446eae86937542f9f698e3c Author: Ashok Menon <[email protected]> Date: Sat Nov 2 01:40:35 2024 +0000 indexer-alt: kv_protocol_configs pipeline ## Description Add a pipeline for indexing protocol configs. ## Test plan Index the first ~1M checkpoints, which should pick up the first protocol version upgrade (from 4 to 5 for epoch 10) and then query `kv_protocol_configs` results after that: ``` sui$ cargo run -p sui-indexer-alt -- \ --database-url postgres://postgres:postgrespw@localhost:5432/sui_indexer_alt \ --remote-store-url https://checkpoints.mainnet.sui.io \ --last-checkpoint 500000 ``` ``` sui_indexer_alt=# SELECT * FROM kv_protocol_configs; ``` commit 97932677f340404a8843ac8fb46ae56ea0b7501c Author: Ashok Menon <[email protected]> Date: Sat Nov 2 01:18:58 2024 +0000 refactor(indexer-alt): configurable processor ## Description Change the pipeline/handler/processor interface to accept a value representing the handler/processor. This allows the processor to include configuration. This will be used by upcoming pipelines for processing protocol configs and feature flags. ## Test plan CI commit ae6275f6957ac936caffe04e82b7762ba192c4f8 Author: Anastasios Kichidis <[email protected]> Date: Wed Nov 13 12:38:22 2024 +0000 [Consensus] resubmit GC'ed transactions (#19572) ## Description Transactions of GC'ed blocks should be retried to remain compliant with the so far system assumptions. Refactored the consensus side to signal back the submitter about the status of the block that the submitted transaction has been included to. Practically two possible outcomes: 1. Sequenced (when the block has been committed and consequently the tx as well) 2. GarbageCollected (when the tx's block has not been committed and passed gc_round) also took the opportunity for a small refactoring around the `SubmitToConsensus` trait and the `ConsensusClient` as intentions started diverging. ## Test plan CI/PT --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API: commit bb0288b9b8de0307690a434eaf62a930cf1280e8 Author: benr-ml <[email protected]> Date: Wed Nov 13 14:35:19 2024 +0200 Enable G1Uncompressed in testnet (#20226) ## Description Describe the changes or additions included in this PR. ## Test plan How did you test the new or updated feature? --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API: commit 6ea6bc9b112bf34a3dcfb136430726924b03e0d0 Author: Mark Logan <[email protected]> Date: Tue Nov 12 23:08:08 2024 -0800 Fixes for overload protection (#20248) - Only check overload queues for mutable shared inputs - Increase max tx queue age to 500ms ## Description Describe the changes or additions included in this PR. ## Test plan How did you test the new or updated feature? --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API: commit 297913c0adcb677eb369d520e9de7c80696fbde3 Author: Mark Logan <[email protected]> Date: Tue Nov 12 21:21:06 2024 -0800 Fix bug with object_by_id inserts. (#20236) The previous version of this code was based on a misunderstanding of the Moka Cache API. It assumed that: let entry = cache.entry(k).or_insert_with(|| v); entry.is_fresh() could be used to detect whether there was a race with cache.insert(k, v) In fact this is not the case! Both threads must use `or_insert_with` in order to handle the race correctly. The bug is instantly reproducible with the following code: use moka::sync::Cache; use std::sync::{Arc, Mutex}; fn monotonic_update(cache: Arc<Cache<u64, Arc<Mutex<u64>>>>, key: u64, value: u64) { let entry = cache .entry(key) .or_insert_with(|| Arc::new(Mutex::new(value))); if !entry.is_fresh() { let mut entry = entry.value().lock().unwrap(); // only update if the new value is greater than the current value if value > *entry { *entry = value; } } } fn blind_write(cache: Arc<Cache<u64, Arc<Mutex<u64>>>>, key: u64, value: u64) { cache.insert(key, Arc::new(Mutex::new(value))); } fn main() { for _ in 0..1000 { let cache = Arc::new(Cache::new(1000)); let cache1 = cache.clone(); let cache2 = cache.clone(); let handle1 = std::thread::spawn(move || { monotonic_update(cache1, 1, 1); }); let handle2 = std::thread::spawn(move || { // Correct way to update the value // monotonic_update(cache2, 1, 2); // Incorrect way to update the value blind_write(cache2, 1, 2); }); handle1.join().unwrap(); handle2.join().unwrap(); let entry = cache.get(&1).unwrap(); let value = entry.lock().unwrap(); assert_eq!(*value, 2); } } ## Description Describe the changes or additions included in this PR. ## Test plan How did you test the new or updated feature? --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API: commit ec6d01642ad68b1dea024a3f34d5966359a3617e Author: Todd Nowacki <[email protected]> Date: Tue Nov 12 15:51:12 2024 -0800 [move-compiler] Improve interactions with Sui mode and interface files (#20196) ## Description - Sui mode used only named addresses for type comparison. This is not sufficient for interface files that might be using only the numerical address. - Switched `is` to use only the numerical address. We might want to evaluate differently if we start using it more outside of Sui mode ## Test plan - Ran tests --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API: commit 7b61e1ff70f4b2fbfd67085db2f795bcaa291105 Author: Avrora <[email protected]> Date: Wed Nov 13 01:42:06 2024 +0200 Docs: Fix typo (#20225) Fix typo ## Description Describe the changes or additions included in this PR. ## Test plan How did you test the new or updated feature? --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API: --------- Co-authored-by: ronny-mysten <[email protected]> commit f716589e9b7d966cea77dbe8145ed892fe4fd0cd Author: ronny-mysten <[email protected]> Date: Tue Nov 12 16:41:39 2024 -0700 [docs] Removed orphaned page (#20228) ## Description Removed orphaned page and updates the link that pointed to it. ## Test plan How did you test the new or updated feature? --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API: commit d5a23d78a9b30ec1c5ae5ac3cafdbe2817d68df6 Author: hayes-mysten <[email protected]> Date: Tue Nov 12 12:49:03 2024 -0800 [ts-sdk] Add tx.object.option for creatnig object options in transact… (#20194) …ion builder ## Description Describe the changes or additions included in this PR. ## Test plan How did you test the new or updated feature? --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API: commit de2c7de6f6b5d643560c0ca343b9009c96e7fefd Author: benr-ml <[email protected]> Date: Tue Nov 12 21:44:54 2024 +0200 Update fastcrypto, including moving some structs between dkg->dkg_v1 (#20224) ## Description Describe the changes or additions included in this PR. ## Test plan How did you test the new or updated feature? --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API: commit 44fe9eb64fed3a10ad5da8d3ad0d542b2f06b281 Author: Andrew Schran <[email protected]> Date: Tue Nov 12 14:44:34 2024 -0500 Revert "Add support for TLS connections with self-signed cert on validator gRPC interface" (#20232) Reverts MystenLabs/sui#19796 due to suspicion it's causing performance issues commit 73dbd005bb0f828a64eb4e811d4f8e2b3c80bdfe Author: sui-merge-bot[bot] <114704316+sui-merge-bot[bot]@users.noreply.github.com> Date: Tue Nov 12 13:38:32 2024 -0500 Version Packages (#20229) This PR was opened by the [Changesets release](https://github.com/changesets/action) GitHub action. When you're ready to do a release, you can merge this and publish to npm yourself or [setup this action to publish automatically](https://github.com/changesets/action#with-publishing). If you're not ready to do a release yet, that's fine, whenever you add more changesets to main, this PR will be updated. # Releases ## @mysten/[email protected] ### Patch Changes - a24d43d: getOrders and decodeOrderId support Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> commit c4afb26067e9a4e9eea75a706032f1764c4c3ded Author: Xun Li <[email protected]> Date: Tue Nov 12 10:17:08 2024 -0800 [indexer-alt] Add checkpoint ingestion lag metrics (#20191) ## Description This PR adds two metrics, one that tracks the ingestion lag on the latest fetched checkpoint, one that tracks histogram. These two combined could give us the full picture on the lag. ## Test plan CI. --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API: commit 4ba4b29a0ca9c473e89a1ba5b01caf71c31c5c21 Author: phoenix <[email protected]> Date: Tue Nov 12 12:47:34 2024 -0500 [kv store] add timeout to bigtable ingestion + backfill mode (#20193) ## Description adds timeouts to the Bigtable ingestion pipeline and introduces a backfill mode that skips most progress updates to DynamoDB during backfilling --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API: commit a24d43d709e1c05f98b092d50c2ac87b5c172489 Author: Tony Lee <[email protected]> Date: Tue Nov 12 12:38:38 2024 -0500 Deepbook SDK Examples (#20227) ## Description Getting balance for balance manager, getting live orders for a balance manager, and price/quantity map for a balance manager. Balance: ``` Manager: MANAGER_1 { coinType: '0x0000000000000000000000000000000000000000000000000000000000000002::sui::SUI', balance: 374527.5686 } { coinType: '0xdba34672e30cb065b1f93e3ab55318768fd6fef66c15942c9f7cb846e2f900e7::usdc::USDC', balance: 1492902.476423 } ``` Live Orders: ``` Manager: MANAGER_1 SUI_USDC [ '170141183460526324404595434778137174870', '170141183460526490425292098164101718871', '170141183460526490425292098164101718872', '56723756473400944927347700', '56502395544516430307955699', '55819866013789176898163698', '56723756473400944927347696' ] DEEP_SUI [ '413576002151014891304428578578', '421323634661972902983148578577', '170141183888249226801011805690925112362', '414498339354700368885228578576' ] ``` Price/Quantity Map: ``` Manager: MANAGER_1 SUI_USDC bid orders: [ [ 3.046, 6.6 ], [ 3.035, 21418.1 ], [ 3.027, 26427.4 ], [ 2.995, 46747.6 ] ] SUI_USDC ask orders: [ [ 3.049, 6.6 ], [ 3.065, 21207.2 ], [ 3.073, 26034.1 ], [ 3.096, 45223.3 ] ] ``` ## Test plan How did you test the new or updated feature? Locally tested using mainnet ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading …
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
introduce mvr-mode flag. If true, enables pruning objects_history, and toggles committer to write only a subset of tables to db. Compatible with the production indexer schema.
Test plan
How did you test the new or updated feature?
Release notes
Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required.
For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates.