[GraphQL] Leverage `objects_version` table. #17543

amnn · 2024-05-07T13:29:38Z

Description

Use the objects_version table to speed up point look-ups (via data loaders) for historical objects (ID + version), and dynamic fields (object look-up bounding version by parent ID).

With this change, the restriction of accessing dynamic fields only within the available range is dropped.

Test plan

sui$ cargo nextest run -p sui-graphql-rpc
sui$ cargo nextest run -p sui-graphql-e2e-tests --features pg_integration.

Perform a query that involves fetching a large number of dynamic fields, which should now be fast. The following example, fetching dynamic fields on a deepbook pool loads 50 dynamic fields in about 5s from cold (which also requires loading packages for resolution), and then 2s from there:

query {
  owner(
    address: "0x029170bfa0a1677054263424fe4f9960c7cf05d359f6241333994c8830772bdb"
  ) {
    dynamicFields {
      pageInfo {
        hasNextPage
        endCursor
      }
      nodes {
        name {
          type {
            repr
          }
          json
        }
        value {
          ... on MoveValue {
            type {
              repr
            }
            json
          }
          ... on MoveObject {
            contents {
              json
              type {
                repr
              }
            }
          }
        }
      }
    }
  }
}

Stack

Release notes

Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required.

For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates.

vercel · 2024-05-07T13:29:41Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
sui-docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Aug 20, 2024 0:24am

3 Skipped Deployments

Name	Status	Preview	Updated (UTC)
multisig-toolkit	⬜️ Ignored (Inspect)	Visit Preview	Aug 20, 2024 0:24am
sui-kiosk	⬜️ Ignored (Inspect)	Visit Preview	Aug 20, 2024 0:24am
sui-typescript-docs	⬜️ Ignored (Inspect)	Visit Preview	Aug 20, 2024 0:24am

stefan-mysten

Nice!

## Description Add a new kind of package point look-up to get the latest version of the package at a given ID (or from another `MovePackage`). For system packages, this is analogous to getting the latest version of the object at that ID, but the versions of other packages all exist at different IDs. ## Test plan New transactional tests: ``` sui$ cargo nextest run -p sui-graphql-e2e-tests \ --features pg_integration \ -- packages/versioning ``` ## Stack - #17686 - #17687 - #17688 - #17689 - #17691 - #17694 - #17695 - #17542 - #17726 - #17543 - #17692 --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [x] GraphQL: Add `Query.latestPackage` and `MovePackage.latest` for fetching the latest version of a package. - [ ] CLI: - [ ] Rust SDK:

## Description Adds a query, `Query.packages` for fetching all packages that were introduced within a given checkpoint range. Useful for fetching package contents in bulk, to do local analyses. ## Test plan New E2E tests: ``` sui$ cargo nextest run -p sui-graphql-e2e-tests \ --features pg_integration \ -- packages/versioning ``` Also tested for performance against a large read replica (the query planner quotes a high estimate for the query but the actual results do not take very long to run because queries on many sub-partitions are eliminated). ## Stack - #17686 - #17687 - #17688 - #17689 - #17691 - #17694 - #17695 - #17542 - #17726 - #17543 - #17692 - #17693 --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [x] GraphQL: Introduces `Query.packages` for paginating through all packages (optionally bounding by the checkpoint the package was introduced in). - [ ] CLI: - [ ] Rust SDK:

## Description Introduce two new queries: `Query.packageVersions` and `MovePackage.versions` for iterating over all the different versions of a given package. This kind of query is useful for understanding package history. These were introduced as a separate query, instead of having a single query for iterating over packages that could optionally take a checkpoint bounds or version bounds because of how system packages interact with the `packages` table: Because system packages are updated in-place, they only have one row in the `packages` table. This makes sense for paginating packages in bulk (e.g. by checkpoint) where the primary aim is to get a snapshot of the packages available at a certain point in time, but doesn't work for answering package version queries for system packages, and it prevents us from creating a combined query. A combined query would also allow someone to create a filter that bounds checkpoints and versions, but doesn't bound the package itself (or would require us to prevent that combination), which is complicated to implement efficiently and not particularly useful. ## Test plan New E2E tests: ``` sui$ cargo nextest run -p sui-graphql-e2e-tests \ --features pg_integration \ -- packages/versioning ``` & Testing against a read replica to make sure system package tests work well, and performance is reasonable. ## Stack - #17686 - #17687 - #17688 - #17689 - #17691 - #17694 - #17695 - #17542 - #17690 - #17543 - #17692 - #17693 - #17696 --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [x] GraphQL: Introduces `Query.packageVersions` and `MovePackage.versions` for paginating over the versions of a particular package. - [ ] CLI: - [ ] Rust SDK:

## Description Remove commands to generate examples, markdown and schema from the main binary as we do not use them: - Instead of generating examples, we have hand-crafted examples in our docs. Removing this code also removes a test that forces regeneration of a markdown file from these docs (which we also were not using). - We also never used the output from the `generate-schema` sub-command, because the schema was always available as a file, or via introspection commands from the running service. - Logic for gathering examples to test has been moved into the test file, to avoid including test-only code in the main library. ## Test plan CI ## Description Describe the changes or additions included in this PR. ## Test plan How did you test the new or updated feature? ## Stack - #17543 - #17692 - #17693 - #17696 - #17697 --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [x] GraphQL: The GraphQL binary no longer supports generating examples, or exporting its own schema as these commands have been unused for some time. - [ ] CLI: - [ ] Rust SDK:

## Description Remove `draft_target_schema.graphql` and promote `current_progress_schema.graphql` to be the canonical schema for the service -- move it to the top-level of the `sui-graphql-rpc` crate to make it easier to find. This is to avoid confusion about source of truth for the GraphQL schema. Because the TS SDK references the schema at multiple GraphQL versions, we will need to cherry-pick this change to release branches when it lands. ## Test plan CI ## Stack - #17543 - #17692 - #17693 - #17696 - #18287 --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [x] GraphQL: The schema file has been moved from `crates/sui-graphql-rpc/schemas/current_progress_schema.graphql` to `crates/sui-graphql-rpc/schema.graphql`. - [ ] CLI: - [ ] Rust SDK:

## Description Add a command for generating a config TOML file for the GraphQL service with all its parameters set to their default values. (We used to have a similar command for the YAML file which we weren't using, but we still use the TOML file). ## Test plan ``` cargo run --bin sui-graphql-rpc -- generate-config /tmp/config.toml ``` ## Stack - #17543 - #17692 - #17693 - #17696 - #18287 - #18288 --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [x] GraphQL: New sub-command for `sui-graphql-rpc`, `generate-config` for creating a TOML config with all default values set. - [ ] CLI: - [ ] Rust SDK:

## Description Gathering all the suggested docs fixes across the stack in #17543 into one PR. ## Test plan :eyes:

ronny-mysten · 2024-08-20T14:30:53Z

Thanks for the review. I'll put up a separate PR to address these because this PR is now a stack, being prepared to land as-is.

Oops. Apologies. I'll start separate PRs after merge instead going forward.

## Description Gathering all the suggested docs fixes across the stack in #17543 into one PR. ## Test plan :eyes: ## Stack - #19047 --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API:

amnn · 2024-08-20T18:56:42Z

Oops. Apologies. I'll start separate PRs after merge instead going forward.

@ronny-mysten, no you're all good! This was a special case because I was landing all the other PRs in the stack into this one so I could then land them all together (it's faster to do it this way -- need to wait for fewer CI runs on the critical path).

I think it's fine to leave the feedback on this PR, I just wanted to explain that I was going to address it in a separate PR, so it wasn't confusing when I resolved the conversation but then landed this PR without you seeing the comments addressed here.

## Description Use the `objects_version` table to speed up point look-ups (via data loaders) for historical objects (ID + version), and dynamic fields (object look-up bounding version by parent ID). With this change, the restriction of accessing dynamic fields only within the available range is dropped. ## Test plan ``` sui$ cargo nextest run -p sui-graphql-rpc sui$ cargo nextest run -p sui-graphql-e2e-tests --features pg_integration. ``` Perform a query that involves fetching a large number of dynamic fields, which should now be fast. The following example, fetching dynamic fields on a deepbook pool loads 50 dynamic fields in about 5s from cold (which also requires loading packages for resolution), and then 2s from there: ``` query { owner( address: "0x029170bfa0a1677054263424fe4f9960c7cf05d359f6241333994c8830772bdb" ) { dynamicFields { pageInfo { hasNextPage endCursor } nodes { name { type { repr } json } value { ... on MoveValue { type { repr } json } ... on MoveObject { contents { json type { repr } } } } } } } } ``` ## Stack - MystenLabs#17686 - MystenLabs#17687 - MystenLabs#17688 - MystenLabs#17689 - MystenLabs#17691 - MystenLabs#17694 - MystenLabs#17695 - MystenLabs#17542 - MystenLabs#17726 --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [x] GraphQL: Dynamic fields can now be looked up on any historical object (not just objects in the available range). - [ ] CLI: - [ ] Rust SDK:

## Description Implement `Query.package` and `MovePackage.atVersion` to query a package at a specific version, using the new fields added to the `packages` table, exposed via some new data loaders. ## Test plan New transactional tests: ``` sui$ cargo nextest run -p sui-graphql-e2e-tests \ --features pg_integration \ -- packages/versioning ``` ## Stack - MystenLabs#17686 - MystenLabs#17687 - MystenLabs#17688 - MystenLabs#17689 - MystenLabs#17691 - MystenLabs#17694 - MystenLabs#17695 - MystenLabs#17542 - MystenLabs#17726 - MystenLabs#17543 - MystenLabs#17692 --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [x] GraphQL: Introduce `Query.package` and `MovePackage.atVersion` to query packages at specific versions. - [ ] CLI: - [ ] Rust SDK:

## Description Add a new kind of package point look-up to get the latest version of the package at a given ID (or from another `MovePackage`). For system packages, this is analogous to getting the latest version of the object at that ID, but the versions of other packages all exist at different IDs. ## Test plan New transactional tests: ``` sui$ cargo nextest run -p sui-graphql-e2e-tests \ --features pg_integration \ -- packages/versioning ``` ## Stack - MystenLabs#17686 - MystenLabs#17687 - MystenLabs#17688 - MystenLabs#17689 - MystenLabs#17691 - MystenLabs#17694 - MystenLabs#17695 - MystenLabs#17542 - MystenLabs#17726 - MystenLabs#17543 - MystenLabs#17692 --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [x] GraphQL: Add `Query.latestPackage` and `MovePackage.latest` for fetching the latest version of a package. - [ ] CLI: - [ ] Rust SDK:

## Description Adds a query, `Query.packages` for fetching all packages that were introduced within a given checkpoint range. Useful for fetching package contents in bulk, to do local analyses. ## Test plan New E2E tests: ``` sui$ cargo nextest run -p sui-graphql-e2e-tests \ --features pg_integration \ -- packages/versioning ``` Also tested for performance against a large read replica (the query planner quotes a high estimate for the query but the actual results do not take very long to run because queries on many sub-partitions are eliminated). ## Stack - MystenLabs#17686 - MystenLabs#17687 - MystenLabs#17688 - MystenLabs#17689 - MystenLabs#17691 - MystenLabs#17694 - MystenLabs#17695 - MystenLabs#17542 - MystenLabs#17726 - MystenLabs#17543 - MystenLabs#17692 - MystenLabs#17693 --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [x] GraphQL: Introduces `Query.packages` for paginating through all packages (optionally bounding by the checkpoint the package was introduced in). - [ ] CLI: - [ ] Rust SDK:

## Description Introduce two new queries: `Query.packageVersions` and `MovePackage.versions` for iterating over all the different versions of a given package. This kind of query is useful for understanding package history. These were introduced as a separate query, instead of having a single query for iterating over packages that could optionally take a checkpoint bounds or version bounds because of how system packages interact with the `packages` table: Because system packages are updated in-place, they only have one row in the `packages` table. This makes sense for paginating packages in bulk (e.g. by checkpoint) where the primary aim is to get a snapshot of the packages available at a certain point in time, but doesn't work for answering package version queries for system packages, and it prevents us from creating a combined query. A combined query would also allow someone to create a filter that bounds checkpoints and versions, but doesn't bound the package itself (or would require us to prevent that combination), which is complicated to implement efficiently and not particularly useful. ## Test plan New E2E tests: ``` sui$ cargo nextest run -p sui-graphql-e2e-tests \ --features pg_integration \ -- packages/versioning ``` & Testing against a read replica to make sure system package tests work well, and performance is reasonable. ## Stack - MystenLabs#17686 - MystenLabs#17687 - MystenLabs#17688 - MystenLabs#17689 - MystenLabs#17691 - MystenLabs#17694 - MystenLabs#17695 - MystenLabs#17542 - MystenLabs#17690 - MystenLabs#17543 - MystenLabs#17692 - MystenLabs#17693 - MystenLabs#17696 --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [x] GraphQL: Introduces `Query.packageVersions` and `MovePackage.versions` for paginating over the versions of a particular package. - [ ] CLI: - [ ] Rust SDK:

## Description Remove commands to generate examples, markdown and schema from the main binary as we do not use them: - Instead of generating examples, we have hand-crafted examples in our docs. Removing this code also removes a test that forces regeneration of a markdown file from these docs (which we also were not using). - We also never used the output from the `generate-schema` sub-command, because the schema was always available as a file, or via introspection commands from the running service. - Logic for gathering examples to test has been moved into the test file, to avoid including test-only code in the main library. ## Test plan CI ## Description Describe the changes or additions included in this PR. ## Test plan How did you test the new or updated feature? ## Stack - MystenLabs#17543 - MystenLabs#17692 - MystenLabs#17693 - MystenLabs#17696 - MystenLabs#17697 --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [x] GraphQL: The GraphQL binary no longer supports generating examples, or exporting its own schema as these commands have been unused for some time. - [ ] CLI: - [ ] Rust SDK:

## Description Remove `draft_target_schema.graphql` and promote `current_progress_schema.graphql` to be the canonical schema for the service -- move it to the top-level of the `sui-graphql-rpc` crate to make it easier to find. This is to avoid confusion about source of truth for the GraphQL schema. Because the TS SDK references the schema at multiple GraphQL versions, we will need to cherry-pick this change to release branches when it lands. ## Test plan CI ## Stack - MystenLabs#17543 - MystenLabs#17692 - MystenLabs#17693 - MystenLabs#17696 - MystenLabs#18287 --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [x] GraphQL: The schema file has been moved from `crates/sui-graphql-rpc/schemas/current_progress_schema.graphql` to `crates/sui-graphql-rpc/schema.graphql`. - [ ] CLI: - [ ] Rust SDK:

## Description Add a command for generating a config TOML file for the GraphQL service with all its parameters set to their default values. (We used to have a similar command for the YAML file which we weren't using, but we still use the TOML file). ## Test plan ``` cargo run --bin sui-graphql-rpc -- generate-config /tmp/config.toml ``` ## Stack - MystenLabs#17543 - MystenLabs#17692 - MystenLabs#17693 - MystenLabs#17696 - MystenLabs#18287 - MystenLabs#18288 --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [x] GraphQL: New sub-command for `sui-graphql-rpc`, `generate-config` for creating a TOML config with all default values set. - [ ] CLI: - [ ] Rust SDK:

## Description Replace the original implementation of the dump-packages command (which requires access to an indexer database) with an implementation that reads from a GraphQL service. The former is not readily accessible, but the latter should be. The new tool is also able to run incrementally: Fetching only packages created before a certain checkpoint, or pick up where a previous invocation took off to fetch new packages that were introduced since. ## Test plan Ran a test invocation, on our experimental read replica. With a max page size of 200, I was able to fetch 17000 packages (all the packages at the time the read replica was created) in 3 minutes. ## Stack - MystenLabs#17543 - MystenLabs#17692 - MystenLabs#17693 - MystenLabs#17696 - MystenLabs#18287 - MystenLabs#18288 - MystenLabs#18336 --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK:

## Description Gathering all the suggested docs fixes across the stack in MystenLabs#17543 into one PR. ## Test plan :eyes: ## Stack - MystenLabs#19047 --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API:

## Description Use the `objects_version` table to speed up point look-ups (via data loaders) for historical objects (ID + version), and dynamic fields (object look-up bounding version by parent ID). With this change, the restriction of accessing dynamic fields only within the available range is dropped. ## Test plan ``` sui$ cargo nextest run -p sui-graphql-rpc sui$ cargo nextest run -p sui-graphql-e2e-tests --features pg_integration. ``` Perform a query that involves fetching a large number of dynamic fields, which should now be fast. The following example, fetching dynamic fields on a deepbook pool loads 50 dynamic fields in about 5s from cold (which also requires loading packages for resolution), and then 2s from there: ``` query { owner( address: "0x029170bfa0a1677054263424fe4f9960c7cf05d359f6241333994c8830772bdb" ) { dynamicFields { pageInfo { hasNextPage endCursor } nodes { name { type { repr } json } value { ... on MoveValue { type { repr } json } ... on MoveObject { contents { json type { repr } } } } } } } } ``` ## Stack - #17686 - #17687 - #17688 - #17689 - #17691 - #17694 - #17695 - #17542 - #17726 --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [x] GraphQL: Dynamic fields can now be looked up on any historical object (not just objects in the available range). - [ ] CLI: - [ ] Rust SDK:

## Description Implement `Query.package` and `MovePackage.atVersion` to query a package at a specific version, using the new fields added to the `packages` table, exposed via some new data loaders. ## Test plan New transactional tests: ``` sui$ cargo nextest run -p sui-graphql-e2e-tests \ --features pg_integration \ -- packages/versioning ``` ## Stack - #17686 - #17687 - #17688 - #17689 - #17691 - #17694 - #17695 - #17542 - #17726 - #17543 - #17692 --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [x] GraphQL: Introduce `Query.package` and `MovePackage.atVersion` to query packages at specific versions. - [ ] CLI: - [ ] Rust SDK:

## Description Add a new kind of package point look-up to get the latest version of the package at a given ID (or from another `MovePackage`). For system packages, this is analogous to getting the latest version of the object at that ID, but the versions of other packages all exist at different IDs. ## Test plan New transactional tests: ``` sui$ cargo nextest run -p sui-graphql-e2e-tests \ --features pg_integration \ -- packages/versioning ``` ## Stack - #17686 - #17687 - #17688 - #17689 - #17691 - #17694 - #17695 - #17542 - #17726 - #17543 - #17692 --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [x] GraphQL: Add `Query.latestPackage` and `MovePackage.latest` for fetching the latest version of a package. - [ ] CLI: - [ ] Rust SDK:

## Description Adds a query, `Query.packages` for fetching all packages that were introduced within a given checkpoint range. Useful for fetching package contents in bulk, to do local analyses. ## Test plan New E2E tests: ``` sui$ cargo nextest run -p sui-graphql-e2e-tests \ --features pg_integration \ -- packages/versioning ``` Also tested for performance against a large read replica (the query planner quotes a high estimate for the query but the actual results do not take very long to run because queries on many sub-partitions are eliminated). ## Stack - #17686 - #17687 - #17688 - #17689 - #17691 - #17694 - #17695 - #17542 - #17726 - #17543 - #17692 - #17693 --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [x] GraphQL: Introduces `Query.packages` for paginating through all packages (optionally bounding by the checkpoint the package was introduced in). - [ ] CLI: - [ ] Rust SDK:

## Description Introduce two new queries: `Query.packageVersions` and `MovePackage.versions` for iterating over all the different versions of a given package. This kind of query is useful for understanding package history. These were introduced as a separate query, instead of having a single query for iterating over packages that could optionally take a checkpoint bounds or version bounds because of how system packages interact with the `packages` table: Because system packages are updated in-place, they only have one row in the `packages` table. This makes sense for paginating packages in bulk (e.g. by checkpoint) where the primary aim is to get a snapshot of the packages available at a certain point in time, but doesn't work for answering package version queries for system packages, and it prevents us from creating a combined query. A combined query would also allow someone to create a filter that bounds checkpoints and versions, but doesn't bound the package itself (or would require us to prevent that combination), which is complicated to implement efficiently and not particularly useful. ## Test plan New E2E tests: ``` sui$ cargo nextest run -p sui-graphql-e2e-tests \ --features pg_integration \ -- packages/versioning ``` & Testing against a read replica to make sure system package tests work well, and performance is reasonable. ## Stack - #17686 - #17687 - #17688 - #17689 - #17691 - #17694 - #17695 - #17542 - #17690 - #17543 - #17692 - #17693 - #17696 --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [x] GraphQL: Introduces `Query.packageVersions` and `MovePackage.versions` for paginating over the versions of a particular package. - [ ] CLI: - [ ] Rust SDK:

## Description Remove commands to generate examples, markdown and schema from the main binary as we do not use them: - Instead of generating examples, we have hand-crafted examples in our docs. Removing this code also removes a test that forces regeneration of a markdown file from these docs (which we also were not using). - We also never used the output from the `generate-schema` sub-command, because the schema was always available as a file, or via introspection commands from the running service. - Logic for gathering examples to test has been moved into the test file, to avoid including test-only code in the main library. ## Test plan CI ## Description Describe the changes or additions included in this PR. ## Test plan How did you test the new or updated feature? ## Stack - #17543 - #17692 - #17693 - #17696 - #17697 --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [x] GraphQL: The GraphQL binary no longer supports generating examples, or exporting its own schema as these commands have been unused for some time. - [ ] CLI: - [ ] Rust SDK:

## Description Remove `draft_target_schema.graphql` and promote `current_progress_schema.graphql` to be the canonical schema for the service -- move it to the top-level of the `sui-graphql-rpc` crate to make it easier to find. This is to avoid confusion about source of truth for the GraphQL schema. Because the TS SDK references the schema at multiple GraphQL versions, we will need to cherry-pick this change to release branches when it lands. ## Test plan CI ## Stack - #17543 - #17692 - #17693 - #17696 - #18287 --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [x] GraphQL: The schema file has been moved from `crates/sui-graphql-rpc/schemas/current_progress_schema.graphql` to `crates/sui-graphql-rpc/schema.graphql`. - [ ] CLI: - [ ] Rust SDK:

## Description Add a command for generating a config TOML file for the GraphQL service with all its parameters set to their default values. (We used to have a similar command for the YAML file which we weren't using, but we still use the TOML file). ## Test plan ``` cargo run --bin sui-graphql-rpc -- generate-config /tmp/config.toml ``` ## Stack - #17543 - #17692 - #17693 - #17696 - #18287 - #18288 --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [x] GraphQL: New sub-command for `sui-graphql-rpc`, `generate-config` for creating a TOML config with all default values set. - [ ] CLI: - [ ] Rust SDK:

## Description Replace the original implementation of the dump-packages command (which requires access to an indexer database) with an implementation that reads from a GraphQL service. The former is not readily accessible, but the latter should be. The new tool is also able to run incrementally: Fetching only packages created before a certain checkpoint, or pick up where a previous invocation took off to fetch new packages that were introduced since. ## Test plan Ran a test invocation, on our experimental read replica. With a max page size of 200, I was able to fetch 17000 packages (all the packages at the time the read replica was created) in 3 minutes. ## Stack - #17543 - #17692 - #17693 - #17696 - #18287 - #18288 - #18336 --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK:

## Description Gathering all the suggested docs fixes across the stack in #17543 into one PR. ## Test plan :eyes: ## Stack - #19047 --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API:

amnn requested review from gegaowp and a team May 7, 2024 13:29

amnn self-assigned this May 7, 2024

amnn requested review from wlmyng, stefan-mysten, emmazzz and suiwombat as code owners May 7, 2024 13:29

stefan-mysten approved these changes May 8, 2024

View reviewed changes

gegaowp approved these changes May 9, 2024

View reviewed changes

amnn force-pushed the amnn/idx-obj-versions branch from 7d41560 to 561c004 Compare May 9, 2024 15:58

amnn force-pushed the amnn/gql-obj-versions branch from 2b132dd to 430d60e Compare May 10, 2024 17:44

vercel bot deployed to Preview – sui-docs May 10, 2024 17:46 View deployment

amnn force-pushed the amnn/idx-obj-versions branch from 9de6ee4 to c50d1ff Compare May 13, 2024 15:14

amnn force-pushed the amnn/gql-obj-versions branch from 430d60e to 4b36d48 Compare May 13, 2024 15:22

vercel bot deployed to Preview – sui-docs May 13, 2024 15:23 View deployment

wlmyng approved these changes May 13, 2024

View reviewed changes

amnn force-pushed the amnn/idx-obj-versions branch from c50d1ff to 057bc99 Compare May 14, 2024 11:15

amnn requested a review from ronny-mysten as a code owner May 14, 2024 11:15

amnn force-pushed the amnn/gql-obj-versions branch from 4b36d48 to 8fd7f36 Compare May 14, 2024 11:15

amnn changed the base branch from amnn/idx-obj-versions to amnn/idx-packages+ May 14, 2024 11:22

vercel bot deployed to Preview – sui-docs May 14, 2024 11:26 View deployment

amnn added a commit that referenced this pull request Aug 20, 2024

fix(graphql): Docs fixes from #17543

a671b11

## Description Gathering all the suggested docs fixes across the stack in #17543 into one PR. ## Test plan :eyes:

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[GraphQL] Leverage `objects_version` table. #17543

[GraphQL] Leverage `objects_version` table. #17543

amnn commented May 7, 2024 •

edited

Loading

vercel bot commented May 7, 2024 •

edited

Loading

stefan-mysten left a comment

ronny-mysten commented Aug 20, 2024 •

edited

Loading

amnn commented Aug 20, 2024

[GraphQL] Leverage objects_version table. #17543

[GraphQL] Leverage objects_version table. #17543

Conversation

amnn commented May 7, 2024 • edited Loading

Description

Test plan

Stack

Release notes

vercel bot commented May 7, 2024 • edited Loading

stefan-mysten left a comment

Choose a reason for hiding this comment

ronny-mysten commented Aug 20, 2024 • edited Loading

amnn commented Aug 20, 2024

[GraphQL] Leverage `objects_version` table. #17543

[GraphQL] Leverage `objects_version` table. #17543

amnn commented May 7, 2024 •

edited

Loading

vercel bot commented May 7, 2024 •

edited

Loading

ronny-mysten commented Aug 20, 2024 •

edited

Loading