[GraphQL/MovePackage] Paginate by checkpoint #17696

amnn · 2024-05-13T15:40:26Z

Description

Adds a query, Query.packages for fetching all packages that were introduced within a given checkpoint range. Useful for fetching package contents in bulk, to do local analyses.

Test plan

New E2E tests:

sui$ cargo nextest run -p sui-graphql-e2e-tests \
  --features pg_integration                     \
  -- packages/versioning

Also tested for performance against a large read replica (the query planner quotes a high estimate for the query but the actual results do not take very long to run because queries on many sub-partitions are eliminated).

Stack

Release notes

Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required.

For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates.

Protocol:
Nodes (Validators and Full nodes):
Indexer:
JSON-RPC:
GraphQL: Introduces Query.packages for paginating through all packages (optionally bounding by the checkpoint the package was introduced in).
CLI:
Rust SDK:

vercel · 2024-05-13T15:40:29Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
sui-docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Aug 19, 2024 10:47am

3 Skipped Deployments

Name	Status	Preview	Updated (UTC)
multisig-toolkit	⬜️ Ignored (Inspect)	Visit Preview	Aug 19, 2024 10:47am
sui-kiosk	⬜️ Ignored (Inspect)	Visit Preview	Aug 19, 2024 10:47am
sui-typescript-docs	⬜️ Ignored (Inspect)	Visit Preview	Aug 19, 2024 10:47am

wlmyng · 2024-05-13T17:16:29Z

crates/sui-graphql-rpc/src/types/move_package.rs

+                    SELECT
+                            p.original_id,
+                            o.*
+                    FROM
+                            packages p
+                    INNER JOIN
+                            objects_history o
+                    ON
+                            p.package_id = o.object_id
+                    AND     p.package_version = o.object_version
+                    AND     p.checkpoint_sequence_number = o.checkpoint_sequence_number


oh no.. did we give up on using the dsl haha

was it the typings or something else that gave you trouble

This was indeed a type issue. Specifically, the way paginate_query is written, it expects the underlying query's source to be the table that the cursor is paginating from, but in this case, the source is a join of two tables, and that messes everything up.

wlmyng · 2024-05-13T17:21:09Z

crates/sui-graphql-rpc/src/types/query.rs

+        after_checkpoint: Option<u64>,
+        before_checkpoint: Option<u64>,


bikeshedding: do we want to tuck any filtering criteria in a filter param?

I don't feel super strongly about it. The reason I didn't do it initially is that I'm introducing two different pagination queries for packages (by checkpoint and by version) and I didn't want to sew confusion by having the two different filter types, but I'm curious to hear what @hayes-mysten has to say on GraphQL ergonomics and what is easier to use in practice.

Not sure if I follow here, but I think having unique filters for different fields isn't a big deal for query ergonomics, but I'm not sure what the filters would be/how they would be different between the 2 query fields

I think I get it now, I don't have a strong preference between the 2 options, but wrapping in a filter would be more consistent with how the other filters work. Is the issue that we don't want to use checkpoint and versioning filters at the same time?

Yes that's right, so if we want to keep these two kinds of filters separate, we would need to introduce two different input types: MovePackageCheckpointFilter and MovePackageVersionFilter each with just a pair of numbers in, so it started to feel a bit overkill, but I buy a consistency argument.

## Description Remove `draft_target_schema.graphql` and promote `current_progress_schema.graphql` to be the canonical schema for the service -- move it to the top-level of the `sui-graphql-rpc` crate to make it easier to find. This is to avoid confusion about source of truth for the GraphQL schema. Because the TS SDK references the schema at multiple GraphQL versions, we will need to cherry-pick this change to release branches when it lands. ## Test plan CI ## Stack - #17543 - #17692 - #17693 - #17696 - #18287 --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [x] GraphQL: The schema file has been moved from `crates/sui-graphql-rpc/schemas/current_progress_schema.graphql` to `crates/sui-graphql-rpc/schema.graphql`. - [ ] CLI: - [ ] Rust SDK:

## Description Introduce two new queries: `Query.packageVersions` and `MovePackage.versions` for iterating over all the different versions of a given package. This kind of query is useful for understanding package history. These were introduced as a separate query, instead of having a single query for iterating over packages that could optionally take a checkpoint bounds or version bounds because of how system packages interact with the `packages` table: Because system packages are updated in-place, they only have one row in the `packages` table. This makes sense for paginating packages in bulk (e.g. by checkpoint) where the primary aim is to get a snapshot of the packages available at a certain point in time, but doesn't work for answering package version queries for system packages, and it prevents us from creating a combined query. A combined query would also allow someone to create a filter that bounds checkpoints and versions, but doesn't bound the package itself (or would require us to prevent that combination), which is complicated to implement efficiently and not particularly useful. ## Test plan New E2E tests: ``` sui$ cargo nextest run -p sui-graphql-e2e-tests \ --features pg_integration \ -- packages/versioning ``` & Testing against a read replica to make sure system package tests work well, and performance is reasonable. ## Stack - #17686 - #17687 - #17688 - #17689 - #17691 - #17694 - #17695 - #17542 - #17690 - #17543 - #17692 - #17693 - #17696 --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [x] GraphQL: Introduces `Query.packageVersions` and `MovePackage.versions` for paginating over the versions of a particular package. - [ ] CLI: - [ ] Rust SDK:

## Description Remove commands to generate examples, markdown and schema from the main binary as we do not use them: - Instead of generating examples, we have hand-crafted examples in our docs. Removing this code also removes a test that forces regeneration of a markdown file from these docs (which we also were not using). - We also never used the output from the `generate-schema` sub-command, because the schema was always available as a file, or via introspection commands from the running service. - Logic for gathering examples to test has been moved into the test file, to avoid including test-only code in the main library. ## Test plan CI ## Description Describe the changes or additions included in this PR. ## Test plan How did you test the new or updated feature? ## Stack - #17543 - #17692 - #17693 - #17696 - #17697 --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [x] GraphQL: The GraphQL binary no longer supports generating examples, or exporting its own schema as these commands have been unused for some time. - [ ] CLI: - [ ] Rust SDK:

## Description Remove `draft_target_schema.graphql` and promote `current_progress_schema.graphql` to be the canonical schema for the service -- move it to the top-level of the `sui-graphql-rpc` crate to make it easier to find. This is to avoid confusion about source of truth for the GraphQL schema. Because the TS SDK references the schema at multiple GraphQL versions, we will need to cherry-pick this change to release branches when it lands. ## Test plan CI ## Stack - #17543 - #17692 - #17693 - #17696 - #18287 --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [x] GraphQL: The schema file has been moved from `crates/sui-graphql-rpc/schemas/current_progress_schema.graphql` to `crates/sui-graphql-rpc/schema.graphql`. - [ ] CLI: - [ ] Rust SDK:

## Description Add a command for generating a config TOML file for the GraphQL service with all its parameters set to their default values. (We used to have a similar command for the YAML file which we weren't using, but we still use the TOML file). ## Test plan ``` cargo run --bin sui-graphql-rpc -- generate-config /tmp/config.toml ``` ## Stack - #17543 - #17692 - #17693 - #17696 - #18287 - #18288 --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [x] GraphQL: New sub-command for `sui-graphql-rpc`, `generate-config` for creating a TOML config with all default values set. - [ ] CLI: - [ ] Rust SDK:

## Description Replace the original implementation of the dump-packages command (which requires access to an indexer database) with an implementation that reads from a GraphQL service. The former is not readily accessible, but the latter should be. The new tool is also able to run incrementally: Fetching only packages created before a certain checkpoint, or pick up where a previous invocation took off to fetch new packages that were introduced since. ## Test plan Ran a test invocation, on our experimental read replica. With a max page size of 200, I was able to fetch 17000 packages (all the packages at the time the read replica was created) in 3 minutes. ## Stack - #17543 - #17692 - #17693 - #17696 - #18287 - #18288 - #18336 --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK:

## Description Adds a query, `Query.packages` for fetching all packages that were introduced within a given checkpoint range. Useful for fetching package contents in bulk, to do local analyses. ## Test plan New E2E tests: ``` sui$ cargo nextest run -p sui-graphql-e2e-tests \ --features pg_integration \ -- packages/versioning ``` Also tested for performance against a large read replica (the query planner quotes a high estimate for the query but the actual results do not take very long to run because queries on many sub-partitions are eliminated). ## Stack - #17686 - #17687 - #17688 - #17689 - #17691 - #17694 - #17695 - #17542 - #17726 - #17543 - #17692 - #17693 --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [x] GraphQL: Introduces `Query.packages` for paginating through all packages (optionally bounding by the checkpoint the package was introduced in). - [ ] CLI: - [ ] Rust SDK: