[GraphQL] Object DataLoader

[amnn]

Description

Implement data loaders for fetching historical object versions, objects bounded by their parent versions, and the latest versions of an object at a given checkpoint.

By implementing an Object DataLoader, we also implicitly get support for data-loading all derived types (MoveObject, MovePackage, MoveModule, DynamicField, Coin, etc).

These implementations (particularly historical queries and queries where the version can be bounded by a parent version) can be made even more efficient with the existence of an index/side table that maps an object's ID and version to the checkpoint it is part of. This change has not been included in this PR, but we will follow up on this as part of Object query benchmarking.

As part of this change, I enabled queries for historical objects outside the available range. Later (with the use of an obj_version index) it will also be possible to enable dynamic field look-ups on historical objects as well.

Test Plan

sui$ cargo nextest run -p sui-graphql-rpc
sui$ cargo nextest run -p sui-graphql-e2e-tests --features pg_integration

Run the following query -- after this change, it takes about 8s to complete on the server, fetching about 80 objects, while previously it would either timeout or squeak in just under the 40s timeout. I expect this number to improve further once we have an efficient way to map object ids and versions to a checkpoint sequence number.

query {
  transactionBlocks(last: 5) {
    nodes {
      effects {
        objectChanges(first: 50) {
          pageInfo { hasNextPage }
          nodes {
            idCreated
            idDeleted
            inputState { asMoveObject { contents { json } } }
            outputState { asMoveObject { contents { json } } }
          }
        }
      }
    }
  }
}

---

Release notes

Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required.

For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates.

• Protocol:
• Nodes (Validators and Full nodes):
• Indexer:
• JSON-RPC:
• GraphQL: Queries for historical versions of objects now return data even if that version of the object is outside the available range.
• CLI:
• Rust SDK:

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
sui-docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Apr 29, 2024 2:05pm

3 Ignored Deployments

Name	Status	Preview	Comments	Updated (UTC)
multisig-toolkit	⬜️ Ignored (Inspect)	Visit Preview		Apr 29, 2024 2:05pm
sui-kiosk	⬜️ Ignored (Inspect)	Visit Preview		Apr 29, 2024 2:05pm
sui-typescript-docs	⬜️ Ignored (Inspect)	Visit Preview		Apr 29, 2024 2:05pm

[amnn]

The change in this test demonstrates how we can now query historical objects, but not their dynamic fields (because the dynamic field query still relies on the consistent object fetching logic).

[amnn]

Returning response for historical queries.

[amnn]

These pub(crate)-s are not strictly part of the data loading PR, but I noticed them here -- can pull them out into their own PR if it's helpful.

[amnn]

This change is also deleting some code that we no longer use, because we no longer fetch directly from the objects table.

[wlmyng]

ptal at the comments regarding using View::Consistent for the LatestAtKey dataloader!

[wlmyng]

I think you can just use View::Historical here; given the changes in this PR perhaps it's a bit of a misnomer, but since we aren't filtering by object_version or some other malleable criteria, and instead solely on object_id, we can avoid the bulky LEFT JOINs. Similarly in the paginated objects query, https://github.com/MystenLabs/sui/blob/main/crates/sui-graphql-rpc/src/types/object.rs#L1330, I believe the check should've been instead for whether the filter has anything that isn't empty or object_ids.

Alternatively, we can also do a single fetch for all object_ids to objects_snapshot, parallel fetches to objects_history per checkpoint bound, and then filter in app for the latest per group from either snapshot or history.

[wlmyng]

In fact, if we're at a cursor for an object that has a yet newer version, using View::Consistent will erroneously filter out the object because a newer version exists, although what we really want is the latest version of the object bounded by the checkpoint

[amnn]

TL;DR: I think View::Historical is pretty broken, so I won't use it here, I'll add a TODO in the codebase and create a task for fixing it).

I see what you're saying (although I had to write out both the historical and consistent queries to double check 😅) -- I think you're right that I can use the Historical query kind here, but I'm in two minds about it (see below).

but since we aren't filtering by object_version or some other malleable criteria,

Note that I am filtering by object_version, but that is what enables use of the historical mode IIRC.

I believe the check should've been instead for whether the filter has anything that isn't empty or object_ids.

I don't think this is right. If you only supply object_ids, then you are implicitly filtering the live object set at whatever checkpoint you're querying at, so historical queries do not work, because you need the behaviour of filtering outdated candidates (objects that match the filters but aren't latest in the checkpoint). It works for object_keys because you are fixing the version of objects you're querying for.

using View::Consistent will erroneously filter out the object because a newer version exists

I don't think this will happen, because the query to find newer versions is also bounded by the available range, and by the apply_parent_bound modifier that I pass to the last argument here.

Finally while View::Historical avoids the left joins:

• It doesn't bound the query on the historical side by the available range (so it's going to look into every partition), while still redundantly querying the snapshot which is a summary of some prefix of object history snapshots.
• In the long run, this loader will be split into two: One that handles object lookups by version, and one that handles latest objects. The former of these will be implemented using the object_versions table directly, so should hopefully be nice and cheap, and the latter is implemented most efficiently as a form of consistent lookup (i.e. avoiding looking into all object history partitions).

This actually makes me think that the Historical mode might be generally broken... imagine a case where you have an ObjectFilter that contains both object_ids and object_keys -- because it has object_keys it will be treated as a historical query, but it also fetching objects by ID, so it's going to fetch them from all partitions in objects_history and then take the latest version without applying a bound by available range, which is going to break consistency.

[wlmyng]

🚢