[indexer] Add full_objects_history #18994
Conversation
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
3 Skipped Deployments
|
|
@lxfind the change looks good but let's not merge to I am merging the last batch of breaking changes into main and reset that branch to latest main too, fixing a couple of minor issues regarding ci. |
|
Got it. Is there a doc that describes the process? For instance, does the oncall usually manually trigger a backfill/migration from the park branch from time to time? How does it identify gaps if the indexer isn't paused? |
lxfind
force-pushed
the
indexer-add-full-object-history
branch
from
55b2e1d
to
f81b98c
Compare
|
|
||
| INSERT INTO full_objects_history (object_id, object_version, object_status, serialized_object) | ||
| SELECT object_id, object_version, object_status, serialized_object | ||
| FROM objects_history; |
There was a problem hiding this comment.
sorry that I actually missed this, I thought this pr only defined the schema. this will not work as INSERT INTO will only run when we set up schema, and objects_history is empty then.
I think instead we will want to
- remove this
INSERT INTO - add separate ingestion codes to directly write to
full_objects_historybesidesobjects_history
and then when we deploy this pr + the pruning policy pr, we
- run both
CREATE TABLE full_objects_historyandINSERT INTOvia psql - update the writer pod with this pr + pruning policy pr and resume writer
- there will be a small gap after the last cp done by
INSERT INTOand the first checkpoint done by new ingestion codes, and we can run aINSERT INTOagain to patch that.
lxfind
force-pushed
the
indexer-add-full-object-history
branch
from
f81b98c
to
c784896
Compare
lxfind
force-pushed
the
indexer-add-full-object-history
branch
from
c784896
to
5314651
Compare
|
Added ingestion change. |
| @@ -572,16 +572,22 @@ impl<T: R2D2Connection + 'static> PgIndexerStore<T> { | |||
| .start_timer(); | |||
| let mut mutated_objects: Vec<StoredHistoryObject> = vec![]; | |||
| let mut object_versions: Vec<StoredObjectVersion> = vec![]; | |||
| let mut deleted_object_ids: Vec<StoredDeletedHistoryObject> = vec![]; | |||
| let mut deleted_objects: Vec<StoredDeletedHistoryObject> = vec![]; | |||
| let mut full_history_objects: Vec<StoredFullHistoryObject> = vec![]; | |||
There was a problem hiding this comment.
objects_history is one of the bottlenecks today, instead of adding more workload here, let's start a new parallel task for this here?
https://github.com/MystenLabs/sui/blob/main/crates/sui-indexer/src/handlers/committer.rs#L147
lxfind
force-pushed
the
indexer-add-full-object-history
branch
2 times, most recently
from
3b9d050
to
f899699
Compare
does this require deserialization of the bcs contents ? |
No, it only need to check if that column is null. |
lxfind
requested review from
stefan-mysten,
suiwombat,
ronny-mysten,
mystenmark,
joyqvq and
patrickkuo
as code owners
lxfind
requested review from
williamrobertson13 and
mystieanwaya
and removed request for
a team
lxfind
removed request for
a team,
wlmyng,
stefan-mysten,
suiwombat,
ronny-mysten,
mystenmark,
joyqvq,
patrickkuo,
williamrobertson13 and
mystieanwaya
lxfind
force-pushed
the
indexer-add-full-object-history
branch
from
f899699
to
a49146b
Compare
## Description This is a redo of #18994, but instead of merging to parking branch, merge it to main. ## Test plan CI --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API:
## Description This is a redo of #18994, but instead of merging to parking branch, merge it to main. ## Test plan CI --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API:
## Description This is a redo of #18994, but instead of merging to parking branch, merge it to main. ## Test plan CI --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API:
Description
We add a separate table that stores full object history that we do not plan to prune.
Test plan
How did you test the new or updated feature?
Release notes
Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required.
For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates.