Archival bucketlist #4403

SirTyson · 2024-07-30T18:38:13Z

Description

Resolves #4394

Depends on XDR changes in stellar/stellar-xdr#200.

This change refactors the BucketList so that we can have multiple Bucket Lists simultaneously. This is necessary for State Archival, where validators will maintain a live BucketList, hot archive BucketList, and cold archive BucketList. This change does not yet support writing the new archival BucketLists to history,

Each BucketList has small differences wrt to the entry types it stores, merge logic, and how lookups are done, but the overall BucketList level logic is the same. Because of this, it seemed easiest to template the Bucket related classes and specialize a few functions. The difference are as follows (I'll be updating the Archival CAP with this info soon)

Entry Types:
Live BucketList: BucketEntry
Hot Archive BucketList: HotArchiveBucketEntry.

This change was necessary due to how LedgerKeys are stored. In the Live BucketList, only DEADENTRY store plain LedgerKey. DEADENTRY is equivalent to null, so these LedgerKeys are dropped at the bottom level bucket.

Hot Archives require two types that store LedgerKeys: HA_LIVE and HA_DELETED. HA_LIVE is equivalent to the "null" type in the Live BucketList, and is dropped in the bottom bucket. However, HA_DELETED needs to be persisted.

Merging:
In the hot archive, LedgerEntry are never updated. They may be overwritten by another HotArchiveBucketEntry type, but the LedgerEntry contents itself never change. This means that the INITENTRY, LIVEENTRY, and DEADENTRY abstraction doesn't really make sense. This makes the merge logic for Hot Archive buckets very simple, the newer entry always overwrites the older entry.

This PR adds the concept of a "Tombstone" entry. A tombstone entry essentially represents null in the given BucketList and can be dropped once it reaches the bottom bucket. On the live BucketList, DEADENTRY is the tombstone type. On the Hot Archive BucketList, HA_LIVE is the tombstone.

BucketListDB lookup:
The live BucketList only ever returns LedgerEntry types when queried. Any DEADENTRY LedgerKey is simply omitted from the return, as this is the "null" tombstone type.

Hot archive lookups must return both LedgerKey and LedgerEntry types. HA_LIVE entries are of type LedgerKey and can be omitted from the return since they are the tombstone type. However, HA_ARCHIVED entries are not tombstone entries and must be returned when found. Do to this, Hot Archive lookups return HotArchiveBucketEntry instead of LedgerEntry when queried.

Future updates will add more functionality to Hot Archives, further distinguishing it from the live BucketList, and add a third BucketList type called the cold archive. This cold archive will add a third BucketEntry type and have different merge logic as well, so I think the templated Bucket classes, while verbose, are warranted.

This is currently a draft until the XDR changes are merged.

Checklist

Reviewed the contributing document
Rebased on top of master (no merge commits)
Ran clang-format v8.0.0 (via make format or the Visual Studio extension)
Compiles
Ran all tests
If change impacts performance, include supporting evidence per the performance document

dmkozh · 2024-08-22T22:51:54Z

Cargo.toml

@@ -3,7 +3,7 @@ members = ["src/rust", "lib/tracy-client-sys"]

 [patch.crates-io]
 tracy-client-sys = { path = "lib/tracy-client-sys" }
-stellar-xdr = { git = "https://github.com/stellar/rs-stellar-xdr", rev = "ae805d0f8c28ca86327a834eea9ce7d29b0a63bb" }
+stellar-xdr = { git = "https://github.com/stellar/rs-stellar-xdr", rev = "9a31b81412712ab2dfbe32cbeaba6bdb754c264b" }


We need to bump this to the most recent revision and wrap all the new XDR uses into the next version macro (or wait until the protocol release)

dmkozh · 2024-08-22T22:53:02Z

src/main/main.cpp

@@ -382,7 +382,7 @@ main(int argc, char* const* argv)
    // built with vnext, causing a curr diff against next. This works now
    // because the xdr is indentical, but the moment that changes this checkk
    // will fail and will need to be fixed.
-    checkXDRFileIdentity();
+    // checkXDRFileIdentity();


Don't forget to revert this

dmkozh · 2024-08-22T23:09:19Z

src/bucket/BucketList.cpp

+template <> BucketListDepth BucketListBase<LiveBucket>::kNumLevels = 11;
+
+// TODO: I made this number up, do some analysis and pick a better value or
+// make this a configurable network config.


Just curious, can we actually change this value at runtime in-between ledgers?

No. We will probably end up making this a config setting at which point I'll change it to a dynamic runtime value, but for now it's easier just to inherit the older BucketList's global const value for this refactor.

dmkozh · 2024-08-22T23:16:25Z

src/bucket/BucketList.cpp

+
+    for (uint32_t i = static_cast<uint32>(mLevels.size()) - 1; i != 0; --i)
+    {
+        /*


Don't forget to remove this (or uncomment if that's universally useful for debuggin)

dmkozh · 2024-08-22T23:21:16Z

src/bucket/BucketManagerImpl.cpp

+        if (protocolVersionStartsFrom(currentHeader.ledgerVersion,
+                                      ProtocolVersion::V_22))
+        {
+            // TODO: Hash Archive Bucket


What's the status of this TODO? I.e. is the PR blocked on this?

dmkozh · 2024-08-22T23:21:46Z

src/bucket/BucketManagerImpl.cpp

@@ -1148,26 +1304,30 @@ BucketManagerImpl::assumeState(HistoryArchiveState const& has,
    ZoneScoped;
    releaseAssertOrThrow(mApp.getConfig().MODE_ENABLES_BUCKETLIST);

-    for (uint32_t i = 0; i < BucketList::kNumLevels; ++i)
+    // TODO: Assume archival bucket state


What about this TODO?

The outstanding TODOs are outside the scope of this PR. I've added some additional context on their dependencies. They require changing the history archive structure, which is a pretty big endeavor.

dmkozh · 2024-08-22T23:27:34Z

src/bucket/BucketSnapshot.cpp

                    {
-                        result.push_back(entryOp->liveEntry());
+                        result.push_back(*entryOp);


I'm not sure I understand this, why do we add entry unconditionally as it seems like it must have been archived at this point? Is that for getLedgerEntry endpoint?

Archival is not enforced at the database level, but at the txn level. Core needs to query about Archival State for a variety of reasons. For example, to check if a new entry creation is valid, core needs to check that an archived version of that key does not exist in the Live bucketlist or the archival bucketlists, so we can't just return null whenever an entry is archived at the db level.

dmkozh · 2024-08-22T23:28:26Z

src/bucket/BucketSnapshot.h

+                  std::vector<BulkLoadReturnT>& result,
+                  LedgerKeyMeter* lkMeter) const;
+
+    // friend struct BucketLevelSnapshot<BucketT>;


Please remove

dmkozh · 2024-08-22T23:48:21Z

src/bucket/LedgerCmp.h

+            if (aty != HOT_ARCHIVE_DELETED && aty != HOT_ARCHIVE_LIVE)
+            {
+                throw std::runtime_error(
+                    "Malformed bucket: expected DELETED/RESTORED key.");


We don't have 'RESTORED' key category, do we?

dmkozh · 2024-08-22T23:56:00Z

src/util/ProtocolVersion.h

@@ -33,7 +33,8 @@ enum class ProtocolVersion : uint32_t
    V_18,
    V_19,
    V_20,
-    V_21
+    V_21,
+    V_22


Don't forget to update this to include v23 and use it where appropriate.
I also would suggest using a named constant instead of the direct version checks in order to make changes easier.

marta-lokhova · 2024-08-29T22:00:33Z

Note: changes in this PR impact BucketListDB quite a bit, I think we need to resolve #4433 to make sure we have good coverage. Wdyt?

SirTyson requested review from marta-lokhova and dmkozh July 30, 2024 18:38

SirTyson force-pushed the archival-bucketlist branch from 76b6594 to 05d6144 Compare August 13, 2024 04:46

SirTyson marked this pull request as ready for review August 13, 2024 04:46

SirTyson force-pushed the archival-bucketlist branch from 5cace22 to ca87d88 Compare August 13, 2024 18:35

dmkozh reviewed Aug 22, 2024

View reviewed changes

SirTyson force-pushed the archival-bucketlist branch 3 times, most recently from 6bc3175 to 2114dbf Compare September 17, 2024 17:56

SirTyson added 14 commits September 17, 2024 13:49

Refactored BucketList to LiveBucketList

c3c5990

Basic Hot Archive Structure

385a5c5

Templated Bucket helper classes

37c5f9a

Template BucketList refactor

2e833f4

Added Hot Archive merge logic

50c9836

BucketList tests for archive

7895e40

Added BucketListDB support for Hot Archive

9808fa5

Minor fixes

01902f1

Added Hot Archive BucketListDB tests and fixes

8d8ff35

Bump xdr

1bc7ff6

Fix BucketList test for versions < p22

1d277b6

Fixed BucketMeta version bug

8206260

Minor fixes

f3e56e2

Cleaned up rebase

b9cf283

SirTyson force-pushed the archival-bucketlist branch 2 times, most recently from 038ece0 to 6ca5887 Compare September 18, 2024 22:36

Add p23 test meta

344647f

SirTyson force-pushed the archival-bucketlist branch from 6ca5887 to 344647f Compare September 19, 2024 17:45

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Archival bucketlist #4403

Archival bucketlist #4403

SirTyson commented Jul 30, 2024 •

edited

Loading

dmkozh Aug 22, 2024

dmkozh Aug 22, 2024

dmkozh Aug 22, 2024

SirTyson Sep 17, 2024

dmkozh Aug 22, 2024

dmkozh Aug 22, 2024

dmkozh Aug 22, 2024

SirTyson Sep 17, 2024

dmkozh Aug 22, 2024

SirTyson Sep 17, 2024

dmkozh Aug 22, 2024

dmkozh Aug 22, 2024

dmkozh Aug 22, 2024

marta-lokhova commented Aug 29, 2024

@@ @@ -33,7 +33,8 @@ enum class ProtocolVersion : uint32_t @@
                   V_18,
                   V_19,
                   V_20,
-                  V_21
+                  V_21,
+                  V_22

Archival bucketlist #4403

Are you sure you want to change the base?

Archival bucketlist #4403

Conversation

SirTyson commented Jul 30, 2024 • edited Loading

Description

Checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

marta-lokhova commented Aug 29, 2024

SirTyson commented Jul 30, 2024 •

edited

Loading