services/horizon: Add cache toggle and use libary for on-disk caching #5197
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Shaptic
force-pushed
the
add-caching-flag
branch
from
7a0224f
to
669eec8
Compare
Shaptic
changed the title
sreuland
reviewed
Feb 7, 2024
sreuland
reviewed
Feb 7, 2024
sreuland
reviewed
Feb 7, 2024
| @@ -107,7 +107,7 @@ func (x *XdrStream) ExpectedHash() ([sha256.Size]byte, bool) { | |||
| func (x *XdrStream) Close() error { | |||
| if x.validateHash { | |||
| // Read all remaining data from rdr | |||
| _, err := io.Copy(ioutil.Discard, x.rdr) | |||
| _, err := io.Copy(io.Discard, x.rdr) | |||
There was a problem hiding this comment.
oh, interesting, was this and ln 144 related to root cause of corruption seen in quickstart/557?
There was a problem hiding this comment.
nah this was just a drive-by fix of deprecated usage - they're functionally equivalent
There was a problem hiding this comment.
ok, how do we know this fixed stellar/quickstart#557, build quickstart locally with this branch for horizon and run with --pubnet ?
There was a problem hiding this comment.
yep, that's how I've been doing it this whole time locally and I can confirm that it no longer encounters the read error when processing ledgers
tamirms
reviewed
Feb 7, 2024
tamirms
reviewed
Feb 7, 2024
tamirms
reviewed
Feb 7, 2024
Shaptic
force-pushed
the
add-caching-flag
branch
from
3e06e8e
to
e0c089e
Compare
tamirms
reviewed
Feb 7, 2024
tamirms
reviewed
Feb 7, 2024
tamirms
reviewed
Feb 7, 2024
tamirms
reviewed
Feb 7, 2024
tamirms
reviewed
Feb 8, 2024
tamirms
approved these changes
Feb 8, 2024
tamirms
reviewed
Feb 8, 2024
urvisavla
pushed a commit
to urvisavla/go
that referenced
this pull request
Feb 13, 2024
…stellar#5197) * Add a `--history-archive-caching` flag (default=true) to toggle behavior * Refactor to use a library: fscache * Hook new metric into prometheus * Add parallel read test to stress cache * Add tests for deadlocking and other misc. scenarios
urvisavla
pushed a commit
to urvisavla/go
that referenced
this pull request
Feb 13, 2024
…stellar#5197) * Add a `--history-archive-caching` flag (default=true) to toggle behavior * Refactor to use a library: fscache * Hook new metric into prometheus * Add parallel read test to stress cache * Add tests for deadlocking and other misc. scenarios
Shaptic
added a commit
to Shaptic/go
that referenced
this pull request
Feb 14, 2024
…stellar#5197) * Add a `--history-archive-caching` flag (default=true) to toggle behavior * Refactor to use a library: fscache * Hook new metric into prometheus * Add parallel read test to stress cache * Add tests for deadlocking and other misc. scenarios
Shaptic
added a commit
to Shaptic/go
that referenced
this pull request
Feb 14, 2024
…stellar#5197) * Add a `--history-archive-caching` flag (default=true) to toggle behavior * Refactor to use a library: fscache * Hook new metric into prometheus * Add parallel read test to stress cache * Add tests for deadlocking and other misc. scenarios
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What
fscache) to avoid the issue we were seeing with the custom implementation.--history-archive-caching=(true|false)flag (default: true) controls whether or not there will be an on-disk cache used for history archive fetches. Most people will not need to set this tofalseunless they're encountering issues. This is to prevent folks from being blocked if issues are encountered with this feature in the future.Why
Battle-tested reliability.
Closes stellar/quickstart#557.
Closes #5186