Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

blockstream store to disk #339

Open
wants to merge 180 commits into
base: main
Choose a base branch
from
Open

blockstream store to disk #339

wants to merge 180 commits into from

Conversation

grooviegermanikus
Copy link
Collaborator

@grooviegermanikus grooviegermanikus commented Feb 26, 2024 •
edited
Loading

Tool to dump some blocks on disk for later replay (e.g. for performance test)

RUST_LOG=info,grpc_dump_blocks=debug,geyser_grpc_connector=debug,solana_lite_rpc_cluster_endpoints::grpc_store_to_disk=debug cargo run --bin grpc_dump_blocks -- --grpc-addr https://api.testnet.rpcpool.com/ --grpc-x-token secrettoken --duration 3 --dump-directory /opt/solana/blocks_on_disk

Usage: grpc_dump_blocks [OPTIONS] --dump-directory <DUMP_DIRECTORY> --duration --grpc-addr <GRPC_ADDR>
Options:
--dump-directory <DUMP_DIRECTORY>
Directory to dump blocks; must exist contain the marker file '.solana-blocks-dump'
--duration
Seconds to run the dumper
--grpc-addr <GRPC_ADDR>
Address of the gRPC server
--grpc-x-token <GRPC_X_TOKEN>
Optional token for gRPC server

2024-02-26T14:42:23.665229Z INFO solana_lite_rpc_cluster_endpoints::grpc_store_to_disk: Wrote block 254984281 with 918423 bytes to disk: /opt/solana/blocks_on_disk/blocks-000254984xxx/block-000254984281-confirmed-1708958543653.dat

@grooviegermanikus
put block sequence invariants in a debugging task
3c34866
@grooviegermanikus
add to main
b919309
@grooviegermanikus
switch to blockhash instead of slot
f16adc8
@grooviegermanikus
update dependencies
3960491
@grooviegermanikus
WIP use autoconnect_task
1dd9ef3
@grooviegermanikus
merge into stream
b372c6d
@grooviegermanikus
handle shutdown
3df4972
@grooviegermanikus
cleanup
a899c0d
@grooviegermanikus
use autoconnect also for blockmeta
009d273
@grooviegermanikus
use new autoconnect
4055183
@grooviegermanikus
deploy experimental
b372bc4
@grooviegermanikus
enable experimental
c56ec56
@grooviegermanikus
reverrt fly log
c28b54c
@grooviegermanikus
wire up slot
2105ed7
@grooviegermanikus
small renamings
fbcd730
@grooviegermanikus
use autoconnect for blocks too
0ddd4db
@grooviegermanikus
remove staging region
c7e7b67
@grooviegermanikus
cleanup task
da0fdee
@grooviegermanikus
FIX panic in multiplex
4f6952c
@grooviegermanikus
improve log
fa9ca19
@grooviegermanikus
handle timeout in slot stream
0fc9b5b
@grooviegermanikus
comment on task termination
d44379b
@grooviegermanikus
cleanup
e681727
@grooviegermanikus
handle poll_vote_accounts_and_cluster_info errors
a64e699
@grooviegermanikus
handle poll_vote_accounts_and_cluster_info errors
62f89f4
@grooviegermanikus
remove debug code
15f3dfa
@grooviegermanikus
add logic to store finalized meta and merge with full processed block
ba40c70
@grooviegermanikus
add debug stream tasks
ae17f20
@grooviegermanikus
check previous_blockhash
cdeb2e2
@grooviegermanikus
improve shutdown
0d1e81c
@grooviegermanikus
set lock_timeout
e2a945d
@grooviegermanikus
use mapping table
69f2947
@grooviegermanikus
add tracing
60d0bfb
@grooviegermanikus
fmt
37fbc34
@grooviegermanikus
increase work_mem:
ece0ad4 
>Sort Method: external merge  Disk: 4672kB
became
> Sort Method: quicksort  Memory: 5228kB
@grooviegermanikus
ONLY 1 write_session
e30c57a
@grooviegermanikus
format elapsed times as 0.00ms
e112402
@grooviegermanikus
refactor chunk logic
2268cd0
@grooviegermanikus
print explain plan
5018824
@grooviegermanikus
add regex
dac6d7b
@grooviegermanikus
MODEL CHANGE: change pkey to (slot,transaction_id)
c43d0f0
@grooviegermanikus
improve write_speed test
e8c2f04
@grooviegermanikus
fix test
49bd29f
@grooviegermanikus
fix logging of timings
2db819a
@grooviegermanikus
comment out execute_explain (citus does not like it)
43ca986
@grooviegermanikus
add metric literpc_blockstore_save_block
6666dda
@grooviegermanikus
remove confusing 'remaining 4 queue elements' log which happend due t…
d137e12 
…o non-confirmed blocks being queued
@grooviegermanikus
set PARALLEL_WRITE_SESSIONS to 4
647734e
@grooviegermanikus
detect CITUS extension
dfed888
@grooviegermanikus
PARALLEL_WRITE_SESSIONS from 4 to 1
2932997
@grooviegermanikus
distribute by transaction_id instead of slot
48ab0d6
@grooviegermanikus
add semikolon to pl sql
3f67375
@grooviegermanikus
distribute by blocks.slot + transaction_ids.transaction_id
10bf2e1
@grooviegermanikus
fix sql
979e3d4
@grooviegermanikus
cannot distribute transaction_ids
3f02f6c
@grooviegermanikus
remove RETURNING from insert (citus does not like it)
a594a40
@grooviegermanikus
comment on work_mem
05ab3eb
@grooviegermanikus
fix insert block check
590b61d
@grooviegermanikus
add sigature as join to transaction_id JOIN not supported by citus
fbbb2d5
@grooviegermanikus
cache epoch schema lookups
2cfd044
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@godmodegalactus godmodegalactus godmodegalactus requested changes

Requested changes must be addressed to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@grooviegermanikus @godmodegalactus