Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor!: ADR-038 go-plugin system #14207

Merged
merged 116 commits into from
Mar 14, 2023
Merged
Show file tree
Hide file tree
Changes from 99 commits
Commits
Show all changes
116 commits
Select commit Hold shift + click to select a range
3f57c9d
use hashicorp go-plugin
Sep 23, 2022
1a6db28
python example
Sep 23, 2022
3db64a0
implement streaming to baseapp
Sep 27, 2022
041d83a
abci listener wiring
Sep 27, 2022
0bf1b54
update test files
Sep 28, 2022
1e4914e
update test files
Sep 28, 2022
02fa1db
kafka plugin python example
Sep 28, 2022
1c5ce62
update package name
Sep 29, 2022
13f750c
update proto and services
Sep 30, 2022
0c4c1bb
tx and kvpair indexing
Oct 1, 2022
040bfa0
tidy up
Oct 1, 2022
7691fc2
add readme
Oct 6, 2022
9ccf4fe
update readme
Oct 6, 2022
db7840f
add testing section
Oct 7, 2022
78e697b
listeners need to be async
Oct 11, 2022
c0871cd
remove goroutine on blocking call
Oct 13, 2022
56c643d
terminate with os.Exit on stopNodeOnErr
Oct 13, 2022
fcd438b
dumb pipe bridge
Oct 23, 2022
460638a
fix: listen state writes on rootmulti store only
yihuang Oct 12, 2022
a022370
introduce memory listener and OnCommit listener
Oct 26, 2022
dcc7915
remove WriteListener and use MemoryListener
Oct 28, 2022
ef0a861
internalize listener init to cache layer
Oct 28, 2022
6a72ea4
determenistic PopStateCache, grpc service
Nov 16, 2022
a67756e
add Go file plugin
Nov 16, 2022
ac5fa45
multiple service support
Nov 18, 2022
c17b5ad
add \n
Nov 18, 2022
3edd320
add proto dir
Nov 18, 2022
55196f4
append to file
Nov 21, 2022
e6c29ce
support in-process listeners
Nov 21, 2022
d1e2ffa
use setStreamingService
Dec 2, 2022
f6d387f
Merge branch 'main' into egaxhaj/adr-038-grpc2
Dec 2, 2022
16ae0c4
Merge branch 'main' into egaxhaj/adr-038-grpc2
Dec 5, 2022
b227ada
enable plugins as managed clients
Dec 5, 2022
f5d9991
add default to toml config
Dec 5, 2022
a8c703d
format toml config
Dec 5, 2022
cbf8c6b
update template
Dec 5, 2022
e38fbd0
update template
Dec 5, 2022
798dde6
update documentation
Dec 6, 2022
fd5739c
fix test
Dec 6, 2022
eebd1a5
write request to single topics
Dec 6, 2022
d296eae
fix test
Dec 7, 2022
736aed6
Merge branch 'main' into egaxhaj/adr-038-grpc2
Dec 7, 2022
1fb020e
fix lint errors
Dec 7, 2022
f813997
remove python examples, CI cannot verify
Dec 7, 2022
bd8993b
fix gosec findings
Dec 8, 2022
5806bc9
Merge branch 'main' into egaxhaj/adr-038-grpc2
egaxhaj Dec 8, 2022
e49fd57
Merge branch 'main' into egaxhaj/adr-038-grpc2
egaxhaj Dec 8, 2022
75aca30
Merge branch 'main' into egaxhaj/adr-038-grpc2
Dec 8, 2022
f6dfe61
move plugin proto with sdk proto files, regen plugin code
Dec 9, 2022
f66dfa5
update readme
Dec 12, 2022
f188f71
Merge branch 'main' into egaxhaj/adr-038-grpc2
Dec 12, 2022
e52f319
use app.kvStoreKeys()
Dec 12, 2022
fea5c87
Merge branch 'main' into egaxhaj/adr-038-grpc2
Dec 22, 2022
e9aa841
Merge branch 'main' into egaxhaj/adr-038-grpc2
Jan 3, 2023
deebf49
fix lint findings
Jan 3, 2023
ef3af41
Merge branch 'main' into egaxhaj/adr-038-grpc2
egaxhaj Jan 4, 2023
3ed0fed
Merge branch 'main' into egaxhaj/adr-038-grpc2
Jan 9, 2023
643a5b3
Merge branch 'main' into egaxhaj/adr-038-grpc2
Jan 11, 2023
1965ed6
group listener props into a StreamingManager struct
Jan 13, 2023
a36f165
Merge branch 'main' into egaxhaj/adr-038-grpc2
egaxhaj Jan 13, 2023
b05e3fa
Merge branch 'main' into egaxhaj/adr-038-grpc2
Jan 13, 2023
5625397
Merge branch 'main' into egaxhaj/adr-038-grpc2
Jan 17, 2023
1e75d5e
Merge branch 'main' into egaxhaj/adr-038-grpc2
egaxhaj Jan 17, 2023
12e1dfb
properly set delivertx res for listeners
Jan 17, 2023
528c082
fix import and proto lint findings
Jan 17, 2023
8fc97eb
Merge branch 'main' into egaxhaj/adr-038-grpc2
Jan 21, 2023
c2165e1
leave async functionality to plugin impl
Jan 21, 2023
961d4ef
fixes based on feedback
Jan 24, 2023
1e9a8dd
Merge branch 'main' into egaxhaj/adr-038-grpc2
Jan 24, 2023
99ff635
move to store/streaming/abci
Jan 25, 2023
2fc36ba
redifine methods to take a pointer
Jan 25, 2023
a211f3f
improve call to popstatecache
Jan 26, 2023
a274311
add locks when accessing to rs.listeners
Jan 26, 2023
e25a74c
update comment on PopStateCache
Jan 26, 2023
028c706
remove sdk dependency from store
Jan 28, 2023
4f7d9be
Merge branch 'main' into egaxhaj/adr-038-grpc2
Feb 2, 2023
b3345ee
use cosmosdk.io mod
Feb 3, 2023
934d52a
Merge branch 'main' into egaxhaj/adr-038-grpc2
Feb 3, 2023
31a9e9c
tidy up and pull streaming dependencies
Feb 3, 2023
e341dcf
use stdout plugin for unit test
Feb 3, 2023
04e409c
fix unexported-return by exported func
Feb 3, 2023
08dfb5d
fix lint finding
Feb 3, 2023
748fd5c
fix examples
Feb 4, 2023
be35b4f
Merge branch 'main' into egaxhaj/adr-038-grpc2
Feb 4, 2023
07326d5
fix comment typos
Feb 6, 2023
9362825
move boilerplate code to baseapp streaming
Feb 7, 2023
fba3119
Merge branch 'main' into egaxhaj/adr-038-grpc2
Feb 8, 2023
5ea9df5
fix lint findings
Feb 8, 2023
d694094
Merge branch 'main' into egaxhaj/adr-038-grpc2
Feb 8, 2023
2e54781
Merge branch 'main' into egaxhaj/adr-038-grpc2
egaxhaj Feb 8, 2023
3f62ee6
correctly log error message
Feb 8, 2023
aa9726b
feedback fixes
Feb 9, 2023
92c5344
Merge branch 'main' into egaxhaj/adr-038-grpc2
Feb 9, 2023
7786fc4
Update store/streaming/README.md
egaxhaj Feb 10, 2023
56bd1e4
Update store/streaming/README.md
egaxhaj Feb 10, 2023
8ad6317
Merge branch 'main' into egaxhaj/adr-038-grpc2
egaxhaj Feb 10, 2023
6ce00ab
remove unused un-exported listeningEnabled method
Feb 10, 2023
10fc6e6
benchmark test cases for state streaming
Feb 10, 2023
e89f01c
Merge branch 'main' into egaxhaj/adr-038-grpc2
Feb 13, 2023
08b6b5e
Merge branch 'main' into egaxhaj/adr-038-grpc2
Feb 14, 2023
9ccaa12
Merge branch 'main' into egaxhaj/adr-038-grpc2
Feb 16, 2023
2c07268
Merge branch 'main' into egaxhaj/adr-038-grpc2
Feb 21, 2023
247179c
fixes after review
Feb 22, 2023
44f30ff
Merge branch 'main' into egaxhaj/adr-038-grpc2
Feb 22, 2023
be7de2a
remove mutex on state listening methods
Feb 23, 2023
a1b04ad
Merge branch 'main' into egaxhaj/adr-038-grpc2
Feb 23, 2023
5195f71
Merge branch 'main' into egaxhaj/adr-038-grpc2
Feb 23, 2023
447912e
Merge branch 'main' into egaxhaj/adr-038-grpc2
Feb 28, 2023
252c350
unit test streaming manager in ctx
Feb 28, 2023
ed1a839
Merge branch 'main' into egaxhaj/adr-038-grpc2
Mar 3, 2023
76b5ee3
Merge branch 'main' into egaxhaj/adr-038-grpc2
Mar 9, 2023
27f68c0
do not commit file plugin binary
Mar 14, 2023
a6d5f88
Merge branch 'main' into egaxhaj/adr-038-grpc2
Mar 14, 2023
45bd990
Merge branch 'main' into egaxhaj/adr-038-grpc2
Mar 14, 2023
4f94ae0
Merge branch 'main' into egaxhaj/adr-038-grpc2
Mar 14, 2023
96e0854
Merge branch 'main' into egaxhaj/adr-038-grpc2
egaxhaj Mar 14, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,8 @@ vagrant
# IDE
.idea
*.iml
*.ipr
*.iws
.dir-locals.el
.vscode

Expand Down
47 changes: 45 additions & 2 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -276,6 +276,20 @@ test-sim-nondeterminism:
@cd ${CURRENT_DIR}/simapp && go test -mod=readonly -run TestAppStateDeterminism -Enabled=true \
-NumBlocks=100 -BlockSize=200 -Commit=true -Period=0 -v -timeout 24h

# Requires an exported plugin. See store/streaming/README.md for documentation.
#
# example:
# export COSMOS_SDK_ABCI_V1=<path-to-plugin-binary>
# make test-sim-nondeterminism-streaming
#
# Using the built-in examples:
# export COSMOS_SDK_ABCI_V1=<path-to-sdk>/store/streaming/abci/examples/file/file
# make test-sim-nondeterminism-streaming
test-sim-nondeterminism-streaming:
@echo "Running non-determinism-streaming test..."
@cd ${CURRENT_DIR}/simapp && go test -mod=readonly -run TestAppStateDeterminism -Enabled=true \
-NumBlocks=100 -BlockSize=200 -Commit=true -Period=0 -v -timeout 24h -EnableStreaming=true

test-sim-custom-genesis-fast:
@echo "Running custom genesis simulation..."
@echo "By default, ${HOME}/.gaiad/config/genesis.json will be used."
Expand Down Expand Up @@ -311,6 +325,7 @@ test-sim-benchmark-invariants:

.PHONY: \
test-sim-nondeterminism \
test-sim-nondeterminism-streaming \
test-sim-custom-genesis-fast \
test-sim-import-export \
test-sim-after-import \
Expand All @@ -325,14 +340,42 @@ SIM_COMMIT ?= true

test-sim-benchmark:
@echo "Running application benchmark for numBlocks=$(SIM_NUM_BLOCKS), blockSize=$(SIM_BLOCK_SIZE). This may take awhile!"
@go test -mod=readonly -benchmem -run=^$$ $(SIMAPP) -bench ^BenchmarkFullAppSimulation$$ \
@cd ${CURRENT_DIR}/simapp && go test -mod=readonly -run=^$$ $(.) -bench ^BenchmarkFullAppSimulation$$ \
-Enabled=true -NumBlocks=$(SIM_NUM_BLOCKS) -BlockSize=$(SIM_BLOCK_SIZE) -Commit=$(SIM_COMMIT) -timeout 24h

# Requires an exported plugin. See store/streaming/README.md for documentation.
#
# example:
# export COSMOS_SDK_ABCI_V1=<path-to-plugin-binary>
# make test-sim-benchmark-streaming
#
# Using the built-in examples:
# export COSMOS_SDK_ABCI_V1=<path-to-sdk>/store/streaming/abci/examples/file/file
# make test-sim-benchmark-streaming
test-sim-benchmark-streaming:
@echo "Running application benchmark for numBlocks=$(SIM_NUM_BLOCKS), blockSize=$(SIM_BLOCK_SIZE). This may take awhile!"
@cd ${CURRENT_DIR}/simapp && go test -mod=readonly -run=^$$ $(.) -bench ^BenchmarkFullAppSimulation$$ \
-Enabled=true -NumBlocks=$(SIM_NUM_BLOCKS) -BlockSize=$(SIM_BLOCK_SIZE) -Commit=$(SIM_COMMIT) -timeout 24h -EnableStreaming=true

test-sim-profile:
@echo "Running application benchmark for numBlocks=$(SIM_NUM_BLOCKS), blockSize=$(SIM_BLOCK_SIZE). This may take awhile!"
@go test -mod=readonly -benchmem -run=^$$ $(SIMAPP) -bench ^BenchmarkFullAppSimulation$$ \
@cd ${CURRENT_DIR}/simapp && go test -mod=readonly -benchmem -run=^$$ $(.) -bench ^BenchmarkFullAppSimulation$$ \
-Enabled=true -NumBlocks=$(SIM_NUM_BLOCKS) -BlockSize=$(SIM_BLOCK_SIZE) -Commit=$(SIM_COMMIT) -timeout 24h -cpuprofile cpu.out -memprofile mem.out

# Requires an exported plugin. See store/streaming/README.md for documentation.
#
# example:
# export COSMOS_SDK_ABCI_V1=<path-to-plugin-binary>
# make test-sim-profile-streaming
#
# Using the built-in examples:
# export COSMOS_SDK_ABCI_V1=<path-to-sdk>/store/streaming/abci/examples/file/file
# make test-sim-profile-streaming
test-sim-profile-streaming:
@echo "Running application benchmark for numBlocks=$(SIM_NUM_BLOCKS), blockSize=$(SIM_BLOCK_SIZE). This may take awhile!"
@cd ${CURRENT_DIR}/simapp && go test -mod=readonly -benchmem -run=^$$ $(.) -bench ^BenchmarkFullAppSimulation$$ \
-Enabled=true -NumBlocks=$(SIM_NUM_BLOCKS) -BlockSize=$(SIM_BLOCK_SIZE) -Commit=$(SIM_COMMIT) -timeout 24h -cpuprofile cpu.out -memprofile mem.out -EnableStreaming=true

.PHONY: test-sim-profile test-sim-benchmark

test-rosetta:
Expand Down
Loading