rpc-v2/tx: Implement transaction_unstable_broadcast and transaction_unstable_stop
#3079
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Alexandru Vasile <[email protected]>
Signed-off-by: Alexandru Vasile <[email protected]>
Signed-off-by: Alexandru Vasile <[email protected]>
Signed-off-by: Alexandru Vasile <[email protected]>
Signed-off-by: Alexandru Vasile <[email protected]>
Signed-off-by: Alexandru Vasile <[email protected]>
Signed-off-by: Alexandru Vasile <[email protected]>
Signed-off-by: Alexandru Vasile <[email protected]>
Signed-off-by: Alexandru Vasile <[email protected]>
Signed-off-by: Alexandru Vasile <[email protected]>
Signed-off-by: Alexandru Vasile <[email protected]>
Signed-off-by: Alexandru Vasile <[email protected]>
Signed-off-by: Alexandru Vasile <[email protected]>
Signed-off-by: Alexandru Vasile <[email protected]>
Signed-off-by: Alexandru Vasile <[email protected]>
Signed-off-by: Alexandru Vasile <[email protected]>
Signed-off-by: Alexandru Vasile <[email protected]>
lexnv
added
I5-enhancement
Signed-off-by: Alexandru Vasile <[email protected]>
Signed-off-by: Alexandru Vasile <[email protected]>
This was referenced Jan 26, 2024
Signed-off-by: Alexandru Vasile <[email protected]>
Signed-off-by: Alexandru Vasile <[email protected]>
Signed-off-by: Alexandru Vasile <[email protected]>
Signed-off-by: Alexandru Vasile <[email protected]>
lexnv
commented
Feb 6, 2024
Comment on lines
+126
to
+136
| while !is_done { | ||
| // Wait for the last block to become available. | ||
| let Some(best_block_hash) = | ||
| last_stream_element(&mut best_block_import_stream).await | ||
| else { | ||
| return; | ||
| }; | ||
|
|
||
| let mut stream = match pool | ||
| .submit_and_watch(best_block_hash, TX_SOURCE, decoded_extrinsic.clone()) | ||
| .await |
There was a problem hiding this comment.
We are effectively trying to resubmit the transaction to the next best block here; if we cannot make progress at this time:
submit_and_watchdirectly returned a recoverable error (ImmediatelyDroped etc)submit_and_watchproduced an event that may be recoverable (Invalid / Dropped etc)
When we get the Invalid event, the transaction-pool will ban it for some time (default 30 mins). In this case, the looping seems redundant unless chain developers manually pick another timeout.
However, I believe we can simplify this and try to submit the transaction only once.
From the transaction_unstable_broadcast, the phrasing states that the server tries to broadcast, but without any guarantees:
JSON-RPC server will try to propagate this transaction
This could be interpreted as we'll submit the transaction to the transaction pool now, and if it fails or is invalid we'll stop.
@skunert would love to get your thoughts on this 🙏
There was a problem hiding this comment.
Yeah sounds like its not enforced by the spec, but IMO the current implementation makes sense. If we can recover and retry in the next block, why not.
skunert
reviewed
Feb 7, 2024
substrate/client/rpc-spec-v2/src/transaction/transaction_broadcast.rs
Outdated
Show resolved
Hide resolved
| // Keep track of this entry and the abortable handle. | ||
| { | ||
| let mut broadcast_ids = self.broadcast_ids.write(); | ||
| broadcast_ids.insert(id.clone(), BroadcastState { handle }); |
There was a problem hiding this comment.
Hmm could someone abuse broadcast_ids by calling broadcast for invalid tx over and over, but never calling stop_broadcast? The future would terminate but the broadcast ids map would grow and never get cleaned up.
There was a problem hiding this comment.
Good catch! I've removed the operationId from our internal mappings as soon as the broadcast future exits.
I believe this is in-line with the spec; since the tx_stop mentions it works for active broadcasts:
A JSON-RPC error is generated if the operationId doesn't correspond to any active transaction_unstable_broadcast operation.
As a side-effect have added a small task executor wrapper for our testing; just to make sure we don't race between tx_stop and broadcast future clearing the internal state
substrate/client/rpc-spec-v2/src/transaction/transaction_broadcast.rs
Outdated
Show resolved
Hide resolved
Comment on lines
+126
to
+136
| while !is_done { | ||
| // Wait for the last block to become available. | ||
| let Some(best_block_hash) = | ||
| last_stream_element(&mut best_block_import_stream).await | ||
| else { | ||
| return; | ||
| }; | ||
|
|
||
| let mut stream = match pool | ||
| .submit_and_watch(best_block_hash, TX_SOURCE, decoded_extrinsic.clone()) | ||
| .await |
There was a problem hiding this comment.
Yeah sounds like its not enforced by the spec, but IMO the current implementation makes sense. If we can recover and retry in the next block, why not.
Signed-off-by: Alexandru Vasile <[email protected]>
Signed-off-by: Alexandru Vasile <[email protected]>
Signed-off-by: Alexandru Vasile <[email protected]>
Signed-off-by: Alexandru Vasile <[email protected]>
…cast.rs Co-authored-by: Sebastian Kunert <[email protected]>
github-merge-queue bot
pushed a commit
that referenced
this pull request
Feb 12, 2024
…lpers (#3215) This PR improves the transaction status documentation. - Added doc references for describing the main states - Extra comment wrt pool ready / future queues - `FinalityTimeout` no longer describes a lagging finality gadget, it signals that the maximum number of finality gadgets has been reached A few helper methods are added to indicate when: - a final event is generated by the transaction pool for a given event - a final event is provided, although the transaction might become valid at a later time and could be re-submitted The helper methods are used and taken from #3079 to help us better keep it in sync. cc @paritytech/subxt-team --------- Signed-off-by: Alexandru Vasile <[email protected]>
Signed-off-by: Alexandru Vasile <[email protected]>
github-merge-queue bot
pushed a commit
that referenced
this pull request
Feb 28, 2024
… tx status (#3193) This PR adds tests for the `transaction_broadcast` method. The testing needs to coordinate the following components: - The `TestApi` marks transactions as invalid and implements `ChainApi::validate_transaction` - this is what dictates if a transaction is valid or not and is called from within the `BasicPool` - The `BasicPool` which maintains the transactions and implements `submit_and_watch` needed by the tx broadcast to submit the transaction - The status of the transaction pool is exposed by mocking the BasicPool - The `ChainHeadMockClient` which mocks the `BlockchainEvents::import_notification_stream` needed by the tx broadcast to know to which blocks the transaction is submitted The following changes have been added to the substrate testing to accommodate this: - `TestApi` gets ` remove_invalid`, counterpart to `add_invalid` to ensure an invalid transaction can become valid again; as well as a priority setter for extrinsics - `BasicPool` test constructor is extended with options for the `PoolRotator` - this mechanism is needed because transactions are banned for 30mins (default) after they are declared invalid - testing bypasses this by providing a `Duration::ZERO` ### Testing Scenarios - Capture the status of the transaction as it is normally broadcasted - `transaction_stop` is valid while the transaction is in progress - A future transaction is handled when the dependencies are completed - Try to resubmit the transaction at a later block (currently invalid) - An invalid transaction status is propagated; the transaction is marked as temporarily banned; then the ban expires and transaction is resubmitted This builds on top of: #3079 Part of: #3084 cc @paritytech/subxt-team --------- Signed-off-by: Alexandru Vasile <[email protected]> Co-authored-by: James Wilson <[email protected]>
bgallois
pushed a commit
to duniter/duniter-polkadot-sdk
that referenced
this pull request
Mar 25, 2024
…lpers (paritytech#3215) This PR improves the transaction status documentation. - Added doc references for describing the main states - Extra comment wrt pool ready / future queues - `FinalityTimeout` no longer describes a lagging finality gadget, it signals that the maximum number of finality gadgets has been reached A few helper methods are added to indicate when: - a final event is generated by the transaction pool for a given event - a final event is provided, although the transaction might become valid at a later time and could be re-submitted The helper methods are used and taken from paritytech#3079 to help us better keep it in sync. cc @paritytech/subxt-team --------- Signed-off-by: Alexandru Vasile <[email protected]>
bgallois
pushed a commit
to duniter/duniter-polkadot-sdk
that referenced
this pull request
Mar 25, 2024
…n_unstable_stop` (paritytech#3079) This PR implements the [transaction_unstable_broadcast](https://github.com/paritytech/json-rpc-interface-spec/blob/main/src/api/transaction_unstable_broadcast.md) and [transaction_unstable_stop](https://github.com/paritytech/json-rpc-interface-spec/blob/main/src/api/transaction_unstable_stop.md). The [transaction_unstable_broadcast](https://github.com/paritytech/json-rpc-interface-spec/blob/main/src/api/transaction_unstable_broadcast.md) submits the provided transaction at the best block of the chain. If the transaction is dropped or declared invalid, the API tries to resubmit the transaction at the next available best block. ### Broadcasting The broadcasting operation continues until either: - the user called `transaction_unstable_stop` with the operation ID that identifies the broadcasting operation - the transaction state is one of the following: - Finalized: the transaction is part of the chain - FinalizedTimeout: we have waited for 256 finalized blocks and timedout - Usurped the transaction has been replaced in the tx pool The broadcasting retires to submit the transaction when the transaction state is: - Invalid: the transaction might become valid at a later time - Dropped: the transaction pool's capacity is full at the moment, but might clear when other transactions are finalized/dropped ### Stopping The `transaction_unstable_broadcast` spawns an abortable future and tracks the abort handler. When the [transaction_unstable_stop](https://github.com/paritytech/json-rpc-interface-spec/blob/main/src/api/transaction_unstable_stop.md) is called with a valid operation ID; the abort handler of the corresponding `transaction_unstable_broadcast` future is called. This behavior ensures the broadcast future is finishes on the next polling. When the `transaction_unstable_stop` is called with an invalid operation ID, an invalid jsonrpc specific error object is returned. ### Testing This PR adds the testing harness of the transaction API and validates two basic scenarios: - transaction enters and exits the transaction pool - transaction stop returns appropriate values when called with valid and invalid operation IDs Closes: paritytech#3039 Note that the API should be enabled after: paritytech#3084. cc @paritytech/subxt-team --------- Signed-off-by: Alexandru Vasile <[email protected]> Co-authored-by: Sebastian Kunert <[email protected]>
bgallois
pushed a commit
to duniter/duniter-polkadot-sdk
that referenced
this pull request
Mar 25, 2024
… tx status (paritytech#3193) This PR adds tests for the `transaction_broadcast` method. The testing needs to coordinate the following components: - The `TestApi` marks transactions as invalid and implements `ChainApi::validate_transaction` - this is what dictates if a transaction is valid or not and is called from within the `BasicPool` - The `BasicPool` which maintains the transactions and implements `submit_and_watch` needed by the tx broadcast to submit the transaction - The status of the transaction pool is exposed by mocking the BasicPool - The `ChainHeadMockClient` which mocks the `BlockchainEvents::import_notification_stream` needed by the tx broadcast to know to which blocks the transaction is submitted The following changes have been added to the substrate testing to accommodate this: - `TestApi` gets ` remove_invalid`, counterpart to `add_invalid` to ensure an invalid transaction can become valid again; as well as a priority setter for extrinsics - `BasicPool` test constructor is extended with options for the `PoolRotator` - this mechanism is needed because transactions are banned for 30mins (default) after they are declared invalid - testing bypasses this by providing a `Duration::ZERO` ### Testing Scenarios - Capture the status of the transaction as it is normally broadcasted - `transaction_stop` is valid while the transaction is in progress - A future transaction is handled when the dependencies are completed - Try to resubmit the transaction at a later block (currently invalid) - An invalid transaction status is propagated; the transaction is marked as temporarily banned; then the ban expires and transaction is resubmitted This builds on top of: paritytech#3079 Part of: paritytech#3084 cc @paritytech/subxt-team --------- Signed-off-by: Alexandru Vasile <[email protected]> Co-authored-by: James Wilson <[email protected]>
This was referenced Jun 5, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR implements the transaction_unstable_broadcast and transaction_unstable_stop.
The transaction_unstable_broadcast submits the provided transaction at the best block of the chain.
If the transaction is dropped or declared invalid, the API tries to resubmit the transaction at the next available best block.
Broadcasting
The broadcasting operation continues until either:
transaction_unstable_stopwith the operation ID that identifies the broadcasting operationThe broadcasting retires to submit the transaction when the transaction state is:
Stopping
The
transaction_unstable_broadcastspawns an abortable future and tracks the abort handler.When the transaction_unstable_stop is called with a valid operation ID; the abort handler of the corresponding
transaction_unstable_broadcastfuture is called. This behavior ensures the broadcast future is finishes on the next polling.When the
transaction_unstable_stopis called with an invalid operation ID, an invalid jsonrpc specific error object is returned.Testing
This PR adds the testing harness of the transaction API and validates two basic scenarios:
Closes: #3039
Note that the API should be enabled after: #3084.
cc @paritytech/subxt-team