Add filtering capability to `parity_pendingTransactions` (issue 8269) #10506

lamafab · 2019-03-21T23:05:12Z

Hi

This pull request adds filtering capabilities for parity_pendingTransactions, as requested in issue #8269.

Changes

MinerService now has a new ready_transactions_filtered method, which can filter transactions by hash, sender or receiver.
The regular ready_transactions method wraps over the filter, therefore ready_transactions_filtered can be called directly from the RPC interface without having to adjust code through the whole codebase.

Considerations
I'm having slight trouble testing the code; ready_transactions (the original implementation) always returns two pending transactions on me, never more. This makes testing inconvenient. It might be related to a detail in the test miner that I'm missing, basing my assumption on this method description:

/// Get a list of all ready transactions either ordered by priority or unordered (cheaper).
///
/// Depending on the settings may look in transaction pool or only in pending block.
/// If you don't need a full set of transactions, you can add max_len and create only a limited set of
/// transactions.

If someone can point me in the right direction I can write some complete test cases (so far, NO testing is being done).

See fn filter_pending_transactions_by_tx_hash() in ethcore/src/miner/miner.rs for more.

Any feedback is welcome!

Upstream merges

parity-cla-bot · 2019-03-21T23:07:17Z

It looks like @lamafab signed our Contributor License Agreement. 👍

Many thanks,

Parity Technologies CLA Bot

Upstream merges

…issue_8269

tomusdrw

Looks good as a starting point, I think I'd go one step further and make the implementation more complete in terms of possible options to filtering but also to avoid breaking backward compatibility (the preoposed filtering object could be made optional with a some default value).

I'm having slight trouble testing the code; ready_transactions (the original implementation) always returns two pending transactions on me, never more.

That's because RPC tests are not using the real MiningService, but rather one for tests.
It would be best to test the serialisation/deserialisation in RPC tests, but the filtering logic deserves a separate test in miner module.

ethcore/src/miner/miner.rs

ethcore/src/miner/mod.rs

ascjones · 2019-06-04T08:44:21Z

@lamafab needs merging with master

lamafab · 2019-06-04T09:38:51Z

@tomusdrw @ascjones How quickly do those changes need to be implemented? I'm currently going through exam week, but I can complete this PR, soon. However, I can not guarantee on exactly when...

Regarding testing: I will skip the direct MiningService tests and use the RPC API, as I did in an other PR.

lamafab · 2019-06-24T19:50:15Z

@tomusdrw @ascjones Alright, I did it! The unified FilterOptions are now implemented in this PR.

pub struct FilterOptions {
    pub sender: FilterOperator<Address>,
    pub receiver: FilterOperator<Address>,
    pub gas: FilterOperator<U256>,
    pub gas_price: FilterOperator<U256>,
    pub value: FilterOperator<U256>,
    pub nonce: FilterOperator<U256>,
}

pub enum FilterOperator<T> {
    Any,
    Eq(T),
    GreaterThan(T),
    LessThan(T),
    ContractCreation, // only used for `receiver`
}

What this PR contains:

The parity_pendingTransactions RPC API can take optional filter operators like the following:

    {
        "sender": {
            "eq": "0x5f3dffcf347944d3739b0805c934d86c8621997f"
        },
        "gas": {
            "gt": "0x493e0"
        }
    }

Since not all operators can be used equally on each filter, the allowed/disallowed operators are caught during the deserialization process with a custom implementation of Deserialize (those tests are included in the same location where FilterOptions is defined).

What's missing

One could argue that there should be GreaterEqualThan and LessEqualThan operators.
While there are deserialization tests, there are no logic tests for the filtering. @tomusdrw I did a code base search and could not find a lot of useful information regarding testing that API (there are some very simple/oversimplified tests). Can you tell me where those logic tests should be placed and what testing data I should use (or to be more specific: how should the RPC API test server be created?). I can hardly use Gitter during the workday, so I'd appreciate it if you could give me some advice here.

Thank you very much (and for the patience!)

tomusdrw

Awesome work! Only few small nits.

ethcore/Cargo.toml

ethcore/src/miner/filter_options.rs

tomusdrw · 2019-06-25T09:11:56Z

ethcore/src/miner/filter_options.rs

+}
+
+#[cfg(test)]
+mod tests {


ethcore/types/src/transaction/transaction.rs

rpc/src/v1/impls/light/parity.rs

seunlanlege · 2019-06-27T07:46:00Z

Amazing work 🤝

ethcore/src/miner/filter_options.rs

seunlanlege

Beautiful work here.

tomusdrw

lgtm

ascjones

👀

lamafab · 2019-06-27T20:42:24Z

Thanks for the nice feedback, everyone. I've added some very small changes (like adding a newline at the end of the file as @seunlanlege suggested).

EDIT: Looks like the failed CI tests are not related to my changes... (?) Last time this happened too, I remember restarting the CI tests solved the issues.

…ckChain * master: revert changes to .gitlab-ci.yml (#10807) Add filtering capability to `parity_pendingTransactions` (issue 8269) (#10506) removed EthEngine alias (#10805) wait a bit longer in should_check_status_of_request_when_its_resolved (#10808) Do not drop the peer with None difficulty (#10772) ethcore-bloom-journal updated to 2018 (#10804) ethcore-light uses bincode 1.1 (#10798) Fix a few typos and unused warnings. (#10803) updated project to ansi_term 0.11 (#10799) added new ropsten-bootnode and removed old one (#10794) updated price-info to edition 2018 (#10801) ethcore-network-devp2p uses igd 0.9 (#10797) updated parity-local-store to edition 2018 and removed redundant Error type (#10800)

…me-parent * master: (24 commits) cargo update -p smallvec (#10822) replace memzero with zeroize crate (#10816) Don't repeat the logic from Default impl (#10813) removed additional_params method (#10818) Add Constantinople eips to the dev (instant_seal) config (#10809) removed redundant fmt::Display implementations (#10806) revert changes to .gitlab-ci.yml (#10807) Add filtering capability to `parity_pendingTransactions` (issue 8269) (#10506) removed EthEngine alias (#10805) wait a bit longer in should_check_status_of_request_when_its_resolved (#10808) Do not drop the peer with None difficulty (#10772) ethcore-bloom-journal updated to 2018 (#10804) ethcore-light uses bincode 1.1 (#10798) Fix a few typos and unused warnings. (#10803) updated project to ansi_term 0.11 (#10799) added new ropsten-bootnode and removed old one (#10794) updated price-info to edition 2018 (#10801) ethcore-network-devp2p uses igd 0.9 (#10797) updated parity-local-store to edition 2018 and removed redundant Error type (#10800) Cleanup unused vm dependencies (#10787) ...

* master: (22 commits) ethcore does not use byteorder (#10829) Better logging when backfilling ancient blocks fail (#10796) depends: Update wordlist to v1.3 (#10823) cargo update -p smallvec (#10822) replace memzero with zeroize crate (#10816) Don't repeat the logic from Default impl (#10813) removed additional_params method (#10818) Add Constantinople eips to the dev (instant_seal) config (#10809) removed redundant fmt::Display implementations (#10806) revert changes to .gitlab-ci.yml (#10807) Add filtering capability to `parity_pendingTransactions` (issue 8269) (#10506) removed EthEngine alias (#10805) wait a bit longer in should_check_status_of_request_when_its_resolved (#10808) Do not drop the peer with None difficulty (#10772) ethcore-bloom-journal updated to 2018 (#10804) ethcore-light uses bincode 1.1 (#10798) Fix a few typos and unused warnings. (#10803) updated project to ansi_term 0.11 (#10799) added new ropsten-bootnode and removed old one (#10794) updated price-info to edition 2018 (#10801) ...

* master: (21 commits) refactor: whisper: Add type aliases and update rustdocs in message.rs (#10812) Break circular dependency between Client and Engine (part 1) (#10833) tests: Relates to #10655: Test instructions for Readme (#10835) refactor: Related #9459 - evmbin: replace untyped json! macro with fully typed serde serialization using Rust structs (#10657) idiomatic changes to PodState (#10834) Allow --nat extip:your.host.here.org (#10830) When updating the client or when called from RPC, sleep should mean sleep (#10814) Remove excessive warning (#10831) Fix typo in README.md (#10828) ethcore does not use byteorder (#10829) Better logging when backfilling ancient blocks fail (#10796) depends: Update wordlist to v1.3 (#10823) cargo update -p smallvec (#10822) replace memzero with zeroize crate (#10816) Don't repeat the logic from Default impl (#10813) removed additional_params method (#10818) Add Constantinople eips to the dev (instant_seal) config (#10809) removed redundant fmt::Display implementations (#10806) revert changes to .gitlab-ci.yml (#10807) Add filtering capability to `parity_pendingTransactions` (issue 8269) (#10506) ...

lamafab added 8 commits March 4, 2019 18:24

Merge pull request #1 from paritytech/master

9169a59

Upstream merges

expand parameters for pending_transactions()

441b71c

move ready_transactions content into filtered method

a2fcaf2

apply filter based on tx_hash, sender or receiver

636e466

call filtered transactions from RPC interface

325eb6f

attempt at testing...

2dc0f32

replace parameters with _ on light client

8d840fb

addes some comments

2862dc8

This comment has been minimized.

Sign in to view

soc1c added this to the 2.5 milestone Mar 22, 2019

soc1c added A0-pleasereview 🤓 Pull request needs code review. M6-rpcapi 📣 RPC API. labels Mar 22, 2019

Merge pull request #2 from paritytech/master

e30326f

Upstream merges

soc1c modified the milestones: 2.5, 2.6 Apr 2, 2019

lamafab added 4 commits April 2, 2019 18:14

removed uncompleted tests in miner.rs

9578d49

attempt at testing, needs more work...

6f1e437

Merge pull request #3 from paritytech/master

c4dbf53

Upstream merges

Merge branch 'issue_8269' of github.com:lamafab/parity-ethereum into …

e7d3428

…issue_8269

folsen requested review from tomusdrw and ascjones May 31, 2019 14:26

tomusdrw reviewed Jun 3, 2019

View reviewed changes

tomusdrw added A5-grumble 🔥 Pull request has minor issues that must be addressed before merging. and removed A0-pleasereview 🤓 Pull request needs code review. labels Jun 3, 2019

lamafab added 3 commits June 24, 2019 18:56

small documentation adjustments

c977666

remove warnings

02a219f

Merge unified FilterOptions implementation

7255a8e

tomusdrw reviewed Jun 25, 2019

View reviewed changes

lamafab added 5 commits June 26, 2019 20:37

replace FilterOperator::ContractCreation with FilterOperator::Eq(None)

9754f1a

implement validate_receiver

bd249e5

make small changes like renames, preamble

7a2e942

cleanup code according to suggestions, add docs

c85699e

Implement suggested changes

5951f59

seunlanlege reviewed Jun 27, 2019

View reviewed changes

ethcore/src/miner/filter_options.rs Outdated Show resolved Hide resolved

seunlanlege approved these changes Jun 27, 2019

View reviewed changes

tomusdrw approved these changes Jun 27, 2019

View reviewed changes

ascjones approved these changes Jun 27, 2019

View reviewed changes

dvdplm added A8-looksgood 🦄 Pull request is reviewed well. and removed A5-grumble 🔥 Pull request has minor issues that must be addressed before merging. labels Jun 27, 2019

ordian added the B7-releasenotes 📜 Changes should be mentioned in the release notes of the next minor version release. label Jun 27, 2019

lamafab added 2 commits June 27, 2019 18:51

small improvements like formatting and newline

5cc8cc0

Merge small changes like error msg formatting and add newline

3be3bde

seunlanlege merged commit 3f61f2d into openethereum:master Jun 28, 2019

lamafab mentioned this pull request Oct 22, 2019

Can no longer filter transactions with RPC API #11189

Closed

ordian mentioned this pull request Oct 22, 2019

Add filtering capability to parity_pendingTransactions #8269

Closed

niklasad1 mentioned this pull request Dec 16, 2019

An error about parity_pendingTransactions. #11340

Closed

ytrezq mentioned this pull request Dec 9, 2020

Can no longer apply filters to parity_pendingtransactions openethereum/openethereum#159

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add filtering capability to `parity_pendingTransactions` (issue 8269) #10506

Add filtering capability to `parity_pendingTransactions` (issue 8269) #10506

lamafab commented Mar 21, 2019

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

parity-cla-bot commented Mar 21, 2019

tomusdrw left a comment

ascjones commented Jun 4, 2019

lamafab commented Jun 4, 2019

lamafab commented Jun 24, 2019

tomusdrw left a comment

tomusdrw Jun 25, 2019

seunlanlege commented Jun 27, 2019

seunlanlege left a comment

tomusdrw left a comment

ascjones left a comment

lamafab commented Jun 27, 2019 •

edited

Loading

Add filtering capability to parity_pendingTransactions (issue 8269) #10506

Add filtering capability to parity_pendingTransactions (issue 8269) #10506

Conversation

lamafab commented Mar 21, 2019

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

parity-cla-bot commented Mar 21, 2019

tomusdrw left a comment

Choose a reason for hiding this comment

ascjones commented Jun 4, 2019

lamafab commented Jun 4, 2019

lamafab commented Jun 24, 2019

tomusdrw left a comment

Choose a reason for hiding this comment

tomusdrw Jun 25, 2019

Choose a reason for hiding this comment

seunlanlege commented Jun 27, 2019

seunlanlege left a comment

Choose a reason for hiding this comment

tomusdrw left a comment

Choose a reason for hiding this comment

ascjones left a comment

Choose a reason for hiding this comment

lamafab commented Jun 27, 2019 • edited Loading

Add filtering capability to `parity_pendingTransactions` (issue 8269) #10506

Add filtering capability to `parity_pendingTransactions` (issue 8269) #10506

lamafab commented Jun 27, 2019 •

edited

Loading