Proposal: return eth_getTransactionCount from the private mempool when RPC request is signed

[ryanschneider]

Proposal: return eth_getTransactionCount from the private mempool when RPC request is signed

This is a proposal to augment the RPC endpoint behavior as follows:

• When a request is made for eth_getTransactionCount targeting the "pending" block specifier.
• And the request is signed with the X-Flashbots-Signature header.
• And the signature address matches the account being queried
• Then the RPC endpoint will return the transaction count from it's internal "private mempool".

Current Behavior

Recall that eth_getTransactionCount takes two arguments:

• The account to query
• The EIP-1898 Block Specifier (e.g. "latest", "pending" or a block number or hash).

And returns the total number of transactions that account has executed at that block specifier. Furthermore, the "pending" specifier is somewhat special-cased, in that in addition to considering all transactions already executed on-chain, it also includes any pending transactions in the node's mempool that are considered executable by the node, where executable refers means that all previous nonces for the account in question are either already on-chain or themselves executable.

Currently, eth_getTransactionCount is treated like any other RPC not directly handled by the endpoint and the request is simply proxied to the PROXY_URL endpoint. However, this endpoint is typically not aware of any "private mempool" transactions sent by the user (e.g. eth_sendRawTransaction requests) so will give an inaccurate response when the "pending" block is queried and the user has any pending private transactions.

Desired Behavior

When a user sends the eth_getTransactionCount request to the RPC endpoint targeting the "pending" block the response should take into consideration any outstanding private transactions.

Security Concerns

However, there's a security concern, in that making this data openly available potentially leaks information about usage of the private mempool.

• For example, a third party could continuously call eth_getTransactionCount targeting the account in question, and see when the user has submitted new transaction(s) to the private mempool.
• If the account in question has known usage patterns this information could be used in an attempt to execute a blind back run or front run on the account.

As such, we propose that information about private mempool transaction counts is only included when the request is signed with the same X-Flashbots-Signature method we use for signing relay requests, and that the users wallet key is used to sign the request.

Implementation Logic

We propose the following implementation logic:

graph TD
    START[Start] --> METHOD{"RPC method is\neth_getTransactionCount?"}
    METHOD -->|Yes| PENDING{"params[1] is pending?"}
    METHOD -->|No| PROXY["Proxy to PROXY_URL"]
    PENDING -->|Yes| HASSIGNATURE{"X-Flashbots-Signature\nheader present?"}
    PENDING -->|No| PROXY
    HASSIGNATURE -->|Yes| VALIDSIGNATURE{"Signature is\nvalid?"}
    HASSIGNATURE -->|No| PROXY
    VALIDSIGNATURE -->|Yes| MATCHES{"params[0] matches\npublic key portion\nof X-Flashbots-Signature?"}
    VALIDSIGNATURE -->|No| ERROR["Return RPC Error"]
    MATCHES -->|Yes| G["Inspect private mempool and return highest executable nonce"]
    MATCHES -->|No| PROXY

Loading

Implementation Details

One potential complication with this approach is that we already have some custom handling of eth_getTransactionCount in place to "trick" MetaMask into correctly reporting dropped transactions, see #31 for details. We should confirm that this trick is still necessary, and either remove it or make sure it's compatible with the above logic when implementing these changes.

Redis Sorted Set Implementation

NOTE: This section will be further fleshed out once we approve the overall approach, for now consider this rough notes on implementation details.

Assuming these complications can be sorted out, we are already tracking the "max nonce" per sender (see RedisState.SetSenderMaxNonce) which gets us mostly there, the only question is how we want to handle "nonce gaps" since a correct implementation of eth_getTransactionCount should only consider executable transactions.

To handle nonce gaps, we can store a per-sender ZSet rather than a single "max nonce", where we use the nonce value as the score, and either the nonce or tx hash as the value. We will enter both the senders "on-chain" nonce (found via eth_getTransaction(sender, "latest") and any new private tx nonces into the ZSet. Then, to compute the "executable" count, we can retrieve the sorted set using ZRangeWithScores and return the highest nonce seen, stopping at either the last item in the set or any gaps. We can continue to use a 24h TTL for this sorted set, and can remove on-chain nonces from the set if it continue to grow beyond a size threshold of say 10 elements.

For example, consider the case where a user has never sent a transaction (next nonce is 0) and the account submits a private tx with nonce 1. Since eth_getTransactionCount(sender, "latest") returns 0, we would insert -1 into the set, alongside the submitted tx nonce of 1. Then their sorted set would contain [-1 1] which we would scan and stop at -1, and thus return 0.

If the user then came back and "filled the gap" a submitted a tx with nonce 0, the set would now contain [-1 0 1] and we would return 2 as the transaction count.

[metachris]

Thanks, great writeup. I believe we need to keep the MetaMask workaround in place.

[ryanschneider]

MVP is now live w/ #151, I'll leave this issue open for tracking the "nonce gap" follow-on work.