Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add better validation around matching response with request information #3140

Merged
merged 4 commits into from
Nov 3, 2023

Conversation

fselmo
Copy link
Collaborator

@fselmo fselmo commented Nov 2, 2023

What was wrong?

  • Add better validation that the cached request information really matches the response we are trying to process. This is done by passing the actual response_id for the matching request once we go back through the middleware onion and the response is present. This area of the code hadn't been iterated on since some of the earlier commits in the building out of the provider. This is a very important and necessary improvement on id matching.

This resolves an issue reported on Discord where await asyncio.gather(one_request, second_request) was guessing the wrong request id for each request since they are basically being triggered / yielding to each other in a shared context.

Steps to reproduce

async def reproducible_example():
    async with AsyncWeb3.persistent_websocket(WebsocketProviderV2(PROVIDER_URI)) as w3:
        (
            latest,
            chain_id,
            block_num,
            pending,
            chain_id2,
            chain_id3,
            finalized,
            balance,
        ) = await asyncio.gather(
            w3.eth.get_block("latest"),
            w3.eth.chain_id,
            w3.eth.block_number,
            w3.eth.get_block("pending"),
            w3.eth.chain_id,
            w3.eth.chain_id,
            w3.eth.get_block("finalized"),
            w3.eth.get_balance("shaq.eth")
        )
        print(f"block num from latest block: {latest['number']}\n")
        print(f"chain_id: {chain_id}\n")
        print(f"block_num: {block_num}\n")
        print(f"block num from pending block: {pending['number']}\n")
        print(f"chain_id2: {chain_id2}\n")
        print(f"chain_id3: {chain_id3}\n")
        print(f"block num from finalized block: {finalized['number']}\n")
        print(f"balance: {balance}\n")

asyncio.run(reproducible_example())

Todo:

Cute Animal Picture

20231102_160925

- Add better validation that the cached request information really matches the response we are trying to process. This is done by passing the response id to get the cached request information for the matching id.

This resolves an issue reported on Discord where ``await asyncio.gather(one_request, second_request)`` was guessing the wrong request id for each request since they are basically being triggered / yielding to each other in a shared context.
fselmo added a commit to fselmo/web3.py that referenced this pull request Nov 2, 2023
@fselmo fselmo marked this pull request as ready for review November 2, 2023 22:00
fselmo added a commit to fselmo/web3.py that referenced this pull request Nov 2, 2023
@fselmo fselmo requested review from pacrob and reedsa November 2, 2023 22:11
- asyncio.gather() will run all tasks, yielding to each other. This makes for a good test case to test that each request information that is cached is matched appropriately to the response that is received and its id.
request_id = next(copy(self._provider.request_counter)) - 1
cache_key = generate_cache_key(request_id)
current_request_cached_info: RequestInformation = (
cache_key = generate_cache_key(response_id)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the examples we were walking through earlier, I think there was a sticky point that some responses didnt appear to include an id. If that happens, generate_cache_key will raise. Should we swallow that exception and just skip the processors here instead?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's a good point. We could still return the raw response and try to not halt things here. I think passing in the response to this method would allow us to retrieve the id if it's there or debug log the response that doesn't have an id and keep moving.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah that sounds like it would work great!

fselmo added a commit to fselmo/web3.py that referenced this pull request Nov 2, 2023
@fselmo fselmo requested a review from reedsa November 3, 2023 17:48
Copy link
Contributor

@reedsa reedsa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lgtm

@fselmo fselmo merged commit 3728df8 into ethereum:main Nov 3, 2023
3 of 89 checks passed
fselmo added a commit that referenced this pull request Nov 3, 2023
@fselmo fselmo deleted the search-radius branch November 3, 2023 19:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants