Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

local_filter_middleware example doesn't work #1244

Closed
jgeary opened this issue Feb 15, 2019 · 7 comments
Closed

local_filter_middleware example doesn't work #1244

jgeary opened this issue Feb 15, 2019 · 7 comments

Comments

@jgeary
Copy link

jgeary commented Feb 15, 2019

  • Version: 5.0.0a5

What was wrong?

I upgraded from 4.8.2 to 5.0.0a5 to use the local filter middleware so I don't have to deal with the node deleting my filters. Unfortunately, I've been unable to use local_filter_middleware, which I believe is due to incorrect / lacking documentation.

According to the docs, here is the proper usage of local_filter_middleware:

from web3.middleware import local_filter_middleware
w3.middleware_onion.add(local_filter_middleware())

When I do this, I get the error local_filter_middleware() missing 2 required positional arguments: 'make_request' and 'w3'. Sure enough, the middleware does require those 2 arguments (source). There are no docs or even tests from the related PR (#732) that ever call local_filter_middleware with w3 or make_request. I have no idea what to pass in for make_request. I found it odd that I was calling local_filter_middleware() at all, but middleware_onion.add(local_request_middleware) doesn't work either.

How can it be fixed?

Please let me know what arguments I'm supposed to pass into local_filter_middleware(), and consider editing the relevant documentation for the next release so others don't need to ask.

@jgeary jgeary changed the title Improve documentation for local_filter_middleware local_filter_middleware example doesn't work Feb 15, 2019
@dylanjw
Copy link
Contributor

dylanjw commented Feb 23, 2019

The doc example shouldn't have the parentheses. It should be:

from web3.middleware import local_filter_middleware
w3.middleware_onion.add(local_filter_middleware)

@dylanjw
Copy link
Contributor

dylanjw commented Feb 23, 2019

middleware_onion.add(local_request_middleware) doesn't work either.

@jgeary What issue did you have? Im not getting an error adding the middleware this way.

Python 3.6.4 (default, Jun  5 2018, 18:04:05) 
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from web3 import Web3, EthereumTesterProvider
>>> from web3.middleware import local_filter_middleware
>>> w3 = Web3(EthereumTesterProvider())
>>> w3.middleware_onion.add(local_filter_middleware)
>>> 

@jgeary
Copy link
Author

jgeary commented Mar 22, 2019

@dylanjw thanks for getting back about this. im using an asyncio event loop to watch for new events from multiple different filters in a continuous loop, and it seems like this middleware isn't compatible with that. here is what im working with: (modified to make it more generic and ignore unnecessary bits):

import web3
from web3 import Web3, WebsocketProvider
from web3.middleware import geth_poa_middleware
from web3.middleware import local_filter_middleware
import asyncio
import aiohttp

w3 = Web3(WebsocketProvider(os.getenv('INFURA_WS_URI')))

async def post_event_loop(event_filter, name):
    all_entries = event_filter.get_all_entries()
    if not all_entries:
        print('no old entries')
    for event in all_entries:
        print('old entry', event.field)
    while True:
        new_entries = event_filter.get_new_entries()
        if not new_entries:
            print ('no new entries')
        for event in new_entries:
            print('new entry', eventfield)
        await asyncio.sleep(10)

def main():
    w3.middleware_onion.inject(geth_poa_middleware, layer=0)
    w3.middleware_onion.add(local_filter_middleware)

    with open('abis/contract.json') as contract_file:
        contract_data = json.load(contract_file)
    Contract = w3.eth.contract(abi=contract_data['abi'], address=os.getenv('CONTRACT_ADDRESS'))
    event_filter1 = Contract.events.EventName1.createFilter(fromBlock=0)
    event_filter2 = Contract.events.EventName2.createFilter(fromBlock=0)

    loop = asyncio.new_event_loop()
    asyncio.set_event_loop(loop)
    try:
        loop.run_until_complete(
            asyncio.gather(
                post_event_loop(event_filter1, 'filter 1'),
                post_event_loop(event_filter2, 'filter 2'),
            )
        )
    finally:
        loop.close()

when i run this script without installing the local filter middleware, it smoothly prints old and new entries. the only issue is it fails every now and then after the node deletes my filters. in production im using a process manager that restarts it after it fails, and the other services dependent on this service are not affected by the redundancy. so this isn't crucial, but i would prefer to use the local filter middleware. but when i run this script as it is above, it hangs for a while (and mysteriously sometimes prints out a single no old entries after a while. when i quit the program with a SIGINT, i get the following trace:

Traceback (most recent call last):
  File "blockchain_service.py", line 184, in <module>
    main()
  File "blockchain_service.py", line 169, in main
    post_event_loop(feed_opened_filter, feed_opened_fields, 'feed_opened'),
  File "/Users/james/.pyenv/versions/3.6.8/lib/python3.6/asyncio/base_events.py", line 471, in run_until_complete
    self.run_forever()
  File "/Users/james/.pyenv/versions/3.6.8/lib/python3.6/asyncio/base_events.py", line 438, in run_forever
    self._run_once()
  File "/Users/james/.pyenv/versions/3.6.8/lib/python3.6/asyncio/base_events.py", line 1451, in _run_once
    handle._run()
  File "/Users/james/.pyenv/versions/3.6.8/lib/python3.6/asyncio/events.py", line 145, in _run
    self._callback(*self._args)
  File "blockchain_service.py", line 54, in post_event_loop
    all_entries = event_filter.get_all_entries()
  File "/Users/james/.pyenv/versions/3.6.8/envs/erasure-bcs/lib/python3.6/site-packages/web3/_utils/filters.py", line 128, in get_all_entries
    log_entries = self._filter_valid_entries(self.web3.eth.getFilterLogs(self.filter_id))
  File "/Users/james/.pyenv/versions/3.6.8/envs/erasure-bcs/lib/python3.6/site-packages/web3/eth.py", line 368, in getFilterLogs
    "eth_getFilterLogs", [filter_id],
  File "/Users/james/.pyenv/versions/3.6.8/envs/erasure-bcs/lib/python3.6/site-packages/web3/manager.py", line 94, in request_blocking
    response = self._make_request(method, params)
  File "/Users/james/.pyenv/versions/3.6.8/envs/erasure-bcs/lib/python3.6/site-packages/web3/manager.py", line 81, in _make_request
    return request_func(method, params)
  File "/Users/james/.pyenv/versions/3.6.8/envs/erasure-bcs/lib/python3.6/site-packages/web3/middleware/filter.py", line 314, in middleware
    return {'result': _filter.get_logs()}
  File "/Users/james/.pyenv/versions/3.6.8/envs/erasure-bcs/lib/python3.6/site-packages/web3/middleware/filter.py", line 240, in get_logs
    max_blocks=MAX_BLOCK_REQUEST)))
  File "/Users/james/.pyenv/versions/3.6.8/envs/erasure-bcs/lib/python3.6/site-packages/web3/middleware/filter.py", line 179, in get_logs_multipart
    drop_items_with_none_value(params))
  File "/Users/james/.pyenv/versions/3.6.8/envs/erasure-bcs/lib/python3.6/site-packages/web3/eth.py", line 373, in getLogs
    "eth_getLogs", [filter_params],
  File "/Users/james/.pyenv/versions/3.6.8/envs/erasure-bcs/lib/python3.6/site-packages/web3/manager.py", line 94, in request_blocking
    response = self._make_request(method, params)
  File "/Users/james/.pyenv/versions/3.6.8/envs/erasure-bcs/lib/python3.6/site-packages/web3/manager.py", line 81, in _make_request
    return request_func(method, params)
  File "/Users/james/.pyenv/versions/3.6.8/envs/erasure-bcs/lib/python3.6/site-packages/web3/middleware/filter.py", line 318, in middleware
    return make_request(method, params)
  File "cytoolz/functoolz.pyx", line 232, in cytoolz.functoolz.curry.__call__
  File "/Users/james/.pyenv/versions/3.6.8/envs/erasure-bcs/lib/python3.6/site-packages/web3/middleware/formatting.py", line 48, in apply_formatters
    response = make_request(method, formatted_params)
  File "/Users/james/.pyenv/versions/3.6.8/envs/erasure-bcs/lib/python3.6/site-packages/web3/middleware/gas_price_strategy.py", line 18, in middleware
    return make_request(method, params)
  File "cytoolz/functoolz.pyx", line 232, in cytoolz.functoolz.curry.__call__
  File "/Users/james/.pyenv/versions/3.6.8/envs/erasure-bcs/lib/python3.6/site-packages/web3/middleware/formatting.py", line 48, in apply_formatters
    response = make_request(method, formatted_params)
  File "/Users/james/.pyenv/versions/3.6.8/envs/erasure-bcs/lib/python3.6/site-packages/web3/middleware/attrdict.py", line 18, in middleware
    response = make_request(method, params)
  File "cytoolz/functoolz.pyx", line 232, in cytoolz.functoolz.curry.__call__
  File "/Users/james/.pyenv/versions/3.6.8/envs/erasure-bcs/lib/python3.6/site-packages/web3/middleware/formatting.py", line 48, in apply_formatters
    response = make_request(method, formatted_params)
  File "/Users/james/.pyenv/versions/3.6.8/envs/erasure-bcs/lib/python3.6/site-packages/web3/middleware/normalize_errors.py", line 9, in middleware
    result = make_request(method, params)
  File "cytoolz/functoolz.pyx", line 232, in cytoolz.functoolz.curry.__call__
  File "/Users/james/.pyenv/versions/3.6.8/envs/erasure-bcs/lib/python3.6/site-packages/web3/middleware/formatting.py", line 50, in apply_formatters
    response = make_request(method, params)
  File "cytoolz/functoolz.pyx", line 232, in cytoolz.functoolz.curry.__call__
  File "/Users/james/.pyenv/versions/3.6.8/envs/erasure-bcs/lib/python3.6/site-packages/web3/middleware/formatting.py", line 48, in apply_formatters
    response = make_request(method, formatted_params)
  File "cytoolz/functoolz.pyx", line 232, in cytoolz.functoolz.curry.__call__
  File "/Users/james/.pyenv/versions/3.6.8/envs/erasure-bcs/lib/python3.6/site-packages/web3/middleware/formatting.py", line 50, in apply_formatters
    response = make_request(method, params)
  File "/Users/james/.pyenv/versions/3.6.8/envs/erasure-bcs/lib/python3.6/site-packages/web3/providers/websocket.py", line 119, in make_request
    return future.result()
  File "/Users/james/.pyenv/versions/3.6.8/lib/python3.6/concurrent/futures/_base.py", line 427, in result
    self._condition.wait(timeout)
  File "/Users/james/.pyenv/versions/3.6.8/lib/python3.6/threading.py", line 295, in wait
    waiter.acquire()
KeyboardInterrupt
Task exception was never retrieved
future: <Task finished coro=<post_event_loop() done, defined at blockchain_service.py:52> exception=KeyboardInterrupt()>
Traceback (most recent call last):
  File "blockchain_service.py", line 184, in <module>
    main()
  File "blockchain_service.py", line 169, in main
    post_event_loop(feed_opened_filter, feed_opened_fields, 'feed_opened'),
  File "/Users/james/.pyenv/versions/3.6.8/lib/python3.6/asyncio/base_events.py", line 471, in run_until_complete
    self.run_forever()
  File "/Users/james/.pyenv/versions/3.6.8/lib/python3.6/asyncio/base_events.py", line 438, in run_forever
    self._run_once()
  File "/Users/james/.pyenv/versions/3.6.8/lib/python3.6/asyncio/base_events.py", line 1451, in _run_once
    handle._run()
  File "/Users/james/.pyenv/versions/3.6.8/lib/python3.6/asyncio/events.py", line 145, in _run
    self._callback(*self._args)
  File "blockchain_service.py", line 54, in post_event_loop
    all_entries = event_filter.get_all_entries()
  File "/Users/james/.pyenv/versions/3.6.8/envs/erasure-bcs/lib/python3.6/site-packages/web3/_utils/filters.py", line 128, in get_all_entries
    log_entries = self._filter_valid_entries(self.web3.eth.getFilterLogs(self.filter_id))
  File "/Users/james/.pyenv/versions/3.6.8/envs/erasure-bcs/lib/python3.6/site-packages/web3/eth.py", line 368, in getFilterLogs
    "eth_getFilterLogs", [filter_id],
  File "/Users/james/.pyenv/versions/3.6.8/envs/erasure-bcs/lib/python3.6/site-packages/web3/manager.py", line 94, in request_blocking
    response = self._make_request(method, params)
  File "/Users/james/.pyenv/versions/3.6.8/envs/erasure-bcs/lib/python3.6/site-packages/web3/manager.py", line 81, in _make_request
    return request_func(method, params)
  File "/Users/james/.pyenv/versions/3.6.8/envs/erasure-bcs/lib/python3.6/site-packages/web3/middleware/filter.py", line 314, in middleware
    return {'result': _filter.get_logs()}
  File "/Users/james/.pyenv/versions/3.6.8/envs/erasure-bcs/lib/python3.6/site-packages/web3/middleware/filter.py", line 240, in get_logs
    max_blocks=MAX_BLOCK_REQUEST)))
  File "/Users/james/.pyenv/versions/3.6.8/envs/erasure-bcs/lib/python3.6/site-packages/web3/middleware/filter.py", line 179, in get_logs_multipart
    drop_items_with_none_value(params))
  File "/Users/james/.pyenv/versions/3.6.8/envs/erasure-bcs/lib/python3.6/site-packages/web3/eth.py", line 373, in getLogs
    "eth_getLogs", [filter_params],
  File "/Users/james/.pyenv/versions/3.6.8/envs/erasure-bcs/lib/python3.6/site-packages/web3/manager.py", line 94, in request_blocking
    response = self._make_request(method, params)
  File "/Users/james/.pyenv/versions/3.6.8/envs/erasure-bcs/lib/python3.6/site-packages/web3/manager.py", line 81, in _make_request
    return request_func(method, params)
  File "/Users/james/.pyenv/versions/3.6.8/envs/erasure-bcs/lib/python3.6/site-packages/web3/middleware/filter.py", line 318, in middleware
    return make_request(method, params)
  File "cytoolz/functoolz.pyx", line 232, in cytoolz.functoolz.curry.__call__
  File "/Users/james/.pyenv/versions/3.6.8/envs/erasure-bcs/lib/python3.6/site-packages/web3/middleware/formatting.py", line 48, in apply_formatters
    response = make_request(method, formatted_params)
  File "/Users/james/.pyenv/versions/3.6.8/envs/erasure-bcs/lib/python3.6/site-packages/web3/middleware/gas_price_strategy.py", line 18, in middleware
    return make_request(method, params)
  File "cytoolz/functoolz.pyx", line 232, in cytoolz.functoolz.curry.__call__
  File "/Users/james/.pyenv/versions/3.6.8/envs/erasure-bcs/lib/python3.6/site-packages/web3/middleware/formatting.py", line 48, in apply_formatters
    response = make_request(method, formatted_params)
  File "/Users/james/.pyenv/versions/3.6.8/envs/erasure-bcs/lib/python3.6/site-packages/web3/middleware/attrdict.py", line 18, in middleware
    response = make_request(method, params)
  File "cytoolz/functoolz.pyx", line 232, in cytoolz.functoolz.curry.__call__
  File "/Users/james/.pyenv/versions/3.6.8/envs/erasure-bcs/lib/python3.6/site-packages/web3/middleware/formatting.py", line 48, in apply_formatters
    response = make_request(method, formatted_params)
  File "/Users/james/.pyenv/versions/3.6.8/envs/erasure-bcs/lib/python3.6/site-packages/web3/middleware/normalize_errors.py", line 9, in middleware
    result = make_request(method, params)
  File "cytoolz/functoolz.pyx", line 232, in cytoolz.functoolz.curry.__call__
  File "/Users/james/.pyenv/versions/3.6.8/envs/erasure-bcs/lib/python3.6/site-packages/web3/middleware/formatting.py", line 50, in apply_formatters
    response = make_request(method, params)
  File "cytoolz/functoolz.pyx", line 232, in cytoolz.functoolz.curry.__call__
  File "/Users/james/.pyenv/versions/3.6.8/envs/erasure-bcs/lib/python3.6/site-packages/web3/middleware/formatting.py", line 48, in apply_formatters
    response = make_request(method, formatted_params)
  File "cytoolz/functoolz.pyx", line 232, in cytoolz.functoolz.curry.__call__
  File "/Users/james/.pyenv/versions/3.6.8/envs/erasure-bcs/lib/python3.6/site-packages/web3/middleware/formatting.py", line 50, in apply_formatters
    response = make_request(method, params)
  File "/Users/james/.pyenv/versions/3.6.8/envs/erasure-bcs/lib/python3.6/site-packages/web3/providers/websocket.py", line 119, in make_request
    return future.result()
  File "/Users/james/.pyenv/versions/3.6.8/lib/python3.6/concurrent/futures/_base.py", line 427, in result
    self._condition.wait(timeout)
  File "/Users/james/.pyenv/versions/3.6.8/lib/python3.6/threading.py", line 295, in wait
    waiter.acquire()
KeyboardInterrupt
sys:1: RuntimeWarning: coroutine 'post_event_loop' was never awaited

@dylanjw
Copy link
Contributor

dylanjw commented Mar 22, 2019

The function post_event_loop is missing the async keyword.

Needs to be:

async def post_event_loop(event_filter, name):

Maybe that is just a typo obfuscating the original issue. If so, can you post the traceback once you have the async keyword added?

@jgeary
Copy link
Author

jgeary commented Mar 22, 2019

@dylanjw good catch, but i actually already had the async and just accidentally didnt copy it into my comment. so, i ensured it's there and edited my original comment, and im still getting the above traceback after sending a keyboard interrupt.

@dylanjw
Copy link
Contributor

dylanjw commented Mar 22, 2019

Can you try to reduce your code to the simplest broken example?

What happens when you run a synchronous version of post_event_loop? My guess is that it is stuck making requests for logs to infura, since it is starting from block 0.

You can try setting the environment variable WEB3_MAX_BLOCK_REQUEST to something higher. The default is 50. With 7 million blocks, that would be making +100,000 requests.

@wolovim
Copy link
Member

wolovim commented Feb 22, 2021

cleaning up stale issues; please reopen/open a new one if appropriate.

@wolovim wolovim closed this as completed Feb 22, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants