Docs and refactoring from discussions on PR #2457

- Add documentation
- Refactor out the provider flag since there is no support for sending transactions w/ CCIP Read support
- Refactor some around the way the ``eth_call`` is made, add flag directly to ``eth_call`` and make ``durin_call()`` and internal ``_durin_call()``
- Refactor data back into post request after linting issues went away
- Make CCIP Read max redirects configurable and add tests
- OffchainLookup should extend from ContractLogicError
- Add some more testing and refactor some more; use web3 ValidationError instead of importing from eth_utils
- Add MultipleFailedRequests if multiple failed requests for similar / same data
- Reconcile changes with rebased async contract
- Raise an exception if CCIP Read urls do not contain expected keyword
- Add tests for ENS offchain resolution via CCIP Read functionality
- CCIP Read support for EthereumTesterProvider / eth-tester

docs/contracts.rst

@@ -743,6 +743,20 @@ Positional and keyword arguments supplied to the contract function subclass
 will be used to find the contract function by signature,
 and forwarded to the contract function when applicable.
 
+`EIP-3668 <https://eips.ethereum.org/EIPS/eip-3668>`_ introduced support for the ``OffchainLookup`` revert /
+CCIP Read support. The ``ccip_read_enabled`` flag is set to ``True`` for calls by default, as recommended in EIP-3668.
+If raising the ``OffchainLookup`` revert is preferred, the flag may be set to ``False`` on a per-call basis.
+
+.. code-block:: python
+
+        >>> myContract.functions.revertsWithOffchainLookup(myData).call(ccip_read_enabled=False)
+        *** web3.exceptions.OffchainLookup
+
+Disabling CCIP Read support can be useful if a transaction needs to be sent to the callback function. In such cases,
+"preflighting" with an ``eth_call``, handling the ``OffchainLookup``, and sending the data via a transaction may be
+necessary. See :ref:`ccip-read-example` in the examples section for how to preflight a transaction with a contract call.
+
+
 Methods
 ~~~~~~~~~~
 

docs/examples.rst

@@ -624,6 +624,44 @@ When someone has an allowance they can transfer those tokens using the
 .. _ERC20: https://github.com/ethereum/EIPs/blob/7f4f0377730f5fc266824084188cc17cf246932e/EIPS/eip-20.md
 
 
+.. _ccip-read-example:
+
+CCIP Read support for offchain lookup
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Contract calls support CCIP Read by default via a ``ccip_read_enabled`` flag that is set to a default value of ``True``.
+The following should work by default without raising the ``OffchainLookup`` and instead handling it appropriately as
+per the specification outlined in `EIP-3668 <https://eips.ethereum.org/EIPS/eip-3668>`_.
+
+.. code-block:: python
+
+    myContract.functions.revertsWithOffchainLookup(myData).call()
+
+If the offchain lookup requires the user to send a transaction rather than make a call, this may be handled
+appropriately in the following way:
+
+.. code-block:: python
+
+    from web3 import Web3, WebsocketProvider
+    from web3.utils import handle_offchain_lookup
+
+    w3 = Web3(WebsocketProvider(...))
+
+    myContract = w3.eth.contract(address=...)
+    myData = b'data for offchain lookup function call'
+
+    # preflight with an `eth_call` and handle the exception
+    try:
+        myContract.functions.revertsWithOffchainLookup(myData).call(ccip_read_enabled=False)
+    except OffchainLookup as ocl:
+        tx = {'to': myContract.address, 'from': my_account}
+        data_for_callback_function = handle_offchain_lookup(ocl.payload)
+        tx['data'] = data_for_callback_function
+
+        # send the built transaction with `eth_sendTransaction` or sign and send with `eth_sendRawTransaction`
+        tx_hash = w3.eth.send_transaction(tx)
+
+
 Contract Unit Tests in Python
 -----------------------------
 

docs/index.rst

@@ -67,6 +67,7 @@ Table of Contents
     web3.miner
     web3.geth
     web3.parity
+    web3.utils
     gas_price
     ens
     constants

docs/web3.eth.rst

@@ -1064,7 +1064,7 @@ The following methods are available on the ``web3.eth`` namespace.
     .. warning:: Deprecated: This property is deprecated in favor of
       :meth:`~web3.eth.Eth.sign_typed_data()`
 
-.. py:method:: Eth.call(transaction, block_identifier=web3.eth.default_block, state_override=None)
+.. py:method:: Eth.call(transaction, block_identifier=web3.eth.default_block, state_override=None, ccip_read_enabled=True)
 
     * Delegates to ``eth_call`` RPC Method
 
@@ -1090,6 +1090,15 @@ The following methods are available on the ``web3.eth`` namespace.
     View their `usage documentation <https://geth.ethereum.org/docs/rpc/ns-eth#3-object---state-override-set>`_
     for a list of possible parameters.
 
+    `EIP-3668 <https://eips.ethereum.org/EIPS/eip-3668>`_ introduced support for the ``OffchainLookup`` revert / CCIP
+    Read support. In order to properly handle a call to a contract function that reverts with an ``OffchainLookup``
+    error for offchain data retrieval, the ``ccip_read_enabled`` flag has been added to the ``eth_call`` method.
+    ``ccip_read_enabled`` is set to ``True`` by default for calls, as recommended in EIP-3668. Therefore, calls to
+    contract functions that revert with an ``OffchainLookup`` will be handled appropriately by default. If the
+    ``ccip_read_enabled`` flag is set to ``False``, the call will raise the ``OffchainLookup`` instead of properly
+    handling the exception according to EIP-3668. This may be useful for "preflighting" a transaction call (see
+    :ref:`ccip-read-example` within the examples section).
+
 
 .. py:method:: Eth.fee_history(block_count, newest_block, reward_percentiles=None)
 

docs/web3.utils.rst

@@ -0,0 +1,17 @@
+Utils
+=====
+
+.. py:module:: web3.utils
+
+The ``utils`` module houses public utility and helper functions.
+
+
+.. py:method:: Utils.handle_offchain_lookup(offchain_lookup_payload, transaction)
+
+    Handle ``OffchainLookup`` reverts on contract function calls manually. For an example, see :ref:`ccip-read-example`
+    within the examples section.
+
+
+.. py:method:: Utils.async_handle_offchain_lookup(offchain_lookup_payload, transaction)
+
+    The async version of the ``handle_offchain_lookup()`` utility method described above.

tests/core/contracts/test_offchain_lookup.py

@@ -0,0 +1,135 @@
+import pytest
+
+from eth_abi import (
+    decode_abi,
+)
+
+from web3._utils.module_testing.module_testing_utils import (
+    mock_offchain_lookup_request_response,
+)
+from web3._utils.module_testing.offchain_lookup_contract import (
+    OFFCHAIN_LOOKUP_ABI,
+    OFFCHAIN_LOOKUP_BYTECODE,
+    OFFCHAIN_LOOKUP_BYTECODE_RUNTIME,
+)
+from web3._utils.type_conversion import (
+    to_hex_if_bytes,
+)
+from web3.exceptions import (
+    TooManyRequests,
+    ValidationError,
+)
+
+# "test offchain lookup" as an abi-encoded string
+OFFCHAIN_LOOKUP_TEST_DATA = '0x0000000000000000000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000000000001474657374206f6666636861696e206c6f6f6b7570000000000000000000000000'  # noqa: E501
+# "web3py" as an abi-encoded string
+WEB3PY_AS_HEXBYTES = '0x000000000000000000000000000000000000000000000000000000000000002000000000000000000000000000000000000000000000000000000000000000067765623370790000000000000000000000000000000000000000000000000000'  # noqa: E501
+
+
+@pytest.fixture
+def OffchainLookup(w3):
+    # compiled from `web3/_utils/module_testing/contract_sources/OffchainLookup.sol`
+    return w3.eth.contract(
+        abi=OFFCHAIN_LOOKUP_ABI,
+        bytecode=OFFCHAIN_LOOKUP_BYTECODE,
+        bytecode_runtime=OFFCHAIN_LOOKUP_BYTECODE_RUNTIME,
+    )
+
+
+@pytest.fixture
+def offchain_lookup_contract(
+    w3, wait_for_block, OffchainLookup, wait_for_transaction, address_conversion_func,
+):
+    wait_for_block(w3)
+    deploy_txn_hash = OffchainLookup.constructor().transact({'gas': 10000000})
+    deploy_receipt = wait_for_transaction(w3, deploy_txn_hash)
+    contract_address = address_conversion_func(deploy_receipt['contractAddress'])
+
+    bytecode = w3.eth.get_code(contract_address)
+    assert bytecode == OffchainLookup.bytecode_runtime
+    deployed_offchain_lookup = OffchainLookup(address=contract_address)
+    assert deployed_offchain_lookup.address == contract_address
+    return deployed_offchain_lookup
+
+
+def test_offchain_lookup_functionality(
+    offchain_lookup_contract, monkeypatch,
+):
+    normalized_address = to_hex_if_bytes(offchain_lookup_contract.address)
+    mock_offchain_lookup_request_response(
+        monkeypatch,
+        mocked_request_url=f'https://web3.py/gateway/{normalized_address}/{OFFCHAIN_LOOKUP_TEST_DATA}.json',  # noqa: E501
+        mocked_json_data=WEB3PY_AS_HEXBYTES,
+    )
+    response = offchain_lookup_contract.caller.testOffchainLookup(OFFCHAIN_LOOKUP_TEST_DATA)
+    assert decode_abi(['string'], response)[0] == 'web3py'
+
+
+def test_offchain_lookup_raises_for_improperly_formatted_rest_request_response(
+    offchain_lookup_contract, monkeypatch,
+):
+    normalized_address = to_hex_if_bytes(offchain_lookup_contract.address)
+    mock_offchain_lookup_request_response(
+        monkeypatch,
+        mocked_request_url=f'https://web3.py/gateway/{normalized_address}/{OFFCHAIN_LOOKUP_TEST_DATA}.json',  # noqa: E501
+        mocked_json_data=WEB3PY_AS_HEXBYTES,
+        json_data_field='not_data',
+    )
+    with pytest.raises(ValidationError, match="missing 'data' field"):
+        offchain_lookup_contract.caller.testOffchainLookup(OFFCHAIN_LOOKUP_TEST_DATA)
+
+
+@pytest.mark.parametrize('status_code_non_4xx_error', [100, 300, 500, 600])
+def test_eth_call_offchain_lookup_tries_next_url_for_non_4xx_error_status_and_tests_POST(
+    offchain_lookup_contract, monkeypatch, status_code_non_4xx_error,
+) -> None:
+    normalized_contract_address = to_hex_if_bytes(offchain_lookup_contract.address).lower()
+
+    # The next url in our test contract doesn't contain '{data}', triggering the POST request
+    # logic. The idea here is to return a bad status for the first url (GET) and a success
+    # status from the second call (POST) to test both that we move on to the next url with
+    # non 4xx status and that the POST logic is also working as expected.
+    mock_offchain_lookup_request_response(
+        monkeypatch,
+        mocked_request_url=f'https://web3.py/gateway/{normalized_contract_address}/{OFFCHAIN_LOOKUP_TEST_DATA}.json',  # noqa: E501
+        mocked_status_code=status_code_non_4xx_error,
+        mocked_json_data=WEB3PY_AS_HEXBYTES,
+    )
+    mock_offchain_lookup_request_response(
+        monkeypatch,
+        http_method='POST',
+        mocked_request_url=f'https://web3.py/gateway/{normalized_contract_address}.json',
+        mocked_status_code=200,
+        mocked_json_data=WEB3PY_AS_HEXBYTES,
+        sender=normalized_contract_address,
+        calldata=OFFCHAIN_LOOKUP_TEST_DATA,
+    )
+    response = offchain_lookup_contract.caller.testOffchainLookup(OFFCHAIN_LOOKUP_TEST_DATA)
+    assert decode_abi(['string'], response)[0] == 'web3py'
+
+
+@pytest.mark.parametrize('status_code_4xx_error', [400, 410, 450, 499])
+def test_eth_call_offchain_lookup_calls_raise_for_status_for_4xx_status_code(
+    offchain_lookup_contract, monkeypatch, status_code_4xx_error,
+) -> None:
+    normalized_contract_address = to_hex_if_bytes(offchain_lookup_contract.address).lower()
+    mock_offchain_lookup_request_response(
+        monkeypatch,
+        mocked_request_url=f'https://web3.py/gateway/{normalized_contract_address}/{OFFCHAIN_LOOKUP_TEST_DATA}.json',  # noqa: E501
+        mocked_status_code=status_code_4xx_error,
+        mocked_json_data=WEB3PY_AS_HEXBYTES,
+    )
+    with pytest.raises(Exception, match="called raise_for_status\\(\\)"):
+        offchain_lookup_contract.caller.testOffchainLookup(OFFCHAIN_LOOKUP_TEST_DATA)
+
+
+def test_offchain_lookup_raises_on_continuous_redirect(
+    offchain_lookup_contract, monkeypatch,
+):
+    normalized_address = to_hex_if_bytes(offchain_lookup_contract.address)
+    mock_offchain_lookup_request_response(
+        monkeypatch,
+        mocked_request_url=f'https://web3.py/gateway/{normalized_address}/0x.json',
+    )
+    with pytest.raises(TooManyRequests, match="Too many CCIP read redirects"):
+        offchain_lookup_contract.caller.continuousOffchainLookup()

tests/core/providers/test_http_provider.py

@@ -39,7 +39,7 @@ def test_user_provided_session():
     w3 = Web3(provider)
     assert w3.manager.provider == provider
 
-    session = request._get_session(URI)
+    session = request.get_session(URI)
     adapter = session.get_adapter(URI)
     assert isinstance(adapter, HTTPAdapter)
     assert adapter._pool_connections == 20

tests/core/utilities/test_request.py

@@ -79,7 +79,7 @@ def test_precached_session(mocker):
     assert len(request._session_cache) == 1
 
     # Ensure the timeout was passed to the request
-    session = request._get_session(URI)
+    session = request.get_session(URI)
     session.post.assert_called_once_with(URI, data=b'request', timeout=60)
 
     # Ensure the adapter parameters match those we specified

tests/core/web3-module/test_providers.py

@@ -23,3 +23,7 @@ def test_auto_provider_none():
     w3.toHex(0) == '0x0'
 
     type(w3.provider) == AutoProvider
+
+
+def test_provider_default_value_for_ccip_read_redirect(w3):
+    assert w3.provider.ccip_read_max_redirects == 4

tests/ens/conftest.py

@@ -5,12 +5,18 @@
 from eth_tester import (
     EthereumTester,
 )
+from eth_utils import (
+    to_checksum_address,
+)
 
 from ens import ENS
 from ens.contract_data import (
     extended_resolver_abi,
     extended_resolver_bytecode,
     extended_resolver_bytecode_runtime,
+    offchain_resolver_abi,
+    offchain_resolver_bytecode,
+    offchain_resolver_bytecode_runtime,
     registrar_abi,
     registrar_bytecode,
     registrar_bytecode_runtime,
@@ -101,6 +107,15 @@ def ExtendedResolver(w3):
     )
 
 
+def OffchainResolver(w3):
+    return w3.eth.contract(
+        bytecode=offchain_resolver_bytecode,
+        bytecode_runtime=offchain_resolver_bytecode_runtime,
+        abi=offchain_resolver_abi,
+        ContractFactoryClass=Contract,
+    )
+
+
 def ENSFactory(w3):
     return w3.eth.contract(
         bytecode="6060604052341561000f57600080fd5b60008080526020527fad3228b676f7d3cd4284a5443f17f1962b36e491b30a40b2405849e597ba5fb58054600160a060020a033316600160a060020a0319909116179055610501806100626000396000f300606060405236156100805763ffffffff7c01000000000000000000000000000000000000000000000000000000006000350416630178b8bf811461008557806302571be3146100b757806306ab5923146100cd57806314ab9038146100f457806316a25cbd146101175780631896f70a1461014a5780635b0fc9c31461016c575b600080fd5b341561009057600080fd5b61009b60043561018e565b604051600160a060020a03909116815260200160405180910390f35b34156100c257600080fd5b61009b6004356101ac565b34156100d857600080fd5b6100f2600435602435600160a060020a03604435166101c7565b005b34156100ff57600080fd5b6100f260043567ffffffffffffffff60243516610289565b341561012257600080fd5b61012d600435610355565b60405167ffffffffffffffff909116815260200160405180910390f35b341561015557600080fd5b6100f2600435600160a060020a036024351661038c565b341561017757600080fd5b6100f2600435600160a060020a0360243516610432565b600090815260208190526040902060010154600160a060020a031690565b600090815260208190526040902054600160a060020a031690565b600083815260208190526040812054849033600160a060020a039081169116146101f057600080fd5b8484604051918252602082015260409081019051908190039020915083857fce0457fe73731f824cc272376169235128c118b49d344817417c6d108d155e8285604051600160a060020a03909116815260200160405180910390a3506000908152602081905260409020805473ffffffffffffffffffffffffffffffffffffffff1916600160a060020a03929092169190911790555050565b600082815260208190526040902054829033600160a060020a039081169116146102b257600080fd5b827f1d4f9bbfc9cab89d66e1a1562f2233ccbf1308cb4f63de2ead5787adddb8fa688360405167ffffffffffffffff909116815260200160405180910390a250600091825260208290526040909120600101805467ffffffffffffffff90921674010000000000000000000000000000000000000000027fffffffff0000000000000000ffffffffffffffffffffffffffffffffffffffff909216919091179055565b60009081526020819052604090206001015474010000000000000000000000000000000000000000900467ffffffffffffffff1690565b600082815260208190526040902054829033600160a060020a039081169116146103b557600080fd5b827f335721b01866dc23fbee8b6b2c7b1e14d6f05c28cd35a2c934239f94095602a083604051600160a060020a03909116815260200160405180910390a250600091825260208290526040909120600101805473ffffffffffffffffffffffffffffffffffffffff1916600160a060020a03909216919091179055565b600082815260208190526040902054829033600160a060020a0390811691161461045b57600080fd5b827fd4735d920b0f87494915f556dd9b54c8f309026070caea5c737245152564d26683604051600160a060020a03909116815260200160405180910390a250600091825260208290526040909120805473ffffffffffffffffffffffffffffffffffffffff1916600160a060020a039092169190911790555600a165627a7a7230582050975b6c54a16d216b563f4c4960d6ebc5881eb1ec73c2ef1f87920a251159530029",  # noqa: E501
@@ -270,6 +285,39 @@ def ens_setup():
         extended_resolver.address
     ).transact({'from': second_account})
 
+    # --- setup offchain resolver example --- #
+
+    # create offchain resolver
+    offchain_resolver = deploy(
+        w3, OffchainResolver, ens_key,
+
+        # use a made up url and mock the call to this endpoint in tests
+        args=[
+            [
+                "https://web3.py/gateway/{sender}/{data}.json",  # for GET request testing
+                "https://web3.py/gateway/{sender}.json",  # for POST request testing
+            ],
+            [to_checksum_address('0x4c40caf7f24a545095299972c381862469b080fb')]
+        ]
+    )
+
+    # set owner of offchainexample.eth to an account controlled by tests
+    ens_contract.functions.setSubnodeOwner(
+        eth_namehash,
+        w3.keccak(text='offchainexample'),
+        second_account
+    ).transact({'from': ens_key})
+
+    # ns.namehash('offchainexample.eth')
+    offchain_example_namehash = bytes32(
+        0x42041b0018edd29d7c17154b0c671acc0502ea0b3693cafbeadf58e6beaaa16c
+    )
+
+    ens_contract.functions.setResolver(
+        offchain_example_namehash,
+        offchain_resolver.address
+    ).transact({'from': second_account})
+
     # --- finish setup --- #
 
     # make the registrar the owner of the 'eth' name