Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dataclasses #3969

Merged
merged 71 commits into from
May 17, 2019
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
71 commits
Select commit Hold shift + click to select a range
46a1416
Introduce dataclass decorator
rakanalh Apr 30, 2019
d2cd3d8
Convert state changes to dataclasses
rakanalh Apr 30, 2019
c8ca385
Convert events to dataclasses
rakanalh Apr 30, 2019
1912b07
Convert identifiers to dataclasses
rakanalh Apr 30, 2019
22b6100
Use dataclasses-json package
rakanalh Apr 30, 2019
091d488
Convert transfer/state.py to dataclasses
rakanalh Apr 30, 2019
8e6765a
Introduce serialization fields for custom types
rakanalh Apr 30, 2019
4595dbc
convert mediate_transfer.events module to dataclasses
rakanalh Apr 30, 2019
b510beb
convert mediate_transfer.state_change module to dataclasses
rakanalh Apr 30, 2019
4982f16
convert mediate_transfer.state module to dataclasses
rakanalh Apr 30, 2019
706c8cd
Fix mypy issues in the state machine
rakanalh May 1, 2019
c60daee
Move tasks to a separate module to prevent import cycles
rakanalh May 2, 2019
22a312a
Fix pylint no-member errors
rakanalh May 2, 2019
867ad0a
Decorated dataclasses with parameters handling
rakanalh May 2, 2019
5813d3c
Fix imports based on the new structure
rakanalh May 2, 2019
530d211
Fix LockedTransferSignedStateProperties
jomuel May 2, 2019
5be3c4d
Initialize signatures
rakanalh May 3, 2019
e136302
Fix wal tests related to dataclass serialization
rakanalh May 6, 2019
371e2e7
re-Blackify after rebase
rakanalh May 6, 2019
16d3f0e
Attach type to stored JSON payload
rakanalh May 6, 2019
64d39d1
Introduce DictSerializer
rakanalh May 6, 2019
b3090cd
Fix unit tests - part 1
rakanalh May 6, 2019
70a48ec
Use DictSerializer and resolve lint errors
rakanalh May 7, 2019
f87bd63
Construct a query to look for balance proof events in transfer
rakanalh May 7, 2019
7447956
Fix unit tests - part 2
rakanalh May 7, 2019
cba49d4
Remove field(init=False) to make fields restorable from JSON payload
rakanalh May 7, 2019
eab8953
Fix lint
rakanalh May 7, 2019
5a1ee42
Rebase and fix lint issues
rakanalh May 8, 2019
da43e32
Make test factories work with dataclasses
jomuel May 8, 2019
ce3d4e6
Fix secrethash initialization
rakanalh May 8, 2019
a921396
Serialization for a tuple of addresses
rakanalh May 8, 2019
e4e8536
Serialize keccack256 properly, fix smoke tests
rakanalh May 8, 2019
f2c69be
Initialize network graph state in fuzz tests
rakanalh May 8, 2019
7b0f9ce
Fix signature initialization for UDP in tests
rakanalh May 8, 2019
2feebbb
Use JSONSerializer in matrix utils
rakanalh May 8, 2019
57bf69e
Upgrade marshmallow and dependencies to dict-supporting version
rakanalh May 8, 2019
0ade4a5
Cache generated schema for classes
rakanalh May 8, 2019
922340f
Exclude PRNG from comparisons
rakanalh May 8, 2019
58e28bc
Support PRNG serialization
rakanalh May 8, 2019
dcce4dc
Organize tasks properly to prevent cyclic imports
rakanalh May 9, 2019
02b7ef1
Add serialization for polymorphic fields
rakanalh May 9, 2019
c6ea7d4
Move TransferTask to the architecture module
rakanalh May 9, 2019
851aeea
Use transfer_state instead of secret reveal event in initiator
rakanalh May 9, 2019
fed6c6e
Remove shortcut and use direct non-quoted type for channel state
rakanalh May 9, 2019
3dfc5dd
Move classes around to unquote types
rakanalh May 9, 2019
74073c8
Upgrade marshmallow polyfield
rakanalh May 9, 2019
65556c4
Attach/Detach _type to every schema dump
rakanalh May 9, 2019
0fb7605
Cache schema generation and attach type to serialized data
rakanalh May 13, 2019
029f63d
Move state restoration code to storage module
rakanalh May 13, 2019
af92ed5
Move BalanceProofSigned/UnsignedState to prevent cyclic imports
rakanalh May 13, 2019
8aa5b7f
Fix serialization of defaultdict
rakanalh May 14, 2019
b54790f
Specific filters to be converted to balance proof queries
rakanalh May 14, 2019
90e0d26
Fix flake8 lint
rakanalh May 14, 2019
aa7d975
Adapt rest module to new marshmallow schema usage
rakanalh May 14, 2019
2ed77ed
Fix test_receive_lockedtransfer_invalidnonce
rakanalh May 15, 2019
a02e473
Fix timeout in test_receive_lockedtransfer_invalidnonce
rakanalh May 15, 2019
19f799f
Fix test_automatic_secret_registration and test_receive_secrethashtra…
rakanalh May 15, 2019
f6ef20a
Pass signature in from_event in messages
rakanalh May 15, 2019
4b372c7
Fix test_regression_multiple_revealsecret
rakanalh May 15, 2019
0ac1488
Fix signature again
rakanalh May 15, 2019
a45ff85
Copy dictionary while before deserializing
rakanalh May 15, 2019
b0fa076
Override hash func for QueueIdentifier
rakanalh May 15, 2019
91e8760
Serialize QueueIdentifier in a custom Field
rakanalh May 15, 2019
70c8c7a
SendMessageEvent serialization
rakanalh May 15, 2019
8cbb340
Custom networkx.Graph serialization
rakanalh May 16, 2019
cb7ae9c
Another signature initialization failure fix
rakanalh May 16, 2019
359f7cd
Reintroduce EventRouteFailed, bad rebase
rakanalh May 16, 2019
0acbc08
Fix lint & Unit tests
rakanalh May 16, 2019
f159a21
Remove serialization module in utils
rakanalh May 16, 2019
3d94101
Use EMPTY_SIGNATURE instead of b""
rakanalh May 16, 2019
7b339b6
Remove message identifier from RetiableMessage and Optional CMDID fro…
rakanalh May 16, 2019
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions constraints.txt
Original file line number Diff line number Diff line change
Expand Up @@ -35,8 +35,9 @@ itsdangerous==0.24
Jinja2==2.10
lru-dict==1.1.6
MarkupSafe==1.0
marshmallow-polyfield==3.2
marshmallow==2.15.4
marshmallow-polyfield==5.5
marshmallow-dataclass==6.0.0c1
marshmallow==3.0.0rc6
matrix-client==0.3.2
miniupnpc==2.0.2
mirakuru==1.0.0
Expand Down Expand Up @@ -69,7 +70,7 @@ toolz==0.9.0
traitlets==4.3.2
urllib3==1.23
web3==4.9.1
webargs==5.1.3
webargs==5.3.1
websockets==6.0
Werkzeug==0.14.1
wrapt==1.11.1
11 changes: 3 additions & 8 deletions raiden/api/python.py
Original file line number Diff line number Diff line change
Expand Up @@ -31,19 +31,14 @@
from raiden.messages import RequestMonitoring
from raiden.settings import DEFAULT_RETRY_TIMEOUT, DEVELOPMENT_CONTRACT_VERSION
from raiden.transfer import architecture, views
from raiden.transfer.architecture import TransferTask
from raiden.transfer.events import (
EventPaymentReceivedSuccess,
EventPaymentSentFailed,
EventPaymentSentSuccess,
)
from raiden.transfer.state import (
BalanceProofSignedState,
InitiatorTask,
MediatorTask,
NettingChannelState,
TargetTask,
TransferTask,
)
from raiden.transfer.mediated_transfer.tasks import InitiatorTask, MediatorTask, TargetTask
from raiden.transfer.state import BalanceProofSignedState, NettingChannelState
from raiden.transfer.state_change import ActionChannelClose
from raiden.utils import pex, typing
from raiden.utils.gas_reserve import has_enough_gas_reserve
Expand Down
21 changes: 10 additions & 11 deletions raiden/api/rest.py
Original file line number Diff line number Diff line change
Expand Up @@ -613,7 +613,7 @@ def open(

result = self.channel_schema.dump(channel_state)

return api_response(result=result.data, status_code=HTTPStatus.CREATED)
return api_response(result=result, status_code=HTTPStatus.CREATED)
hackaugusto marked this conversation as resolved.
Show resolved Hide resolved

def connect(
self,
Expand Down Expand Up @@ -656,7 +656,7 @@ def leave(self, registry_address: typing.PaymentNetworkID, token_address: typing
)
closed_channels = self.raiden_api.token_network_leave(registry_address, token_address)
closed_channels = [
self.channel_schema.dump(channel_state).data for channel_state in closed_channels
self.channel_schema.dump(channel_state) for channel_state in closed_channels
]
return api_response(result=closed_channels)

Expand Down Expand Up @@ -718,8 +718,7 @@ def get_channel_list(
)
assert isinstance(raiden_service_result, list)
result = [
self.channel_schema.dump(channel_schema).data
for channel_schema in raiden_service_result
self.channel_schema.dump(channel_schema) for channel_schema in raiden_service_result
]
return api_response(result=result)

Expand All @@ -733,7 +732,7 @@ def get_tokens_list(self, registry_address: typing.PaymentNetworkID):
assert isinstance(raiden_service_result, list)
tokens_list = AddressList(raiden_service_result)
result = self.address_list_schema.dump(tokens_list)
return api_response(result=result.data)
return api_response(result=result)

def get_token_network_for_token(
self, registry_address: typing.PaymentNetworkID, token_address: typing.TokenAddress
Expand Down Expand Up @@ -839,7 +838,7 @@ def get_raiden_events_payment_history_with_timestamps(
unexpected_event=event.wrapped_event,
)

result.append(serialized_event.data)
result.append(serialized_event)
return api_response(result=result)

def get_raiden_internal_events_with_timestamps(self, limit, offset):
Expand Down Expand Up @@ -898,7 +897,7 @@ def get_channel(
partner_address=partner_address,
)
result = self.channel_schema.dump(channel_state)
return api_response(result=result.data)
return api_response(result=result)
except ChannelNotFound as e:
return api_error(errors=str(e), status_code=HTTPStatus.NOT_FOUND)

Expand Down Expand Up @@ -934,7 +933,7 @@ def get_partners_by_token(

schema_list = PartnersPerTokenList(return_list)
result = self.partner_per_token_list_schema.dump(schema_list)
return api_response(result=result.data)
return api_response(result=result)

def initiate_payment(
self,
Expand Down Expand Up @@ -1003,7 +1002,7 @@ def initiate_payment(
"secret_hash": sha3(secret),
}
result = self.payment_schema.dump(payment)
return api_response(result=result.data)
return api_response(result=result)

def _deposit(
self,
Expand Down Expand Up @@ -1044,7 +1043,7 @@ def _deposit(
)

result = self.channel_schema.dump(updated_channel_state)
return api_response(result=result.data)
return api_response(result=result)

def _close(
self, registry_address: typing.PaymentNetworkID, channel_state: NettingChannelState
Expand Down Expand Up @@ -1074,7 +1073,7 @@ def _close(
)

result = self.channel_schema.dump(updated_channel_state)
return api_response(result=result.data)
return api_response(result=result)

def patch_channel(
self,
Expand Down
4 changes: 2 additions & 2 deletions raiden/api/v1/encoding.py
Original file line number Diff line number Diff line change
Expand Up @@ -150,8 +150,8 @@ class BaseOpts(SchemaOpts):
This allows for having the Object the Schema encodes to inside of the class Meta
"""

def __init__(self, meta):
SchemaOpts.__init__(self, meta)
def __init__(self, meta, ordered):
SchemaOpts.__init__(self, meta, ordered=ordered)
self.decoding_class = getattr(meta, "decoding_class", None)


Expand Down
22 changes: 16 additions & 6 deletions raiden/blockchain_events_handler.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,10 +7,18 @@
from raiden.blockchain.state import get_channel_state
from raiden.connection_manager import ConnectionManager
from raiden.network.proxies.utils import get_onchain_locksroots
from raiden.storage.restore import (
get_event_with_balance_proof_by_locksroot,
get_state_change_with_balance_proof_by_locksroot,
)
from raiden.transfer import views
from raiden.transfer.architecture import StateChange
from raiden.transfer.identifiers import CanonicalIdentifier
from raiden.transfer.state import TokenNetworkState, TransactionChannelNewBalance
from raiden.transfer.state import (
TokenNetworkGraphState,
TokenNetworkState,
TransactionChannelNewBalance,
)
from raiden.transfer.state_change import (
ContractReceiveChannelBatchUnlock,
ContractReceiveChannelClosed,
Expand All @@ -23,10 +31,6 @@
ContractReceiveSecretReveal,
ContractReceiveUpdateTransfer,
)
from raiden.transfer.utils import (
get_event_with_balance_proof_by_locksroot,
get_state_change_with_balance_proof_by_locksroot,
)
from raiden.utils import pex, typing
from raiden_contracts.constants import (
EVENT_SECRET_REVEALED,
Expand Down Expand Up @@ -58,7 +62,12 @@ def handle_tokennetwork_new(raiden: "RaidenService", event: Event):
from_block=block_number,
)

token_network_state = TokenNetworkState(token_network_address, token_address)
token_network_graph_state = TokenNetworkGraphState(token_network_address)
token_network_state = TokenNetworkState(
address=token_network_address,
token_address=token_address,
network_graph=token_network_graph_state,
)

transaction_hash = event.event_data["transaction_hash"]

Expand Down Expand Up @@ -111,6 +120,7 @@ def handle_channel_new(raiden: "RaidenService", event: Event):
)
raiden.handle_and_track_state_change(new_channel)

# pylint: disable=E1101
partner_address = channel_state.partner_state.address
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So, in our case this happens when field(repr=False) is used and the field that is another dataclass. Shouldn't we just remove the field(repr=False) and have the lint working?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removing field(repr=False) in the case where we only disable repr for these fields is possible. However, there will be many different places where we need to override the field with something else.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, but this would fix roughly 32 instances that are just repr=True. Counted with:

ag --python 'field(repr=False)' --nobreak --nofilename | wc

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I checked this point out. Most of the disable=E1101 were added because partner_state and our_state are defined as field(repr=False). However, i would argue that setting them to be included in repr would be a bit too much because of the amount of data that will end up being logged.
Do you think we should accept this and remove field assignment to those attributes?


if ConnectionManager.BOOTSTRAP_ADDR != partner_address:
Expand Down
4 changes: 4 additions & 0 deletions raiden/constants.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,11 @@
BalanceHash,
BlockHash,
BlockNumber,
LockHash,
Locksroot,
RaidenProtocolVersion,
Secret,
SecretHash,
Signature,
TokenAmount,
)
Expand Down Expand Up @@ -44,11 +46,13 @@

EMPTY_HASH = BlockHash(bytes(32))
EMPTY_BALANCE_HASH = BalanceHash(bytes(32))
EMPTY_LOCK_HASH = LockHash(bytes(32))
EMPTY_MESSAGE_HASH = AdditionalHash(bytes(32))
EMPTY_HASH_KECCAK = keccak(EMPTY_HASH)
EMPTY_SIGNATURE = Signature(bytes(65))
EMPTY_MERKLE_ROOT = Locksroot(bytes(32))
EMPTY_SECRET = Secret(b"")
EMPTY_SECRETHASH = SecretHash(bytes(32))
ZERO_TOKENS = TokenAmount(0)

SECRET_LENGTH = 32
Expand Down
29 changes: 20 additions & 9 deletions raiden/message_handler.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,12 +11,12 @@
RevealSecret,
SecretRequest,
Unlock,
lockedtransfersigned_from_message,
)
from raiden.raiden_service import RaidenService
from raiden.routing import get_best_routes
from raiden.transfer import views
from raiden.transfer.architecture import StateChange
from raiden.transfer.mediated_transfer.state import lockedtransfersigned_from_message
from raiden.transfer.mediated_transfer.state_change import (
ReceiveLockExpired,
ReceiveSecretRequest,
Expand Down Expand Up @@ -73,17 +73,17 @@ def on_message(self, raiden: RaidenService, message: Message) -> None:
@staticmethod
def handle_message_secretrequest(raiden: RaidenService, message: SecretRequest) -> None:
secret_request = ReceiveSecretRequest(
message.payment_identifier,
message.amount,
message.expiration,
message.secrethash,
message.sender,
payment_identifier=message.payment_identifier,
amount=message.amount,
expiration=message.expiration,
secrethash=message.secrethash,
sender=message.sender,
)
raiden.handle_and_track_state_change(secret_request)

@staticmethod
def handle_message_revealsecret(raiden: RaidenService, message: RevealSecret) -> None:
state_change = ReceiveSecretReveal(message.secret, message.sender)
state_change = ReceiveSecretReveal(secret=message.secret, sender=message.sender)
raiden.handle_and_track_state_change(state_change)

@staticmethod
Expand All @@ -93,13 +93,15 @@ def handle_message_unlock(raiden: RaidenService, message: Unlock) -> None:
message_identifier=message.message_identifier,
secret=message.secret,
balance_proof=balance_proof,
sender=balance_proof.sender,
)
raiden.handle_and_track_state_change(state_change)

@staticmethod
def handle_message_lockexpired(raiden: RaidenService, message: LockExpired) -> None:
balance_proof = balanceproof_from_envelope(message)
state_change = ReceiveLockExpired(
sender=balance_proof.sender,
balance_proof=balance_proof,
secrethash=message.secrethash,
message_identifier=message.message_identifier,
Expand Down Expand Up @@ -140,10 +142,19 @@ def handle_message_refundtransfer(raiden: RaidenService, message: RefundTransfer

secret = random_secret()
state_change = ReceiveTransferRefundCancelRoute(
routes=routes, transfer=from_transfer, secret=secret
routes=routes,
transfer=from_transfer,
balance_proof=from_transfer.balance_proof,
sender=from_transfer.balance_proof.sender, # pylint: disable=no-member
secret=secret,
)
else:
state_change = ReceiveTransferRefund(transfer=from_transfer, routes=routes)
state_change = ReceiveTransferRefund(
transfer=from_transfer,
balance_proof=from_transfer.balance_proof,
sender=from_transfer.balance_proof.sender, # pylint: disable=no-member
routes=routes,
)

raiden.handle_and_track_state_change(state_change)

Expand Down
Loading