TypeError: 'URL' is not JSON serialisable

[darkfishy]

This happens when an object of type yarl.URL is in the value of any given key of any dictionary object.

For instance:

>>> import json
>>> import yarl
>>> error = {'response': {'code': 400, 'status': 'BAD_REQUEST', 'reason': 'something is not right'}, 'request': yarl.URL('https://example.com/some/api/endpoint')}
>>> error
{'response': 'something', 'json': {'code': 400, 'status': 'BAD_REQUEST'}, 'request': URL('https://fm-data.herokuapp.com/api/marketplaces/436/currentHolding')}
>>> json.dumps(error)
Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "/Users/dev/.pyenv/versions/3.6.8/lib/python3.6/json/__init__.py", line 231, in dumps
    return _default_encoder.encode(obj)
  File "/Users/dev/.pyenv/versions/3.6.8/lib/python3.6/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/Users/dev/.pyenv/versions/3.6.8/lib/python3.6/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/Users/dev/.pyenv/versions/3.6.8/lib/python3.6/json/encoder.py", line 180, in default
    o.__class__.__name__)
TypeError: Object of type 'URL' is not JSON serializable

Would this be an error with yarl.URL class or json library? It seems like either __str__() or __repr__() doesn't return a string?

[aio-libs-bot]

GitMate.io thinks possibly related issues are #24 (TypeError in URL.with_query), #272 (bool(URL()) should be False), #33 (Parse URL parameter), #84 (Incorrect handling of '..' in url path), and #280 (Allow scheme replacement for relative urls).

[bsolomon1124]

@darkfishy For what it's worth, this has nothing to do with yarl.URL implementing __str__() or __repr__(), and it doesn't matter if those methods return a str either. (Though they both are implemented and return str as they should for yarl.URL.)

Consider a similar example with a datetime object, which does define __str__ and __repr__ but throws the same TypeError:

>>> import json
>>> import datetime
>>> now = datetime.datetime.now()
>>> json.dumps({"timestamp": now})
# ...
TypeError: Object of type datetime is not JSON serializable
>>> hasattr(now, "__str__")
True
>>> hasattr(now, "__repr__")
True

The other difficulty in this case is that, though its common to serialize using the instance's __dict__ attribute, yarl.URL uses __slots__ and so has no __dict__ attribute.

Suggestion: use either u.human_repr() (this is what __str__ return in the first place) or build a custom encoder + decoder. Example of the latter:

>>> from yarl import URL
>>> u = URL("https://fm-data.herokuapp.com/api/marketplaces/436/currentHolding")
>>> import json
>>> json.dumps({"url": u}, default=lambda x: x.human_repr())
'{"url": "https://fm-data.herokuapp.com/api/marketplaces/436/currentHolding"}'

[bsolomon1124]

@asvetlov this issue seems like it is not a bug and should probably be closed; it's common for custom object instances to not be serializable, and definitely applies in this case because URL does not inherit from str.

[asvetlov]

Agree