Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add copy_from method for field assignment #215

Merged
merged 3 commits into from
Mar 16, 2021
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
66 changes: 66 additions & 0 deletions docs/messages.rst
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,72 @@ Instantiate messages using either keyword arguments or a :class:`dict`
>>> song.title
'Canon in D'


Assigning to Fields
-------------------

One of the goals of proto-plus is to make protobufs feel as much like regular python
objects as possible. It is possible to update a message's field by assigning to it,
just as if it were a regular python object.

.. code-block:: python
song = Song()
song.composer = Composer(given_name="Johann", family_name="Bach")

# Can also assign from a dictionary as a convenience.
song.composer = {"given_name": "Claude", "family_name": "Debussy"}

# Repeated fields can also be assigned
class Album(proto.Message):
songs = proto.RepeatedField(Song, number=1)

a = Album()
songs = [Song(title="Canon in D"), Song(title="Little Fugue")]
a.songs = songs

.. note::

Assigning to a proto-plus message field works by making copies, not by updating references.
This is necessary because of memory layout requirements of protocol buffers.
These memory constraints are maintained by the protocol buffers runtime.
This behavior can be surprising under certain circumstances, e.g. trying to save
an alias to a nested field.

:class:`proto.Message` defines a helper message, :meth:`~.Message.copy_from` to
help make the distinction clear when reading code.
The semantics of :meth:`~.Message.copy_from` are addentical to regular assignment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

addentical should be identical

And perhaps ... identical to the field assignment behavior described above. would help clarify, as "regular assignment" could imply the behavior most would expect outside the context of protobufs.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doh! Good catch. Yes, the proposed wording is more clear.


.. code-block:: python

composer = Composer(given_name="Johann", family_name="Bach")
song = Song(title="Tocatta and Fugue in D Minor", composer=composer)
composer.given_name = "Wilhelm"

# 'composer' is NOT a reference to song.composer
assert song.composer.given_name == "Johann"

# We CAN update the song's composer by assignment.
song.composer = composer
composer.given_name = "Carl"

# 'composer' is STILL not a referene to song.composer.
assert song.composer.given_name == "Wilhelm"

# It does work in reverse, though,
# if we want a reference we can access then update.
composer = song.composer
composer.given_name = "Gottfried"

assert song.composer.given_name == "Gottfried"

# We can use 'copy_from' if we're concerned that the code
# implies that assignment involves references.
composer = Composer(given_name="Elisabeth", family_name="Bach")
Composer.copy_from(song.composer, composer)

assert song.composer.given_name == "Elisabeth"


Enums
-----

Expand Down
31 changes: 30 additions & 1 deletion proto/message.py
Original file line number Diff line number Diff line change
Expand Up @@ -436,7 +436,7 @@ def __init__(self, mapping=None, *, ignore_unknown_fields=False, **kwargs):
#
# The `wrap` method on the metaclass is the public API for taking
# ownership of the passed in protobuf objet.
mapping = copy.copy(mapping)
mapping = copy.deepcopy(mapping)
if kwargs:
mapping.MergeFrom(self._meta.pb(**kwargs))

Expand Down Expand Up @@ -602,6 +602,35 @@ def __setattr__(self, key, value):
if pb_value is not None:
self._pb.MergeFrom(self._meta.pb(**{key: pb_value}))

def copy_from(self, other):
"""Equivalent for protobuf.Message.CopyFrom

Args:
other: (Union[dict, ~.Message):
A dictionary or message to reinitialize the values for this message.
"""
if isinstance(other, type(self)):
# Just want the underlying proto.
other = Message.pb(other)

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is equivalent to doing other = other._pb right?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it is equivalent. The code (and documentation) tries to use the accessor instead of touching the attribute directly.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍🏻

elif isinstance(other, type(Message.pb(self))):

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why does it pass in this scenario?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because there's no transformation that needs to occur. It's just here to catch the input type and make sure that it doesn't generate the type mismatch exception below.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍🏻

# Don't need to do anything.
pass
elif isinstance(other, collections.abc.Mapping):

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this is in case of a repeated field?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's to catch the dictionary syntax:

Message.copy_from(message.submessage, {"name": "Lawrence", "armor_class": 15})

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh good to know that's possible!

# Coerce into a proto
other = type(Message.pb(self))(**other)
else:
raise TypeError(
"invalid argument type to copy to {}: {}".format(
self.__class__.__name__, other.__class__.__name__
)
)

# Note: we can't just run self.__init__ because this may be a message field
# for a higher order proto; the memory layout for protos is NOT LIKE the
# python memory model. We cannot rely on just setting things by reference.
# Non-trivial complexity is (partially) hidden by the protobuf runtime.
self._pb.CopyFrom(other)


class _MessageInfo:
"""Metadata about a message.
Expand Down
24 changes: 24 additions & 0 deletions tests/test_message.py
Original file line number Diff line number Diff line change
Expand Up @@ -317,3 +317,27 @@ class Squid(proto.Message):

s = Squid({"mass_kg": 20, "length_cm": 100}, ignore_unknown_fields=True)
assert not hasattr(s, "length_cm")


def test_copy_from():
class Mollusc(proto.Message):
class Squid(proto.Message):
mass_kg = proto.Field(proto.INT32, number=1)

squid = proto.Field(Squid, number=1)

m = Mollusc()
s = Mollusc.Squid(mass_kg=20)
Mollusc.Squid.copy_from(m.squid, s)

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The reason we can't have s.squid.copy_from is because of potential collisions with the proto definition, right?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct. The general problem is that we have to be very, very careful about adding fields and methods to the message class types because of potential name collisions with fields, and the general solution is to add the method (or field) to the metaclass. This means that method invocation becomes a little more indirect: we invoke it from the class and pass in the instance as the first parameter.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍🏻

assert m.squid is not s
assert m.squid == s

s.mass_kg = 30
Mollusc.Squid.copy_from(m.squid, Mollusc.Squid.pb(s))
assert m.squid == s

Mollusc.Squid.copy_from(m.squid, {"mass_kg": 10})
assert m.squid.mass_kg == 10

with pytest.raises(TypeError):
Mollusc.Squid.copy_from(m.squid, (("mass_kg", 20)))