Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Schema Registry: Change to use BinaryContent and change Encoder to Serializer #27598

Closed
wants to merge 16 commits into from
Closed
2 changes: 1 addition & 1 deletion eng/versioning/version_client.txt
Original file line number Diff line number Diff line change
Expand Up @@ -354,8 +354,8 @@ unreleased_com.azure:azure-aot-graalvm-support;1.0.0-beta.1
unreleased_com.azure:azure-aot-graalvm-support-netty;1.0.0-beta.1
unreleased_com.azure:azure-aot-graalvm-perf;1.0.0-beta.1


unreleased_com.azure:azure-core-amqp;2.5.0-beta.1
unreleased_com.azure:azure-core-experimental;1.0.0-beta.26

# Released Beta dependencies: Copy the entry from above, prepend "beta_", remove the current
# version and set the version to the released beta. Released beta dependencies are only valid
Expand Down
2 changes: 2 additions & 0 deletions sdk/core/azure-core-experimental/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,8 @@

### Breaking Changes

- Changed `com.azure.core.experimental.models.MessageWithMetadata` to `BinaryContent`.

### Bugs Fixed

### Other Changes
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@
* An abstraction for a message containing a content type along with its data.
*/
@Fluent
public class MessageWithMetadata {
public class BinaryContent {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We would now have BinaryData and BinaryContent as public types and another implementation type i.e. BinaryDataContent. This is likely going to confuse us in future :)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, this is going to be confusing. BinaryData and BinaryContent are very similar.

Copy link
Member Author

@conniey conniey Mar 11, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was based on architect feedback. The other languages also have moved their names to this and also have BinaryData. I can add you to the thread if you want.

https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/core/Azure.Core.Experimental/api/Azure.Core.Experimental.netstandard2.0.cs#L9

Copy link
Member

@JoshLove-msft JoshLove-msft Mar 11, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The idea is that BinaryContent contains BinaryData along with the content type. We didn't think it would be confusing as they are two related and similarly named types, rather than being just two similarly named types.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/cc @tg-msft

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pallavit, @srnagar, could you please elaborate what's confusing here and what concrete implications of this confusion you see? There are many types in all platforms that are similar, yet they are not considered confusing, e.g. Int32 and Int64 are super similar.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I set-up a quick chat on Monday. Might be easier to get all the thoughts in the same room.

Copy link
Member

@srnagar srnagar Mar 12, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Java already has BinaryDataContent which is wrapped by BinaryData but that's currently internal. Also, when deciding which type to use, would there be a scenario where we want to use BinaryData and not BinaryContent given that content type in BinaryContent is optional?

One option was to move content type property to BinaryData but this may not be possible as we want to make this the super type for Event Hubs and Service Bus event and message types. If this type is intended for extension in Event Hubs and Service Bus, should we make this type abstract?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It was initially abstract, but we made it concrete to make it easier for use with Schema Registry that didn't involve Service Bus/Event Hubs. However, I think we can still make this scenario easy by having a private implementation that we return if the user doesn't specify a more specific BinaryContent type. This would allow us to move the type back to being abstract which I believe would address your concerns around any confusion that might arise from how this type is meant to be used.

Copy link
Member

@JoshLove-msft JoshLove-msft Mar 12, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here is what this change would look like in .NET - Azure/azure-sdk-for-net#27525

After making the class abstract, I needed to add two additional generic overloads that only took the TData instead of both the TEnvelope and TData. This was necessary in order to leave in our generic type constraint that TEnvelope have an empty public constructor. When using the new overloads we would construct the internal DefaultBinaryContent and return that to the user. The new overloads make the sample code simpler since now the generic arguments can be completely inferred based on the input data.

private BinaryData binaryData;
private String contentType;

Expand All @@ -28,9 +28,9 @@ public BinaryData getBodyAsBinaryData() {
*
* @param binaryData The message body.
*
* @return The updated {@link MessageWithMetadata} object.
* @return The updated {@link BinaryContent} object.
*/
public MessageWithMetadata setBodyAsBinaryData(BinaryData binaryData) {
public BinaryContent setBodyAsBinaryData(BinaryData binaryData) {
this.binaryData = binaryData;
return this;
}
Expand All @@ -49,9 +49,9 @@ public String getContentType() {
*
* @param contentType The content type.
*
* @return The updated {@link MessageWithMetadata} object.
* @return The updated {@link BinaryContent} object.
*/
public MessageWithMetadata setContentType(String contentType) {
public BinaryContent setContentType(String contentType) {
this.contentType = contentType;
return this;
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,16 +9,16 @@
import static org.junit.jupiter.api.Assertions.assertNull;

/**
* Tests for {@link MessageWithMetadata}
* Tests for {@link BinaryContent}
*/
public class MessageWithMetadataTest {
public class BinaryContentTest {
/**
* Verify default parameters.
*/
@Test
public void initialize() {
// Act
final MessageWithMetadata message = new MessageWithMetadata();
final BinaryContent message = new BinaryContent();

// Assert
assertNull(message.getBodyAsBinaryData(), "'body' should initially be null.");
Expand All @@ -33,10 +33,10 @@ public void settingProperties() {
// Arrange
final BinaryData binaryData = BinaryData.fromString("foo.bar.baz");
final String contentType = "some-content";
final MessageWithMetadata message = new MessageWithMetadata();
final BinaryContent message = new BinaryContent();

// Act
final MessageWithMetadata actual = message.setContentType(contentType)
final BinaryContent actual = message.setContentType(contentType)
.setBodyAsBinaryData(binaryData);

// Assert
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,11 @@

### Breaking Changes

- Changed `SchemaRegistryApacheAvroEncoder` to deserialize `BinaryContent`.
- Changed `SchemaRegistryApacheAvroEncoder` to `SchemaRegistryApacheAvroSerializer`.
- Changed `decodeMessageData` and `decodeMessageDataAsync` to `deserializeMessageData` and `deserializeMessageDataAsync`.
- Changed `encodeMessageData` and `encodeMessageDataAsync` to `serializeMessageData` and `serializeMessageDataAsync`.

### Bugs Fixed

### Other Changes
Expand All @@ -14,7 +19,7 @@

### Features Added

- Changed `SchemaRegistryApacheAvroEncoder` to deserialize `MessageWithMetadata` rather than tied to a binary format
- Changed `SchemaRegistryApacheAvroEncoder` to deserialize `MessageWithMetadata` rather than tied to a binary format
with preamble. Backwards compatibility with preamble format supported for this release. See issue #26449.

### Breaking Changes
Expand Down
20 changes: 10 additions & 10 deletions sdk/schemaregistry/azure-data-schemaregistry-apacheavro/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,11 +72,11 @@ SchemaRegistryAsyncClient schemaRegistryAsyncClient = new SchemaRegistryClientBu

#### Create `SchemaRegistryAvroSerializer` through the builder

```java readme-sample-createSchemaRegistryAvroEncoder
SchemaRegistryApacheAvroEncoder encoder = new SchemaRegistryApacheAvroEncoderBuilder()
```java readme-sample-createSchemaRegistryAvroSerializer
SchemaRegistryApacheAvroSerializer serializer = new SchemaRegistryApacheAvroSerializerBuilder()
.schemaRegistryAsyncClient(schemaRegistryAsyncClient)
.schemaGroup("{schema-group}")
.buildEncoder();
.buildSerializer();
```

## Key concepts
Expand Down Expand Up @@ -105,14 +105,14 @@ The serializer in this library creates messages in a wire format. The format is
### Serialize
Serialize a strongly-typed object into Schema Registry-compatible avro payload.

```java readme-sample-encodeSample
```java readme-sample-serializeSample
PlayingCard playingCard = new PlayingCard();
playingCard.setPlayingCardSuit(PlayingCardSuit.SPADES);
playingCard.setIsFaceCard(false);
playingCard.setCardValue(5);

MessageWithMetadata message = encoder.encodeMessageData(playingCard,
TypeReference.createInstance(MessageWithMetadata.class));
BinaryContent message = serializer.serializeMessageData(playingCard,
TypeReference.createInstance(BinaryContent.class));
```

The avro type `PlayingCard` is available in samples package
Expand All @@ -121,10 +121,10 @@ The avro type `PlayingCard` is available in samples package
### Deserialize
Deserialize a Schema Registry-compatible avro payload into a strongly-type object.

```java readme-sample-decodeSample
SchemaRegistryApacheAvroEncoder encoder = createAvroSchemaRegistryEncoder();
MessageWithMetadata message = getSchemaRegistryAvroMessage();
PlayingCard playingCard = encoder.decodeMessageData(message, TypeReference.createInstance(PlayingCard.class));
```java readme-sample-deserializeSample
SchemaRegistryApacheAvroSerializer serializer = createAvroSchemaRegistrySerializer();
BinaryContent message = getSchemaRegistryAvroMessage();
PlayingCard playingCard = serializer.deserializeMessageData(message, TypeReference.createInstance(PlayingCard.class));
```

## Troubleshooting
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -44,10 +44,15 @@
</properties>

<dependencies>
<dependency>
<groupId>com.azure</groupId>
<artifactId>azure-core</artifactId>
<version>1.26.0</version> <!-- {x-version-update;com.azure:azure-core;dependency} -->
</dependency>
<dependency>
<groupId>com.azure</groupId>
<artifactId>azure-core-experimental</artifactId>
<version>1.0.0-beta.25</version> <!-- {x-version-update;com.azure:azure-core-experimental;dependency} -->
<version>1.0.0-beta.26</version> <!-- {x-version-update;unreleased_com.azure:azure-core-experimental;dependency} -->
</dependency>
<dependency>
<groupId>com.azure</groupId>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@

package com.azure.data.schemaregistry.apacheavro;

import com.azure.core.experimental.models.MessageWithMetadata;
import com.azure.core.experimental.models.BinaryContent;
import com.azure.core.util.BinaryData;
import com.azure.core.util.CoreUtils;
import com.azure.core.util.logging.ClientLogger;
Expand All @@ -28,13 +28,13 @@
/**
* Schema Registry-based serializer implementation for Avro data format using Apache Avro.
*/
public final class SchemaRegistryApacheAvroEncoder {
public final class SchemaRegistryApacheAvroSerializer {
static final String AVRO_MIME_TYPE = "avro/binary";
static final byte[] RECORD_FORMAT_INDICATOR = new byte[]{0x00, 0x00, 0x00, 0x00};
static final int RECORD_FORMAT_INDICATOR_SIZE = RECORD_FORMAT_INDICATOR.length;
static final int SCHEMA_ID_SIZE = 32;

private final ClientLogger logger = new ClientLogger(SchemaRegistryApacheAvroEncoder.class);
private final ClientLogger logger = new ClientLogger(SchemaRegistryApacheAvroSerializer.class);
private final SchemaRegistryAsyncClient schemaRegistryClient;
private final AvroSerializer avroSerializer;
private final SerializerOptions serializerOptions;
Expand All @@ -46,7 +46,7 @@ public final class SchemaRegistryApacheAvroEncoder {
* @param avroSerializer Serializer implemented using Apache Avro.
* @param serializerOptions Options to configure the serializer with.
*/
SchemaRegistryApacheAvroEncoder(SchemaRegistryAsyncClient schemaRegistryClient,
SchemaRegistryApacheAvroSerializer(SchemaRegistryAsyncClient schemaRegistryClient,
AvroSerializer avroSerializer, SerializerOptions serializerOptions) {
this.schemaRegistryClient = Objects.requireNonNull(schemaRegistryClient,
"'schemaRegistryClient' cannot be null.");
Expand All @@ -60,7 +60,7 @@ public final class SchemaRegistryApacheAvroEncoder {
*
* @param object Object to encode.
* @param typeReference Type of message to create.
* @param <T> Concrete type of {@link MessageWithMetadata}.
* @param <T> Concrete type of {@link BinaryContent}.
*
* @return The message encoded or {@code null} if the message could not be encoded.
*
Expand All @@ -70,8 +70,8 @@ public final class SchemaRegistryApacheAvroEncoder {
* encoding the object.
* @throws NullPointerException if the {@code object} is null or {@code typeReference} is null.
*/
public <T extends MessageWithMetadata> T encodeMessageData(Object object, TypeReference<T> typeReference) {
return encodeMessageDataAsync(object, typeReference).block();
public <T extends BinaryContent> T serializeMessageData(Object object, TypeReference<T> typeReference) {
return serializeMessageDataAsync(object, typeReference).block();
}

/**
Expand All @@ -80,7 +80,7 @@ public <T extends MessageWithMetadata> T encodeMessageData(Object object, TypeRe
* @param object Object to encode.
* @param typeReference Type of message to create.
* @param messageFactory Factory to create an instance given the serialized Avro.
* @param <T> Concrete type of {@link MessageWithMetadata}.
* @param <T> Concrete type of {@link BinaryContent}.
*
* @return The message encoded or {@code null} if the message could not be encoded.
*
Expand All @@ -90,17 +90,17 @@ public <T extends MessageWithMetadata> T encodeMessageData(Object object, TypeRe
* encoding the object.
* @throws NullPointerException if the {@code object} is null or {@code typeReference} is null.
*/
public <T extends MessageWithMetadata> T encodeMessageData(Object object, TypeReference<T> typeReference,
public <T extends BinaryContent> T serializeMessageData(Object object, TypeReference<T> typeReference,
Function<BinaryData, T> messageFactory) {
return encodeMessageDataAsync(object, typeReference, messageFactory).block();
return serializeMessageDataAsync(object, typeReference, messageFactory).block();
}

/**
* Encodes an object into a message.
*
* @param object Object to encode.
* @param typeReference Type of message to create.
* @param <T> Concrete type of {@link MessageWithMetadata}.
* @param <T> Concrete type of {@link BinaryContent}.
*
* @return A Mono that completes with the encoded message.
*
Expand All @@ -110,10 +110,10 @@ public <T extends MessageWithMetadata> T encodeMessageData(Object object, TypeRe
* encoding the object.
* @throws NullPointerException if the {@code object} is null or {@code typeReference} is null.
*/
public <T extends MessageWithMetadata> Mono<T> encodeMessageDataAsync(Object object,
public <T extends BinaryContent> Mono<T> serializeMessageDataAsync(Object object,
TypeReference<T> typeReference) {

return encodeMessageDataAsync(object, typeReference, null);
return serializeMessageDataAsync(object, typeReference, null);
}

/**
Expand All @@ -123,7 +123,7 @@ public <T extends MessageWithMetadata> Mono<T> encodeMessageDataAsync(Object obj
* @param typeReference Type of message to create.
* @param messageFactory Factory to create an instance given the serialized Avro. If null is passed in, then the
* no argument constructor will be used.
* @param <T> Concrete type of {@link MessageWithMetadata}.
* @param <T> Concrete type of {@link BinaryContent}.
*
* @return A Mono that completes with the encoded message.
*
Expand All @@ -133,7 +133,7 @@ public <T extends MessageWithMetadata> Mono<T> encodeMessageDataAsync(Object obj
* encoding the object.
* @throws NullPointerException if the {@code object} is null or {@code typeReference} is null.
*/
public <T extends MessageWithMetadata> Mono<T> encodeMessageDataAsync(Object object,
public <T extends BinaryContent> Mono<T> serializeMessageDataAsync(Object object,
TypeReference<T> typeReference, Function<BinaryData, T> messageFactory) {

if (object == null) {
Expand Down Expand Up @@ -193,29 +193,29 @@ public <T extends MessageWithMetadata> Mono<T> encodeMessageDataAsync(Object obj
*
* @param message Object to encode.
* @param typeReference Message to encode to.
* @param <T> Concrete type of {@link MessageWithMetadata}.
* @param <T> Concrete type of {@link BinaryContent}.
*
* @return The message encoded.
*
* @throws NullPointerException if {@code message} or {@code typeReference} is null.
*/
public <T> T decodeMessageData(MessageWithMetadata message, TypeReference<T> typeReference) {
return decodeMessageDataAsync(message, typeReference).block();
public <T> T deserializeMessageData(BinaryContent message, TypeReference<T> typeReference) {
return deserializeMessageDataAsync(message, typeReference).block();
}

/**
* Decodes a message into its object.
*
* @param message Object to encode.
* @param typeReference Message to encode to.
* @param <T> Concrete type of {@link MessageWithMetadata}.
* @param <T> Concrete type of {@link BinaryContent}.
*
* @return A Mono that completes when the message encoded. If {@code message.getBodyAsBinaryData()} is null or
* empty, then an empty Mono is returned.
*
* @throws NullPointerException if {@code message} or {@code typeReference} is null.
*/
public <T> Mono<T> decodeMessageDataAsync(MessageWithMetadata message, TypeReference<T> typeReference) {
public <T> Mono<T> deserializeMessageDataAsync(BinaryContent message, TypeReference<T> typeReference) {
if (message == null) {
return monoError(logger, new NullPointerException("'message' cannot be null."));
} else if (typeReference == null) {
Expand Down Expand Up @@ -282,10 +282,10 @@ public <T> Mono<T> decodeMessageDataAsync(MessageWithMetadata message, TypeRefer
contents.reset();
}

return decodeMessageDataAsync(schemaId, contents, typeReference);
return deserializeMessageDataAsync(schemaId, contents, typeReference);
}

private <T> Mono<T> decodeMessageDataAsync(String schemaId, ByteBuffer buffer, TypeReference<T> typeReference) {
private <T> Mono<T> deserializeMessageDataAsync(String schemaId, ByteBuffer buffer, TypeReference<T> typeReference) {
return this.schemaRegistryClient.getSchema(schemaId)
.handle((registryObject, sink) -> {
final byte[] payloadSchema = registryObject.getDefinition().getBytes(StandardCharsets.UTF_8);
Expand All @@ -305,7 +305,7 @@ private <T> Mono<T> decodeMessageDataAsync(String schemaId, ByteBuffer buffer, T
* @throws RuntimeException if an instance of {@code T} could not be instantiated.
*/
@SuppressWarnings("unchecked")
private static <T extends MessageWithMetadata> T createNoArgumentInstance(TypeReference<T> typeReference) {
private static <T extends BinaryContent> T createNoArgumentInstance(TypeReference<T> typeReference) {

final Optional<Constructor<?>> constructor =
Arrays.stream(typeReference.getJavaClass().getDeclaredConstructors())
Expand Down
Loading