AvroGen: Generate Encoder/Decoder

[Khalid-Nowaf]

Including an encoder/decoder function for each generated schema would be a great idea, where the functions will have all the information to operate, such as the schema. Therefore the end user will use the function with no extra parameters.

[nrwiersma]

This is not actually true. As you would parse the schema in as an .avsc, the tool has no way of knowing where the schema would reside in code. Further, in complex multi file schemas, this would break down really quickly IMO.

[Khalid-Nowaf]

I was thinking to embed the schema as a string in the code, and generate the decoder/encoder for the root record in the schema.

Here is an example I'm working to implement in my code:

Giving the following schema

{
  "name": "some.org.device.Device",
  "type": "record",
  "fields": [
    {
      "name": "RecordA",
      "type": [
        {
          "name": "some.org.device.InfoType",
          "type": "record",
          "fields": [
            {
              "name": "name",
              "type": "string"
            },
            {
              "name": "id",
              "type": "long"
            }
          ]
        },
        {
          "name": "some.org.device.LocType",
          "type": "record",
          "fields": [
            {
              "name": "latitude",
              "type": "long"
            },
            {
              "name": "longitude",
              "type": "long"
            },
            {
              "name": "altitude",
              "type": "long"
            }
          ]
        },
        {
          "name": "some.org.device.MetaType",
          "type": "record",
          "fields": [
            {
              "name": "priority",
              "type": "int"
            },
            {
              "name": "weight",
              "type": "int"
            }
          ]
        }
      ]
    },
    {
      "name": "RecordB",
      "type": [
        "some.org.device.InfoType",
        "some.org.device.LocType",
        "some.org.device.MetaType"
      ]
    },
    {
      "name": "RecordC",
      "type": [
        null,
        "some.org.device.InfoType",
        "some.org.device.LocType",
        "some.org.device.MetaType"
      ]
    }
  ]
}

The generated file should be something like this

package schema

import (
	"bytes"

	"github.com/hamba/avro/v2"
)

type InfoType struct {
	Name string `avro:"name"`
	ID   int64  `avro:"id"`
}

type LocType struct {
	Latitude  int64 `avro:"latitude"`
	Longitude int64 `avro:"longitude"`
	Altitude  int64 `avro:"altitude"`
}

type MetaType struct {
	Priority int `avro:"priority"`
	Weight   int `avro:"weight"`
}

type Device struct {
	RecordA any `avro:"RecordA"`
	RecordB any `avro:"RecordB"`
	RecordC any `avro:"RecordC"`
}

func DeviceDecoder() func([]byte) (*Device, error) {
	schema := `{"name":"some.org.device.Device","type":"record","fields":[{"name":"RecordA","type":[{"name":"some.org.device.InfoType","type":"record","fields":[{"name":"name","type":"string"},{"name":"id","type":"long"}]},{"name":"some.org.device.LocType","type":"record","fields":[{"name":"latitude","type":"long"},{"name":"longitude","type":"long"},{"name":"altitude","type":"long"}]},{"name":"some.org.device.MetaType","type":"record","fields":[{"name":"priority","type":"int"},{"name":"weight","type":"int"}]}]},{"name":"RecordB","type":["some.org.device.InfoType","some.org.device.LocType","some.org.device.MetaType"]},{"name":"RecordC","type":[null,"some.org.device.InfoType","some.org.device.LocType","some.org.device.MetaType"]}]}`
	r := bytes.NewReader([]byte{})

	decoder, err := avro.NewDecoder(schema, r)
	if err != nil {
		panic(err)
	}

	return func(encoded []byte) (*Device, error) {
		decoded := &Device{}
		r.Reset(encoded)
		if err := decoder.Decode(decoded); err != nil {
			return nil, err
		}
		return decoded, nil

	}
}

func DeviceEncoder() func(*Device) ([]byte, error) {
	schema := `{"name":"some.org.device.Device","type":"record","fields":[{"name":"RecordA","type":[{"name":"some.org.device.InfoType","type":"record","fields":[{"name":"name","type":"string"},{"name":"id","type":"long"}]},{"name":"some.org.device.LocType","type":"record","fields":[{"name":"latitude","type":"long"},{"name":"longitude","type":"long"},{"name":"altitude","type":"long"}]},{"name":"some.org.device.MetaType","type":"record","fields":[{"name":"priority","type":"int"},{"name":"weight","type":"int"}]}]},{"name":"RecordB","type":["some.org.device.InfoType","some.org.device.LocType","some.org.device.MetaType"]},{"name":"RecordC","type":[null,"some.org.device.InfoType","some.org.device.LocType","some.org.device.MetaType"]}]}`
	w := bytes.NewBuffer([]byte{})

	encoder, err := avro.NewEncoder(schema, w)
	if err != nil {
		panic(err)
	}

	return func(decoded *Device) ([]byte, error) {
		w.Reset()
		if err = encoder.Encode(decoded); err != nil {
			return nil, err
		}
		return w.Bytes(), nil
	}

}

And the usage would be like this:

device := schema.Device{
		RecordA: &schema.InfoType{Name: "a", ID: 123},
		RecordB: &schema.LocType{Latitude: 123, Altitude: 1, Longitude: 321},
		RecordC: nil,
	}
	decode := schema.DeviceDecoder()
	encode := schema.DeviceEncoder()

	encodedValue, _ = encode(&device)
	decodedValue, _ = decode(encodedValue)

[nrwiersma]

Can you explain how this idea works with a multi-file schema?

[Khalid-Nowaf]

I do not have a muti-file schema for my current project, so I'm not entirely familiar with the edge cases that come with multi-file schema.

I'm thinking out loud here; each Record type could have its own embedded schema as string and encoder/decoder function. I know that's easier said than done. (I would go back to Java Avro-tool as a reference and sees how they are dealing with such case)

@nrwiersma What do you think about this? Do you believe this package internals can accomplish such a thing, or are there some design limitations?

[nrwiersma]

The multi-file schema is an issue, and it is an actively supported case of the generator. Will have to give it some thought.

[nrwiersma]

Having looked at this, it is not possible with the current generator design.

[nrwiersma]

There is a work around I am trying out for this, which is to add encoders to everything. It will for sure bloat the application with large structures, but I dont see any other way: #292.

[Khalid-Nowaf]

This looks promising and great!

usually, any code generator will bloat the codebase :).

Thanks, @nrwiersma for the hard work.