How to dynamically write avro data with a schema that has unions defined

[rchapin]

I found this example on how to write avro data to via the ocf writer.

This works when the schema is strictly defined for only one type per field AND the data that is being written out is known at compile time.

I have a program where I am reading JSON data from a REST API and then writing it to an avro schema in which every field is defined as a union of ["null", <type>] with a default value of null to ensure forwards and backwards compatibility.

I tried modifying the example above such that the struct was defined with interface{} as a type, but that meant that I needed to cast the values defined in the struct passed in to the writer.

I also tried defining a map[string]interface{} as follows, but had the same issue; I needed to cast the values to be able to encode them as avro.

	record := map[string]interface{}{
		"a": int64(10),
		"b": "hello",
	}

Any ideas how I could use this library to pass in a JSON object that is a map[string]interface{} along with the avro schema and then dynamically build either a struct or map that I could pass to the ocfwriter?

Thanks,
-- Ryan

[nrwiersma]

Hi,

Do you know the exact format of json/avro, or are you trying to define the schema for it all on the fly?

Perhaps a small code example would be helpful.

[rchapin]

That is actually the problem; I will not know the exact structure of the JSON or the avro schema at compile time.

I think that I answered my own question by writing up a sample program.

It looks like what I'm going to need to do is walk the schema and then check the types of the data/fields to which I'm going to write and then dynamically build map[string]interface{} instances for each "JSON" record that I want to encode.

Any additional thoughts would be welcome. Thanks!

package main

import (
	"fmt"
	"log"
	"os"

	"github.com/hamba/avro/ocf"
)

var schema string = `
{
	"type": "record",
	"name": "simple",
	"namespace": "org.hamba.avro",
	"fields" : [
	{"name": "a", "type": ["null", "long"], "default": null},
	{"name": "b", "type": ["null", "string"], "default": null}
	]
}`

type JsonRecord map[string]interface{}

// This is how I will be getting data from the REST API.
var jsonFromApi = []JsonRecord{
	{
		"a": map[string]interface{}{"long": int64(1234567890)},
		"b": "works",
	},
	{
		"a": nil,
		"b": "works",
	},
	{
		"a": 987654321,
		"b": "nope",
	},
	{
		"a": int64(647),
		"b": "nope",
	},
}

func main() {

	f, err := os.Create("/var/tmp/hamba.avro")
	if err != nil {
		log.Fatal(err)
	}
	defer f.Close()

	enc, err := ocf.NewEncoder(schema, f)
	if err != nil {
		log.Fatal(err)
	}

	for _, r := range jsonFromApi {
		/*
			Since this is a data driven application, and I will not know the schema or the structure
			of the JSON data that I am getting at compile time, I cannot use a struct and have to
			use a map.
		*/
		fmt.Printf("encoding r=[%v]\n", r)
		err = enc.Encode(r)
		if err != nil {
			fmt.Printf("cannot encode r=[%v], err=[%v]\n", r, err)
		} else {
			fmt.Println("success!")
		}
	}

	if err := enc.Flush(); err != nil {
		log.Fatal(err)
	}

	if err := f.Sync(); err != nil {
		log.Fatal(err)
	}
}

[nrwiersma]

Yeah. Keep in mind that unions on maps are different. They are themselves a map[string]interface{} with a single key of the type name with the value attached. This is described in the README.

[rchapin]

Yep, understood. Thanks for your input :)