Specify Vector version in the Vector config file

[binarylogic]

After talking with @a-rodin he had the following idea:

My thoughts about release process were mostly questioning the need to add version to the config file, similarly to how docker-compose files have version in the beginning. I see both benefits and downsides of this, but haven’t decided yet would it be better to have it or not.

He also mentioned:

I agree that it is possible to make breaking changes between 0.y.z releases according to semver. However, it could be nice to use configs from older versions in newer ones without manual changes, which can be done by running a special command which would take config from an older version and generate a config for the current version.

Something like running vector config upgrade which would bump the version in config and make necessary fixes, for example renaming config options or transforms names.

I don't think we should address this during the pre-1.0 phase of development, but I wanted to open this issue for discussion.

[LucioFranco]

I think the one issue that arises with this is our use of serde and defining the config via set of structs. I'm not sure this method supports the ability to go backwards with previous versions. We may want to consider more custom config loading methods using an external schema like vector.spec.toml

[ghost]

I'm not sure this method supports the ability to go backwards with previous versions.

It might be not necessary to support previous versions, we can just load TOML, check version, and if the version is not the same as Vectors, exit with an error. And the "migration" steps for implementation of vector config upgrade command could be defined manually, similar to how database migrations are usually defined.

[LucioFranco]

@a-rodin I am totally on board with that! I think this would provide a much better UX.

[binarylogic]

I'm removing the needs: approval on this. I think this is worth supporting and will provide a much better UX in the face of changes.

[binarylogic]

Noting, we are deferring this until we actually need it.

[jszwedko]

Idea from @bruceg : could version configuration at the component level.