[processor/schematransformerprocessor] Initial PR #8371

MovieStoreGuy · 2022-03-10T06:26:01Z

Description:
As part of the accepted specification, there needs to be a processor within the collector that is able to allow for transformations of semantic convention versions . This is my initial draft, I wrote this before https://github.com/open-telemetry/opentelemetry-collector-contrib/pull/8344/files shared their solution to it.

Link to tracking Issue:
#8495
Original raised in the collector but should be shifted to the contrib a I can tell.

Testing:
It currently does nothing and there is the initial testing set up within the components.

Documentation:
Added a minimal README, will expand upon it once more work has been added to the collector.

MovieStoreGuy · 2022-03-15T03:56:43Z

I have avoided adding any implementation detail here to keep the high level PR in, this PR can should be fine to merge in and run (since it will only just validate the configuration, passthrough exactly what it receives).

Not entirely sure how to resolve the issue within the gomoddownload step, @codeboten if you have any ideas please let me know :)

After this initial PR goes in, the next logical flow I have in mind is:

Figure out the best way to cache schemas
Determine schema versions and applying changes with minimal efforts
Adopt @tigrannajaryan compiled schemas to reduce runtime efforts
Add in directory extension support to write schemas to disk

go.mod

processor/schematransformerprocessor/transformer_test.go

Aneurysm9 · 2022-03-16T01:43:21Z

You'll also need to add a replace directive to cmd/configschema/go.mod as it depends on the root module and replace directives are only honored from the current go.mod for any operation. It's super annoying and we're working on a tool to make this a non-issue, but until then some hand-crafting is necessary.

MovieStoreGuy · 2022-03-16T03:10:56Z

Thanks @Aneurysm9, it looks like it managed to get past the gomoddownload part and getting onto the meaningful tests :)

MovieStoreGuy · 2022-03-16T04:29:01Z

Nobody move... the load tests somehow passed.

@codeboten , @jpkrohling, @Aneurysm9 , if you have time to have a look over this pull request?

jpkrohling

Sorry about that, but we just had a new release, so, the go.mod/sum files here have to be changed.

CHANGELOG.md

processor/schematransformerprocessor/config.go

jpkrohling · 2022-03-16T16:57:35Z

processor/schematransformerprocessor/factory_test.go

+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package schematransformerprocessor_test


Are tests coming to this file in the next PR?

Yes, since the transformer is currently just an empty struct, I wasn't exactly sure how the transformer would look like so I didn't want to burden reviewers with more work than required.

processor/schematransformerprocessor/internal/schema/indentifer.go

processor/schematransformerprocessor/internal/schema/indentifer_test.go

processor/schematransformerprocessor/testdata/schema.yml

MovieStoreGuy · 2022-03-17T01:36:32Z

Lame, the loadtest flaked out on me.

MovieStoreGuy · 2022-03-18T05:00:28Z

The stairs have aligned once again! Once I get a chance, I will have a stab to see if I can help reduce the flake in the those tests.

cmd/configschema/go.mod

processor/schematransformerprocessor/internal/schema/indentifer.go

processor/schematransformerprocessor/internal/schema/indentifer_test.go

processor/schematransformerprocessor/config.go

processor/schematransformerprocessor/README.md

processor/schematransformerprocessor/testdata/config.yml

processor/schematransformerprocessor/testdata/schema.yml

processor/schematransformerprocessor/README.md

MovieStoreGuy · 2022-03-30T06:52:57Z

Sorry @tigrannajaryan,

Finally got a chance to get back to this :)
If you have a moment to look over this it would be greatly appreciated 🙏🏽

codeboten

Just a couple of suggestions and questions. Please add a note about what constitutes a family in the readme file, it can be a copy paste of the definition in the schema overview, but i think it will help users of this processor

CHANGELOG.md

codeboten · 2022-04-22T15:16:55Z

processor/schemaprocessor/README.md

+
+## Caching Schema Translation Files
+
+In order to improve effeicency of the processor, the `precache` option allows the processor to start downloading and preparing


Suggested change

In order to improve effeicency of the processor, the `precache` option allows the processor to start downloading and preparing

In order to improve efficency of the processor, the `precache` option allows the processor to start downloading and preparing

Not sure if that was a mistake, but misspell is reporting it as a typo.

processor/schemaprocessor/README.md

processor/schemaprocessor/factory.go

processor/schemaprocessor/testdata/config.yml

jpkrohling

LGTM. Just clarify to users what happens when conflicting versions of the schemas are used, and fix a couple of typos.

processor/schemaprocessor/README.md

processor/schemaprocessor/config.go

Retracting, due to the changelog changes.

jpkrohling · 2022-04-22T19:59:28Z

Sorry, I'm retracting my approval because of the unwanted changes to the changelog.

jpkrohling

Apart from the couple of typos, LGTM!

processor/schemaprocessor/config.go

processor/schemaprocessor/internal/schema/identifier_test.go

processor/schemaprocessor/transformer_test.go

MovieStoreGuy · 2022-04-29T00:40:30Z

@jpkrohling 🙏🏽

Could I get this merged please?

jpkrohling · 2022-04-29T12:20:52Z

I tried to update this branch, but got an error :-( The changelog needs updating, and there's a conflict with the go.sum versions. Once this is fixed and the build is green, I'll merge (if this happens today).

tigrannajaryan · 2022-04-29T13:14:53Z

@MovieStoreGuy ping me when the conflicts are resolved, I will merge.

codeboten

Thanks for addressing my comments, please rebase and we'll get this merge right away

This is the initial commit into the collector to add in the component to help enforce a standard semantic convention for all signals.

tigrannajaryan · 2022-05-02T13:28:16Z

Thank you @MovieStoreGuy
Looking forward to the continuation PRs!

**Description:** As part of the [accepted specification](https://github.com/open-telemetry/opentelemetry-specification/blob/e16e1a781bdf62a8fcfa1a12963844749de23e7a/specification/schemas/overview.md#collector-assisted-schema-transformation), there needs to be a processor within the collector that is able to allow for transformations of semantic convention versions . This is my initial draft, I wrote this before https://github.com/open-telemetry/opentelemetry-collector-contrib/pull/8344/files shared their solution to it. **Link to tracking Issue:** open-telemetry#8495 Original raised in the collector but should be shifted to the contrib a I can tell. **Testing:** It currently does nothing and there is the initial testing set up within the components. **Documentation:** Added a minimal README, will expand upon it once more work has been added to the collector.

tigrannajaryan · 2022-05-27T18:44:33Z

@MovieStoreGuy do you plan to continue working on the schema processor?

MovieStoreGuy · 2022-05-29T12:59:30Z

Yes, sorry. I have been meaning to get back into it but I have been working on rollout infra changes to adopt the otel collector for resource metrics, so that has taken a lot of brain space atm.

I intend to get back to this this week

tigrannajaryan · 2022-05-30T13:53:22Z

Sounds good, thanks!

MovieStoreGuy marked this pull request as ready for review March 11, 2022 01:30

MovieStoreGuy requested review from a team and jpkrohling March 11, 2022 01:30

github-actions bot assigned codeboten Mar 11, 2022

MovieStoreGuy force-pushed the msg/schema-transformer-processor branch 2 times, most recently from 6ac440e to 5e039b9 Compare March 15, 2022 02:58

Aneurysm9 reviewed Mar 16, 2022

View reviewed changes

go.mod Outdated Show resolved Hide resolved

MovieStoreGuy force-pushed the msg/schema-transformer-processor branch from b1483ad to e5136ea Compare March 16, 2022 01:13

MovieStoreGuy commented Mar 16, 2022

View reviewed changes

processor/schematransformerprocessor/transformer_test.go Outdated Show resolved Hide resolved

MovieStoreGuy force-pushed the msg/schema-transformer-processor branch from a7abf33 to b532668 Compare March 16, 2022 02:44

jpkrohling reviewed Mar 16, 2022

View reviewed changes

MovieStoreGuy force-pushed the msg/schema-transformer-processor branch 2 times, most recently from f6d815f to 73551e6 Compare March 17, 2022 01:03

MovieStoreGuy force-pushed the msg/schema-transformer-processor branch 2 times, most recently from 81c7ae5 to c1006e6 Compare March 18, 2022 03:11

jpkrohling reviewed Mar 18, 2022

View reviewed changes

cmd/configschema/go.mod Outdated Show resolved Hide resolved

processor/schematransformerprocessor/internal/schema/indentifer.go Outdated Show resolved Hide resolved

processor/schematransformerprocessor/internal/schema/indentifer_test.go Outdated Show resolved Hide resolved

tigrannajaryan reviewed Mar 18, 2022

View reviewed changes

MovieStoreGuy force-pushed the msg/schema-transformer-processor branch from c1006e6 to 8e293cd Compare March 21, 2022 01:31

MovieStoreGuy force-pushed the msg/schema-transformer-processor branch 2 times, most recently from b19fafd to e2c8a9b Compare March 30, 2022 06:46

MovieStoreGuy force-pushed the msg/schema-transformer-processor branch 3 times, most recently from 5b48af9 to 8c4d72f Compare March 30, 2022 08:01

MovieStoreGuy force-pushed the msg/schema-transformer-processor branch 2 times, most recently from 401ff1b to d364da8 Compare April 22, 2022 03:11

codeboten reviewed Apr 22, 2022

View reviewed changes

jpkrohling previously approved these changes Apr 22, 2022

View reviewed changes

processor/schemaprocessor/README.md Outdated Show resolved Hide resolved

processor/schemaprocessor/README.md Outdated Show resolved Hide resolved

processor/schemaprocessor/config.go Show resolved Hide resolved

MovieStoreGuy force-pushed the msg/schema-transformer-processor branch from 4dfbbae to 8ada9d1 Compare April 25, 2022 23:26

MovieStoreGuy requested review from jpkrohling and codeboten April 26, 2022 01:19

MovieStoreGuy force-pushed the msg/schema-transformer-processor branch from 9cd8112 to 92b91ad Compare April 26, 2022 01:37

jpkrohling reviewed Apr 26, 2022

View reviewed changes

MovieStoreGuy force-pushed the msg/schema-transformer-processor branch from 92b91ad to 6610e95 Compare April 26, 2022 22:59

jpkrohling approved these changes Apr 27, 2022

View reviewed changes

MovieStoreGuy force-pushed the msg/schema-transformer-processor branch from 6610e95 to 2be9365 Compare April 28, 2022 23:34

codeboten approved these changes Apr 29, 2022

View reviewed changes

MovieStoreGuy force-pushed the msg/schema-transformer-processor branch from 2be9365 to 64ccae4 Compare May 1, 2022 23:53

Adding new component: Schema Processor

5f83e82

This is the initial commit into the collector to add in the component to help enforce a standard semantic convention for all signals.

MovieStoreGuy force-pushed the msg/schema-transformer-processor branch from 64ccae4 to 5f83e82 Compare May 1, 2022 23:54

MovieStoreGuy added 2 commits May 2, 2022 11:54

Update CHANGELOG.md

8133143

Update CHANGELOG.md

c594e1d

tigrannajaryan merged commit 35db7b8 into open-telemetry:main May 2, 2022

ahayworth mentioned this pull request May 16, 2022

Update to otel-contrib-col v0.51.0 Shopify/opentelemetry-collector-contrib#796

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[processor/schematransformerprocessor] Initial PR #8371

[processor/schematransformerprocessor] Initial PR #8371

MovieStoreGuy commented Mar 10, 2022

MovieStoreGuy commented Mar 15, 2022

Aneurysm9 commented Mar 16, 2022

MovieStoreGuy commented Mar 16, 2022

MovieStoreGuy commented Mar 16, 2022

jpkrohling left a comment

jpkrohling Mar 16, 2022

MovieStoreGuy Mar 16, 2022

MovieStoreGuy commented Mar 17, 2022

MovieStoreGuy commented Mar 18, 2022

MovieStoreGuy commented Mar 30, 2022

codeboten left a comment

codeboten Apr 22, 2022

MovieStoreGuy Apr 26, 2022

jpkrohling left a comment

jpkrohling commented Apr 22, 2022

jpkrohling left a comment

MovieStoreGuy commented Apr 29, 2022

jpkrohling commented Apr 29, 2022

tigrannajaryan commented Apr 29, 2022

codeboten left a comment

tigrannajaryan commented May 2, 2022

tigrannajaryan commented May 27, 2022

MovieStoreGuy commented May 29, 2022

tigrannajaryan commented May 30, 2022


		## Caching Schema Translation Files

		In order to improve effeicency of the processor, the `precache` option allows the processor to start downloading and preparing

	In order to improve effeicency of the processor, the `precache` option allows the processor to start downloading and preparing
	In order to improve efficency of the processor, the `precache` option allows the processor to start downloading and preparing

[processor/schematransformerprocessor] Initial PR #8371

[processor/schematransformerprocessor] Initial PR #8371

Conversation

MovieStoreGuy commented Mar 10, 2022

MovieStoreGuy commented Mar 15, 2022

Aneurysm9 commented Mar 16, 2022

MovieStoreGuy commented Mar 16, 2022

MovieStoreGuy commented Mar 16, 2022

jpkrohling left a comment

Choose a reason for hiding this comment

jpkrohling Mar 16, 2022

Choose a reason for hiding this comment

MovieStoreGuy Mar 16, 2022

Choose a reason for hiding this comment

MovieStoreGuy commented Mar 17, 2022

MovieStoreGuy commented Mar 18, 2022

MovieStoreGuy commented Mar 30, 2022

codeboten left a comment

Choose a reason for hiding this comment

codeboten Apr 22, 2022

Choose a reason for hiding this comment

MovieStoreGuy Apr 26, 2022

Choose a reason for hiding this comment

jpkrohling left a comment

Choose a reason for hiding this comment

jpkrohling commented Apr 22, 2022

jpkrohling left a comment

Choose a reason for hiding this comment

MovieStoreGuy commented Apr 29, 2022

jpkrohling commented Apr 29, 2022

tigrannajaryan commented Apr 29, 2022

codeboten left a comment

Choose a reason for hiding this comment

tigrannajaryan commented May 2, 2022

tigrannajaryan commented May 27, 2022

MovieStoreGuy commented May 29, 2022

tigrannajaryan commented May 30, 2022