Rewritable Tracer

[safris]

Abstract

The Rewritable Tracer allows one to rewrite data in the spans created by Integrations without having to modify the source code. A JSON config file defines a set of rules that modify trace information in the layer between the integration, and the exporter.

Introduction

The Rewritable Tracer is an idea inspired by the OpenTracing Rewrite module that was originally suggested by @zeitlinger for the SpecialAgent project.

Specification
The specification below applies to the OT API, and needs to be adapted to the OTel API.

1. Context
   The Rewritable Tracer will sit in the Span Processor layer, to be able to modify trace data generated by integrations in real time.

2. Inputs and Outputs
   Rules contain input (required) and output (optional) properties.

   • Input (required)
     The input property is required, as it defines the rule by which span data is matched. The input object contains 3 properties:

     1. type: The type of data to match (tag, log, or operationName).
     2. key: The key of the data (a string).
     3. value: The value of the data (a boolean, number, or string -- if a string, then the value is interpreted as a regular expression).

     The input property can contain a value with a single input object, or an array with multiple input objects. If multiple input objects are provided, they will be considered with OR relationship.

     Output (optional)
     If output is omitted, then the matched data will be dropped.

     If output is provided, then the matched data will be rewritten.

     Like the input object, the output object follows the same schema:

     1. type: The type of data to which the matched data is to be rewritten (tag, log, or operationName).
     2. key: The key of the data to which the matched data is to be rewritten (a string).
        If key is omitted, then the key of the matched data will not be changed.
     3. value: The value of the data (a boolean, number, or string -- if a string, then the value is interpreted as a regular expression).
        If value is omitted, then the value of the matched data will not be changed.
        The value string may contain back references (i.e. $1, $2, etc) if the value in the input defines a regular expression with matching groups.

     The output property can contain a value with a single output object, or an array with multiple output objects. If multiple output objects are provided, they will be considered with AND relationship.

This is a draft proposal, intended to spur discussion so we can come to a proposal for its v1 implementation.

Other similar prior art:

1. java-api-extensions
2. opentracing-proxy

[zeitlinger]

@safris I've just checked the integration points of this project. It seems that RewritableTracer could be a standalone project:

1. TracerProvider can be customized
   1. which gives us the possibility to provide different Tracer instances to different instrumentations
   2. this would allow us to customize spans as they are created (the current strategy)
2. SpanProcessor can be customized
   1. which gives us the possibility to modify spans before they are exported
   2. we can still identify the instrumentation that created the span
   3. it would be convenient to be able to specify multiple exporter.jar entries

Both options would work as far as I can see.
The only reference I found doesn't seem to suggest which of the options should be preferred.

[zeitlinger]

I added this to the Java SIG agenda.

Please tell me if you want to discuss this also in the auto-inst SIG.

[safris]

@zeitlinger, I believe the SpanProcessor would give us the granularity of control we'd need to implement a well-featured RewritableTracer. Thank you for adding the topic of this issue to the Java SIG. I'll make sure to attend it on April 17th. Currently, this issue is pinned to the auto-instr project, and we can put it on the agenda for this coming Friday.

[zeitlinger]

@safris following up on the suggestion to include this functionality in the collector, I've created open-telemetry/opentelemetry-collector#928 and open-telemetry/opentelemetry-collector#929 - which should cover all use cases that were covered by rewritable tracer.

Hence I think this issue can be closed.

[safris]

Thanks for following up on this @zeitlinger. We'll continue the development in the collector issues you provided.