Add json_array_parser parser and assign_keys transformer

[RoeiDimi]

Description:

We are using otel-collector as an infrastructure and receive many types of data from a client. The client's sent data is always a form of json and in one use case the json is a simple headerless jarray and so we need a way to parse it and match headers to each field (something similar to what csv_parser does but also supports types supported in a json format and nested objects)

Link to tracking Issue:
#30321

Testing:

• unittests
  All the tests found in csv_parser were copied and adjusted adding test scenarios for different types (numbers, booleans, null) as well as a test for parsing a nested object as a part of the jarray
• End to end tests
  Used generated traffic on a running otel collector thats using the parser and verified the data is as expected in the end table

Documentation:

• json_array_parser.md
• assign_keys.md

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: RoeiDimi (07aa655, 1fe4ff7, 2d2fde1, de83043, fce1fff, 6c29ede, 488d4d4, 4dd2117, e00f4d7, 7d54086, 0088093, 3c23e50, 91fea1d, 9a3897f, b325b8f, f68cd9a, fe40b11, 9ac4971, b2b26f2, 76c1fe0, beacea4, 4cf11dd, 9e17cea, c371559, 4e7dc24, 89dae82, 1a140ff)
• ✅ login: djaglowski / name: Daniel Jaglowski (947cbe9, 4e3af6f, 7f93ec8)

[djaglowski]

Thanks @RoeiDimi, the design feels right. I left a lot of comments but mostly just nits. A few minor things I think we need to take care of though.

[djaglowski]

@RoeiDimi, I was able to resolve the issue by adding the new dependency directly to the root module. Thanks for the new features and for iterating on the design with me.

[RoeiDimi]

Thanks @djaglowski, it's been a pleasure working together!

One last question though - assign_keys can't really parse the values straight into attributes (the way csv_parser can).
can you think of an easy way to make that happen?

i mean, I now understand that all this may not solve my initial issue completely.
with the initial design i could do this -

- type: csv_parser
      header: TimeGenerated,SourceIP,SourcePort,DestinationIP,DestinationPort,Protocol,SentBytes,ReceivedBytes,ExtID
      parse_from: body
      parse_to: attributes
      is_jarray: true

we have a generic exporter code that reads all the key-values from attributes and sends them.

Can you think of a way I could use the json_array_parser -> assign_keys design to achieve that?

[djaglowski]

@RoeiDimi, looks like we lost track of the goal somewhat but luckily you should be able to flatten after assigning keys:

    operators:
      - type: json_array_parser
        parse_from: body
        parse_to: attributes.tmp
      - type: assign_keys
        keys: ["a", "b", "c"]
        field: attributes.tmp
      - type: flatten
        field: attributes.tmp

Edit: I should point out this is not entirely by luck, but rather is a natural benefit of the highly decomposed approach we've always pursued with these operators. With a robust set of simple operations, we can typically compose them to do what's needed.

[RoeiDimi]

@djaglowski yeah that's pretty cool, thank you so much! :)

I apologize but I have another question :
About the feature gate we added - how do I enable it? I saw this link

so I tried running the collector with all the following variations but none worked -

• --feature-gates=logs.assignKeys,logs.jsonParserArray
• --feature-gates=assignKeys,+jsonParserArray
• --feature-gates=assignKeys,jsonParserArray
• --feature-gates=+logs.assignKeys,+logs.jsonParserArray

[sumo-drosiek]

@djaglowski, @RoeiDimi Shouldn't we have the functionality in OTTL language? I was able to achive the same functionality I believe using transformprocessor:

exporters:
  logging:
    verbosity: detailed
processors:
  transform/parsejson:
    log_statements:
      - context: log
        statements:
          // ParseJSON expects `{}` so I need to build it manually
          - set(body, ParseJSON(Concat(["{\"tmp\":", body, "}"], "")))
          - set(body["key1"], body["tmp"][0])
          - set(body["key2"], body["tmp"][1])
          - set(body["key3"], body["tmp"][2])
          - delete_key(body, "tmp")
receivers:
  filelog/containers:
    start_at: beginning
    include:
      - ./tmp/logs/5.json
service:
  pipelines:
    logs/containers:
      exporters:
      - logging
      processors:
      - transform/parsejson
      receivers:
      - filelog/containers

./tmp/logs/5.json

["value1", "value2", "value3"]

output:

LogRecord #0
ObservedTimestamp: 2024-01-24 08:11:49.193363736 +0000 UTC
Timestamp: 1970-01-01 00:00:00 +0000 UTC
SeverityText: 
SeverityNumber: Unspecified(0)
Body: Map({"key1":"value1","key2":"value2","key3":"value3"})
Attributes:
     -> log.file.name: Str(5.json)
Trace ID: 
Span ID: 
Flags: 0
        {"kind": "exporter", "data_type": "logs", "name": "logging"}

[RoeiDimi]

@sumo-drosiek @djaglowski , this suggestion may be interesting as after i ran large-scale stress tests it seems that using the 3 operators together (json_array_parser, assign_keys, flatten) result in a massive performance hit. Seems like chaining operators is not very efficient in general as the initial design with a single operator (headerless_jarray_parser) that does the same logic as in json_array_parser+assign_keys together was significantly better performance-wise.

anyway @sumo-drosiek I'd like to test your suggestion, i tried adding this to the builder config -

processors:
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/transformprocessor v0.90.0

but im getting and error saying the revision is wrong -
go: github.com/open-telemetry/opentelemetry-collector-contrib/[email protected]: reading github.com/open-telemetry/opentelemetry-collector-contrib/transformprocessor/go.mod at revision transformprocessor/v0.90.0: unknown revision transformprocessor/v0.90.0

is it not v0.90.0? can you please share the revision you used?

Thanks guys for helping me through this entirely new world of OTEL collector hehe..

[sumo-drosiek]

@RoeiDimi It should be github.com/open-telemetry/opentelemetry-collector-contrib/processor/transformprocessor

[RoeiDimi]

Hi @djaglowski
I know this is closed as completed but after trying to use the solution we went for, I see too big of a performance hit to be able to use it. Using these 3 operators is at least 2X slower than using the original headerless_jarray_parser alone.
I also tried @sumo-drosiek 's suggestion (thank you for trying to help!) and it shows roughly the same numbers as using the 3 operators

For my benchmarks I'm using Azure kubernetes, single node, single pod, 5 cpus. I tested the following:

1. headerless-jarray-parser can take ~10-11 mil events per minute
2. 3 operators chained (or the processor suggestion) ~4-5 mil events per minute
3. Just for testing i removed the flatten to see if this is the only issue but its just a part of it, i got ~6-7 mil events per minute
4. json_array_parser alone got me back to the ~10-11 mil events per minute

It also makes sense in a way as headerless-jarray-parser could iterate only once on the input array, create a map and then replace attributes with that map..

I totally get the refactor we went for from a modularity point of view but is there any chance you'd agree to now also adding headerless-jarray-parser for performance reasons?

[djaglowski]

@RoeiDimi, thanks for providing detailed performance numbers.

I think given the notable performance benefit, it's reasonable to revisit the combined operator.

json_array_parser and assign_keys are both useful in their own right, even if composition affects performance. I'd like to keep these capabilities in place regardless.

If we were to apply the same terminology update as we arrived at, we are talking about a headerless_json_array_parser. Rather than having two closely related parsers, headerless_json_array_parser and json_array_parser, we should consider what this looks like to support both use cases. In other words, pull the header/less part into the config. I imagine this could work as follows:

• Re-add header settings, equivalent to those in the csv parser.
• If the neither header setting is used, then the parser behaves exactly as it does today. That is, the result placed in parse_to is expected to be []any.
• If either header setting is used, then we perform the parsing step and then apply the header, and ultimately place an object in parse_to.

The benefit of this approach is that the operator just has the ability to apply an optional behavior, but one that is closely related to the first step of parsing the array. WDYT?

[RoeiDimi]

@djaglowski yes, that's a good, clean idea
I guess we can now continue here - #30814
:)