Binary search for sorted arrays

[bn-shuber]

Component(s)

pkg/ottl

Is your feature request related to a problem? Please describe.

I have a list of 10.000 to 100.000 unique sorted IDs. And I want to filter logs depending on whether an attribute is in that list or not. Without optimization, I would need to compare each log to each ID. A binary search algorithm would reduce the number of needed comparisons.

Describe the solution you'd like

A new function, or a new flag for IsMatch, that allows optimization.

Describe alternatives you've considered

No response

Additional context

No response

[github-actions]

Pinging code owners:

• pkg/ottl: @TylerHelmuth @kentquirk @bogdandrutu @evan-bradley

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[TylerHelmuth]

@Frodatima will you end up building the list of ids by hand in the statement?

[bn-shuber]

I would rather not because the list will change over time. I want to provide it via a confmap provider

[evan-bradley]

@Frodatima Could you provide an example of how you'd like to provide the list? The only way I see to do it currently using a confmap provider would be through e.g. the file provider by providing a Collector config fragment, which from OTTL's perspective is equivalent to providing it by hand.

[bn-shuber]

Yes, exactly. I want to deploy them through the file provider. So I want to provide them by hand

[TylerHelmuth]

I do believe a function that checks if a given item is in a list is a good feature. I can think of 2 ways of doing this:

1. a generic In Converter that can check if the given item is in a Slice or Map or String
2. type-specific Converters like InSlice or InMap or InString.

[bn-shuber]

I found a solution for my specific problem.
My list doesn't change that often, so I decided to create a script that calculates a function from it. It looks like this
((5>x and ((2>x and (1==x)) or (2<=x and ((2==x or x==3))))) or (5<=x and ((7>x and ((5==x or x==6))) or (7<=x and ((7==x or x==9))))))
It just creates a binary tree like structure oft of the list. I then update it via the confmap provider. Of cores this only works when the list does not change quite often.
I will close this issue.
Thank you for your help