[jaeger-v2] add Badger storage backend integration test #5281

Pushkarm029 · 2024-03-18T06:47:45Z

Which problem is this PR solving?

Part of [jaeger-v2] Storage backend integration tests #5254

Description of the changes

Added integration tests for Badger.
Modified GH Workflow to perform integration tests for jaeger-v2 also.

How was this change tested?

./scripts/badger-integration-test.sh

Checklist

I have read https://github.com/jaegertracing/jaeger/blob/master/CONTRIBUTING_GUIDELINES.md
I have signed all commits
I have added unit tests for the new functionality
I have run lint and test steps successfully
- for jaeger: make lint test
- for jaeger-ui: yarn lint and yarn test

Signed-off-by: Pushkar Mishra <[email protected]>

Pushkarm029 · 2024-03-18T06:51:21Z

@james-ryans, this test is not able to receive data. any idea, how to tackle this?
log:

2024/03/18 12:02:01 Starting load generator at 1024 items/sec.
2024/03/18 12:02:03  | Sent:       539 traces (275/sec) | Received:         0 items (0/sec)
2024/03/18 12:02:06  | Sent:     1,533 traces (309/sec) | Received:         0 items (0/sec)
2024/03/18 12:02:08 Stopped generator. Sent:     1,533 traces (219/sec)
2024/03/18 12:02:09  | Sent:     1,533 traces (192/sec) | Received:         0 items (0/sec)
2024/03/18 12:02:12  | Sent:     1,533 traces (139/sec) | Received:         0 items (0/sec)
2024/03/18 12:02:15  | Sent:     1,533 traces (109/sec) | Received:         0 items (0/sec)
2024/03/18 12:02:18  | Sent:     1,533 traces (90/sec) | Received:         0 items (0/sec)
2024/03/18 12:02:18 Time out waiting for all data items received
    test_case.go:289: Time out waiting for all data items received

codecov · 2024-03-18T06:56:04Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 95.00%. Comparing base (53592b3) to head (2dfd9b9).
Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #5281      +/-   ##
==========================================
- Coverage   95.08%   95.00%   -0.08%     
==========================================
  Files         340      340              
  Lines       16626    16626              
==========================================
- Hits        15809    15796      -13     
- Misses        628      642      +14     
+ Partials      189      188       -1

Flag	Coverage Δ
badger	`12.74% <ø> (-0.52%)`	⬇️
cassandra-3.x	`26.44% <ø> (ø)`
cassandra-4.x	`26.44% <ø> (ø)`
elasticsearch-5.x	`21.70% <ø> (+0.01%)`	⬆️
elasticsearch-6.x	`21.68% <ø> (ø)`
elasticsearch-7.x	`?`
elasticsearch-8.x	`?`
grpc	`11.01% <ø> (+0.03%)`	⬆️
kafka	`14.73% <ø> (ø)`
opensearch-1.x	`21.77% <ø> (ø)`
opensearch-2.x	`21.77% <ø> (-0.02%)`	⬇️
unittests	`92.25% <ø> (-0.02%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

james-ryans · 2024-03-18T13:51:30Z

I've just learned that Badger is an embedded local storage. In Testbed, the collector and the data receiver are both running as separated instances. That being said the data receiver doesn't have access to anything inside the collector, that's why we can't test the memory storage backend. But from my perspective, it seems your Badger config writes and stores the spans in the /tmp/jaeger directory. Since both the collector and data receiver are running on the same computer, the data receiver should be able to read those spans from /tmp/jaeger directory. I'll try to take a look at how Badger works and update to you what I can do.

Also, that means you probably don't need to have a script to run a remote storage.

yurishkuro · 2024-03-18T14:01:03Z

If you point two badger instances to the same file system they may run into synchronization issues. I don't think it's recommended.

The way to test badger and memory is by using jaeger remote storage, a binary that can run one instance to which you connect as grpc storage. I believe our crossdock tests are using that.

james-ryans · 2024-03-18T14:24:08Z

If you point two badger instances to the same file system they may run into synchronization issues.

I think Yuri is right, I've just tried to run the Badger integration test with ephemeral: false in the config to read & write from the directory_key and directory_value. But I encountered this error.

panic: failed to start extensions: failed to initialize badger storage badger_main: Cannot acquire directory lock on "/tmp/jaeger".  Another process is using this Badger database. error: resource temporarily unavailable

The way to test badger and memory is by using jaeger remote storage

Yes, in the previous PR for gRPC integration test, I actually used memory span storage type for the remote storage. You may also follow that configuration at grpc_config.yaml. But for the script, we're actually trying to improve that by build a binary/container remote storage from "this" version of repository not the published image #5266.

Pushkarm029 · 2024-03-18T19:53:08Z

Thanks for the suggestions. I am trying it out, I will update you soon.

Pushkarm029 · 2024-03-19T13:04:29Z

~~Update: This method is working fine locally :~~ sorry, it's not working locally also, throwing same dir error. it was working previously due to some misconfiguration of env variables.

However, replicating this in docker using a script throws this error because user_uid is set to 1001(non_root) in remote-storage.

"error":"Error Creating Dir: \"/go/bin/data/keys\" error: mkdir /go/bin/data: permission denied,"

Signed-off-by: Pushkar Mishra <[email protected]>

yurishkuro · 2024-03-19T14:43:42Z

You don't want /go/bin as the data dir for badger. See #5282 about permissions

Signed-off-by: Pushkar Mishra <[email protected]>

Pushkarm029 · 2024-03-20T15:45:11Z

Fixed this permission error, Still same :

2024/03/20 20:56:49 Starting load generator at 1024 items/sec.
2024/03/20 20:56:51  | Sent:       460 traces (231/sec) | Received:         0 items (0/sec)
2024/03/20 20:56:54  | Sent:     1,533 traces (307/sec) | Received:         0 items (0/sec)
2024/03/20 20:56:56 Stopped generator. Sent:     1,839 traces (268/sec)
2024/03/20 20:56:57  | Sent:     1,839 traces (230/sec) | Received:         0 items (0/sec)
2024/03/20 20:57:00  | Sent:     1,839 traces (167/sec) | Received:         0 items (0/sec)
2024/03/20 20:57:03  | Sent:     1,839 traces (131/sec) | Received:         0 items (0/sec)
2024/03/20 20:57:06  | Sent:     1,839 traces (108/sec) | Received:         0 items (0/sec)

Any other approach or fix @yashrsharma44

Pushkarm029 · 2024-03-21T13:57:06Z

dbentries.txt
I recovered the test database to verify that traces are stored in the remote badger database and tried to read key value pair. Got this result. Please check if it is the same as what we are supposed to store in it. @james-ryans

yurishkuro · 2024-03-21T15:24:15Z

@Pushkarm029 I doubt you can compare binary files like that, @james-ryans might have completely different binary representation because of timestamps. You can query db via jaeger-query/apiv3 JSON interface

Pushkarm029 · 2024-03-21T16:11:13Z

traces are being stored in the database, so I think there may be some problem with the data receiver.

yurishkuro · 2024-03-21T17:02:55Z

traces are being stored in the database, so I think there may be some problem with the data receiver.

perhaps we can add logging to the data receiver that shows which query it is executing against the backend. Then try to run the same query manually - if it returns the data it means you likely have some connectivity issue rather than data issue.

james-ryans · 2024-03-21T18:20:56Z

perhaps we can add logging to the data receiver that shows which query it is executing against the backend.

Yes, I'm trying to help figure out how to add logging similar to the OTEL collector's components log to the data receiver.

I just finished debugging the issue, it was sourced from the artificial jaeger receiver. In Badger, the spanReader.FindTraces query params require StartTimeMin and StartTimeMax which causes the error below. This might also happen to @akagami-harsh's PR at #5287.

Failed to consume traces from consumer	{"error": "stream error: rpc error: code = Unknown desc = start and end time must be set"}

I'll create a new PR to fix this issue and also add logging to the data receiver. Sorry for the inconvenience.

Pushkarm029 · 2024-03-21T19:13:15Z

Sorry for the inconvenience.

Don't feel sorry; after all, we all are here to learn❤️

Signed-off-by: Pushkar Mishra <[email protected]>

… function. (#5288) ## Which problem is this PR solving? - Fixes #5281 issue for not being able to consume traces ## Description of the changes - Added `StartTimeMin` and `StartTimeMax` query params with 1h lookback to `consumeTraces` function. - Added logging to the data receivers with the OTelCol codes below as the references. - https://github.com/open-telemetry/opentelemetry-collector/blob/main/otelcol/unmarshaler.go#L45 - https://github.com/open-telemetry/opentelemetry-collector/blob/main/service/telemetry/telemetry.go#L118 ## How was this change tested? - Tested manually. ## Checklist - [x] I have read https://github.com/jaegertracing/jaeger/blob/master/CONTRIBUTING_GUIDELINES.md - [x] I have signed all commits - [x] I have added unit tests for the new functionality - [x] I have run lint and test steps successfully - for `jaeger`: `make lint test` - for `jaeger-ui`: `yarn lint` and `yarn test` --------- Signed-off-by: James Ryans <[email protected]>

Pushkarm029 · 2024-03-24T05:41:04Z

Why closed?

yurishkuro · 2024-03-24T06:02:13Z

Auto-closed because the other PR had the text Fixes #5281 issue, need to be careful with those.

Signed-off-by: Pushkar Mishra <[email protected]>

cmd/jaeger/internal/integration/fixtures/badger_config.yaml

Signed-off-by: Pushkar Mishra <[email protected]>

yurishkuro · 2024-05-03T17:24:53Z

@Pushkarm029 do we still need this PR or is it covered by #5355 ?

Pushkarm029 · 2024-05-03T17:26:57Z

Nope, I'm closing it now.

init

d4cebe2

Signed-off-by: Pushkar Mishra <[email protected]>

Pushkarm029 changed the title ~~[jaeger-v2] add GRPC storage backend integration test~~ [jaeger-v2] add Badger storage backend integration test Mar 18, 2024

working locally

206dc33

Signed-off-by: Pushkar Mishra <[email protected]>

Pushkarm029 added 2 commits March 20, 2024 20:12

fix badger mkdir issue

dbf4032

Signed-off-by: Pushkar Mishra <[email protected]>

fix

bab56b5

Signed-off-by: Pushkar Mishra <[email protected]>

yurishkuro added the changelog:test Change that's adding missing tests or correcting existing tests label Mar 21, 2024

james-ryans mentioned this pull request Mar 21, 2024

[jaeger-v2] Add StartTimeMin and StartTimeMax params to consumeTraces function. #5288

Merged

4 tasks

some fixes

6bcd218

Signed-off-by: Pushkar Mishra <[email protected]>

Pushkarm029 marked this pull request as ready for review March 21, 2024 19:32

Pushkarm029 requested a review from a team as a code owner March 21, 2024 19:32

Pushkarm029 requested a review from albertteoh March 21, 2024 19:32

yurishkuro closed this in #5288 Mar 23, 2024

yurishkuro reopened this Mar 24, 2024

Pushkarm029 added 2 commits March 24, 2024 11:40

Merge branch 'main' into badger_integration

52b6ac5

fix gh cli error

6c70cea

Signed-off-by: Pushkar Mishra <[email protected]>

yurishkuro mentioned this pull request Mar 24, 2024

[jaeger-v2] Storage backend integration tests #5254

Closed

5 tasks

yurishkuro reviewed Mar 24, 2024

View reviewed changes

cmd/jaeger/internal/integration/fixtures/badger_config.yaml Outdated Show resolved Hide resolved

fix

2dfd9b9

Signed-off-by: Pushkar Mishra <[email protected]>

Pushkarm029 closed this May 3, 2024

Pushkarm029 deleted the badger_integration branch May 3, 2024 17:27

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[jaeger-v2] add Badger storage backend integration test #5281

[jaeger-v2] add Badger storage backend integration test #5281

Pushkarm029 commented Mar 18, 2024

Pushkarm029 commented Mar 18, 2024 •

edited

Loading

codecov bot commented Mar 18, 2024 •

edited

Loading

james-ryans commented Mar 18, 2024

yurishkuro commented Mar 18, 2024

james-ryans commented Mar 18, 2024

Pushkarm029 commented Mar 18, 2024

Pushkarm029 commented Mar 19, 2024 •

edited

Loading

yurishkuro commented Mar 19, 2024

Pushkarm029 commented Mar 20, 2024

Pushkarm029 commented Mar 21, 2024

yurishkuro commented Mar 21, 2024

Pushkarm029 commented Mar 21, 2024 •

edited

Loading

yurishkuro commented Mar 21, 2024

james-ryans commented Mar 21, 2024

Pushkarm029 commented Mar 21, 2024 •

edited

Loading

Pushkarm029 commented Mar 24, 2024

yurishkuro commented Mar 24, 2024

yurishkuro commented May 3, 2024

Pushkarm029 commented May 3, 2024

[jaeger-v2] add Badger storage backend integration test #5281

[jaeger-v2] add Badger storage backend integration test #5281

Conversation

Pushkarm029 commented Mar 18, 2024

Which problem is this PR solving?

Description of the changes

How was this change tested?

Checklist

Pushkarm029 commented Mar 18, 2024 • edited Loading

codecov bot commented Mar 18, 2024 • edited Loading

Codecov Report

james-ryans commented Mar 18, 2024

yurishkuro commented Mar 18, 2024

james-ryans commented Mar 18, 2024

Pushkarm029 commented Mar 18, 2024

Pushkarm029 commented Mar 19, 2024 • edited Loading

yurishkuro commented Mar 19, 2024

Pushkarm029 commented Mar 20, 2024

Pushkarm029 commented Mar 21, 2024

yurishkuro commented Mar 21, 2024

Pushkarm029 commented Mar 21, 2024 • edited Loading

yurishkuro commented Mar 21, 2024

james-ryans commented Mar 21, 2024

Pushkarm029 commented Mar 21, 2024 • edited Loading

Pushkarm029 commented Mar 24, 2024

yurishkuro commented Mar 24, 2024

yurishkuro commented May 3, 2024

Pushkarm029 commented May 3, 2024

Pushkarm029 commented Mar 18, 2024 •

edited

Loading

codecov bot commented Mar 18, 2024 •

edited

Loading

Pushkarm029 commented Mar 19, 2024 •

edited

Loading

Pushkarm029 commented Mar 21, 2024 •

edited

Loading

Pushkarm029 commented Mar 21, 2024 •

edited

Loading