Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CI Flaky TestTransmittedSpansShouldHaveASampleRateOfAtLeastOne #897

Closed
VinozzZ opened this issue Nov 3, 2023 · 0 comments · Fixed by #934
Closed

CI Flaky TestTransmittedSpansShouldHaveASampleRateOfAtLeastOne #897

VinozzZ opened this issue Nov 3, 2023 · 0 comments · Fixed by #934
Assignees
Labels
type: bug Something isn't working
Milestone

Comments

@VinozzZ
Copy link
Contributor

VinozzZ commented Nov 3, 2023

Test Name

TestTransmittedSpansShouldHaveASampleRateOfAtLeastOne

Link to failure build

Test Output

TestTransmittedSpansShouldHaveASampleRateOfAtLeastOne
collect_test.go:249:
Error Trace: /home/circleci/project/collect/collect_test.go:249
Error: Not equal:
expected: 1
actual : 0
Test: TestTransmittedSpansShouldHaveASampleRateOfAtLeastOne
Messages: should be some events transmitted
panic: test timed out after 1m0s
running tests:
TestTransmittedSpansShouldHaveASampleRateOfAtLeastOne (1m0s)

goroutine 23 [running]:
testing.(*M).startAlarm.func1()
/usr/local/go/src/testing/testing.go:2241 +0x3c5
created by time.goFunc
/usr/local/go/src/time/sleep.go:176 +0x32

goroutine 1 [chan receive]:
testing.(*T).Run(0xc000102b60, {0xb2ffbf?, 0x52d5a5?}, 0xb57548)
/usr/local/go/src/testing/testing.go:1630 +0x405
testing.runTests.func1(0x101ff60?)
/usr/local/go/src/testing/testing.go:2036 +0x45
testing.tRunner(0xc000102b60, 0xc0002efc88)
/usr/local/go/src/testing/testing.go:1576 +0x10b
testing.runTests(0xc000140500?, {0x100d4a0, 0xf, 0xf}, {0x20?, 0x100c000119098?, 0x101f6c0?})
/usr/local/go/src/testing/testing.go:2034 +0x489
testing.(*M).Run(0xc000140500)
/usr/local/go/src/testing/testing.go:1906 +0x63a
main.main()
_testmain.go:75 +0x1aa

goroutine 7 [sync.Mutex.Lock]:
sync.runtime_SemacquireMutex(0xc000102b60?, 0x0?, 0xc00010e750?)
/usr/local/go/src/runtime/sema.go:77 +0x26
sync.(*Mutex).lockSlow(0xc000382bbc)
/usr/local/go/src/sync/mutex.go:171 +0x165
sync.(*Mutex).Lock(...)
/usr/local/go/src/sync/mutex.go:90
sync.(*RWMutex).Lock(0xc00011f260?)
/usr/local/go/src/sync/rwmutex.go:147 +0x36
github.com/honeycombio/refinery/collect.(*InMemCollector).Stop(0xc000382b60)
/home/circleci/project/collect/collect.go:662 +0x5f
panic({0xac5620, 0xc00003c810})
/usr/local/go/src/runtime/panic.go:884 +0x213
github.com/honeycombio/refinery/collect.TestTransmittedSpansShouldHaveASampleRateOfAtLeastOne(0x0?)
/home/circleci/project/collect/collect_test.go:250 +0x731
testing.tRunner(0xc000103380, 0xb57548)
/usr/local/go/src/testing/testing.go:1576 +0x10b
created by testing.(*T).Run
/usr/local/go/src/testing/testing.go:1629 +0x3ea

goroutine 20 [sleep]:
time.Sleep(0x186a0)
/usr/local/go/src/runtime/time.go:195 +0x135
github.com/honeycombio/refinery/collect/cache.NewCuckooTraceChecker.func1()
/home/circleci/project/collect/cache/cuckoo.go:63 +0x45
created by github.com/honeycombio/refinery/collect/cache.NewCuckooTraceChecker
/home/circleci/project/collect/cache/cuckoo.go:58 +0x116

goroutine 35 [sleep]:
time.Sleep(0x186a0)
/usr/local/go/src/runtime/time.go:195 +0x135
github.com/honeycombio/refinery/collect/cache.NewCuckooTraceChecker.func1()
/home/circleci/project/collect/cache/cuckoo.go:63 +0x45
created by github.com/honeycombio/refinery/collect/cache.NewCuckooTraceChecker
/home/circleci/project/collect/cache/cuckoo.go:58 +0x116

goroutine 8 [sleep]:
time.Sleep(0x186a0)
/usr/local/go/src/runtime/time.go:195 +0x135
github.com/honeycombio/refinery/collect/cache.NewCuckooTraceChecker.func1()
/home/circleci/project/collect/cache/cuckoo.go:63 +0x45
created by github.com/honeycombio/refinery/collect/cache.NewCuckooTraceChecker
/home/circleci/project/collect/cache/cuckoo.go:58 +0x116

goroutine 9 [select]:
github.com/honeycombio/refinery/collect/cache.(*cuckooSentCache).monitor(0xc000128940)
/home/circleci/project/collect/cache/cuckooSentCache.go:108 +0x85
created by github.com/honeycombio/refinery/collect/cache.NewCuckooSentCache
/home/circleci/project/collect/cache/cuckooSentCache.go:100 +0x145

goroutine 10 [sync.RWMutex.Lock]:
sync.runtime_SemacquireRWMutex(0xc0000bacb8?, 0x10?, 0x40feaa?)
/usr/local/go/src/runtime/sema.go:87 +0x26
sync.(*RWMutex).Lock(0xc0000bace8?)
/usr/local/go/src/sync/rwmutex.go:152 +0x71
github.com/honeycombio/refinery/transmit.(*MockTransmission).EnqueueSpan(0xc000114990, 0xc0003b62c0)
/home/circleci/project/transmit/mock.go:25 +0x49
github.com/honeycombio/refinery/collect.(*InMemCollector).send(0xc000382b60, 0xc0003b6370, {0xb18610, 0x13})
/home/circleci/project/collect/collect.go:654 +0x978
github.com/honeycombio/refinery/collect.(*InMemCollector).sendTracesInCache(0xc000382b60, {0xc0000a8f10?, 0x0?, 0x101f6c0?})
/home/circleci/project/collect/collect.go:370 +0x87
github.com/honeycombio/refinery/collect.(*InMemCollector).collect(0xc000382b60)
/home/circleci/project/collect/collect.go:338 +0x4c5
created by github.com/honeycombio/refinery/collect.TestTransmittedSpansShouldHaveASampleRateOfAtLeastOne
/home/circleci/project/collect/collect_test.go:231 +0x476
FAIL github.com/honeycombio/refinery/collect 60.012s

@VinozzZ VinozzZ added the type: bug Something isn't working label Nov 3, 2023
@kentquirk kentquirk added this to the v2.2 milestone Nov 21, 2023
@kentquirk kentquirk modified the milestones: v2.2, v2.3 Nov 28, 2023
@kentquirk kentquirk self-assigned this Dec 6, 2023
kentquirk added a commit that referenced this issue Dec 6, 2023
## Which problem is this PR solving?

We've had some problematic flaky tests. 
* Some were because of probabalistic assertions relating to samplers --
we now run the worst of those a second time if they fail.
* Some were waiting for events on another goroutine that might not have
been scheduled, and these have been adjusted by using
`assert.Eventually` which is actually pretty useful.

Fixes #896
Fixes #897
Fixes #901
Fixes #902

Plus a couple of other tests that hadn't gotten their own issue.

This might also make the tests run a little bit faster overall.

You might want to turn off whitespace when reviewing this, because the
retry loop changed indentation.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type: bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants