Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix TestReadNilTimeRanges #3299

Closed
miroswan opened this issue Feb 28, 2021 · 1 comment · Fixed by #3300
Closed

Fix TestReadNilTimeRanges #3299

miroswan opened this issue Feb 28, 2021 · 1 comment · Fixed by #3300
Assignees
@gibbscullen

Comments

@miroswan
Copy link
Contributor

miroswan commented Feb 28, 2021 •
edited
Loading

A build in my PR is failing due to a flaky test exhibiting intermittent data racing.

Steps to reproduce:

  • cd into src/dbnode/storage/bootstrap/bootstrapper/fs
  • execute while go test -race -v -count=1 -run TestReadNilTimeRanges; do : ; done until it fails

Output during failure:

go test -race -v -count=1  -run TestReadNilTimeRanges 
=== RUN   TestReadNilTimeRanges
{"level":"info","ts":1614505803.672145,"msg":"bootstrapping time series data start","bootstrapper":"filesystem","cachePolicy":"all"}
==================
WARNING: DATA RACE
Read at 0x00c06bbac750 by goroutine 34:
  runtime.mapaccess1_faststr()
      /usr/local/Cellar/go/1.13.5/libexec/src/runtime/map_faststr.go:12 +0x0
  github.com/m3db/m3/src/dbnode/storage/bootstrap.(*TestDataAccumulator).checkoutSeriesWithLock.func1()
      /Users/dswan/go/modules/m3/src/dbnode/storage/bootstrap/util.go:249 +0x2d9
  runtime.call64()
      /usr/local/Cellar/go/1.13.5/libexec/src/runtime/asm_amd64.s:540 +0x3a
  reflect.Value.Call()
      /usr/local/Cellar/go/1.13.5/libexec/src/reflect/value.go:321 +0xd3
  github.com/golang/mock/gomock.(*Call).DoAndReturn.func1()
      /Users/dswan/go/pkg/mod/github.com/golang/[email protected]/gomock/call.go:123 +0x4ef
  github.com/golang/mock/gomock.(*Controller).Call()
      /Users/dswan/go/pkg/mod/github.com/golang/[email protected]/gomock/controller.go:221 +0x144
  github.com/m3db/m3/src/dbnode/storage/series.(*MockDatabaseSeries).LoadBlock()
      /Users/dswan/go/modules/m3/src/dbnode/storage/series/series_mock.go:213 +0x1a3
  github.com/m3db/m3/src/dbnode/storage/bootstrap/bootstrapper/fs.(*fileSystemSource).readNextEntryAndRecordBlock()
      /Users/dswan/go/modules/m3/src/dbnode/storage/bootstrap/bootstrapper/fs/source.go:704 +0x559
  github.com/m3db/m3/src/dbnode/storage/bootstrap/bootstrapper/fs.(*fileSystemSource).loadShardReadersDataIntoShardResult()
      /Users/dswan/go/modules/m3/src/dbnode/storage/bootstrap/bootstrapper/fs/source.go:432 +0xae3
  github.com/m3db/m3/src/dbnode/storage/bootstrap/bootstrapper/fs.(*fileSystemSource).bootstrapFromReaders()
      /Users/dswan/go/modules/m3/src/dbnode/storage/bootstrap/bootstrapper/fs/source.go:324 +0x208
  github.com/m3db/m3/src/dbnode/storage/bootstrap/bootstrapper/fs.(*fileSystemSource).read.func3()
      /Users/dswan/go/modules/m3/src/dbnode/storage/bootstrap/bootstrapper/fs/source.go:868 +0x206

Previous write at 0x00c06bbac750 by goroutine 32:
  runtime.mapassign_faststr()
      /usr/local/Cellar/go/1.13.5/libexec/src/runtime/map_faststr.go:202 +0x0
  github.com/m3db/m3/src/dbnode/storage/bootstrap.(*TestDataAccumulator).checkoutSeriesWithLock.func1()
      /Users/dswan/go/modules/m3/src/dbnode/storage/bootstrap/util.go:249 +0x4de
  runtime.call64()
      /usr/local/Cellar/go/1.13.5/libexec/src/runtime/asm_amd64.s:540 +0x3a
  reflect.Value.Call()
      /usr/local/Cellar/go/1.13.5/libexec/src/reflect/value.go:321 +0xd3
  github.com/golang/mock/gomock.(*Call).DoAndReturn.func1()
      /Users/dswan/go/pkg/mod/github.com/golang/[email protected]/gomock/call.go:123 +0x4ef
  github.com/golang/mock/gomock.(*Controller).Call()
      /Users/dswan/go/pkg/mod/github.com/golang/[email protected]/gomock/controller.go:221 +0x144
  github.com/m3db/m3/src/dbnode/storage/series.(*MockDatabaseSeries).LoadBlock()
      /Users/dswan/go/modules/m3/src/dbnode/storage/series/series_mock.go:213 +0x1a3
  github.com/m3db/m3/src/dbnode/storage/bootstrap/bootstrapper/fs.(*fileSystemSource).readNextEntryAndRecordBlock()
      /Users/dswan/go/modules/m3/src/dbnode/storage/bootstrap/bootstrapper/fs/source.go:704 +0x559
  github.com/m3db/m3/src/dbnode/storage/bootstrap/bootstrapper/fs.(*fileSystemSource).loadShardReadersDataIntoShardResult()
      /Users/dswan/go/modules/m3/src/dbnode/storage/bootstrap/bootstrapper/fs/source.go:432 +0xae3
  github.com/m3db/m3/src/dbnode/storage/bootstrap/bootstrapper/fs.(*fileSystemSource).bootstrapFromReaders()
      /Users/dswan/go/modules/m3/src/dbnode/storage/bootstrap/bootstrapper/fs/source.go:324 +0x208
  github.com/m3db/m3/src/dbnode/storage/bootstrap/bootstrapper/fs.(*fileSystemSource).read.func3()
      /Users/dswan/go/modules/m3/src/dbnode/storage/bootstrap/bootstrapper/fs/source.go:868 +0x206

Goroutine 34 (running) created at:
  github.com/m3db/m3/src/dbnode/storage/bootstrap/bootstrapper/fs.(*fileSystemSource).read()
      /Users/dswan/go/modules/m3/src/dbnode/storage/bootstrap/bootstrapper/fs/source.go:867 +0x875
  github.com/m3db/m3/src/dbnode/storage/bootstrap/bootstrapper/fs.(*fileSystemSource).Read()
      /Users/dswan/go/modules/m3/src/dbnode/storage/bootstrap/bootstrapper/fs/source.go:165 +0x624
  github.com/m3db/m3/src/dbnode/storage/bootstrap.(*NamespacesTester).TestReadWith()
      /Users/dswan/go/modules/m3/src/dbnode/storage/bootstrap/util.go:582 +0x1dc
  github.com/m3db/m3/src/dbnode/storage/bootstrap/bootstrapper/fs.validateReadResults()
      /Users/dswan/go/modules/m3/src/dbnode/storage/bootstrap/bootstrapper/fs/source_data_test.go:545 +0x27c
  github.com/m3db/m3/src/dbnode/storage/bootstrap/bootstrapper/fs.TestReadNilTimeRanges()
      /Users/dswan/go/modules/m3/src/dbnode/storage/bootstrap/bootstrapper/fs/source_data_test.go:588 +0x2c7
  testing.tRunner()
      /usr/local/Cellar/go/1.13.5/libexec/src/testing/testing.go:909 +0x199

Goroutine 32 (finished) created at:
  github.com/m3db/m3/src/dbnode/storage/bootstrap/bootstrapper/fs.(*fileSystemSource).read()
      /Users/dswan/go/modules/m3/src/dbnode/storage/bootstrap/bootstrapper/fs/source.go:867 +0x875
  github.com/m3db/m3/src/dbnode/storage/bootstrap/bootstrapper/fs.(*fileSystemSource).Read()
      /Users/dswan/go/modules/m3/src/dbnode/storage/bootstrap/bootstrapper/fs/source.go:165 +0x624
  github.com/m3db/m3/src/dbnode/storage/bootstrap.(*NamespacesTester).TestReadWith()
      /Users/dswan/go/modules/m3/src/dbnode/storage/bootstrap/util.go:582 +0x1dc
  github.com/m3db/m3/src/dbnode/storage/bootstrap/bootstrapper/fs.validateReadResults()
      /Users/dswan/go/modules/m3/src/dbnode/storage/bootstrap/bootstrapper/fs/source_data_test.go:545 +0x27c
  github.com/m3db/m3/src/dbnode/storage/bootstrap/bootstrapper/fs.TestReadNilTimeRanges()
      /Users/dswan/go/modules/m3/src/dbnode/storage/bootstrap/bootstrapper/fs/source_data_test.go:588 +0x2c7
  testing.tRunner()
      /usr/local/Cellar/go/1.13.5/libexec/src/testing/testing.go:909 +0x199
==================
{"level":"info","ts":1614505803.857261,"msg":"bootstrapping time series data success","bootstrapper":"filesystem","cachePolicy":"all","took":0.184990811}
{"level":"info","ts":1614505803.857339,"msg":"bootstrapping index metadata start","bootstrapper":"filesystem"}
{"level":"info","ts":1614505803.948724,"msg":"building in-memory index segment","bootstrapper":"filesystem","namespace":"test_namespace","shouldBuildSegment":true,"noneRemaining":false,"overlapsWithInitialIndexRange":false,"totalEntries":1,"requestedRangesMinMax":"2021-02-28 00:00:00 -0800 PST - 2021-02-28 04:00:00 -0800 PST","remainingRangesMinMax":"2021-02-28 02:00:00 -0800 PST - 2021-02-28 04:00:00 -0800 PST","remainingRanges":"{0: 2h0m0s}","totalFulfilledRangesMinMax":"2021-02-28 00:00:00 -0800 PST - 2021-02-28 02:00:00 -0800 PST","totalFulfilledRanges":"{0: 2h0m0s}","initialIndexRange":"2021-02-26 00:00:00 -0800 PST - 2021-02-26 04:00:00 -0800 PST","shouldFlush":false,"satisfiedFlushRanges":false}
{"level":"info","ts":1614505803.9621031,"msg":"building in-memory index segment","bootstrapper":"filesystem","namespace":"test_namespace","shouldBuildSegment":true,"noneRemaining":false,"overlapsWithInitialIndexRange":false,"totalEntries":1,"requestedRangesMinMax":"2021-02-28 08:00:00 -0800 PST - 2021-02-28 11:00:00 -0800 PST","remainingRangesMinMax":"2021-02-28 08:00:00 -0800 PST - 2021-02-28 10:00:00 -0800 PST","remainingRanges":"{0: 2h0m0s}","totalFulfilledRangesMinMax":"2021-02-28 10:00:00 -0800 PST - 2021-02-28 12:00:00 -0800 PST","totalFulfilledRanges":"{0: 2h0m0s}","initialIndexRange":"2021-02-26 00:00:00 -0800 PST - 2021-02-26 04:00:00 -0800 PST","shouldFlush":false,"satisfiedFlushRanges":false}
{"level":"info","ts":1614505804.0092459,"msg":"bootstrapping index metadata success","bootstrapper":"filesystem","took":0.151868242}
--- FAIL: TestReadNilTimeRanges (0.97s)
    testing.go:853: race detected during execution of test
FAIL
exit status 1
FAIL    github.com/m3db/m3/src/dbnode/storage/bootstrap/bootstrapper/fs 3.224s
@miroswan miroswan mentioned this issue Feb 28, 2021
Merged
@gibbscullen gibbscullen self-assigned this Mar 2, 2021
@gibbscullen
Copy link
Collaborator

@miroswan -- Thanks for looking into this and for the PR contribution! We will look into this and get it merged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@gibbscullen gibbscullen

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[donode] Fix Data Race in checkoutSeriesWithLock during TestReadNilTimeRanges
2 participants
@miroswan @gibbscullen