Coordinator dies with nil pointer exception when sending remote_write before initialization

[juliusv]

I was trying out M3 using https://github.com/m3db/m3/blob/master/scripts/development/m3_stack/start_m3.sh and tried to remove the aggregated second namespace by just removing these lines:

m3/scripts/development/m3_stack/start_m3.sh

Lines 208 to 243 in dc6eadf

	curl -vvvsSf -X POST localhost:7201/api/v1/services/m3db/namespace -d '{
	"name": "metrics_30s_24h",
	"options": {
	"bootstrapEnabled": true,
	"flushEnabled": true,
	"writesToCommitLog": true,
	"cleanupEnabled": true,
	"snapshotEnabled": true,
	"repairEnabled": false,
	"retentionOptions": {
	"retentionPeriodDuration": "24h",
	"blockSizeDuration": "2h",
	"bufferFutureDuration": "10m",
	"bufferPastDuration": "10m",
	"blockDataExpiry": true,
	"blockDataExpiryAfterNotAccessPeriodDuration": "5m"
	},
	"indexOptions": {
	"enabled": true,
	"blockSizeDuration": "2h"
	},
	"aggregationOptions": {
	"aggregations": [
	{
	"aggregated": true,
	"attributes": {
	"resolutionDuration": "30s"
	}
	}
	]
	},
	"stagingState": {
	"status": "INITIALIZING"
	}
	}
	}'

But I didn't notice that I also had to remove the subsequent lines that checked for validity and readiness of that namespace, and thus the script got stuck and the coordinator never got fully initialized.

I was not aware of that and pointed a Prometheus server at it to send remote_write requests, and the coordinator died with:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x1104820]

goroutine 8686 [running]:
github.com/m3db/m3/src/query/storage/m3.(*m3storage).writeSingle(0xc008e4a210, 0x1cbda00, 0xc009c69e60, 0xc0084779a0, 0x1c5f2f00, 0xed7530e19, 0x27d5a60, 0x0, 0x1ccbcc0, 0xc00b3eabc0, ...)
/home/julius/tmp/m3/src/query/storage/m3/storage.go:720 +0x120
github.com/m3db/m3/src/query/storage/m3.(*m3storage).Write(0xc008e4a210, 0x1cbda00, 0xc009c69e60, 0xc0084779a0, 0x1ca2900, 0xc0084779a0)
/home/julius/tmp/m3/src/query/storage/m3/storage.go:635 +0x7ba
github.com/m3db/m3/src/query/storage/fanout.(*fanoutStorage).Write(0xc008613360, 0x1cbda00, 0xc009c69e60, 0xc0084779a0, 0x7, 0x0)
/home/julius/tmp/m3/src/query/storage/fanout/storage.go:422 +0x2ef
github.com/m3db/m3/src/cmd/services/m3coordinator/ingest.(*downsamplerAndWriter).WriteBatch.func2()
/home/julius/tmp/m3/src/cmd/services/m3coordinator/ingest/write.go:416 +0x274
github.com/m3db/m3/src/x/sync.(*pooledWorkerPool).spawnWorker.func1(0xc005d93500, 0x0, 0x957f1712957f1712, 0xc000de8fc0, 0x1a02501)
/home/julius/tmp/m3/src/x/sync/pooled_worker_pool.go:177 +0x18a
created by github.com/m3db/m3/src/x/sync.(*pooledWorkerPool).spawnWorker
/home/julius/tmp/m3/src/x/sync/pooled_worker_pool.go:156 +0x66

I assume the coordinator shouldn't die when sending a remote_write request before it is initialized.

The coordinator Docker image is tagged dev / with image ID 54a4f1377d80.

[juliusv]

Btw., in this half-initialized state the PromQL HTTP API also returns nil pointer exceptions when sending queries to it.