Using Scan: Cannot read property '0' of undefined

[toddbluhm]

We run a bunch of worker processes that use redis to store some temporary state and we use scan streams to pull out any remaining redis records that might still remain in redis and process those before terminating the worker. We run this worker about 1-2k times a week and I have noticed that a couple of our workers are failing with this error being thrown by bluebird and leading back into ioredis scan streaming code. Here is the full error:

/mnt/task/node_modules/bluebird/js/main/async.js:43
        fn = function () { throw arg; };
                           ^

TypeError: Cannot read property '0' of undefined
    at /mnt/task/node_modules/ioredis/lib/scan_stream.js:37:30
    at tryCatcher (/mnt/task/node_modules/bluebird/js/main/util.js:26:23)
    at Promise.errorAdapter (/mnt/task/node_modules/bluebird/js/main/nodeify.js:36:34)
    at Promise._settlePromiseAt (/mnt/task/node_modules/bluebird/js/main/promise.js:579:21)
    at Promise._settlePromises (/mnt/task/node_modules/bluebird/js/main/promise.js:697:14)
    at Async._drainQueue (/mnt/task/node_modules/bluebird/js/main/async.js:123:16)
    at Async._drainQueues (/mnt/task/node_modules/bluebird/js/main/async.js:133:10)
    at Immediate.Async.drainQueues [as _onImmediate] (/mnt/task/node_modules/bluebird/js/main/async.js:15:14)
    at processImmediate [as _immediateCallback] (timers.js:368:17)

The piece of ioredis code that it is pointing to is here:

var _this = this;
this.opt.redis[this.opt.command](args, function (_, res) {
  _this._redisCursor = (res[0] instanceof Buffer) ? res[0].toString() : res[0];
  if (_this._redisCursor === '0') {
    _this._redisDrained = true;
  }
  _this.push(res[1]);
});

The error is occurring because I believe res is either undefined or is not an array so it can't be accessed by using [0]. I believe this is probably some sort of race condition maybe or something weird like that because it only happens occasionally (3 out of 1-2k) and when the system auto retries those workers after a failure it always succeeds (so not easily reproducible).

My thought is maybe we should be checking to see if res exists first before doing the [0] index, but I didn't write this lib so maybe res should never be null and this should be handled elsewhere outside the code.

Any help or clarity on this issue would be much appreciated.

[luin]

I think the problem is ioredis doesn't handle the error. res will be undefined when error occurs. I'll handle it in next version. It is possible to log the error in your code so I can know why the command is failed? Thank you!

this.opt.redis[this.opt.command](args, function (_, res) {
  console.log('ScanStream receives an error', _);

[toddbluhm]

Yeah I will certainly do that. I will let you know next Monday if I run into any errors with this, and what those errors are. We run the workers over the weekend so I won't know before then.

[toddbluhm]

Okay so here is the error that I am getting. I received this error 2 times out of all the jobs we ran +2k. Again the retry on the jobs worked just fine.

ScanStream receives an error [Error: Stream isn't writeable and enableOfflineQueue options is false]

Note: I do have offline queue set to false as I don't want to queue any data up (our worker has a hard ram limit and its small so I can't afford to buffer/queue very many items, or it will force kill the worker immediately).

[luin]

This error will be emitted when the connection is down and enableOfflineQueue is false. I think the way to handle this is to use another Redis instance to do the scanning and enable the offline queue.

[toddbluhm]

Yeah sounds good. Just let me know when you have this error getting forwarded to the error handler so I can catch it and then do as you suggested 😄

[luin]

Released in 1.12.1 😆

var stream = redis.scanstream({ count: 1 });
stream.on('data', function (data) {
});
stream.on('error', function (err) {
});

[toddbluhm]

Thanks!