Graceful handling of low memory states

[NathanSavageUoP]

Is your feature request related to a problem? Please describe.
Im using bullmq to schedule and handle web data scraping for a project. It involves downloading html pages as well as binary document types such as DOCX and PDF, running a feature extraction job against them, shrinking and compressing any images and then finally saving them to a DB. Recently I have been having trouble with running out of memory in the JS heap and in redis with jobs with large payloads or return values.

Describe the solution you'd like
There are a few ways i can think of to improve handling of large payloads and specificly buffers.

1. Provide an option to delegate loading the job payload. This would allow the job to perform custom resource availability checks before loading the payload.
2. Store buffers more effeciently in redis. Currently, buffers in payloads or results are stored in a CSV array of string numbers between 0 and 255. I can see why this was done to make it JSON compliant and easy to decode but this is quite wasteful. Redis supports storing binary data natively. In a test i did, downloading a 1.1mb pdf, the return value ended up taking up 5.58mb in redis.
3. Handle OOM errors gracefully. In the program that was having issues, I was using sandboxed processors. I was wondering it if would be possible in a worst case senario for these processors to crash without taking out the main program with them? This is probably quite out of scope though.

Describe alternatives you've considered
Im going to implement a "pass by reference" type system for the binary data im passing between jobs. this should solve issues 1 and 2 hopefully. Im also going to implement a system that pauses particular queues based on the amount of available JS and redis memory and unpauses them once the jobs have completed and memory consumption gone down again.

[manast]

Yes, in general you should not store big chunks of data in the jobs as RAM is much more expensive than disk, so you should offload the heavy elements to a different storage system or disk based database. Regarding 3., do you have a callstack where this happens, in theory if you handle all the errors appropriately it should not crash the process, also there are some hints here that may be useful for you: https://docs.bullmq.io/guide/going-to-production and specifically this one: https://docs.bullmq.io/guide/going-to-production#unhandled-exceptions-and-rejections

[NathanSavageUoP]

Hi Manast,

Thanks for those links, I have had a look through them. I will try and put together a minimal repo for point three. I have also put together a tiny demo for points one and two

import { Queue, Worker, Job } from "bullmq";
import { randomUUID } from "crypto";
import Redis from 'ioredis'

const redis = new Redis({ maxRetriesPerRequest: null })
const demoQueue = new Queue('demo', { connection: redis })

export async function queueJob(src: Buffer) {
    const payloadKeyUUID = randomUUID()
    const job = await demoQueue.add('demoJob', {payloadKey: payloadKeyUUID})
     // saving large buffer under a different key in the same hash table
    await redis.hset(`${job.queueQualifiedName}:${job.id!}`, payloadKeyUUID, src)
}

const demoWorker = new Worker('demo', 
    async (job: Job<{ payloadKey: string }>) => {
        //retrieving the buffer back
        const src = await redis.hgetBuffer(`${job.queueQualifiedName}:${job.id!}`, job.data.payloadKey) as Buffer
    },
{ connection: redis })

I tested this with a large PDF with an original size of 26.4MB. The default bullmq payload took up nearly 97MB

wheras storing it as a binary string didnt introduce any overhead

I dont want to commit the files to disk as I only need them transitively untill the feature extraction job is performed at which point they can be discarded.