Why when using sandbox processors, the concurrency is not working

[geosigno]

I have this below code

if (process.env.NODE_ENV === "production") {
  worker = new Worker(
    "transcoder-queue",
    path.join(__dirname, "worker/index.js"),
    {
      connection,
      // stalledInterval: 1000 * 60 * 60,
      lockDuration: 1000 * 60 * 30,
      concurrency: MAX_CONCURRENCY,
    }
  );
} else {
  worker = new Worker(
    "transcoder-queue",
    async (job: JobCustom) => {
      const jobType = job.name;
      switch (jobType) {
        case "getAudioLow":
          await processAudio(job);
          break;
        case "getAudioHigh":
          await processAudio(job);
          break;
        case "getMediaInfos":
          await processMediaInfos(job);
          break;
        case "getThumbnails":
          await processThumbnails(job);
          break;
        case "getVideo360":
          await processVideo(job);
          break;
        case "getVideo480":
          await processVideo(job);
          break;
        case "getVideo720":
          await processVideo(job);
          break;
        case "getVideo1080":
          await processVideo(job);
          break;
        case "getClip":
          await processClip(job);
          break;
        default:
          throw new Error(`Unknown job type: ${jobType}`);
      }
    },
    {
      connection,
      // stalledInterval: 1000 * 60 * 60,
      lockDuration: 1000 * 60 * 30,
      concurrency: MAX_CONCURRENCY,
    }
  );
}

I'm encountering a problem with my job processing setup. In my local environment, I'm not using the sandbox processor and the worker is processing jobs with the MAX_CONCURRENCY set to 3. However, on production, I want to use the sandbox processor because I need to encode very heavy videos. The issue is that when I switch to the sandbox processor, it takes all the jobs at the same time without considering the concurrency setting.

Any idea why?

[ShootingStar91]

I'm having the exact same problem. Did you ever get this resolved?

[manast]

This is most likely the result of a processor that is not properly returning a promise. We have tests for sandboxed concurrency that shows it is working. You are welcome to produce a simple test that shows this behavior and we will certainly look deeper into it.

[ShootingStar91]

I think I found the cause. It was that I had a circular dependency which was not recognized and warned about, because of the way that the processor file is imported (not imported traditionally but used runtime, as far as I understand). So, the processor file which is called by worker in file "worker" to be a sandboxed processor, imported file "functions", which imported the file "worker". When I replicated this setting in a minimum example app, the problem happened only if the circular import was there, but disappeared when I removed that.

So the problem is resolved for me. I guess this is not a "bug" since there probably shouldn't be circular dependencies. If it's not expected behaviour, I'm happy to deliver my minimal example for this.

I made an issue before noticing there were discussions, I closed it now: #2317