Vercel add Job : TypeError: client.addJob is not a function #469

ghost · 2021-04-13T18:22:35Z

Thanks for developing this open source. I really enjoy it.
I have encountered this problem when running on vercel serverless.
The source code works perfectly fine on local.

{
  "errorType": "Runtime.UnhandledPromiseRejection",
  "errorMessage": "TypeError: client.addJob is not a function",
  "reason": {
    "errorType": "TypeError",
    "errorMessage": "client.addJob is not a function",
    "stack": [
      "TypeError: client.addJob is not a function",
      " at Function.addJob (/var/task/node_modules/bullmq/dist/classes/scripts.js:52:23)",
      " at Job.addJob (/var/task/node_modules/bullmq/dist/classes/job.js:352:34)",
      " at Function.create (/var/task/node_modules/bullmq/dist/classes/job.js:34:28)",
      " at processTicksAndRejections (internal/process/task_queues.js:93:5)",
      " at Queue.add (/var/task/node_modules/bullmq/dist/classes/queue.js:38:25)"
    ]
  },
  "promise": {},
  "stack": [
    "Runtime.UnhandledPromiseRejection: TypeError: client.addJob is not a function",
    " at process.<anonymous> (/var/runtime/index.js:35:15)",
    " at process.emit (events.js:327:22)",
    " at process.emit (/var/task/___vc_sourcemap_support.js:587:21)",
    " at processPromiseRejections (internal/process/promises.js:245:33)",
    " at processTicksAndRejections (internal/process/task_queues.js:94:32)"
  ]
}

Example Code:

const connection = new IORedis(process.env.REDIS_URI);
const queue = new Queue('SENDEMAIL', { connection });
const job = await queue.add('SEND',  { email });

I found a workaround for this while customizing the vercel.json file. Hope to be able to help anyone who is having this problem.
    "functions": {
        "api/serverless.ts": {
            "includeFiles": "node_modules/bullmq/dist/commands/**",
        }
    },

The text was updated successfully, but these errors were encountered:

krinoid · 2021-04-13T19:33:19Z

I'm encountering the same error (works locally, fails on Vercel). This issue seems somehow related: #79

krinoid · 2021-04-13T20:37:28Z

I've done some research, and my guess right now is that the *.lua files are not bundled and can't be used in the Vercel's serverless environment.

I've tried to use vercel.json config with includeFiles as suggested in [0], but since I'm using redwood.js, this avenue won't work, as includeFiles won't work, per Vercel's own reply [1].

I'll continue looking for a solution/workaround later in the week.

Related:
[0] OptimalBits/bull#1418
[1] redwoodjs/redwood#1664

manast · 2021-04-13T21:09:26Z

I do not know how vercel works but my feeling is that the deployment is missing the lua files: https://github.com/taskforcesh/bullmq/tree/master/src/commands
which also is placed on the dist directory of the npm module at /bullmq/dist/commands
If those lua scripts are in the correct place when any of BullMQ main classes are instantiated they will be loaded and you should not get any error.

krinoid · 2021-04-13T21:17:17Z

I do not know how vercel works but my feeling is that the deployment is missing the lua files: https://github.com/taskforcesh/bullmq/tree/master/src/commands
which also is placed on the dist directory of the npm module at /bullmq/dist/commands
If those lua scripts are in the correct place when any of BullMQ main classes are instantiated they will be loaded and you should not get any error.

Yes, I think that's correct. I've tried different workarounds but had issues including the lua files along the functions. For now, I've downgraded from bullmq to bull and it works as expected on Vercel.

ghost · 2021-04-14T06:06:08Z

I do not know how vercel works but my feeling is that the deployment is missing the lua files: https://github.com/taskforcesh/bullmq/tree/master/src/commands
which also is placed on the dist directory of the npm module at /bullmq/dist/commands
If those lua scripts are in the correct place when any of BullMQ main classes are instantiated they will be loaded and you should not get any error.

Yes, I think that's correct. I've tried different workarounds but had issues including the lua files along the functions. For now, I've downgraded from bullmq to bull and it works as expected on Vercel.

https://github.com/OptimalBits/bull/tree/develop/lib/commands
https://github.com/taskforcesh/bullmq/tree/master/src/commands

They all use lua files, why bull works but bullmq doesn't?

ghost · 2021-04-14T08:36:27Z

I found a workaround for this while customizing the vercel.json file. Hope to be able to help anyone who is having this problem.

    "functions": {
        "api/serverless.ts": {
            "includeFiles": "node_modules/bullmq/dist/commands/**",
        }
    },

krinoid · 2021-04-14T17:39:38Z

I do not know how vercel works but my feeling is that the deployment is missing the lua files: https://github.com/taskforcesh/bullmq/tree/master/src/commands
which also is placed on the dist directory of the npm module at /bullmq/dist/commands
If those lua scripts are in the correct place when any of BullMQ main classes are instantiated they will be loaded and you should not get any error.

Yes, I think that's correct. I've tried different workarounds but had issues including the lua files along the functions. For now, I've downgraded from bullmq to bull and it works as expected on Vercel.

https://github.com/OptimalBits/bull/tree/develop/lib/commands
https://github.com/taskforcesh/bullmq/tree/master/src/commands

They all use lua files, why bull works but bullmq doesn't?

Yeah, that's very strange! I don't have enough time to dive deep into the issue but that seems like a useful data point during debugging.

I've seen vercel.json-based workaround in some other issue but it doesn't work in every case (e.g. when you're using redwood) 👈 for those in the same boat as me :).

prescience-data · 2021-04-17T07:48:48Z

@manast @naicoi2407
For additional context, this also occurs when bundling with esbuild

I've found that when this occurs it is either due to:
a) a circular dependency (which can be ferreted out with madge), or;
b) a lazy-load style require

BullMQ (and Bull) are the final package I have not been able to bundle in my stack.

The bundling process generally always completes, but when you run the application you receive:

TypeError: client.addJob is not a function

prescience-data · 2021-04-17T07:57:47Z

@manast

Just did a quick check, if it's possible to repair these, it will likely fix this issue:

$ madge  --circular dist/index.js

manast · 2021-04-17T08:02:32Z

repair what? I do not get the problem.

prescience-data · 2021-04-17T08:06:56Z

When a bundler runs (ie webpack, rollup, esbuild), either directly or as Vercel must also be doing in the background, it cannot handle / resolve circular dependencies so it creates these orphaned stubs that register as undefined at runtime.

As an example of some on that list, these are the root cause:
https://github.com/taskforcesh/bullmq/blob/master/src/classes/job.ts#L6
https://github.com/taskforcesh/bullmq/blob/master/src/classes/queue-scheduler.ts#L3
https://github.com/taskforcesh/bullmq/blob/master/src/classes/repeat.ts#L3
https://github.com/taskforcesh/bullmq/blob/master/src/classes/queue.ts#L5
https://github.com/taskforcesh/bullmq/blob/master/src/classes/worker.ts#L5

(In fact, looking at that list, the above should solve most of them).

If interested in making sure the package is able to be bundled, I have found it super useful to run madge in the CI build loop to immediately detect problems.
ie

{
  "scripts": {
    "madge": "madge --orphans -j dist/index.js && madge --circular -j dist/index.js"
   }
}

or

      - name: Release
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          NPM_TOKEN: ${{ secrets.NPM_TOKEN }}
        run: |
          yarn build
          npx madge --orphans -j dist/index.js && madge --circular -j dist/index.js
          npx semantic-release

manast · 2021-04-17T14:44:13Z

hmm, I cannot see the circular dependency, for example in the first case: https://github.com/taskforcesh/bullmq/blob/master/src/classes/job.ts#L6
where is the circular dependency?

prescience-data · 2021-04-17T16:51:29Z

hmm, I cannot see the circular dependency, for example in the first case: https://github.com/taskforcesh/bullmq/blob/master/src/classes/job.ts#L6
where is the circular dependency?

It is occuring when the code path attempts to import from "./" (shorthand for "./index.ts") as the path for "job.ts" which is importing from "./" is itself exported from "./"

If you switch that line to import each class from it's own relative path it will remove the circular reference.

Eg

import { Backoffs } from "./backoffs"
import { QueueEvents } from "./queue-events"

manast · 2021-04-17T19:27:32Z

ok, but thats not circular dependencies, thats sounds more like a bug in webpack that is not handling file references correctly. Nevertheless we can do that change to workaround the problem.

prescience-data · 2021-04-17T21:36:42Z

https://stackoverflow.com/a/49117079
(it's due to the order of the exports, ie when it flattens the export list, the packages that are erroring have not been required yet)

## [1.20.4](v1.20.3...v1.20.4) (2021-04-23) ### Bug Fixes * remove internal deps on barrel fixes [#469](#469) ([#495](#495)) ([60dbeed](60dbeed))

github-actions · 2021-04-23T16:40:24Z

🎉 This issue has been resolved in version 1.20.4 🎉

The release is available on:

Your semantic-release bot 📦🚀

gendronb · 2021-05-04T18:58:09Z

I had the exact same problem (NextJS API routes on Vercel), and tried to upgrade to 1.20.4 as mentioned, but the release does not solve the issue at all. The workaround (custom vercel.json) does not work either.

[GET] /api/test/5 14:55:11:74 2021-05-04T18:55:11.806Z d44e978c-1c74-4f9c-84d6-d358912702cf ERROR Unhandled Promise Rejection {"errorType":"Runtime.UnhandledPromiseRejection","errorMessage":"TypeError: client.addJob is not a function","reason":{"errorType":"TypeError","errorMessage":"client.addJob is not a function","stack":["TypeError: client.addJob is not a function"," at Function.addJob (/var/task/node_modules/bullmq/dist/classes/scripts.js:54:23)"," at Job.addJob (/var/task/node_modules/bullmq/dist/classes/job.js:378:34)"," at Function.create (/var/task/node_modules/bullmq/dist/classes/job.js:36:28)"," at processTicksAndRejections (internal/process/task_queues.js:93:5)"," at async Queue.add (/var/task/node_modules/bullmq/dist/classes/queue.js:40:25)"]},"promise":{},"stack":["Runtime.UnhandledPromiseRejection: TypeError: client.addJob is not a function"," at process.<anonymous> (/var/runtime/index.js:35:15)"," at process.emit (events.js:327:22)"," at processPromiseRejections (internal/process/promises.js:245:33)"," at processTicksAndRejections (internal/process/task_queues.js:94:32)"]} Unknown application error occurred

manast · 2021-05-04T22:20:52Z

@gendronb I am not a vercel user, but the error as mentioned above is due .lua files not being copied correctly, the workaround should work but you can check if the files were copied or not to the correct place.

gendronb · 2021-05-05T12:27:15Z

@manast thanks for your quick reply, I have reached the same conclusion (missing lua files)... I will to ask Vercel for help, as these files are not present in the deployed files.

gendronb · 2021-05-05T12:28:16Z

Have to mention that I'm using NextJS wirth Vercel, so it may be a situation particular to NextJS...

krinoid · 2021-05-07T12:17:10Z

@manast thanks for your quick reply, I have reached the same conclusion (missing lua files)... I will to ask Vercel for help, as these files are not present in the deployed files.

@gendronb Have you tried using the previous version (bull vs bullmq)? I'd encourage you to just try with a single function. I know it's not a proper solution but if you're on a schedule, it might help you in the short term.

Also, if you'll hear back from Vercel on that issue, I'd appreciate if you could repost the response here, I'm also interested. Thanks!

gendronb · 2021-05-07T19:06:13Z

@gendronb Have you tried using the previous version (bull vs bullmq)? I'd encourage you to just try with a single function. I know it's not a proper solution but if you're on a schedule, it might help you in the short term.

Also, if you'll hear back from Vercel on that issue, I'd appreciate if you could repost the response here, I'm also interested. Thanks!

@manast will do (downgrade to bull) and keep you informed, thanks again for your help!

Kounda-Dartagnan · 2021-05-21T09:59:51Z

I had the same issue with esbuild. I solved it by declaring bullmq as external
esbuild --external:bullmq

ctrlaltdylan · 2022-02-04T02:44:00Z

Confirming this workaround for bull on Next.js v11:

vercel.json

    "functions": {
        "pages/api/**/*.js": {
            "includeFiles": "node_modules/bull/lib/commands/**",
        }
    },

Why this works? Have little to no idea.

I had 0 issues with Heroku running NextJS with Bull for over a year, but Vercel's build must be more strict and it's parsing out these Lua files.

To make matters worse, I didn't get a nice error message in the Vercel logs at all.

Sorry, this problem was a tough one. Spent too much time on it.

manast · 2022-02-04T09:17:37Z

@ctrlaltdylan any chances to report this issue to the Vercel team?

gendronb · 2022-02-04T15:28:51Z

@gendronb Have you tried using the previous version (bull vs bullmq)? I'd encourage you to just try with a single function. I know it's not a proper solution but if you're on a schedule, it might help you in the short term.

@manast sorry for taking so long to reply...

I did downgrade to bull (v4.5), and it's working perfectly on Vercel (with Next 11) without any workaround.

ctrlaltdylan · 2022-02-04T19:12:35Z

@ctrlaltdylan any chances to report this issue to the Vercel team?

Yes will do.

ctrlaltdylan · 2022-02-04T19:13:42Z

Weird @

@gendronb Have you tried using the previous version (bull vs bullmq)? I'd encourage you to just try with a single function. I know it's not a proper solution but if you're on a schedule, it might help you in the short term.

@manast sorry for taking so long to reply...

I did downgrade to bull (v4.5), and it's working perfectly on Vercel (with Next 11) without any workaround.

Weird, I'm using bull: ^4.5.0 myself with Next 11 on Vercel and it wouldn't add jobs without this lua scripts inclusion workaround.

gendronb · 2022-02-04T19:42:51Z

Weird, I'm using bull: ^4.5.0 myself with Next 11 on Vercel and it wouldn't add jobs without this lua scripts inclusion workaround.

@ctrlaltdylan Weird indeed...

My repo is https://github.com/gendronb/next-bullmq/tree/bull, very simple, a single API route (GET /api/jobs/add) adding a job to a Upstash-backed REDIS database, without any kind of post-processing.

NOTE : use the bull branch, no the master branch.

Actual deployment: https://next-bullmq-a6hrefhj5-gendronb.vercel.app/api/jobs/add

I can see the jobs being added in the Upstash console.

ctrlaltdylan · 2022-02-04T20:53:03Z

Yup I did something similar in my app. I would get a 200 response, I await'd the queue.addJob, but alas no actual job added to the queue until I did that lua script inclusion update. Very odd.

styfle · 2022-02-10T19:33:18Z

The issue is this dynamic readdir() that loads the lua files:

bullmq/src/commands/script-loader.ts

Line 395 in 86e84df

const files = await readdir(dir);

If this could be changed to const files = await readdir(__dirname) then it can be static analyzed at build time since the __dirname is known at build time.

manast · 2022-02-13T15:57:38Z

If this could be changed to const files = await readdir(__dirname) then it can be static analyzed at build time since the __dirname is known at build time.

This cannot be changed currently because loadScripts accepts a "dir" as argument. It is unfortunate that vercel does not support to manually specify a dir and then it could just copy those .lua files to the correct place.

barbinbrad · 2023-03-02T02:54:10Z

Weird, I'm using bull: ^4.5.0 myself with Next 11 on Vercel and it wouldn't add jobs without this lua scripts inclusion workaround.

@ctrlaltdylan Weird indeed...

My repo is https://github.com/gendronb/next-bullmq/tree/bull, very simple, a single API route (GET /api/jobs/add) adding a job to a Upstash-backed REDIS database, without any kind of post-processing.

NOTE : use the bull branch, no the master branch.

Actual deployment: https://next-bullmq-a6hrefhj5-gendronb.vercel.app/api/jobs/add

I can see the jobs being added in the Upstash console.

I had a similar error, and similarly wound up in this thread. But my problem was actually that Upstash doesn't support bullmq -- and this was causing the serverless function to crash. There wasn't any error being logged in Vercel, but as soon as I changed the environment variable to point to RedisLabs, the problem went away.

manast · 2023-03-02T13:14:26Z

Upstash is going to be compatible very soon, when they support Redis Streams which is almost ready.

manast added a commit that referenced this issue Apr 23, 2021

fix: remove internal deps on barrel fixes #469

d71ce35

manast closed this as completed in 60dbeed Apr 23, 2021

github-actions bot pushed a commit that referenced this issue Apr 23, 2021

chore(release): 1.20.4 [skip ci]

9cf71f6

## [1.20.4](v1.20.3...v1.20.4) (2021-04-23) ### Bug Fixes * remove internal deps on barrel fixes [#469](#469) ([#495](#495)) ([60dbeed](60dbeed))

github-actions bot added the released label Apr 23, 2021

voronin-ivan mentioned this issue Feb 5, 2022

Tracing .lua files vercel/nft#265

Closed

roggervalf mentioned this issue Feb 10, 2022

Feat transform commands #958

Closed

styfle mentioned this issue Feb 23, 2022

chore: add test for bullmq vercel/nft#271

Merged

roggervalf mentioned this issue Oct 11, 2022

perf(scripts): pre-build scripts #1441

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Vercel add Job : TypeError: client.addJob is not a function #469

Vercel add Job : TypeError: client.addJob is not a function #469

ghost commented Apr 13, 2021 •

edited by ghost

Loading

krinoid commented Apr 13, 2021

krinoid commented Apr 13, 2021

manast commented Apr 13, 2021

krinoid commented Apr 13, 2021

ghost commented Apr 14, 2021

ghost commented Apr 14, 2021

krinoid commented Apr 14, 2021

prescience-data commented Apr 17, 2021 •

edited

Loading

prescience-data commented Apr 17, 2021

manast commented Apr 17, 2021

prescience-data commented Apr 17, 2021 •

edited

Loading

manast commented Apr 17, 2021

prescience-data commented Apr 17, 2021 •

edited

Loading

manast commented Apr 17, 2021

prescience-data commented Apr 17, 2021 •

edited

Loading

github-actions bot commented Apr 23, 2021

gendronb commented May 4, 2021

manast commented May 4, 2021 •

edited

Loading

gendronb commented May 5, 2021

gendronb commented May 5, 2021

krinoid commented May 7, 2021

gendronb commented May 7, 2021

Kounda-Dartagnan commented May 21, 2021 •

edited

Loading

ctrlaltdylan commented Feb 4, 2022 •

edited

Loading

manast commented Feb 4, 2022

gendronb commented Feb 4, 2022 •

edited

Loading

ctrlaltdylan commented Feb 4, 2022

ctrlaltdylan commented Feb 4, 2022

gendronb commented Feb 4, 2022 •

edited

Loading

ctrlaltdylan commented Feb 4, 2022

styfle commented Feb 10, 2022 •

edited

Loading

manast commented Feb 13, 2022

barbinbrad commented Mar 2, 2023 •

edited

Loading

manast commented Mar 2, 2023

Vercel add Job : TypeError: client.addJob is not a function #469

Vercel add Job : TypeError: client.addJob is not a function #469

Comments

ghost commented Apr 13, 2021 • edited by ghost Loading

krinoid commented Apr 13, 2021

krinoid commented Apr 13, 2021

manast commented Apr 13, 2021

krinoid commented Apr 13, 2021

ghost commented Apr 14, 2021

ghost commented Apr 14, 2021

krinoid commented Apr 14, 2021

prescience-data commented Apr 17, 2021 • edited Loading

prescience-data commented Apr 17, 2021

manast commented Apr 17, 2021

prescience-data commented Apr 17, 2021 • edited Loading

manast commented Apr 17, 2021

prescience-data commented Apr 17, 2021 • edited Loading

manast commented Apr 17, 2021

prescience-data commented Apr 17, 2021 • edited Loading

github-actions bot commented Apr 23, 2021

gendronb commented May 4, 2021

manast commented May 4, 2021 • edited Loading

gendronb commented May 5, 2021

gendronb commented May 5, 2021

krinoid commented May 7, 2021

gendronb commented May 7, 2021

Kounda-Dartagnan commented May 21, 2021 • edited Loading

ctrlaltdylan commented Feb 4, 2022 • edited Loading

manast commented Feb 4, 2022

gendronb commented Feb 4, 2022 • edited Loading

ctrlaltdylan commented Feb 4, 2022

ctrlaltdylan commented Feb 4, 2022

gendronb commented Feb 4, 2022 • edited Loading

ctrlaltdylan commented Feb 4, 2022

styfle commented Feb 10, 2022 • edited Loading

manast commented Feb 13, 2022

barbinbrad commented Mar 2, 2023 • edited Loading

manast commented Mar 2, 2023

ghost commented Apr 13, 2021 •

edited by ghost

Loading

prescience-data commented Apr 17, 2021 •

edited

Loading

prescience-data commented Apr 17, 2021 •

edited

Loading

prescience-data commented Apr 17, 2021 •

edited

Loading

prescience-data commented Apr 17, 2021 •

edited

Loading

manast commented May 4, 2021 •

edited

Loading

Kounda-Dartagnan commented May 21, 2021 •

edited

Loading

ctrlaltdylan commented Feb 4, 2022 •

edited

Loading

gendronb commented Feb 4, 2022 •

edited

Loading

gendronb commented Feb 4, 2022 •

edited

Loading

styfle commented Feb 10, 2022 •

edited

Loading

barbinbrad commented Mar 2, 2023 •

edited

Loading