Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add debug information for queue handler timeouts caused by lingering waituntils #1708

Merged
merged 1 commit into from
Feb 26, 2024

Conversation

jbwcloudflare
Copy link
Contributor

@jbwcloudflare jbwcloudflare commented Feb 21, 2024

@a-robinson and I are trying to add some debug information to help identify the cause of an issue where queue events seem to hang for 15 minutes (the timeout duration).

This info should tell us:

  • Is the user's queue() handler function actually finishing?
  • Are there other waitUntil tasks that are causing the event to run for 15 minutes then timeout?

Copy link
Member

@a-robinson a-robinson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let me know if you could use help testing this out locally!

src/workerd/api/queue.c++ Outdated Show resolved Hide resolved
src/workerd/api/queue.c++ Outdated Show resolved Hide resolved
src/workerd/api/queue.c++ Outdated Show resolved Hide resolved
src/workerd/api/queue.c++ Outdated Show resolved Hide resolved
Copy link
Member

@a-robinson a-robinson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a couple nits, but LGTM otherwise.

src/workerd/api/queue.c++ Show resolved Hide resolved
src/workerd/api/queue.c++ Outdated Show resolved Hide resolved
@jbwcloudflare jbwcloudflare changed the title WIP: add debug information for queue handler timeouts caused by lingering waituntils Add debug information for queue handler timeouts caused by lingering waituntils Feb 23, 2024
@jbwcloudflare
Copy link
Contributor Author

Tested this out with some local ew-test-bins, logs are looking good:

1m10	4c6539480 warning workerd/api/queue.c++:556 NOSENTRY queue event timed out; status = Incomplete; tasks = task: src/edgeworker/server/edgeworker@64c3f20|task: src/edgeworker/server/edgeworker@64c3f20|task: src/edgeworker/server/edgeworker@64c3f20 src/edgeworker/server/edgeworker@83e6f10; ownerId = 1212; zoneId = zone; scriptId = exceeded-cpu-queue-handler; cordon = paid; process = rpc

1m10	4c6539480 warning workerd/api/queue.c++:556 NOSENTRY queue event timed out; status = Completed Succesfully; tasks = task: src/edgeworker/server/edgeworker@64c3f20|task: src/edgeworker/server/edgeworker@64c3f20; ownerId = 1212; zoneId = zone; scriptId = exceeded-cpu-queue-handler; cordon = paid; process = rpc

1m10	4c6539480 warning workerd/api/queue.c++:556 NOSENTRY queue event timed out; status = Completed with error:workerd/jsg/_virtual_includes/jsg/workerd/jsg/value.h:1334: failed: jsg.Error: sorry|stack: src/edgeworker/server/edgeworker@83e6f10; tasks = task: src/edgeworker/server/edgeworker@64c3f20|task: src/edgeworker/server/edgeworker@64c3f20; ownerId = 1212; zoneId = zone; scriptId = exceeded-cpu-queue-handler; cordon = paid; process = rpc

@jbwcloudflare
Copy link
Contributor Author

@dom96 / @jasnell this is ready for review when you have time 🙏

@jbwcloudflare
Copy link
Contributor Author

bump, sorry to bug you guys @jasnell / @dom96, could you take a look please? 🙏

@a-robinson a-robinson merged commit 5f07bb7 into cloudflare:main Feb 26, 2024
10 of 11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants