Improve Webserver Response Under Load

[anthonylavado]

Hi all!
We (the company I work for) are currently testing this chart, and ran into some (small) issues.

When we're running a bunch of DAGs, the webserver can occasionally become unresponsive, and fail it's livenessprobe, thus causing it to be restarted, which fails the readinessprobe, and it just ends up in a crash loop. Obviously if I disable those probes it seems to be okay, it just takes a much longer time for things to load.

What I'm wondering is what I can do to improve the webserver's response when the node is running all these other tasks. Should I be setting up a web replica? Should I be setting resource requests for the web pod? I apologize if this is a little k8s 101, but I figure there may be suggestions here that will be a little more relevant.

Some details:

• Running on Azure AKS
• Have two workers running
• Using CeleryExecutor
• Using external Postgres, embedded Redis, and embedded PGBouncer
• Chart Version: 8.5.3

[thesuperzapper]

@anthonylavado it's hard to know what is causing the webserver to become unresponsive without more context.

Questions:

• What version of airflow are you running?
• How many people are accessing the webserver?
• Are you using any of the REST APIs provided by the webserver (to run dags for example)?
• Are you using a PVC to store logs?
• Are your worker Pods running on the same nodes as your webserver pods (if so, this indicates resource starvation and the need for larger resource requests).

Answers:

• Yes, always set resource requests for pods (otherwise K8S will de-prioritize those pods when allocating resources)
• Yes, increasing the replica count of the webserver will improve performance (but only if the bottleneck is too many connections to the webserver, which seems unlikely from your description)

[anthonylavado]

Hey! Thanks for responding.

* What version of airflow are you running? 
* How many people are accessing the webserver?
* Are you using any of the REST APIs provided by the webserver (to run dags for example)?

It's running version 2.2.4. It's likely two people at most, but it is when they kick off around 15k runs. The REST APIs don't appear to be in use at this time.

* Are you using a PVC to store logs?
* Are your worker Pods running on the same nodes as your webserver pods (if so, this indicates resource starvation and the need for larger resource requests).

No PVC for the logs at this time. The worker pods are sharing the same nodes. Since my first post yesterday I've looked further and noticed that there were no resource requests put in place by the person who set it up (not me), so it looks like that was our first mistake. I'll be addressing that right away.

Answers:

* Yes, always set resource requests for pods (otherwise K8S will de-prioritize those pods when allocating resources)

* Yes, increasing the replica count of the webserver will improve performance (but only if the bottleneck is too many connections to the webserver, which seems unlikely from your description)

This all makes sense and I think right now we're going to make sure resource requests are in place and go from there.