Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Chart: refactor webserver and flower networkpolicy #16619

Merged
merged 1 commit into from
Jun 30, 2021
Merged

Chart: refactor webserver and flower networkpolicy #16619

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions chart/UPDATING.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,12 @@ The default Airflow version that is installed with the Chart is now ``2.1.0``, p

This chart has dropped support for [Helm 2 as it has been deprecated](https://helm.sh/blog/helm-v2-deprecation-timeline/) and no longer receiving security updates since November 2020.

### `webserver.extraNetworkPolicies` and `flower.extraNetworkPolicies` parameters have been renamed

`webserver.extraNetworkPolicies` and `flower.extraNetworkPolicies` have been renamed to `webserver.networkPolicy.ingress.from` and `flower.networkPolicy.ingress.from`, respectively. Their values and behavior are the same.

The old parameter names will continue to work, however support for them will be removed in a future release so please update your values file.

### Removed `dags.gitSync.root`, `dags.gitSync.dest`, and `dags.gitSync.excludeWebserver` parameters

The `dags.gitSync.root` and `dags.gitSync.dest` parameters didn't provide any useful behaviors to chart users so they have been removed.
Expand Down
17 changes: 17 additions & 0 deletions chart/templates/NOTES.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,3 +79,20 @@ Information on how to set knownHosts can be found here:
https://airflow.apache.org/docs/helm-chart/latest/production-guide.html#knownhosts

{{- end }}

{{- if .Values.flower.extraNetworkPolicies }}

DEPRECATION WARNING:
`flower.extraNetworkPolicies` has been renamed to `flower.networkPolicy.peers`.
Please change your values as support for the old name will be dropped in a future release.

{{- end }}


{{- if .Values.webserver.extraNetworkPolicies }}

DEPRECATION WARNING:
`webserver.extraNetworkPolicies` has been renamed to `webserver.networkPolicy.peers`.
Please change your values as support for the old name will be dropped in a future release.

{{- end }}
13 changes: 9 additions & 4 deletions chart/templates/flower/flower-networkpolicy.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@
{{- if .Values.flower.enabled }}
{{- $celery_executors := list "CeleryExecutor" "CeleryKubernetesExecutor"}}
{{- if and .Values.networkPolicies.enabled (has .Values.executor $celery_executors) }}
{{- $from := or .Values.flower.networkPolicy.ingress.from .Values.flower.extraNetworkPolicies }}
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
Expand All @@ -42,13 +43,17 @@ spec:
release: {{ .Release.Name }}
policyTypes:
- Ingress
{{- if .Values.flower.extraNetworkPolicies }}
{{- if $from }}
ingress:
- from:
{{ toYaml .Values.flower.extraNetworkPolicies | indent 6 }}
{{ toYaml $from | indent 6 }}
ports:
- protocol: TCP
port: {{ .Values.ports.flowerUI }}
{{ range .Values.flower.networkPolicy.ingress.ports }}
-
{{- range $key, $val := . }}
{{ $key }}: {{ tpl (toString $val) $ }}
{{- end }}
{{- end }}
{{- end }}
{{- end }}
{{- end }}
13 changes: 9 additions & 4 deletions chart/templates/webserver/webserver-networkpolicy.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@
## Airflow Webserver NetworkPolicy
#################################
{{- if .Values.networkPolicies.enabled }}
{{- $from := or .Values.webserver.networkPolicy.ingress.from .Values.webserver.extraNetworkPolicies }}
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
Expand All @@ -40,12 +41,16 @@ spec:
release: {{ .Release.Name }}
policyTypes:
- Ingress
{{- if .Values.webserver.extraNetworkPolicies }}
{{- if $from }}
ingress:
- from:
{{ toYaml .Values.webserver.extraNetworkPolicies | indent 6 }}
{{ toYaml $from | indent 6 }}
ports:
- protocol: TCP
port: {{ .Values.ports.airflowUI }}
{{ range .Values.webserver.networkPolicy.ingress.ports }}
-
{{- range $key, $val := . }}
{{ $key }}: {{ tpl (toString $val) $ }}
{{- end }}
{{- end }}
{{- end }}
{{- end }}
83 changes: 83 additions & 0 deletions chart/tests/test_flower.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -321,3 +321,86 @@ def test_ports_overrides(self, ports, expected_ports):
)

assert expected_ports == jmespath.search("spec.ports", docs[0])


class TestFlowerNetworkPolicy:
def test_off_by_default(self):
docs = render_chart(
show_only=["templates/flower/flower-networkpolicy.yaml"],
)
assert 0 == len(docs)

def test_defaults(self):
docs = render_chart(
values={
"networkPolicies": {"enabled": True},
"flower": {
"networkPolicy": {
"ingress": {
"from": [{"namespaceSelector": {"matchLabels": {"release": "myrelease"}}}]
}
}
},
},
show_only=["templates/flower/flower-networkpolicy.yaml"],
)

assert 1 == len(docs)
assert "NetworkPolicy" == docs[0]["kind"]
assert [{"namespaceSelector": {"matchLabels": {"release": "myrelease"}}}] == jmespath.search(
"spec.ingress[0].from", docs[0]
)
assert [{"port": "flower-ui"}] == jmespath.search("spec.ingress[0].ports", docs[0])

@pytest.mark.parametrize(
"ports, expected_ports",
[
(
[{"name": "{{ .Release.Name }}", "protocol": "UDP", "port": "{{ .Values.ports.flowerUI }}"}],
[{"name": "RELEASE-NAME", "protocol": "UDP", "port": 5555}],
),
([{"name": "only_sidecar", "port": "sidecar"}], [{"name": "only_sidecar", "port": "sidecar"}]),
(
[
{"name": "flower-ui", "port": "{{ .Values.ports.flowerUI }}"},
{"name": "sidecar", "port": 80},
],
[
{"name": "flower-ui", "port": 5555},
{"name": "sidecar", "port": 80},
],
),
],
)
def test_ports_overrides(self, ports, expected_ports):
docs = render_chart(
values={
"networkPolicies": {"enabled": True},
"flower": {
"networkPolicy": {
"ingress": {
"from": [{"namespaceSelector": {"matchLabels": {"release": "myrelease"}}}],
"ports": ports,
}
}
},
},
show_only=["templates/flower/flower-networkpolicy.yaml"],
)

assert expected_ports == jmespath.search("spec.ingress[0].ports", docs[0])

def test_deprecated_from_param(self):
docs = render_chart(
values={
"networkPolicies": {"enabled": True},
"flower": {
"extraNetworkPolicies": [{"namespaceSelector": {"matchLabels": {"release": "myrelease"}}}]
},
},
show_only=["templates/flower/flower-networkpolicy.yaml"],
)

assert [{"namespaceSelector": {"matchLabels": {"release": "myrelease"}}}] == jmespath.search(
"spec.ingress[0].from", docs[0]
)
82 changes: 82 additions & 0 deletions chart/tests/test_webserver.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -511,3 +511,85 @@ def test_webserver_config_configmap(self):
"CSRF_ENABLED = True # RELEASE-NAME"
== jmespath.search('data."webserver_config.py"', docs[0]).strip()
)


class WebserverNetworkPolicyTest(unittest.TestCase):
def test_off_by_default(self):
docs = render_chart(
show_only=["templates/webserver/webserver-networkpolicy.yaml"],
)
assert 0 == len(docs)

def test_defaults(self):
docs = render_chart(
values={
"networkPolicies": {"enabled": True},
"webserver": {
"networkPolicy": {
"ingress": {
"from": [{"namespaceSelector": {"matchLabels": {"release": "myrelease"}}}]
}
}
},
},
show_only=["templates/webserver/webserver-networkpolicy.yaml"],
)

assert 1 == len(docs)
assert "NetworkPolicy" == docs[0]["kind"]
assert [{"namespaceSelector": {"matchLabels": {"release": "myrelease"}}}] == jmespath.search(
"spec.ingress[0].from", docs[0]
)
assert [{"port": "airflow-ui"}] == jmespath.search("spec.ingress[0].ports", docs[0])

@parameterized.expand(
[
(
[{"name": "{{ .Release.Name }}", "protocol": "UDP", "port": "{{ .Values.ports.airflowUI }}"}],
[{"name": "RELEASE-NAME", "protocol": "UDP", "port": 8080}],
),
([{"name": "only_sidecar", "port": "sidecar"}], [{"name": "only_sidecar", "port": "sidecar"}]),
(
[
{"name": "flower-ui", "port": "{{ .Values.ports.airflowUI }}"},
{"name": "sidecar", "port": 80},
],
[
{"name": "flower-ui", "port": 8080},
{"name": "sidecar", "port": 80},
],
),
],
)
def test_ports_overrides(self, ports, expected_ports):
docs = render_chart(
values={
"networkPolicies": {"enabled": True},
"webserver": {
"networkPolicy": {
"ingress": {
"from": [{"namespaceSelector": {"matchLabels": {"release": "myrelease"}}}],
"ports": ports,
}
}
},
},
show_only=["templates/webserver/webserver-networkpolicy.yaml"],
)

assert expected_ports == jmespath.search("spec.ingress[0].ports", docs[0])

def test_deprecated_from_param(self):
docs = render_chart(
values={
"networkPolicies": {"enabled": True},
"webserver": {
"extraNetworkPolicies": [{"namespaceSelector": {"matchLabels": {"release": "myrelease"}}}]
},
},
show_only=["templates/webserver/webserver-networkpolicy.yaml"],
)

assert [{"namespaceSelector": {"matchLabels": {"release": "myrelease"}}}] == jmespath.search(
"spec.ingress[0].from", docs[0]
)
70 changes: 68 additions & 2 deletions chart/values.schema.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1626,10 +1626,43 @@
}
},
"extraNetworkPolicies": {
"description": "Additional NetworkPolicies as needed.",
"description": "Additional NetworkPolicies as needed (Deprecated - renamed to `webserver.networkPolicy.ingress.from`).",
"type": "array",
"default": []
},
"networkPolicy": {
"description": "Webserver NetworkPolicy configuration",
"type": "object",
"additionalProperties": false,
"properties": {
"ingress": {
"description": "Webserver NetworkPolicy ingress configuration",
"type": "object",
"additionalProperties": false,
"properties": {
"from": {
"description": "Peers for webserver NetworkPolicy ingress.",
"type": "array",
"default": []
},
"ports": {
"description": "Ports for webserver NetworkPolicy ingress (if `from` is set).",
"type": "array",
"default": [
{
"port": "airflow-ui"
}
],
"examples": [
{
"port": "sidecar"
}
]
}
}
}
}
},
"resources": {
"description": "Resources for webserver pods.",
"type": "object",
Expand Down Expand Up @@ -1827,10 +1860,43 @@
]
},
"extraNetworkPolicies": {
"description": "Additional NetworkPolicies as needed.",
"description": "Additional NetworkPolicies as needed (Deprecated - renamed to `flower.networkPolicy.ingress.from`).",
"type": "array",
"default": []
},
"networkPolicy": {
"description": "Flower NetworkPolicy configuration",
"type": "object",
"additionalProperties": false,
"properties": {
"ingress": {
"description": "Flower NetworkPolicy ingress configuration",
"type": "object",
"additionalProperties": false,
"properties": {
"from": {
"description": "Peers for flower NetworkPolicy ingress.",
"type": "array",
"default": []
},
"ports": {
"description": "Ports for flower NetworkPolicy ingress (if `from` is set).",
"type": "array",
"default": [
{
"port": "flower-ui"
}
],
"examples": [
{
"port": "sidecar"
}
]
}
}
}
}
},
"resources": {
"description": "Resources for Flower pods.",
"type": "object",
Expand Down
19 changes: 17 additions & 2 deletions chart/values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -628,8 +628,15 @@ webserver:
# Allow overriding Update Strategy for Webserver
strategy: ~

# Additional network policies as needed
# Additional network policies as needed (Deprecated - renamed to `webserver.networkPolicy.ingress.from`)
extraNetworkPolicies: []
networkPolicy:
ingress:
# Peers for webserver NetworkPolicy ingress
from: []
# Ports for webserver NetworkPolicy ingress (if `from` is set)
ports:
- port: airflow-ui

resources: {}
# limits:
Expand Down Expand Up @@ -720,8 +727,16 @@ flower:
exec \
airflow {{ semverCompare ">=2.0.0" .Values.airflowVersion | ternary "celery flower" "flower" }}

# Additional network policies as needed
# Additional network policies as needed (Deprecated - renamed to `flower.networkPolicy.ingress.from`)
extraNetworkPolicies: []
networkPolicy:
ingress:
# Peers for flower NetworkPolicy ingress
from: []
# Ports for flower NetworkPolicy ingress (if ingressPeers is set)
ports:
- port: flower-ui

resources: {}
# limits:
# cpu: 100m
Expand Down