Use Nginx + Basic Auth to secure OSS Airbyte (airbytehq#17694)

* Use Nginx + Basic Auth to secure OSS Airbyte * use local passwords * Use gradle builds * K8s setup and source values from ENV * note about disabling * add back defaults * custom 401 page * update http message * update docs * remove kube files * additional doc updates * Add a test suite * fix failure exit codes * doc updates * Add docs * bump to re-test * add more sleep in tests for CI * better sleep in test * Update docs/operator-guides/security.md Co-authored-by: Davin Chia <[email protected]> * PR updates * test comment * change test host on CI * update tests and nginx to boot without backend * proxy updates for docker DNS * simpler test for uptime * acceptance test skips PWs * remove resolver madness * fixup tests * more proxy_pass revert * update acceptance test exit codes * relax test expectations * add temporal mount back for testing * Update docs/operator-guides/security.md Co-authored-by: swyx <[email protected]> * Update airbyte-proxy/401.html Co-authored-by: swyx <[email protected]> * more doc updates * Octavia CLI uses Basic Auth (airbytehq#17982) * [WIP] Octavia CLI uses Basic Auth * readme * augustin: add basic auth headers to clien * augustin: add basic auth headers to client * tests passing * lint * docs * Move monkey patch to test * coerce headers into strings * monkey patch get_basic_auth_token Co-authored-by: alafanechere <[email protected]> * fix launch permissions * Keep worker port internal * more readme Co-authored-by: Davin Chia <[email protected]> Co-authored-by: swyx <[email protected]> Co-authored-by: alafanechere <[email protected]>
jhammarstedt · Oct 31, 2022 · dfaba18 · dfaba18
1 parent d393301
commit dfaba18
Show file tree

Hide file tree

Showing 28 changed files with 25,412 additions and 23,571 deletions.
diff --git a/.bumpversion.cfg b/.bumpversion.cfg
@@ -3,7 +3,7 @@ current_version = 0.40.15
 commit = False
 tag = False
 parse = (?P<major>\d+)\.(?P<minor>\d+)\.(?P<patch>\d+)(\-[a-z]+)?
-serialize = 
+serialize =
 	{major}.{minor}.{patch}
 
 [bumpversion:file:.bumpversion.cfg]
@@ -18,42 +18,44 @@ serialize =
 
 [bumpversion:file:airbyte-metrics/reporter/Dockerfile]
 
-[bumpversion:file:airbyte-server/Dockerfile]
+[bumpversion:file:airbyte-proxy/Dockerfile]
 
-[bumpversion:file:airbyte-webapp/package.json]
+[bumpversion:file:airbyte-server/Dockerfile]
 
 [bumpversion:file:airbyte-webapp/package-lock.json]
 
+[bumpversion:file:airbyte-webapp/package.json]
+
 [bumpversion:file:airbyte-workers/Dockerfile]
 
-[bumpversion:file:charts/airbyte/Chart.yaml]
+[bumpversion:file:charts/airbyte-bootloader/Chart.yaml]
 
-[bumpversion:file:charts/airbyte-worker/Chart.yaml]
+[bumpversion:file:charts/airbyte-server/Chart.yaml]
 
 [bumpversion:file:charts/airbyte-temporal/Chart.yaml]
 
 [bumpversion:file:charts/airbyte-webapp/Chart.yaml]
 
-[bumpversion:file:charts/airbyte-server/Chart.yaml]
-
-[bumpversion:file:charts/airbyte-bootloader/Chart.yaml]
+[bumpversion:file:charts/airbyte-worker/Chart.yaml]
 
 [bumpversion:file:charts/airbyte/README.md]
 
-[bumpversion:file:docs/operator-guides/upgrading-airbyte.md]
+[bumpversion:file:charts/airbyte/values.yaml]
 
-[bumpversion:file:kube/overlays/stable/.env]
-
-[bumpversion:file:kube/overlays/stable/kustomization.yaml]
+[bumpversion:file:docs/operator-guides/upgrading-airbyte.md]
 
 [bumpversion:file:kube/overlays/stable-with-resource-limits/.env]
 
 [bumpversion:file:kube/overlays/stable-with-resource-limits/kustomization.yaml]
 
-[bumpversion:file:octavia-cli/install.sh]
+[bumpversion:file:kube/overlays/stable/.env]
 
-[bumpversion:file:octavia-cli/README.md]
+[bumpversion:file:kube/overlays/stable/kustomization.yaml]
 
 [bumpversion:file:octavia-cli/Dockerfile]
 
+[bumpversion:file:octavia-cli/README.md]
+
+[bumpversion:file:octavia-cli/install.sh]
+
 [bumpversion:file:octavia-cli/setup.py]
diff --git a/.env b/.env
@@ -32,6 +32,10 @@ LOCAL_DOCKER_MOUNT=/tmp/airbyte_local
 # Issue: https://github.com/airbytehq/airbyte/issues/577
 HACK_LOCAL_ROOT_PARENT=/tmp
 
+# Proxy Configuration
+# Set to empty values, e.g. "" to disable basic auth
+BASIC_AUTH_USERNAME=airbyte
+BASIC_AUTH_PASSWORD=password
 
 ### DATABASE ###
 # Airbyte Internal Job Database, see https://docs.airbyte.io/operator-guides/configuring-airbyte-db

diff --git a/airbyte-proxy/401.html b/airbyte-proxy/401.html
@@ -0,0 +1,19 @@
+<html>
+  <head>
+    <meta charset="utf-8" />
+    <title>Airbyte - Access Denied</title>
+  </head>
+  <body>
+    <h1>🐙 Nope.</h1>
+    <h2><code>HTTP Error Code: 401</code></h2>
+
+    <p>
+      This deployment of Airbyte is protected by HTTP Basic Authentication.
+      Please refer to the <a href="https://docs.airbyte.com">Airbyte docs</a> to learn more about:
+      <ul>
+	      <li><a href="https://docs.airbyte.com/quickstart/deploy-airbyte">the default username/password</a></li>
+	      <li><a href="https://docs.airbyte.com/operator-guides/security">securing your Airbyte deployment</a> including turning off or changing the default username/password.</li>
+      </ul>
+    </p>
+  </body>
+</html>
diff --git a/airbyte-proxy/Dockerfile b/airbyte-proxy/Dockerfile
@@ -0,0 +1,27 @@
+# Inspired by https://medium.com/pernod-ricard-tech/adding-basic-authentication-with-nginx-as-a-reverse-proxy-a229f9d12b73
+
+FROM nginx:latest
+
+ARG VERSION=0.40.12
+
+ENV APPLICATION airbyte-proxy
+ENV VERSION ${VERSION}
+
+RUN apt-get update -y && apt-get install -y apache2-utils && rm -rf /var/lib/apt/lists/*
+
+# This variable can be used to update the destintion containers that Nginx proxies to.
+ENV PROXY_PASS_WEB "http://airbyte-webapp:80"
+ENV PROXY_PASS_API "http://airbyte-server:8001"
+
+# Nginx config file
+WORKDIR /
+RUN mkdir -p /etc/nginx/templates
+COPY nginx-auth.conf.template /etc/nginx/templates/nginx-auth.conf.template
+COPY nginx-no-auth.conf.template /etc/nginx/templates/nginx-no-auth.conf.template
+COPY 401.html /etc/nginx/401.html
+
+# Startup script
+COPY run.sh ./
+RUN chmod 0755 ./run.sh
+CMD [ "./run.sh" ]
+ENTRYPOINT ["./run.sh"]
diff --git a/airbyte-proxy/LICENSE b/airbyte-proxy/LICENSE
@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2022 Airbyte, Inc.
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
diff --git a/airbyte-proxy/README.md b/airbyte-proxy/README.md
@@ -0,0 +1,14 @@
+# Airbyte Proxy
+
+This service uses Nginx to front the Aribyte `webapp` and `server` services to add Authentication via HTTP basic auth.
+
+Authentication is controlled by 2 environment variables, `BASIC_AUTH_USERNAME` and `BASIC_AUTH_PASSWORD` which can be modified in the `.env` file for your Airbyte deployment. You can disable authentication by setting both `BASIC_AUTH_USERNAME` and `BASIC_AUTH_PASSWORD` to empty strings. Changes in your environment variables will be applied when the service (re)boots.
+
+This service is intended to work in conjunction with the `airbyte_internal` network defined in the default docker compose file. By default, this application forwards requesting coming in on 8000 and 8001 to the PROXY_PASS_WEB and PROXY_PASS_API accordingly - which are also configured by environment variables within this container (see Dockerfile). The deafults are configured to work with the default `docker-compose.yaml` file for Airbyte OSS deployments.
+
+```
+ENV PROXY_PASS_WEB "http://airbyte-webapp:80"
+ENV PROXY_PASS_API "http://airbyte-server:8001"
+```
+
+🐙
diff --git a/airbyte-proxy/build.gradle b/airbyte-proxy/build.gradle
@@ -0,0 +1,21 @@
+task prepareBuild(type: Copy) {
+    from layout.projectDirectory.file("nginx-auth.conf.template")
+    from layout.projectDirectory.file("nginx-no-auth.conf.template")
+    from layout.projectDirectory.file("run.sh")
+    from layout.projectDirectory.file("401.html")
+
+    into layout.buildDirectory.dir("docker")
+}
+
+tasks.named("buildDockerImage") {
+    dependsOn prepareBuild
+    dependsOn copyDocker
+}
+
+task bashTest(type: Exec) {
+    dependsOn buildDockerImage
+    commandLine "./test.sh"
+}
+
+// we can't override the 'test' command, so we can make our bash test a dependency
+test.dependsOn(project.tasks.bashTest)
diff --git a/airbyte-proxy/gradle.properties b/airbyte-proxy/gradle.properties
@@ -0,0 +1 @@
+dockerImageName=proxy
diff --git a/airbyte-proxy/nginx-auth.conf.template b/airbyte-proxy/nginx-auth.conf.template
@@ -0,0 +1,45 @@
+events {}
+
+http {
+  server {
+    listen 8000 default_server;
+
+    location / {
+      proxy_set_header Host $host;
+      proxy_set_header X-Real-IP $remote_addr;
+      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+
+      auth_basic "Welcome to Airbyte";
+      auth_basic_user_file /etc/nginx/.htpasswd;
+
+      proxy_pass "${PROXY_PASS_WEB}";
+
+      error_page 401 /etc/nginx/401.html;
+      location ~ (401.html)$ {
+        alias /etc/nginx/$1;
+        auth_basic off;
+      }
+    }
+  }
+
+  server {
+    listen 8001;
+
+    location / {
+      proxy_set_header Host $host;
+      proxy_set_header X-Real-IP $remote_addr;
+      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+
+      auth_basic "Welcome to Airbyte";
+      auth_basic_user_file /etc/nginx/.htpasswd;
+
+      proxy_pass "${PROXY_PASS_API}";
+
+      error_page 401 /etc/nginx/401.html;
+      location ~ (401.html)$ {
+        alias /etc/nginx/$1;
+        auth_basic off;
+      }
+    }
+  }
+}
diff --git a/airbyte-proxy/nginx-no-auth.conf.template b/airbyte-proxy/nginx-no-auth.conf.template
@@ -0,0 +1,27 @@
+events {}
+
+http {
+  server {
+    listen 8000 default_server;
+
+    location / {
+      proxy_set_header Host $host;
+      proxy_set_header X-Real-IP $remote_addr;
+      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+
+      proxy_pass "${PROXY_PASS_WEB}";
+    }
+  }
+
+  server {
+    listen 8001;
+
+    location / {
+      proxy_set_header Host $host;
+      proxy_set_header X-Real-IP $remote_addr;
+      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+
+      proxy_pass "${PROXY_PASS_API}";
+    }
+  }
+}
diff --git a/airbyte-proxy/run.sh b/airbyte-proxy/run.sh
@@ -0,0 +1,23 @@
+#!/bin/bash
+
+rm /etc/nginx/nginx.conf
+
+if [[ -z "${BASIC_AUTH_USERNAME}" ]]; then
+  echo "BASIC_AUTH_USERNAME is not set, skipping nginx auth"
+
+  TEMPLATE_PATH="/etc/nginx/templates/nginx-no-auth.conf.template"
+else
+  echo "BASIC_AUTH_USERNAME is set, requiring auth for user '$BASIC_AUTH_USERNAME'"
+
+  # htpasswd for basic authentication
+  rm -rf /etc/nginx/.htpasswd
+  htpasswd -c -b /etc/nginx/.htpasswd $BASIC_AUTH_USERNAME $BASIC_AUTH_PASSWORD
+
+  TEMPLATE_PATH="/etc/nginx/templates/nginx-auth.conf.template"
+fi
+
+envsubst '${PROXY_PASS_WEB} ${PROXY_PASS_API} ${PROXY_PASS_RESOLVER}' < $TEMPLATE_PATH > /etc/nginx/nginx.conf
+
+echo "starting nginx..."
+nginx -v
+nginx -g "daemon off;"