502 gateway error (nginx)

[jeffshead]

I had CanaryTokens up and running 7-10 months ago but I recently lost my entire server. I kept detailed notes so I rebuilt everything but I can't get things to work with the new v3 docker image of canarytokens-docker.

I am using a AlmaLinux 9.3 server with Portainer and everything is behind the Nginx Proxy Manager (NPM) reverse-proxy Docker container which is hosted on the same server.

• Docker version 24.0.7, build afdd53b
• Docker Compose version v2.23.3

If I pull thinkst/canarytokens:v2_latest, everything works.

However, when I try the latest v3 image, I'm getting a 502 gateway error (nginx).

In Portainer, these are the internal IP's assigned to the containers:

• frontend 172.18.0.5
• nginx 172.18.0.7
• nginxproxymanager-app-1 (NPM) 172.18.0.4
• redis 172.18.0.2
• portainer 172.18.0.3
• switchboard NONE LISTED

In NPM, I have the following for the destination in my proxy host for canarytokens-docker: http://nginx:80
This works with canary-tokens v2.

Canarytokens-docker nginx container log:

2023/12/02 13:36:36 [error] 7#7: *1 connect() failed (113: No route to host) while connecting to upstream, client: 172.18.0.4, server: _, request: "GET / HTTP/1.1", upstream: "http://172.18.0.5:8082/", host: "sub.example.com"
2023/12/02 13:36:39 [error] 7#7: *3 connect() failed (113: No route to host) while connecting to upstream, client: 172.18.0.4, server: _, request: "GET / HTTP/1.1", upstream: "http://172.18.0.5:8082/", host: "sub.example.com"
172.18.0.4 - - [02/Dec/2023:13:36:39 +0000] "GET / HTTP/1.1" 502 150 "-" "Mozilla/5.0 (Windows NT 10.0; rv:109.0) Gecko/20100101 Firefox/115.0"

Canarytokens-docker frontend container log:

    return ctx.invoke(self.callback, **ctx.params)
  File "/srv/.venv/lib/python3.9/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/srv/.venv/lib/python3.9/site-packages/uvicorn/main.py", line 437, in main
    run(app, **kwargs)
  File "/srv/.venv/lib/python3.9/site-packages/uvicorn/main.py", line 463, in run
    server.run()
  File "/srv/.venv/lib/python3.9/site-packages/uvicorn/server.py", line 60, in run
    return asyncio.run(self.serve(sockets=sockets))
  File "/usr/local/lib/python3.9/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.9/asyncio/base_events.py", line 647, in run_until_complete
    return future.result()
  File "/srv/.venv/lib/python3.9/site-packages/uvicorn/server.py", line 67, in serve
    config.load()
  File "/srv/.venv/lib/python3.9/site-packages/uvicorn/config.py", line 458, in load
    self.loaded_app = import_from_string(self.app)
  File "/srv/.venv/lib/python3.9/site-packages/uvicorn/importer.py", line 21, in import_from_string
    module = importlib.import_module(module_str)
  File "/usr/local/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/srv/frontend/./app.py", line 35, in <module>
    from canarytokens import extendtoken, kubeconfig, msreg, queries
  File "/srv/canarytokens/kubeconfig.py", line 12, in <module>
    from canarytokens.channel_input_mtls import mTLS
  File "/srv/canarytokens/channel_input_mtls.py", line 26, in <module>
    from canarytokens import queries
  File "/srv/canarytokens/queries.py", line 16, in <module>
    from canarytokens import canarydrop as cand
  File "/srv/canarytokens/canarydrop.py", line 23, in <module>
    from canarytokens import queries, tokens
  File "/srv/canarytokens/tokens.py", line 72, in <module>
    switchboard_settings = SwitchboardSettings()
  File "pydantic/env_settings.py", line 39, in pydantic.env_settings.BaseSettings.__init__
  File "pydantic/main.py", line 341, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 1 validation error for SwitchboardSettings
PUBLIC_DOMAIN
  field required (type=value_error.missing)
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/local/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/srv/.venv/lib/python3.9/site-packages/uvicorn/__main__.py", line 4, in <module>
    uvicorn.main()
  File "/srv/.venv/lib/python3.9/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/srv/.venv/lib/python3.9/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/srv/.venv/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/srv/.venv/lib/python3.9/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/srv/.venv/lib/python3.9/site-packages/uvicorn/main.py", line 437, in main
    run(app, **kwargs)
  File "/srv/.venv/lib/python3.9/site-packages/uvicorn/main.py", line 463, in run
    server.run()
  File "/srv/.venv/lib/python3.9/site-packages/uvicorn/server.py", line 60, in run
    return asyncio.run(self.serve(sockets=sockets))
  File "/usr/local/lib/python3.9/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.9/asyncio/base_events.py", line 647, in run_until_complete
    return future.result()
  File "/srv/.venv/lib/python3.9/site-packages/uvicorn/server.py", line 67, in serve
    config.load()
  File "/srv/.venv/lib/python3.9/site-packages/uvicorn/config.py", line 458, in load
    self.loaded_app = import_from_string(self.app)
  File "/srv/.venv/lib/python3.9/site-packages/uvicorn/importer.py", line 21, in import_from_string
    module = importlib.import_module(module_str)
  File "/usr/local/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/srv/frontend/./app.py", line 35, in <module>
    from canarytokens import extendtoken, kubeconfig, msreg, queries
  File "/srv/canarytokens/kubeconfig.py", line 12, in <module>
    from canarytokens.channel_input_mtls import mTLS
  File "/srv/canarytokens/channel_input_mtls.py", line 26, in <module>
    from canarytokens import queries
  File "/srv/canarytokens/queries.py", line 16, in <module>
    from canarytokens import canarydrop as cand
  File "/srv/canarytokens/canarydrop.py", line 23, in <module>
    from canarytokens import queries, tokens
  File "/srv/canarytokens/tokens.py", line 72, in <module>
    switchboard_settings = SwitchboardSettings()
  File "pydantic/env_settings.py", line 39, in pydantic.env_settings.BaseSettings.__init__
  File "pydantic/main.py", line 341, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 1 validation error for SwitchboardSettings
PUBLIC_DOMAIN
  field required (type=value_error.missing)

Canarytokens-docker redis container log:

1:C 02 Dec 2023 13:34:31.252 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:C 02 Dec 2023 13:34:31.252 # Redis version=7.0.10, bits=64, commit=00000000, modified=0, pid=1, just started
1:C 02 Dec 2023 13:34:31.252 # Configuration loaded
1:M 02 Dec 2023 13:34:31.253 * monotonic clock: POSIX clock_gettime
1:M 02 Dec 2023 13:34:31.263 # Warning: Could not create server TCP listening socket ::*:6379: unable to bind socket, errno: 97
1:M 02 Dec 2023 13:34:31.265 * Running mode=standalone, port=6379.
1:M 02 Dec 2023 13:34:31.265 # Server initialized
1:M 02 Dec 2023 13:34:31.265 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
1:M 02 Dec 2023 13:34:31.268 * Creating AOF base file appendonly.aof.1.base.rdb on server start
1:M 02 Dec 2023 13:34:31.271 * Creating AOF incr file appendonly.aof.1.incr.aof on server start
1:M 02 Dec 2023 13:34:31.271 * Ready to accept connections

Canarytokens-docker switchboard container log:

Failed to load application: 1 validation error for FrontendSettings
DOMAINS
  field required (type=value_error.missing)
Unhandled Error
Traceback (most recent call last):
  File "/srv/.venv/lib/python3.9/site-packages/twisted/application/app.py", line 674, in run
    runApp(config)
  File "/srv/.venv/lib/python3.9/site-packages/twisted/scripts/twistd.py", line 29, in runApp
    runner.run()
  File "/srv/.venv/lib/python3.9/site-packages/twisted/application/app.py", line 370, in run
    self.application = self.createOrGetApplication()
  File "/srv/.venv/lib/python3.9/site-packages/twisted/application/app.py", line 437, in createOrGetApplication
    application = getApplication(self.config, passphrase)
--- <exception caught here> ---
  File "/srv/.venv/lib/python3.9/site-packages/twisted/application/app.py", line 446, in getApplication
    application = service.loadApplication(filename, style, passphrase)
  File "/srv/.venv/lib/python3.9/site-packages/twisted/application/service.py", line 404, in loadApplication
    application = sob.loadValueFromFile(filename, "application")
  File "/srv/.venv/lib/python3.9/site-packages/twisted/persisted/sob.py", line 177, in loadValueFromFile
    eval(codeObj, d, d)
  File "switchboard.tac", line 42, in <module>
    frontend_settings = FrontendSettings()
  File "pydantic/env_settings.py", line 39, in pydantic.env_settings.BaseSettings.__init__
    
  File "pydantic/main.py", line 341, in pydantic.main.BaseModel.__init__
    
pydantic.error_wrappers.ValidationError: 1 validation error for FrontendSettings
DOMAINS
  field required (type=value_error.missing)
Failed to load application: 1 validation error for FrontendSettings
DOMAINS
  field required (type=value_error.missing)
Unhandled Error
Traceback (most recent call last):
  File "/srv/.venv/lib/python3.9/site-packages/twisted/application/app.py", line 674, in run
    runApp(config)
  File "/srv/.venv/lib/python3.9/site-packages/twisted/scripts/twistd.py", line 29, in runApp
    runner.run()
  File "/srv/.venv/lib/python3.9/site-packages/twisted/application/app.py", line 370, in run
    self.application = self.createOrGetApplication()
  File "/srv/.venv/lib/python3.9/site-packages/twisted/application/app.py", line 437, in createOrGetApplication
    application = getApplication(self.config, passphrase)
--- <exception caught here> ---
  File "/srv/.venv/lib/python3.9/site-packages/twisted/application/app.py", line 446, in getApplication
    application = service.loadApplication(filename, style, passphrase)
  File "/srv/.venv/lib/python3.9/site-packages/twisted/application/service.py", line 404, in loadApplication
    application = sob.loadValueFromFile(filename, "application")
  File "/srv/.venv/lib/python3.9/site-packages/twisted/persisted/sob.py", line 177, in loadValueFromFile
    eval(codeObj, d, d)
  File "switchboard.tac", line 42, in <module>
    frontend_settings = FrontendSettings()
  File "pydantic/env_settings.py", line 39, in pydantic.env_settings.BaseSettings.__init__
    
  File "pydantic/main.py", line 341, in pydantic.main.BaseModel.__init__
    
pydantic.error_wrappers.ValidationError: 1 validation error for FrontendSettings
DOMAINS
  field required (type=value_error.missing)
Failed to load application: 1 validation error for FrontendSettings
DOMAINS
  field required (type=value_error.missing)
Unhandled Error
Traceback (most recent call last):
  File "/srv/.venv/lib/python3.9/site-packages/twisted/application/app.py", line 674, in run
    runApp(config)
  File "/srv/.venv/lib/python3.9/site-packages/twisted/scripts/twistd.py", line 29, in runApp
    runner.run()
  File "/srv/.venv/lib/python3.9/site-packages/twisted/application/app.py", line 370, in run
    self.application = self.createOrGetApplication()
  File "/srv/.venv/lib/python3.9/site-packages/twisted/application/app.py", line 437, in createOrGetApplication
    application = getApplication(self.config, passphrase)
--- <exception caught here> ---
  File "/srv/.venv/lib/python3.9/site-packages/twisted/application/app.py", line 446, in getApplication
    application = service.loadApplication(filename, style, passphrase)
  File "/srv/.venv/lib/python3.9/site-packages/twisted/application/service.py", line 404, in loadApplication
    application = sob.loadValueFromFile(filename, "application")
  File "/srv/.venv/lib/python3.9/site-packages/twisted/persisted/sob.py", line 177, in loadValueFromFile
    eval(codeObj, d, d)
  File "switchboard.tac", line 42, in <module>
    frontend_settings = FrontendSettings()
  File "pydantic/env_settings.py", line 39, in pydantic.env_settings.BaseSettings.__init__
    
  File "pydantic/main.py", line 341, in pydantic.main.BaseModel.__init__
    
pydantic.error_wrappers.ValidationError: 1 validation error for FrontendSettings
DOMAINS
  field required (type=value_error.missing)
Failed to load application: 1 validation error for FrontendSettings
DOMAINS
  field required (type=value_error.missing)

Can v3 be made to work with my server setup? If so, what needs to be changed?

[jeffshead]

The following two sections of code in the docker-compose-v3.yml file do not work for me:

...
env_file:
     - frontend.env
    volumes:
     - ./frontend.env:/srv/frontend/frontend.env:ro
     - ./switchboard.env:/srv/switchboard/switchboard.env:ro
...
env_file:
     - switchboard.env
    volumes:
     - ./frontend.env:/srv/frontend/frontend.env:ro
     - ./switchboard.env:/srv/switchboard/switchboard.env:ro
...

I tried changing the paths but nothing worked. I ended up replacing each section of code with an environment declaration and included the contents of each .env file directly in the Portainer stack. For example:

...
environment:      
      - CANARY_DOMAINS=sub.example.com
      - CANARY_NXDOMAINS=sub2.example.com
      - CANARY_PUBLIC_IP=1.1.1.1
...

I was able to get things running by making sure all of the following exist in BOTH the frontend and switchboard environments (.env files):

• CANARY_DOMAINS=sub.example.com
• CANARY_NXDOMAINS=sub2.example.com
• CANARY_PUBLIC_IP=1.1.1.1
• CANARY_PUBLIC_DOMAIN=sub.example.com
• CANARY_WG_PRIVATE_KEY_SEED=jdjdy473if845
• CANARY_WEB_IMAGE_UPLOAD_PATH=/uploads

This should be documented somewhere that ALL of the above need to be included in both environments.

However, I'm still not getting alerts. I opened #159 to report this issue.

As of this post, I switched back to v2 and everything is working but I would like to use v3.

[jayjb]

Hi @jeffshead,

Sorry for the delay here. We haven't been keeping a sharp eye on the discussions (but we've made notes that we should so hopefully better going forward).

I just want to check if you still hitting this issue? We did a bit of an overhaul end of last year to get everything running smoothly.

[jeffshead]

I just want to check if you still hitting this issue?

I resolved this issue and posted what I did in #159.