Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docker option to pass CA file (for self-signed CA) #479

Closed
jlim0930 opened this issue Jul 16, 2019 · 25 comments
Closed

Docker option to pass CA file (for self-signed CA) #479

jlim0930 opened this issue Jul 16, 2019 · 25 comments

Comments

@jlim0930
Copy link

General information

  • OS Platform and Distribution (e.g., Linux Ubuntu 16.04): CentOS 7.4
  • ElasticHQ Version: 3.5
  • Elasticsearch Version: 7.2
  • Python version: 3.6.8
  • Browser Vendor and Version (if applicable): firefox

Issue Description

running ElasticHQ in docker container and passed both HQ_ENABLE_SSL & HQ_CA_CERTS as docker run -p 5000:5000 -e HQ_ENABLE_SSL=true -e HQ_CA_CERTS=/certs/ca.crt -v /config/certs/ca/ca.crt:/certs/ca.crt elastichq/elasticsearch-hq

Source Code / Logs

urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='elastic.local', port=9200): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)'),))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/src/elastichq/service/ConnectionService.py", line 71, in create_connection
response = requests.get(scheme + "://" + ip + ":" + port, timeout=REQUEST_TIMEOUT)
File "/usr/local/lib/python3.6/site-packages/requests/api.py", line 75, in get
return request('get', url, params=params, **kwargs)
File "/usr/local/lib/python3.6/site-packages/requests/api.py", line 60, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 514, in send
raise SSLError(e, request=request)

@valentin-fischer
Copy link

Having the same issue, haven't found a solution yet...

@royrusso
Copy link
Member

Yes. This seems related to #471

Hope to get this patched this week.

@qakart
Copy link

qakart commented Oct 7, 2019

@royrusso thanks for closing this issue and i am running Docker image
I did build my own image using the latest code to get this above fix but still can't able to see the issue fixed.

(Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)'),))",),),) --- Logging error --- Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 601, in urlopen chunked=chunked) File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 346, in _make_request self._validate_conn(conn) File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 850, in _validate_conn conn.connect() File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 326, in connect ssl_context=context) File "/usr/local/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 329, in ssl_wrap_socket return context.wrap_socket(sock, server_hostname=server_hostname) File "/usr/local/lib/python3.6/site-packages/eventlet/green/ssl.py", line 438, in wrap_socket return GreenSSLSocket(sock, *a, _context=self, **kw) File "/usr/local/lib/python3.6/site-packages/eventlet/green/ssl.py", line 128, in __init__ self.do_handshake() File "/usr/local/lib/python3.6/site-packages/eventlet/green/ssl.py", line 301, in do_handshake super(GreenSSLSocket, self).do_handshake) File "/usr/local/lib/python3.6/site-packages/eventlet/green/ssl.py", line 150, in _call_trampolining return func(*a, **kw) File "/usr/local/lib/python3.6/ssl.py", line 1077, in do_handshake self._sslobj.do_handshake() File "/usr/local/lib/python3.6/ssl.py", line 689, in do_handshake self._sslobj.do_handshake() ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)

Me to facing the same issue and looking for the fix or workaround

@gittygoo
Copy link

gittygoo commented Nov 7, 2019

This also does not work for me either with the latest version.
I am passing the following env vars:

  • HQ_DEFAULT_URL = https://elastic.internal:9200 (works fine)
  • HQ_VERIFY_CERTS = False (does not work, i keep getting "CERTIFICATE_VERIFY_FAILED")

@gplechuck
Copy link

gplechuck commented Nov 11, 2019

Also getting a [SSL: CERTIFICATE_VERIFY_FAILED] error. Specified a HQ_CA_CERTS var when running the container and this can be seen in the output from 'env' within the container, but it doesn't appear that the custom CA cert location passed by the var is being read when a connection is attempted.

Ran opensnoop on the container host and observed which files were being opened in the container when attempting to access a https enabled cluster -

~# opensnoop-bpfcc             
PID    COMM               FD ERR PATH            
...
10589  gunicorn           13   0 /etc/hosts
10589  gunicorn           14   0 /usr/local/lib/python3.6/site-packages/certifi/cacert.pem

The default cacert.pem bundle is opened, but the custom ca cert is not accessed.

Haven't looked into it further, but as a dirty workaround have replaced the container CMD with a chain of commands that will cat the custom ca cert to the default ca bundle before ElasticHQ is started. Docker compose file looks something like this -

version: '3.4'
services:
    elastichq:
        image: elastichq/elasticsearch-hq
        restart: always
        deploy:
            replicas: 1
            restart_policy:
                condition: on-failure
        container_name: elastichq
        environment:
          - HQ_ENABLE_SSL=True
          - HQ_VERIFY_CERTS=False
#          - HQ_CA_CERTS=/usr/local/share/ca-certificates/MYCA.crt
        volumes:
          - "/usr/local/share/ca-certificates:/usr/local/share/ca-certificates:ro"
        ports:
            - "5000:5000/tcp"
        command: bash -c 'cat /usr/local/share/ca-certificates/MYCA.crt >> /usr/local/lib/python3.6/site-packages/certifi/cacert.pem && supervisord -c /etc/supervisor/supervisord.conf'

@gittygoo
Copy link

gittygoo commented Nov 12, 2019

@royrusso can this issue please be reopened? not being able to test elastichq atm due to this

@royrusso royrusso reopened this Nov 12, 2019
@royrusso
Copy link
Member

royrusso commented Nov 12, 2019

I see the bug now:

application.config['HQ_VERIFY_CERTS'] = os.environ.get('HQ_VERIFY_CERTS', default_verify_certs)

The internal name used in code is simply VERIFY_CERTS.

@royrusso royrusso modified the milestones: 3.5.2, 3.5.5 Nov 13, 2019
@gittygoo
Copy link

@royrusso i tried using VERIFY_CERTS also that didnt work, should it?

@royrusso royrusso reopened this Nov 13, 2019
@royrusso
Copy link
Member

Working on a new release. I'll update this when it's out.

@royrusso royrusso modified the milestones: 3.5.5, 3.5.6 Nov 13, 2019
@royrusso
Copy link
Member

v3.5.6 is released and merged to master. Cross your fingers, because I can't exactly test this feature. ;-)

@gittygoo
Copy link

I can test it tomorrow and feedback you here if there is a docker container update for it also (on the phone at the moment so cant check) but will feedback tomorrow if so :)

@gittygoo
Copy link

gittygoo commented Nov 14, 2019

@royrusso I'm still getting the same:
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)

options im using
HQ_DEBUG = True
HQ_DEFAULT_URL = https://XXXX:[email protected]:31425
HQ_ENABLE_SSL = True
HQ_VERIFY_CERTS = False
VERIFY_CERTS = False

@royrusso
Copy link
Member

Are you setting the HQ_CA_CERTS value? If you have SSL enabled, you need to pass a cert.

@royrusso royrusso reopened this Nov 14, 2019
@gittygoo
Copy link

gittygoo commented Nov 14, 2019

Yes, here is some more debugging info that might help

LOG.error(message, ex)
Message: 'Unable to create connection to: https://XXXX:[email protected]:31425'
Arguments: (SSLError(MaxRetryError("HTTPSConnectionPool(host='my.host', port=31425): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)'),))",),),)
--- Logging error ---
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 601, in urlopen
chunked=chunked)
File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 346, in _make_request
self._validate_conn(conn)
File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 850, in _validate_conn
conn.connect()
File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 326, in connect
ssl_context=context)
File "/usr/local/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 329, in ssl_wrap_socket
return context.wrap_socket(sock, server_hostname=server_hostname)
File "/usr/local/lib/python3.6/site-packages/eventlet/green/ssl.py", line 438, in wrap_socket
return GreenSSLSocket(sock, *a, _context=self, **kw)
File "/usr/local/lib/python3.6/site-packages/eventlet/green/ssl.py", line 128, in __init__
self.do_handshake()
File "/usr/local/lib/python3.6/site-packages/eventlet/green/ssl.py", line 301, in do_handshake
super(GreenSSLSocket, self).do_handshake)
File "/usr/local/lib/python3.6/site-packages/eventlet/green/ssl.py", line 150, in _call_trampolining
return func(*a, **kw)
File "/usr/local/lib/python3.6/ssl.py", line 1077, in do_handshake
self._sslobj.do_handshake()
File "/usr/local/lib/python3.6/ssl.py", line 689, in do_handshake
self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)```

@royrusso
Copy link
Member

royrusso commented Nov 14, 2019

Make sure you are running 3.5.6. The stack trace I think should include info messages like "SSL is True". Can you post that or the whole log file?

@gittygoo
Copy link

gittygoo commented Nov 14, 2019

The version on the gui says it is correct: "You are on the current version: 3.5.6"
By the way, i am of course redacting the host , ports and credentials on the logs so i'm not trying with "my.host" :)

here you go the full log

2019-11-14 14:25:40,392 INFO supervisord started with pid 1
2019-11-14 14:25:41,397 INFO spawned: 'gunicorn' with pid 8
2019-11-14 14:25:42,401 INFO success: gunicorn entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
loading config /src/elastichq/config/logger.json
2019-11-14 14:25:44,138 INFO engineio server.__init__:140 Server initialized for eventlet.
loading config /src/elastichq/config/logger_debug.json
2019-11-14 14:26:15,129 DEBUG sqlalchemy.orm.path_registry path_registry.set:63 set 'memoized_setups' on path 'EntityRegistry((<Mapper at 0x7fbbed7d7cc0; ClusterModel>,))' to '{}'
2019-11-14 14:26:15,131 DEBUG sqlalchemy.pool.NullPool pool.__connect:668 Created new connection <sqlite3.Connection object at 0x7fbbed15e490>
2019-11-14 14:26:15,132 DEBUG sqlalchemy.pool.NullPool pool.checkout:550 Connection <sqlite3.Connection object at 0x7fbbed15e490> checked out from pool
2019-11-14 14:26:15,132 INFO sqlalchemy.engine.base.Engine base._begin_impl:682 BEGIN (implicit)
2019-11-14 14:26:15,134 INFO sqlalchemy.engine.base.Engine base._execute_context:1151 SELECT cluster.cluster_name AS cluster_cluster_name, cluster.cluster_ip AS cluster_cluster_ip, cluster.cluster_port AS cluster_cluster_port, cluster.cluster_scheme AS cluster_cluster_scheme, cluster.cluster_version AS cluster_cluster_version, cluster.cluster_username AS cluster_cluster_username, cluster.cluster_password AS cluster_cluster_password
FROM cluster
2019-11-14 14:26:15,134 INFO sqlalchemy.engine.base.Engine base._execute_context:1154 ()
2019-11-14 14:26:15,136 DEBUG sqlalchemy.engine.base.Engine result._init_metadata:681 Col ('cluster_cluster_name', 'cluster_cluster_ip', 'cluster_cluster_port', 'cluster_cluster_scheme', 'cluster_cluster_version', 'cluster_cluster_username', 'cluster_cluster_password')
2019-11-14 14:26:15,142 DEBUG sqlalchemy.pool.NullPool pool._finalize_fairy:697 Connection <sqlite3.Connection object at 0x7fbbed15e490> being returned to pool
2019-11-14 14:26:15,142 DEBUG sqlalchemy.pool.NullPool pool._reset:869 Connection <sqlite3.Connection object at 0x7fbbed15e490> rollback-on-return, via agent
2019-11-14 14:26:15,143 INFO sqlalchemy.engine.base.Engine base._rollback_impl:702 ROLLBACK
2019-11-14 14:26:15,143 DEBUG sqlalchemy.pool.NullPool pool._close_connection:314 Closing connection <sqlite3.Connection object at 0x7fbbed15e490>
2019-11-14 14:26:15,165 DEBUG sqlalchemy.orm.path_registry path_registry.set:63 set 'memoized_setups' on path 'EntityRegistry((<Mapper at 0x7fbbed7d7cc0; ClusterModel>,))' to '{}'
2019-11-14 14:26:15,167 DEBUG sqlalchemy.pool.NullPool pool.__connect:668 Created new connection <sqlite3.Connection object at 0x7fbbed0ad2d0>
2019-11-14 14:26:15,167 DEBUG sqlalchemy.pool.NullPool pool.checkout:550 Connection <sqlite3.Connection object at 0x7fbbed0ad2d0> checked out from pool
2019-11-14 14:26:15,168 INFO sqlalchemy.engine.base.Engine base._begin_impl:682 BEGIN (implicit)
2019-11-14 14:26:15,169 INFO sqlalchemy.engine.base.Engine base._execute_context:1151 SELECT cluster.cluster_name AS cluster_cluster_name, cluster.cluster_ip AS cluster_cluster_ip, cluster.cluster_port AS cluster_cluster_port, cluster.cluster_scheme AS cluster_cluster_scheme, cluster.cluster_version AS cluster_cluster_version, cluster.cluster_username AS cluster_cluster_username, cluster.cluster_password AS cluster_cluster_password
FROM cluster
2019-11-14 14:26:15,170 INFO sqlalchemy.engine.base.Engine base._execute_context:1154 ()
2019-11-14 14:26:15,171 DEBUG sqlalchemy.engine.base.Engine result._init_metadata:681 Col ('cluster_cluster_name', 'cluster_cluster_ip', 'cluster_cluster_port', 'cluster_cluster_scheme', 'cluster_cluster_version', 'cluster_cluster_username', 'cluster_cluster_password')
2019-11-14 14:26:15,174 DEBUG sqlalchemy.pool.NullPool pool._finalize_fairy:697 Connection <sqlite3.Connection object at 0x7fbbed0ad2d0> being returned to pool
2019-11-14 14:26:15,175 DEBUG sqlalchemy.pool.NullPool pool._reset:869 Connection <sqlite3.Connection object at 0x7fbbed0ad2d0> rollback-on-return, via agent
2019-11-14 14:26:15,175 INFO sqlalchemy.engine.base.Engine base._rollback_impl:702 ROLLBACK
2019-11-14 14:26:15,176 DEBUG sqlalchemy.pool.NullPool pool._close_connection:314 Closing connection <sqlite3.Connection object at 0x7fbbed0ad2d0>
2019-11-14 14:26:15,218 DEBUG sqlalchemy.orm.path_registry path_registry.set:63 set 'memoized_setups' on path 'EntityRegistry((<Mapper at 0x7fbbed7d7cc0; ClusterModel>,))' to '{}'
2019-11-14 14:26:15,219 DEBUG sqlalchemy.pool.NullPool pool.__connect:668 Created new connection <sqlite3.Connection object at 0x7fbbed0ad9d0>
2019-11-14 14:26:15,220 DEBUG sqlalchemy.pool.NullPool pool.checkout:550 Connection <sqlite3.Connection object at 0x7fbbed0ad9d0> checked out from pool
2019-11-14 14:26:15,221 INFO sqlalchemy.engine.base.Engine base._begin_impl:682 BEGIN (implicit)
2019-11-14 14:26:15,222 INFO sqlalchemy.engine.base.Engine base._execute_context:1151 SELECT cluster.cluster_name AS cluster_cluster_name, cluster.cluster_ip AS cluster_cluster_ip, cluster.cluster_port AS cluster_cluster_port, cluster.cluster_scheme AS cluster_cluster_scheme, cluster.cluster_version AS cluster_cluster_version, cluster.cluster_username AS cluster_cluster_username, cluster.cluster_password AS cluster_cluster_password
FROM cluster
2019-11-14 14:26:15,222 INFO sqlalchemy.engine.base.Engine base._execute_context:1154 ()
2019-11-14 14:26:15,224 DEBUG sqlalchemy.engine.base.Engine result._init_metadata:681 Col ('cluster_cluster_name', 'cluster_cluster_ip', 'cluster_cluster_port', 'cluster_cluster_scheme', 'cluster_cluster_version', 'cluster_cluster_username', 'cluster_cluster_password')
2019-11-14 14:26:15,225 DEBUG elastichq status.get:63 {"name": "ElasticHQ", "installed_version": "3.5.6", "current_stable_version": "3.5.6", "tagline": "You know, for Elasticsearch", "clusters": [], "default_url": "https://XXXX:[email protected]:31425"}
2019-11-14 14:26:15,227 DEBUG sqlalchemy.pool.NullPool pool._finalize_fairy:697 Connection <sqlite3.Connection object at 0x7fbbed0ad9d0> being returned to pool
2019-11-14 14:26:15,227 DEBUG sqlalchemy.pool.NullPool pool._reset:869 Connection <sqlite3.Connection object at 0x7fbbed0ad9d0> rollback-on-return, via agent
2019-11-14 14:26:15,228 INFO sqlalchemy.engine.base.Engine base._rollback_impl:702 ROLLBACK
2019-11-14 14:26:15,228 DEBUG sqlalchemy.pool.NullPool pool._close_connection:314 Closing connection <sqlite3.Connection object at 0x7fbbed0ad9d0>
None
None
2019-11-14 14:26:40,524 INFO elastichq ConnectionService.create_connection:69 Basic Auth is True
2019-11-14 14:26:40,526 INFO elastichq ConnectionService.create_connection:71 SSL enabled
config settings.json not found, searched /etc/elastic-hq/settings.json,~/settings.json,/src/settings.json,/src/elastichq/settings.json,/src/elastichq/config/settings.json
--- Logging error ---
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 601, in urlopen
chunked=chunked)
File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 346, in _make_request
self._validate_conn(conn)
File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 850, in _validate_conn
conn.connect()
File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 326, in connect
ssl_context=context)
File "/usr/local/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 329, in ssl_wrap_socket
return context.wrap_socket(sock, server_hostname=server_hostname)
File "/usr/local/lib/python3.6/site-packages/eventlet/green/ssl.py", line 438, in wrap_socket
return GreenSSLSocket(sock, *a, _context=self, **kw)
File "/usr/local/lib/python3.6/site-packages/eventlet/green/ssl.py", line 128, in __init__
self.do_handshake()
File "/usr/local/lib/python3.6/site-packages/eventlet/green/ssl.py", line 301, in do_handshake
super(GreenSSLSocket, self).do_handshake)
File "/usr/local/lib/python3.6/site-packages/eventlet/green/ssl.py", line 150, in _call_trampolining
return func(*a, **kw)
File "/usr/local/lib/python3.6/ssl.py", line 1077, in do_handshake
self._sslobj.do_handshake()
File "/usr/local/lib/python3.6/ssl.py", line 689, in do_handshake
self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
timeout=timeout
File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 639, in urlopen
_stacktrace=sys.exc_info()[2])
File "/usr/local/lib/python3.6/site-packages/urllib3/util/retry.py", line 388, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='my.host', port=31425): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)'),))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/src/elastichq/service/ConnectionService.py", line 73, in create_connection
timeout=REQUEST_TIMEOUT, verify=ca_certs, cert=client_cert_credentials)
File "/usr/local/lib/python3.6/site-packages/requests/api.py", line 75, in get
return request('get', url, params=params, **kwargs)
File "/usr/local/lib/python3.6/site-packages/requests/api.py", line 60, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 514, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='my.host', port=31425): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)'),))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/logging/__init__.py", line 994, in emit
msg = self.format(record)
File "/usr/local/lib/python3.6/logging/__init__.py", line 840, in format
return fmt.format(record)
File "/usr/local/lib/python3.6/logging/__init__.py", line 577, in format
record.message = record.getMessage()
File "/usr/local/lib/python3.6/logging/__init__.py", line 338, in getMessage
msg = msg % self.args
TypeError: not all arguments converted during string formatting
Call stack:
File "/usr/local/lib/python3.6/site-packages/eventlet/greenthread.py", line 221, in main
result = function(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/gunicorn/workers/geventlet.py", line 116, in handle
super(EventletWorker, self).handle(listener, client, addr)
File "/usr/local/lib/python3.6/site-packages/gunicorn/workers/async.py", line 56, in handle
self.handle_request(listener_name, req, client, addr)
File "/usr/local/lib/python3.6/site-packages/gunicorn/workers/async.py", line 107, in handle_request
respiter = self.wsgi(environ, resp.start_response)
File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1997, in __call__
return self.wsgi_app(environ, start_response)
File "/usr/local/lib/python3.6/site-packages/flask_socketio/__init__.py", line 43, in __call__
start_response)
File "/usr/local/lib/python3.6/site-packages/engineio/middleware.py", line 67, in __call__
return self.wsgi_app(environ, start_response)
File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1982, in wsgi_app
response = self.full_dispatch_request()
File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1612, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1598, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/usr/local/lib/python3.6/site-packages/flask_restful/__init__.py", line 480, in wrapper
resp = resource(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/flask/views.py", line 84, in view
return self.dispatch_request(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/flask_restful/__init__.py", line 595, in dispatch_request
resp = meth(*args, **kwargs)
File "/src/elastichq/common/exceptions.py", line 29, in _request_wrapper
return functor(*args, **kwargs)
File "/src/elastichq/api/clusters.py", line 129, in post
client_cert=client_cert)
File "/src/elastichq/service/ConnectionService.py", line 133, in create_connection
LOG.error(message, ex)
Message: 'Unable to create connection to: https://my.host:31425'
Arguments: (SSLError(MaxRetryError("HTTPSConnectionPool(host='my.host', port=31425): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)'),))",),),)
--- Logging error ---
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 601, in urlopen
chunked=chunked)
File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 346, in _make_request
self._validate_conn(conn)
File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 850, in _validate_conn
conn.connect()
File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 326, in connect
ssl_context=context)
File "/usr/local/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 329, in ssl_wrap_socket
return context.wrap_socket(sock, server_hostname=server_hostname)
File "/usr/local/lib/python3.6/site-packages/eventlet/green/ssl.py", line 438, in wrap_socket
return GreenSSLSocket(sock, *a, _context=self, **kw)
File "/usr/local/lib/python3.6/site-packages/eventlet/green/ssl.py", line 128, in __init__
self.do_handshake()
File "/usr/local/lib/python3.6/site-packages/eventlet/green/ssl.py", line 301, in do_handshake
super(GreenSSLSocket, self).do_handshake)
File "/usr/local/lib/python3.6/site-packages/eventlet/green/ssl.py", line 150, in _call_trampolining
return func(*a, **kw)
File "/usr/local/lib/python3.6/ssl.py", line 1077, in do_handshake
self._sslobj.do_handshake()
File "/usr/local/lib/python3.6/ssl.py", line 689, in do_handshake
self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)
During handling of the above exception, another exception occurred:```

@royrusso royrusso modified the milestones: 3.5.6, 3.5.7 Nov 14, 2019
royrusso added a commit that referenced this issue Nov 14, 2019
@royrusso
Copy link
Member

I added more logging and a new docker is being built. I may have found the problem, but again... difficult for me to test locally.

@royrusso royrusso reopened this Nov 14, 2019
@gittygoo
Copy link

Im guessing this is the part you are looking for?

2019-11-14 16:31:06,759 INFO elastichq ConnectionService.create_connection:58 Cert File: /certs/tls.crt
2019-11-14 16:31:06,759 INFO elastichq ConnectionService.create_connection:73 Basic Auth is True
2019-11-14 16:31:06,760 INFO elastichq ConnectionService.create_connection:75 SSL enabled
2019-11-14 16:31:06,760 INFO elastichq ConnectionService.create_connection:82 Verify Certs is True
config settings.json not found, searched /etc/elastic-hq/settings.json,~/settings.json,/src/settings.json,/src/elastichq/settings.json,/src/elastichq/config/settings.json```

@royrusso
Copy link
Member

Yeah. I just added that. So that's the most recent code changes. Is it working? If not, stack trace?

@gittygoo
Copy link

I had to run with :development on the docker image to be able to pull this one
On the bottom says "Installed Version: 3.5.7. Latest version: 3.5.6 Consider upgrading!"

But no luck yet :(

2019-11-14 16:36:32,339 INFO supervisord started with pid 1
2019-11-14 16:36:33,344 INFO spawned: 'gunicorn' with pid 8
2019-11-14 16:36:34,348 INFO success: gunicorn entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
loading config /src/elastichq/config/logger.json
2019-11-14 16:36:36,082 INFO engineio server.__init__:140 Server initialized for eventlet.
loading config /src/elastichq/config/logger_debug.json
2019-11-14 16:36:39,972 DEBUG sqlalchemy.orm.path_registry path_registry.set:63 set 'memoized_setups' on path 'EntityRegistry((<Mapper at 0x7f5551fd2cc0; ClusterModel>,))' to '{}'
2019-11-14 16:36:39,974 DEBUG sqlalchemy.pool.NullPool pool.__connect:668 Created new connection <sqlite3.Connection object at 0x7f55519599d0>
2019-11-14 16:36:39,975 DEBUG sqlalchemy.pool.NullPool pool.checkout:550 Connection <sqlite3.Connection object at 0x7f55519599d0> checked out from pool
2019-11-14 16:36:39,976 INFO sqlalchemy.engine.base.Engine base._begin_impl:682 BEGIN (implicit)
2019-11-14 16:36:39,977 INFO sqlalchemy.engine.base.Engine base._execute_context:1151 SELECT cluster.cluster_name AS cluster_cluster_name, cluster.cluster_ip AS cluster_cluster_ip, cluster.cluster_port AS cluster_cluster_port, cluster.cluster_scheme AS cluster_cluster_scheme, cluster.cluster_version AS cluster_cluster_version, cluster.cluster_username AS cluster_cluster_username, cluster.cluster_password AS cluster_cluster_password
FROM cluster
2019-11-14 16:36:39,977 INFO sqlalchemy.engine.base.Engine base._execute_context:1154 ()
2019-11-14 16:36:39,979 DEBUG sqlalchemy.engine.base.Engine result._init_metadata:681 Col ('cluster_cluster_name', 'cluster_cluster_ip', 'cluster_cluster_port', 'cluster_cluster_scheme', 'cluster_cluster_version', 'cluster_cluster_username', 'cluster_cluster_password')
2019-11-14 16:36:39,984 DEBUG sqlalchemy.pool.NullPool pool._finalize_fairy:697 Connection <sqlite3.Connection object at 0x7f55519599d0> being returned to pool
2019-11-14 16:36:39,984 DEBUG sqlalchemy.pool.NullPool pool._reset:869 Connection <sqlite3.Connection object at 0x7f55519599d0> rollback-on-return, via agent
2019-11-14 16:36:39,985 INFO sqlalchemy.engine.base.Engine base._rollback_impl:702 ROLLBACK
2019-11-14 16:36:39,985 DEBUG sqlalchemy.pool.NullPool pool._close_connection:314 Closing connection <sqlite3.Connection object at 0x7f55519599d0>
2019-11-14 16:36:40,016 DEBUG sqlalchemy.orm.path_registry path_registry.set:63 set 'memoized_setups' on path 'EntityRegistry((<Mapper at 0x7f5551fd2cc0; ClusterModel>,))' to '{}'
2019-11-14 16:36:40,018 DEBUG sqlalchemy.pool.NullPool pool.__connect:668 Created new connection <sqlite3.Connection object at 0x7f55518a11f0>
2019-11-14 16:36:40,018 DEBUG sqlalchemy.pool.NullPool pool.checkout:550 Connection <sqlite3.Connection object at 0x7f55518a11f0> checked out from pool
2019-11-14 16:36:40,019 INFO sqlalchemy.engine.base.Engine base._begin_impl:682 BEGIN (implicit)
2019-11-14 16:36:40,020 INFO sqlalchemy.engine.base.Engine base._execute_context:1151 SELECT cluster.cluster_name AS cluster_cluster_name, cluster.cluster_ip AS cluster_cluster_ip, cluster.cluster_port AS cluster_cluster_port, cluster.cluster_scheme AS cluster_cluster_scheme, cluster.cluster_version AS cluster_cluster_version, cluster.cluster_username AS cluster_cluster_username, cluster.cluster_password AS cluster_cluster_password
FROM cluster
2019-11-14 16:36:40,020 INFO sqlalchemy.engine.base.Engine base._execute_context:1154 ()
2019-11-14 16:36:40,022 DEBUG sqlalchemy.engine.base.Engine result._init_metadata:681 Col ('cluster_cluster_name', 'cluster_cluster_ip', 'cluster_cluster_port', 'cluster_cluster_scheme', 'cluster_cluster_version', 'cluster_cluster_username', 'cluster_cluster_password')
2019-11-14 16:36:40,024 DEBUG sqlalchemy.pool.NullPool pool._finalize_fairy:697 Connection <sqlite3.Connection object at 0x7f55518a11f0> being returned to pool
2019-11-14 16:36:40,024 DEBUG sqlalchemy.pool.NullPool pool._reset:869 Connection <sqlite3.Connection object at 0x7f55518a11f0> rollback-on-return, via agent
2019-11-14 16:36:40,025 INFO sqlalchemy.engine.base.Engine base._rollback_impl:702 ROLLBACK
2019-11-14 16:36:40,025 DEBUG sqlalchemy.pool.NullPool pool._close_connection:314 Closing connection <sqlite3.Connection object at 0x7f55518a11f0>
2019-11-14 16:36:40,072 DEBUG sqlalchemy.orm.path_registry path_registry.set:63 set 'memoized_setups' on path 'EntityRegistry((<Mapper at 0x7f5551fd2cc0; ClusterModel>,))' to '{}'
2019-11-14 16:36:40,074 DEBUG sqlalchemy.pool.NullPool pool.__connect:668 Created new connection <sqlite3.Connection object at 0x7f55518b31f0>
2019-11-14 16:36:40,074 DEBUG sqlalchemy.pool.NullPool pool.checkout:550 Connection <sqlite3.Connection object at 0x7f55518b31f0> checked out from pool
2019-11-14 16:36:40,075 INFO sqlalchemy.engine.base.Engine base._begin_impl:682 BEGIN (implicit)
2019-11-14 16:36:40,076 INFO sqlalchemy.engine.base.Engine base._execute_context:1151 SELECT cluster.cluster_name AS cluster_cluster_name, cluster.cluster_ip AS cluster_cluster_ip, cluster.cluster_port AS cluster_cluster_port, cluster.cluster_scheme AS cluster_cluster_scheme, cluster.cluster_version AS cluster_cluster_version, cluster.cluster_username AS cluster_cluster_username, cluster.cluster_password AS cluster_cluster_password
FROM cluster
2019-11-14 16:36:40,076 INFO sqlalchemy.engine.base.Engine base._execute_context:1154 ()
2019-11-14 16:36:40,078 DEBUG sqlalchemy.engine.base.Engine result._init_metadata:681 Col ('cluster_cluster_name', 'cluster_cluster_ip', 'cluster_cluster_port', 'cluster_cluster_scheme', 'cluster_cluster_version', 'cluster_cluster_username', 'cluster_cluster_password')
2019-11-14 16:36:40,079 DEBUG elastichq status.get:63 {"name": "ElasticHQ", "installed_version": "3.5.7", "current_stable_version": "3.5.7", "tagline": "You know, for Elasticsearch", "clusters": [], "default_url": "https://XXX:[email protected]:31425"}
2019-11-14 16:36:40,081 DEBUG sqlalchemy.pool.NullPool pool._finalize_fairy:697 Connection <sqlite3.Connection object at 0x7f55518b31f0> being returned to pool
2019-11-14 16:36:40,081 DEBUG sqlalchemy.pool.NullPool pool._reset:869 Connection <sqlite3.Connection object at 0x7f55518b31f0> rollback-on-return, via agent
2019-11-14 16:36:40,082 INFO sqlalchemy.engine.base.Engine base._rollback_impl:702 ROLLBACK
2019-11-14 16:36:40,082 DEBUG sqlalchemy.pool.NullPool pool._close_connection:314 Closing connection <sqlite3.Connection object at 0x7f55518b31f0>
None
None
2019-11-14 16:36:43,111 INFO elastichq ConnectionService.create_connection:57 Verify: False
2019-11-14 16:36:43,112 INFO elastichq ConnectionService.create_connection:58 Cert File: /certs/tls.crt
2019-11-14 16:36:43,112 INFO elastichq ConnectionService.create_connection:73 Basic Auth is True
2019-11-14 16:36:43,113 INFO elastichq ConnectionService.create_connection:75 SSL enabled
2019-11-14 16:36:43,113 INFO elastichq ConnectionService.create_connection:82 Verify Certs is True
config settings.json not found, searched /etc/elastic-hq/settings.json,~/settings.json,/src/settings.json,/src/elastichq/settings.json,/src/elastichq/config/settings.json
--- Logging error ---
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 601, in urlopen
chunked=chunked)
File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 346, in _make_request
self._validate_conn(conn)
File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 850, in _validate_conn
conn.connect()
File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 326, in connect
ssl_context=context)
File "/usr/local/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 329, in ssl_wrap_socket
return context.wrap_socket(sock, server_hostname=server_hostname)
File "/usr/local/lib/python3.6/site-packages/eventlet/green/ssl.py", line 438, in wrap_socket
return GreenSSLSocket(sock, *a, _context=self, **kw)
File "/usr/local/lib/python3.6/site-packages/eventlet/green/ssl.py", line 128, in __init__
self.do_handshake()
File "/usr/local/lib/python3.6/site-packages/eventlet/green/ssl.py", line 301, in do_handshake
super(GreenSSLSocket, self).do_handshake)
File "/usr/local/lib/python3.6/site-packages/eventlet/green/ssl.py", line 150, in _call_trampolining
return func(*a, **kw)
File "/usr/local/lib/python3.6/ssl.py", line 1077, in do_handshake
self._sslobj.do_handshake()
File "/usr/local/lib/python3.6/ssl.py", line 689, in do_handshake
self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
timeout=timeout
File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 639, in urlopen
_stacktrace=sys.exc_info()[2])
File "/usr/local/lib/python3.6/site-packages/urllib3/util/retry.py", line 388, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='my.host', port=31425): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)'),))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/src/elastichq/service/ConnectionService.py", line 84, in create_connection
timeout=REQUEST_TIMEOUT, verify=ca_certs, cert=client_cert_credentials)
File "/usr/local/lib/python3.6/site-packages/requests/api.py", line 75, in get
return request('get', url, params=params, **kwargs)
File "/usr/local/lib/python3.6/site-packages/requests/api.py", line 60, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 514, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='my.host', port=31425): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)'),))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/logging/__init__.py", line 994, in emit
msg = self.format(record)
File "/usr/local/lib/python3.6/logging/__init__.py", line 840, in format
return fmt.format(record)
File "/usr/local/lib/python3.6/logging/__init__.py", line 577, in format
record.message = record.getMessage()
File "/usr/local/lib/python3.6/logging/__init__.py", line 338, in getMessage
msg = msg % self.args
TypeError: not all arguments converted during string formatting
Call stack:
File "/usr/local/lib/python3.6/site-packages/eventlet/greenthread.py", line 221, in main
result = function(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/gunicorn/workers/geventlet.py", line 116, in handle
super(EventletWorker, self).handle(listener, client, addr)
File "/usr/local/lib/python3.6/site-packages/gunicorn/workers/async.py", line 56, in handle
self.handle_request(listener_name, req, client, addr)
File "/usr/local/lib/python3.6/site-packages/gunicorn/workers/async.py", line 107, in handle_request
respiter = self.wsgi(environ, resp.start_response)
File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1997, in __call__
return self.wsgi_app(environ, start_response)
File "/usr/local/lib/python3.6/site-packages/flask_socketio/__init__.py", line 43, in __call__
start_response)
File "/usr/local/lib/python3.6/site-packages/engineio/middleware.py", line 67, in __call__
return self.wsgi_app(environ, start_response)
File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1982, in wsgi_app
response = self.full_dispatch_request()
File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1612, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1598, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/usr/local/lib/python3.6/site-packages/flask_restful/__init__.py", line 480, in wrapper
resp = resource(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/flask/views.py", line 84, in view
return self.dispatch_request(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/flask_restful/__init__.py", line 595, in dispatch_request
resp = meth(*args, **kwargs)
File "/src/elastichq/common/exceptions.py", line 29, in _request_wrapper
return functor(*args, **kwargs)
File "/src/elastichq/api/clusters.py", line 129, in post
client_cert=client_cert)
File "/src/elastichq/service/ConnectionService.py", line 152, in create_connection
LOG.error(message, ex)
Message: 'Unable to create connection to: https://my.host:31425'
Arguments: (SSLError(MaxRetryError("HTTPSConnectionPool(host='my.host', port=31425): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)'),))",),),)
--- Logging error ---
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 601, in urlopen
chunked=chunked)
File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 346, in _make_request
self._validate_conn(conn)
File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 850, in _validate_conn
conn.connect()
File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 326, in connect
ssl_context=context)
File "/usr/local/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 329, in ssl_wrap_socket
return context.wrap_socket(sock, server_hostname=server_hostname)
File "/usr/local/lib/python3.6/site-packages/eventlet/green/ssl.py", line 438, in wrap_socket
return GreenSSLSocket(sock, *a, _context=self, **kw)
File "/usr/local/lib/python3.6/site-packages/eventlet/green/ssl.py", line 128, in __init__
self.do_handshake()
File "/usr/local/lib/python3.6/site-packages/eventlet/green/ssl.py", line 301, in do_handshake
super(GreenSSLSocket, self).do_handshake)
File "/usr/local/lib/python3.6/site-packages/eventlet/green/ssl.py", line 150, in _call_trampolining
return func(*a, **kw)
File "/usr/local/lib/python3.6/ssl.py", line 1077, in do_handshake
self._sslobj.do_handshake()
File "/usr/local/lib/python3.6/ssl.py", line 689, in do_handshake
self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)
During handling of the above exception, another exception occurred:```

@royrusso
Copy link
Member

Can you post your ENV vars? Looks like you're running with HQ_VERIFY_CERTS=False

@royrusso royrusso reopened this Nov 14, 2019
@gittygoo
Copy link

Yes i am, posted them here previously, here they are again

options im using
HQ_DEBUG = True
HQ_DEFAULT_URL = https://XXXX:[email protected]:31425
HQ_ENABLE_SSL = True
HQ_VERIFY_CERTS = False
VERIFY_CERTS = False
HQ_CA_CERTS = /certs/tls.crt

@royrusso royrusso modified the milestones: 3.5.7, 3.5.12 Nov 14, 2019
@acarsercan
Copy link

acarsercan commented Dec 9, 2019

Hi,

I'm having the problem above with 3.5.12

Passing ENV:

1   │         env:
  22   │         - name: HQ_VERIFY_CERTS
  23   │           value: "False"

elasticsearch/es-hq-69b59cdb48-rprvs[es-hq]: Arguments: (SSLError(MaxRetryError("HTTPSConnectionPool(host='XXXX', port=9200): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)'),))",),),)

@mgiammarco
Copy link

Please reopen I have the same problem with latest version

@sangdrax8
Copy link

I am trying to test this now, and can't get the docker to read my cert or turn off verification.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

9 participants