Unable to run HUE using the latest Helm chart on EKS

[matousek-martin]

Is the issue already present in https://github.com/cloudera/hue/issues or discussed in the forum https://discourse.gethue.com?
No, I have been googling for the past 2 days.

Describe the bug:
After deploying the helm chart to my EKS cluster, pods run, but logs show the following error. Could anyone point me in the right direction, please? Thank you in advance.

11/Nov/2022 03:26:10 -0800] models       ERROR    Failed to create home directory for user hue: HTTPConnectionPool(host='localhost', port=50070): Max retries exceeded with url: /webhdfs/v1?user.name=hue&doas=hue&op=GETHOMEDIRECTORY (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fa595f08748>: Failed to establish a new connection: [Errno 111] Connection refused',))
Traceback (most recent call last):
  File "/usr/share/hue/build/env/lib/python3.6/site-packages/urllib3/connection.py", line 170, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw
  File "/usr/share/hue/build/env/lib/python3.6/site-packages/urllib3/util/connection.py", line 96, in create_connection
    raise err
  File "/usr/share/hue/build/env/lib/python3.6/site-packages/urllib3/util/connection.py", line 86, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/share/hue/build/env/lib/python3.6/site-packages/urllib3/connectionpool.py", line 706, in urlopen
    chunked=chunked,
  File "/usr/share/hue/build/env/lib/python3.6/site-packages/urllib3/connectionpool.py", line 394, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/share/hue/build/env/lib/python3.6/site-packages/urllib3/connection.py", line 234, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/usr/lib/python3.6/http/client.py", line 1281, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.6/http/client.py", line 1327, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.6/http/client.py", line 1276, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.6/http/client.py", line 1042, in _send_output
    self.send(msg)
  File "/usr/lib/python3.6/http/client.py", line 980, in send
    self.connect()
  File "/usr/share/hue/build/env/lib/python3.6/site-packages/urllib3/connection.py", line 200, in connect
    conn = self._new_conn()
  File "/usr/share/hue/build/env/lib/python3.6/site-packages/urllib3/connection.py", line 182, in _new_conn
    self, "Failed to establish a new connection: %s" % e
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7fa595f08748>: Failed to establish a new connection: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/share/hue/build/env/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/usr/share/hue/build/env/lib/python3.6/site-packages/urllib3/connectionpool.py", line 756, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/usr/share/hue/build/env/lib/python3.6/site-packages/urllib3/util/retry.py", line 574, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=50070): Max retries exceeded with url: /webhdfs/v1?user.name=hue&doas=hue&op=GETHOMEDIRECTORY (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fa595f08748>: Failed to establish a new connection: [Errno 111] Connection refused',))

The error is a bit longer, but repetitive. I think I captured the important part.

Steps to reproduce it?
I am running this values.yaml

image:
  registry: "gethue"
  tag: 4.10.0
  pullPolicy: IfNotPresent
hue:
  replicas: 2
  database:
    create: false
    persist: false
    engine: "postgresql_psycopg2"
    host: "${POSTGRES_HOST}"
    port: 5432
    user: "dbadmin"
    password: "${POSTGRES_PASSWORD}"
    name: "postgres"
#    storageName: "microk8s-hostpath"
  interpreters: |
    [[[postgresql]]]
      name = postgresql
      interface=sqlalchemy
      options='{"url": "postgresql://dbadmin:${POSTGRES_PASSWORD}@${POSTGRES_HOST}:5432/postgres"}'
  ini: |
    [desktop]
    secret_key=hue123
    app_blacklist=filebrowser,search,hbase,security,jobbrowser,oozie
    django_debug_mode=false
    gunicorn_work_class=sync
    enable_prometheus=true
    [[task_server]]
    enabled=false
    broker_url=redis://redis:6379/0
    result_cache='{"BACKEND": "django_redis.cache.RedisCache", "LOCATION": "redis://redis:6379/0", "OPTIONS": {"CLIENT_CLASS": "django_redis.client.DefaultClient"},"KEY_PREFIX": "queries"}'
    celery_result_backend=redis://redis:6379/0
balancer:
  enabled: true
api:
  enabled: false
  domain: api.gethue.com
workers:
  enabled: false
  replicas: 1
scheduler:
  enabled: false
websocket:
  enabled: false
  replicas: 1
monitoring:
  enabled: false
alerting:
  enabled: false
tracing:
  enabled: false
cron:
  enabled: false
  emailStats: "0 13 * * WED,SUN"
ingress:
  create: true
  hasAuth: false
  type: "alb"
  domain: <host>
  path: "/hue*"
  pathType: ImplementationSpecific
  annotations: {
      alb.ingress.kubernetes.io/backend-protocol: HTTP,
      alb.ingress.kubernetes.io/certificate-arn: <cert>,
      alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS":443}]',
      alb.ingress.kubernetes.io/ssl-redirect: '443',
      external-dns.alpha.kubernetes.io/hostname: <host>,
      alb.ingress.kubernetes.io/group.name: main.ingress,
  }
  email: <mail>
  loadBalancerIp: null
aws:
  accessKeyId: "${AWS_ACCESS_KEY_ID}"
  secretAccessKey: "${AWS_SECRET_ACCESS_KEY}"
  awsRegion: "${AWS_REGION}"
hive:
  site: |
    <!--
    <property>
      <name>hive.support.concurrency</name>
      <value>false</value>
      <description>Whether hive supports concurrency or not. A zookeeper instance must be up and running for the default hive lock manager to support read-write locks.</description>
    </property>
    -->
## Pod Annotations
# podAnnotations: {}

## Pod Labels
# podLabels: {}

Hue version or source? (e.g. open source 4.5, CDH 5.16, CDP 1.0...). System info (e.g. OS, Browser...).
Helm chart version == 1.0.3
Hue == 4.10.0
EKS == 1.21

[Harshg999]

Hi @matousek-martin, looking at the error:

HTTPConnectionPool(host='localhost', port=50070): Max retries exceeded with url: /webhdfs/v1?user.name=hue&doas=hue&op=GETHOMEDIRECTORY (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fa595f08748>: Failed to establish a new connection: [Errno 111] Connection refused',))

It feels like you're trying to enable HDFS by configuring under [[hdfs_clusters]] I think but Hue is not able to find the WebHDFS URL to talk to HDFS. Therefore, it's defaulting back to localhost:50070 URL and retrying.

hue/desktop/libs/hadoop/src/hadoop/conf.py

Lines 89 to 94 in b629542

	WEBHDFS_URL=Config(
	"webhdfs_url",
	help="The URL to WebHDFS/HttpFS service. Defaults to the WebHDFS URL on the NameNode.",
	type=str,
	default="http://localhost:50070/webhdfs/v1"
	),

[matousek-martin]

That seems to be the case, but I do not have HDFS specified, as we do not have a HDFS cluster. I even tried to explicitly disable it via:

[hadoop]
[[hdfs_clusters]]
[[[default]]]
enabled=false

inside interpreter and ini, but it didn't help. Is there any way to disable it or is Hue reliant on Hadoop? I only need it to connect to S3 and SQL.

[Harshg999]

If connecting to HDFS is not your use case, then can you once try commenting out everything under [hadoop] and try again? Also try removing any HDFS config under [interpreters] as well.

Also, do you have connectors enabled? If yes, then remove any HDFS config if by any chance present there as well.

[matousek-martin]

That is exactly where I am stuck at. These are my interpreters and ini

  interpreters: |
    [[[postgresql]]]
      name = postgresql
      interface=sqlalchemy
      options='{"url": "postgresql://dbadmin:${POSTGRES_PASSWORD}@${POSTGRES_HOST}:5432/postgres"}'
  ini: |
    [desktop]
    secret_key=hue123
    app_blacklist=filebrowser,search,hbase,security,jobbrowser,oozie
    django_debug_mode=false
    gunicorn_work_class=sync
    enable_prometheus=true
    [[task_server]]
    enabled=false
    broker_url=redis://redis:6379/0
    result_cache='{"BACKEND": "django_redis.cache.RedisCache", "LOCATION": "redis://redis:6379/0", "OPTIONS": {"CLIENT_CLASS": "django_redis.client.DefaultClient"},"KEY_PREFIX": "queries"}'
    celery_result_backend=redis://redis:6379/0

There are only postgres in interpreters and desktop + task_servers in ini. They were there by default by the helm chart. I am not setting HDFS, despite that it is trying to connect to it.

[Harshg999]

Maybe this is enabling it? Once try commenting it out in your cluster.

hue/tools/docker/hue/conf3/z-hue-overrides.ini

Lines 189 to 206 in b629542

	[hadoop]
	# Configuration for HDFS NameNode
	# ------------------------------------------------------------------------
	[[hdfs_clusters]]
	# HA support by using HttpFs
	# [[[default]]]
	# Enter the filesystem uri
	## fs_defaultfs=hdfs://localhost:8020
	# Use WebHdfs/HttpFs as the communication mechanism.
	# Domain should be the NameNode or HttpFs host.
	# Default port is 14000 for HttpFs.
	## webhdfs_url=http://localhost:50070/webhdfs/v1
	## is_enabled=false

On a side note, if your use case is to connect to S3, then I think you have to remove the filebrowser app from the app_blacklist.

[matousek-martin]

Thanks for the tip about the file browser! I checked inside the pod for that file. It is disabled there as well. Do you have any other ideas?

[Harshg999]

Comment out [[hdfs_clusters]] and [hadoop] as well and once check again?

[matousek-martin]

Still the same issue. Also encountered this in hue.ini

# Whether Hue should list this HDFS cluster. For historical reason there is no way to disable HDFS.
## is_enabled=true

Is what I am attempting even possible?

[Harshg999]

Possibly this is true and might be required for some historical compatibility issues. The last idea which comes to my mind, is to suppress the source of these errors if they are not related to your use-case (generally not-recommended but Hue is customisable and open-sourced to play around 🙂).

You can try it by commenting-out the code sections which you feel are initiating these calls (for example, it looks like it's trying to create the home directory for every user in Hue in HDFS).

[y2s82]

If disabling hdfs_cluster is your intent, also try uncommenting and setting:
fs_defaultfs=
(left intentionally empty)

This helped me not get more errors regarding hdfs when setting up HUE for non-hadoop use.

[Harshg999]

Interesting, that can be a possible workaround as well. Setting fs_defaultfs and webhdfs_url to blank can remove its default value and the call might not go to HDFS to initiate the retries.
Worth a try, thanks @y2s82!

[matousek-martin]

Thanks to both of you for all the help. It turns out I am an idiot and Hue was not working for me due to a poorly configured ingress. Apologies and thanks again.

FYI, the error still persisted even after Setting fs_defaultfs and webhdfs_url to blank. Only the error change slightly, because it was connecting to :50700 instead of localhost:50700