Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HA Component Errors: Bath and North East Somerset #347

Closed
RealGandy opened this issue Sep 26, 2023 · 17 comments
Closed

HA Component Errors: Bath and North East Somerset #347

RealGandy opened this issue Sep 26, 2023 · 17 comments
Assignees
Labels
bug Something isn't working home assistant custom component Something related to the home assistant custom component

Comments

@RealGandy
Copy link

RealGandy commented Sep 26, 2023

Hi

Just downloaded via HACS and tried to set up for my local council [Bath and North East Somerset] but getting no data.

The following errors are thrown in the homeassistant.log file, suggesting that the HACS install hasn't added all the required files?

I don't see any council.py files in the custom_components/uk_bin_collection directory, or should they be somewhere else?

2023-09-26 10:00:41.083 ERROR (MainThread) [custom_components.uk_bin_collection.sensor] Unexpected error fetching Spruce Way data: No module named 'BathAndNorthEastSomersetCouncil'
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 290, in _async_refresh
    self.data = await self._async_update_data()
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/uk_bin_collection/sensor.py", line 137, in _async_update_data
    data = await self.hass.async_add_executor_job(self.ukbcd.run)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/uk_bin_collection/uk_bin_collection/collect_data.py", line 64, in run
    council_module = self.get_council_module(council_module_str)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/uk_bin_collection/uk_bin_collection/collect_data.py", line 51, in get_council_module
    return importlib.import_module(council_module_str)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1140, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'BathAndNorthEastSomersetCouncil'
2023-09-26 10:00:41.090 ERROR (MainThread) [homeassistant.components.sensor] Error while setting up uk_bin_collection platform for sensor
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 290, in _async_refresh
    self.data = await self._async_update_data()
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/uk_bin_collection/sensor.py", line 137, in _async_update_data
    data = await self.hass.async_add_executor_job(self.ukbcd.run)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/uk_bin_collection/uk_bin_collection/collect_data.py", line 64, in run
    council_module = self.get_council_module(council_module_str)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/uk_bin_collection/uk_bin_collection/collect_data.py", line 51, in get_council_module
    return importlib.import_module(council_module_str)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1140, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'BathAndNorthEastSomersetCouncil'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 359, in _async_setup_platform
    await asyncio.shield(task)
  File "/config/custom_components/uk_bin_collection/sensor.py", line 68, in async_setup_entry
    await coordinator.async_config_entry_first_refresh()
  File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 262, in async_config_entry_first_refresh
    raise ex
homeassistant.exceptions.ConfigEntryNotReady: No module named 'BathAndNorthEastSomersetCouncil'
@OliverCullimore
Copy link
Collaborator

@RealGandy there's currently an issue with releases being published which means that councils that were recently added or fixed won't yet work from HACS.

Thanks in advance for your patience while we resolve this.

@OliverCullimore
Copy link
Collaborator

OliverCullimore commented Sep 27, 2023

@RealGandy the releases have now been fixed so if you can update the custom component via HACS and ensure it's shows "2181ae2" as the version downloaded when you click into it.

image

Then remove and re-add the integration if it doesn't automatically start working.

@RealGandy
Copy link
Author

RealGandy commented Sep 28, 2023

@OliverCullimore

Thanks.

Updated and tried again .. fully removed and re-downloaded that version. Confirmed that was the same.

Configured my council OK, no errors thrown now, but it doesn't display any info, just shows 'No devices or entities'.

HACS just offered me a newer version (a552211) which I also downloaded, removed original config and tried again.

Still shows 'No devices or entities'

How can I help find the issue?

Thanks

@RealGandy
Copy link
Author

So.. logged onto HASS terminal and tried the python script from the command line and get the following errors.
(I used a valid UPRN in the call, not XXXXX)

Looks like there is an SSL negotiation issue going on here.?

hass:/usr/local/lib/python3.11/site-packages/uk_bin_collection/uk_bin_collection# python collect_data.py BathAndNorthEastSomersetCouncil https://www.bathnes.gov.uk/webforms/waste/collectionday/ -s -u xxxxxxxxxxx
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 714, in urlopen
    httplib_response = self._make_request(
                       ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 403, in _make_request
    self._validate_conn(conn)
  File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 1053, in _validate_conn
    conn.connect()
  File "/usr/local/lib/python3.11/site-packages/urllib3/connection.py", line 419, in connect
    self.sock = ssl_wrap_socket(
                ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/util/ssl_.py", line 449, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(
               ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/ssl.py", line 517, in wrap_socket
    return self.sslsocket_class._create(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/ssl.py", line 1108, in _create
    self.do_handshake()
  File "/usr/local/lib/python3.11/ssl.py", line 1379, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: UNSAFE_LEGACY_RENEGOTIATION_DISABLED] unsafe legacy renegotiation disabled (_ssl.c:1006)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/requests/adapters.py", line 486, in send
    resp = conn.urlopen(
           ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 798, in urlopen
    retries = retries.increment(
              ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/util/retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='www.bathnes.gov.uk', port=443): Max retries exceeded with url: /webapi/api/BinsAPI/v2/getbartecroute/xxxxxxxx/true (Caused by SSLError(SSLError(1, '[SSL: UNSAFE_LEGACY_RENEGOTIATION_DISABLED] unsafe legacy renegotiation disabled (_ssl.c:1006)')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/uk_bin_collection/uk_bin_collection/collect_data.py", line 88, in <module>
    data = app.run()
           ^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/uk_bin_collection/uk_bin_collection/collect_data.py", line 71, in run
    return self.client_code(
           ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/uk_bin_collection/uk_bin_collection/collect_data.py", line 59, in client_code
    return get_bin_data_class.template_method(address_url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/uk_bin_collection/uk_bin_collection/get_bin_data.py", line 70, in template_method
    bin_data_dict = self.parse_data(
                    ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/uk_bin_collection/uk_bin_collection/councils/BathAndNorthEastSomersetCouncil.py", line 37, in parse_data
    response = requests.get(
               ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/requests/api.py", line 73, in get
    return request("get", url, params=params, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/requests/adapters.py", line 517, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='www.bathnes.gov.uk', port=443): Max retries exceeded with url: /webapi/api/BinsAPI/v2/getbartecroute/xxxxxxxxxx/true (Caused by SSLError(SSLError(1, '[SSL: UNSAFE_LEGACY_RENEGOTIATION_DISABLED] unsafe legacy renegotiation disabled (_ssl.c:1006)')))
hass:/usr/local/lib/python3.11/site-packages/uk_bin_collection/uk_bin_collection#

@OliverCullimore OliverCullimore self-assigned this Sep 29, 2023
@OliverCullimore OliverCullimore added the bug Something isn't working label Sep 29, 2023
@OliverCullimore
Copy link
Collaborator

@RealGandy sorry disregard that last message I was looking into 2 things at once and had the wrong council's files open.

Can you instead firstly check your UPRN is correct by putting it in the UPRN Search field of https://www.findmyaddress.co.uk/search and checking the returned address is the expected one please?

If that's all good can you try going to the following URL, replacing the X's with your UPRN please?

https://www.bathnes.gov.uk/webapi/api/BinsAPI/v2/getbartecroute/XXXXXXXXXXXX/true

You should see it load a JSON string starting with {"urpn":XXXXXXXXXXXX.0

Otherwise, it may appear as if it hasn't loaded anything if you enter an invalid UPRN.

Our test UPRN of 100120000855 for Bath is working as expected in my home assistant and is instantly populating 3 entities.

@RealGandy
Copy link
Author

RealGandy commented Sep 29, 2023

@OliverCullimore

Yes, UPRN is correct - I found it via findmyaddress.co.uk and have double checked.

I plug it into that URL and get a valid JSON payload.

I'm running hass as a container, so I can log onto the terminal to run the manual code to collect data and I get the errors thrown as shown on my previous post. It looks to be an SSL negotiation error.

I get the same error if I try to connect to the BANES URL just by importing requests in python running in the hass terminal.

hass:/usr/local/lib/python3.11/site-packages/uk_bin_collection/uk_bin_collection# python
Python 3.11.5 (main, Aug 28 2023, 16:01:57) [GCC 12.2.1 20220924] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> r = requests.get('https://www.bathnes.gov.uk/webapi/api/BinsAPI/v2/getbartecroute/XXXXXXXXXXX/true')
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 714, in urlopen
    httplib_response = self._make_request(
                       ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 403, in _make_request
    self._validate_conn(conn)
  File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 1053, in _validate_conn
    conn.connect()
  File "/usr/local/lib/python3.11/site-packages/urllib3/connection.py", line 419, in connect
    self.sock = ssl_wrap_socket(
                ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/util/ssl_.py", line 449, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(
               ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/ssl.py", line 517, in wrap_socket
    return self.sslsocket_class._create(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/ssl.py", line 1108, in _create
    self.do_handshake()
  File "/usr/local/lib/python3.11/ssl.py", line 1379, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: UNSAFE_LEGACY_RENEGOTIATION_DISABLED] unsafe legacy renegotiation disabled (_ssl.c:1006)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/requests/adapters.py", line 486, in send
    resp = conn.urlopen(
           ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 798, in urlopen
    retries = retries.increment(
              ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/util/retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='www.bathnes.gov.uk', port=443): Max retries exceeded with url: /webapi/api/BinsAPI/v2/getbartecroute/XXXXXXXXXXX/true (Caused by SSLError(SSLError(1, '[SSL: UNSAFE_LEGACY_RENEGOTIATION_DISABLED] unsafe legacy renegotiation disabled (_ssl.c:1006)')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.11/site-packages/requests/api.py", line 73, in get
    return request("get", url, params=params, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/requests/adapters.py", line 517, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='www.bathnes.gov.uk', port=443): Max retries exceeded with url: /webapi/api/BinsAPI/v2/getbartecroute/XXXXXXXXXXX/true (Caused by SSLError(SSLError(1, '[SSL: UNSAFE_LEGACY_RENEGOTIATION_DISABLED] unsafe legacy renegotiation disabled (_ssl.c:1006)')))

I've seen something similar before with the IMAP integration which was fixed by home-assistant/core#91896
In that case, python or whatever library was being used was refusing to successfully negotiate with the server using what it considered to be insecure cipher suites and this change allowed the user to select different levels until a working one was found.

I've also found this which gives a suggested workaround by using urllib3:
urllib3/urllib3#2653 (comment)

I've tested that example and it works fine for me, maybe you can look at this?

Thanks

@OliverCullimore
Copy link
Collaborator

@RealGandy can you confirm which version number and type (e.g. Home Assistant OS/Core/Container/Supervised) you're running?

It just seems odd that the same version of the custom component is working fine on the Home Assistant container (2023.8.0.dev0) and my production instance (Home Assistant 2023.4.6, Supervisor 2023.09.2) too.

@OliverCullimore
Copy link
Collaborator

@RealGandy you can also give using requests via terminal another go but if you can run requests.packages.urllib3.disable_warnings() before running the requests.get command that's what the script uses to get around those SSL warnings

@OliverCullimore
Copy link
Collaborator

OliverCullimore commented Sep 29, 2023

I've redacted your UPRN for you also

@RealGandy
Copy link
Author

Running 2023.9.3 container.

Tried adding that line just before the .get and get the same error message.

I would have expected that error to be thrown in the hass log as well, but there are no errors relating to this component now.

Yes, very odd that you can connect fine yet my system cannot. I don't have any man-in-middle proxies that could be trying to break and remake the connection that might be causing this.

@OliverCullimore
Copy link
Collaborator

@RealGandy thanks for confirming, I've just set up a container using version 2023.9.3 and am experiencing the same where the entities don't get populated.

I can only assume the dev version I was using before may have some fixes/is more relaxed on this. I'll look to submit a PR with that workaround you suggested and we can see if that makes this work, I imagine other councils will likely run into the same as they're all using very similar code.

@RealGandy
Copy link
Author

Good to hear that you have reproduced it - was beginning to think it was me!

@OliverCullimore OliverCullimore added the home assistant custom component Something related to the home assistant custom component label Oct 11, 2023
@OliverCullimore OliverCullimore changed the title Errors thrown following setup routine, no data collected HA Component Errors: Bath and North East Somerset Oct 11, 2023
@OliverCullimore
Copy link
Collaborator

@RealGandy Looking like the fixes in #361 might fix this one for you 🤞

@RealGandy
Copy link
Author

RealGandy commented Oct 15, 2023

@OliverCullimore
Unfortunately not :(

Just updated to d8ccbcd, but now get the following error and the integration does not load.


2023-10-15 19:38:16.353 ERROR (SyncWorker_9) [homeassistant.util.package] Unable to install package uk-bin-collection>=0.13.1: ERROR: Cannot install uk-bin-collection==0.13.1 because these package versions have conflicting dependencies.
ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts

[notice] A new release of pip is available: 23.2.1 -> 23.3
[notice] To update, run: pip install --upgrade pip
2023-10-15 19:38:20.724 ERROR (SyncWorker_9) [homeassistant.util.package] Unable to install package uk-bin-collection>=0.13.1: ERROR: Cannot install uk-bin-collection==0.13.1 because these package versions have conflicting dependencies.
ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts

[notice] A new release of pip is available: 23.2.1 -> 23.3
[notice] To update, run: pip install --upgrade pip
2023-10-15 19:38:25.633 ERROR (SyncWorker_9) [homeassistant.util.package] Unable to install package uk-bin-collection>=0.13.1: ERROR: Cannot install uk-bin-collection==0.13.1 because these package versions have conflicting dependencies.
ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts

[notice] A new release of pip is available: 23.2.1 -> 23.3
[notice] To update, run: pip install --upgrade pip
2023-10-15 19:38:25.634 ERROR (MainThread) [homeassistant.setup] Setup failed for custom integration uk_bin_collection: Requirements for uk_bin_collection not found: ['uk-bin-collection>=0.13.1'].

@OliverCullimore
Copy link
Collaborator

@RealGandy thanks for those logs, I've replicated the same and submitted another PR ref #365 (comment)

@OliverCullimore
Copy link
Collaborator

@RealGandy the latest version appears to be working for me now if you can update and give it another go

@RealGandy
Copy link
Author

Yes. All working now!
Excellent.

Many thanks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working home assistant custom component Something related to the home assistant custom component
Projects
None yet
Development

No branches or pull requests

2 participants