add flag to MQTT Settings to use the old v3.4 entity ID names in Home Assistant #804

proddy · 2022-12-14T13:08:43Z

There may be unhappy users responding to the new format of the MQTT Entity ID (unqiueid in the MQTT Discovery config topic) that was introduced in 3.5.0b11 and uses the shortname (as seen in the Configuration screen and MQTT payloads) instead of the fullname which is the English translation used in previous versions.

The suggestion is to add a flag to enable the old format from v3.4.

If we get more votes I'll add it as a feature.

The text was updated successfully, but these errors were encountered:

sciurius · 2022-12-14T13:33:10Z

I have a number of HomeAssistant scripts that would require updating. Then there are some dashboards.
This is the least problematic.

More important is that the entries are logged (recorded) in the database that, on its turn, is used by a lot of grafana graphs and other external tools. Hunting these down is interesting. Preventing discontinuity in the logged data is challenging.

Yes, the amount of work to be done is finite. The impact is just bigger than I anticipated.

So I would very much appreciate a compatibility settings as suggested by proddy.

bbqkees · 2022-12-14T14:37:27Z

@tp1de would also like this.

tp1de · 2022-12-14T14:45:07Z

@tp1de would also like this.

... I have adjusted HA dashboards and scripts in the meantime. Anyhow it is the recommendation from HA to keep unique_id and object_id identical. But several users in Discord are now facing the consequences. Whenever 3.5 is going to be final version I would expect quite some more questions /complaints.

proddy · 2022-12-14T14:50:27Z

an alternative is to create a script using HA's CLI (https://github.com/home-assistant-ecosystem/home-assistant-cli) that renames all entities back to the old format.

tp1de · 2022-12-14T15:06:49Z

I am still in the "learning phase" of HA and recognized quite some behavior I do not like.
Any renaming of entities leaves orphaned statistics data in the database. I did a lot of testing to write statistics data directly with SQL statements to include the energy usage data from km200 and my PV into the energy dashboard and recognized that the data does not seem to be reorganized when entities do no exist anymore or are renamed.

A script to rename might be only a partial solution. But you are the experts ....

sciurius · 2022-12-14T15:25:10Z

an alternative is to create a script using HA's CLI (https://github.com/home-assistant-ecosystem/home-assistant-cli) that renames all entities back to the old format.

The names for the entities come from autodetection. Changing the 3.5 names to 3.4 (using the cli or whatever means) would not work since the 3.5 names will come back eventually.

proddy · 2022-12-14T18:52:09Z

The names for the entities come from autodetection. Changing the 3.5 names to 3.4 (using the cli or whatever means) would not work since the 3.5 names will come back eventually.

The script will change the v3.4 entity names to the newer v3.5 names, so retaining the history. You'll still need to update the scripts and Grafana, but it'll probably take you less time than for me to write a script or change the code to add a new flag ;)

sciurius · 2022-12-14T20:07:51Z

I misunderstood -- I thought the name changing was from 3.5 to 3.4.
Interesting suggestion. Changing the names in the database and updating the gateway to v3.5 will provide continuity in the database.
Thereafter there is all time to chase name mismatches in scripts and graphs.

sciurius · 2022-12-15T15:36:14Z

So I bit the bullet. This is what I did:

Create a rename script for the database entries (let me know if you're interested).
Modified EMS-ESP to not use MQTT (actually, I routed it to a test broker).
In the MQTT broker, I removed all autodiscovery items for EMS-ESP.
In HA, removed the EMS-ESP, boiler and thermostat from the MQTT integration.
Run the rename sql to update all database entries.
Reflashed the gateway with v3.5b11.
Re-attached to the boiler and verify it is working.
Modify EMS-ESP to use the real MQTT broker and enable autodiscovery.
Hunt down YAML files and change entity names.

Maybe some of the steps are unneccessary, but this worked for me.
Unfortunately, due to a recent change in HA the 'more info' popup will only show new data. 'show more' will show all data.

proddy · 2022-12-15T16:51:54Z

i did a PR last night with the flag to support <3.5 MQTT entity formatting. It needs some changes which I'll do tonight, so that would be another solution to keep the old naming.

#808

sciurius · 2022-12-16T07:06:07Z

Great. Even though I could deal with the changes I'm not sure the above procedure would be safe in the hands of average users.

proddy · 2022-12-16T07:09:37Z

Great. Even though I could deal with the changes I'm not sure the above procedure would be safe in the hands of average users.

would still be great to see the scripts if you don't mind sharing.

sciurius · 2022-12-16T07:22:41Z

Sure. I made a markdown document with all the entities and a small Perl program that reads the md file to produce a SQL rename script. I use PostgreSQL but I think the script will work with other SQL databases as well.

rename.zip

Note that the program generates a rename sql that groups everything in a single transaction, and rolls back at the end. This is for testing. If testing succeeds, or if you are reckless, change ROLLBACK to COMMIT and go.

mtc716 · 2022-12-17T07:27:39Z

So I bit the bullet. This is what I did:

Create a rename script for the database entries (let me know if you're interested).

Modified EMS-ESP to not use MQTT (actually, I routed it to a test broker).

In the MQTT broker, I removed all autodiscovery items for EMS-ESP.

In HA, removed the EMS-ESP, boiler and thermostat from the MQTT integration.

Run the rename sql to update all database entries.

Reflashed the gateway with v3.5b11.

Re-attached to the boiler and verify it is working.

Modify EMS-ESP to use the real MQTT broker and enable autodiscovery.

Hunt down YAML files and change entity names.

Maybe some of the steps are unneccessary, but this worked for me. Unfortunately, due to a recent change in HA the 'more info' popup will only show new data. 'show more' will show all data.

I did more or less the same by stopping recorder, modifying SQL DB and changing dashboard and yaml files. And it worked like charm. In the DB there are roughly 6 or 7 table columns at which you need to change the old to new name. And if you do not see the short term statistics, you need to check under developer settings and statistics. There might be the problem reported and by solving it there your statistics will be back.

proddy · 2022-12-17T08:17:02Z

That's great we have a solution that others can use. Did you try using the CLI tool like hass-cli entity rename <old> <new> ?

sciurius · 2022-12-17T09:35:29Z

In the DB there are roughly 6 or 7 table columns at which you need to change the old to new name.

Can you elaborate?

proddy · 2022-12-17T10:05:33Z

I've added an option to the MQTT Settings that enables the 'old' entity ID way as it was in v3.4. For backwards compatibility

proddy · 2022-12-17T10:09:59Z

@sciurius @mtc716 I'm closing this issue as we're preparing for v3.5 and moved the renaming HA chat to a new GH discussion here: #814

mtc716 · 2022-12-17T12:27:52Z

In the DB there are roughly 6 or 7 table columns at which you need to change the old to new name.

Can you elaborate?

Just check in #814 , I added my SQL ;-)

support 3 types of MQTT discovery entity id - #804

proddy · 2022-12-17T19:26:25Z

I added 3 options now

no base, use long descriptive entity names (v3.4 compatible)
no base, using the short names for the entity ID, matching the MQTT payloads
base prefixed, using short names as above supporting multiple instances of EMS-ESP

in b12.

mtc716 · 2022-12-17T20:14:27Z

Just checked the new b12 and it works like charm.
Only in the web, the dropdown field does not show the selected option once selected.

proddy · 2022-12-17T20:16:18Z

yeah just spotted that myself and fixed it. I should really test before I commit, doh :(

mtc716 · 2022-12-17T20:18:13Z

yeah just spotted that myself and fixed it. I should really test before I commit, doh :(

No big deal, well done anyhow...

sciurius · 2022-12-17T20:48:42Z

What is the default value for this flag?

proddy · 2022-12-17T20:56:35Z

What is the default value for this flag?

the 2nd/middle option - no prefixed mqtt base name and using the shorter mqtt like entity names.

HansRemmerswaal · 2022-12-20T07:35:24Z

I guess the v3.4 backwards compatibility mode is not future proof and will disappear over time? If so then it would be great if there is a kind of procedure to migrate from long name to short name. I would be more than happy to support in this.

Is that script created by @mtc716 complete? I'm running HA on a Raspberry Pi, how to execute the script from there?
(I'm an expert in creating SQL on Oracle SQL but didn't manage yet to connect to the database on the Raspberry Pi)

mtc716 · 2022-12-20T07:57:41Z

I guess the v3.4 backwards compatibility mode is not future proof and will disappear over time? If so then it would be great if there is a kind of procedure to migrate from long name to short name. I would be more than happy to support in this.

Is that script created by @mtc716 complete? I'm running HA on a Raspberry Pi, how to execute the script from there? (I'm an expert in creating SQL on Oracle SQL but didn't manage yet to connect to the database on the Raspberry Pi)

The perl script is made by @sciurius and I just checked the database at which the entity id's are used and provided the SQL Replace Statement.

So the easiest way to connect to the database is using the addon phpmysqladmin. By this you have direct access and can execute whatever SQL you need.

BTW: I am using the same configuration.

HansRemmerswaal · 2022-12-20T08:30:06Z

In my HA I do not see the phpmysqladmin addon. I do see in the "Home Assistant Community Add-ons" the phpMyAdmin. But I guess that can't be use as my database is SQLite (sqlite:////config/home-assistant_v2.db)

sciurius · 2022-12-20T08:39:10Z

Please note that my script generates SQL out of a description of entities (in the zip names.md). These are the entities for my Topline GB162 boiler and Moduline 400 thermostat. If you have different devices you may want to adapt the description file. And let us know.
And beware: the rename operation can take a long time (couple of hours)!

HansRemmerswaal · 2022-12-20T11:52:46Z

But I guess that @proddy has the complete x-ref between the v3.4 and v3.5 names? As it's somehow used in the MQTT option.

HansRemmerswaal · 2022-12-20T17:41:31Z

Hmm it will not be that easy. Different hardware, different databases behind it. Best would be something that is running in HA itself to convert the data.

zibous · 2022-12-22T17:10:26Z

I have the same problem, so I can hardly install the newer versions because the name changes have a problem with my automation and scripts.

In addition, I lose the history in the Influx database and in the Grafans representations.

The only remedy would be the compatibility with previous version v3.5.0.

Alternatively, I would have to switch off Homeassistant Disvovery and create a own one to map the new names to the old one.

proddy · 2022-12-23T06:58:22Z

I have the same problem, so I can hardly install the newer versions because the name changes have a problem with my automation and scripts.

In addition, I lose the history in the Influx database and in the Grafans representations.

The only remedy would be the compatibility with previous version v3.5.0.

Alternatively, I would have to switch off Homeassistant Disvovery and create a own one to map the new names to the old one.

In one of the last beta releases of v3.5 you can choose to use the old v3.4 MQTT naming so nothing should change in Home Assistant. If that is not the case let me know please.

sciurius · 2022-12-23T07:06:09Z

Upgrading retains settings. Won't there be the risk that after the upgrade EMS-ESP contacts the MQTT server and spawns the new names to HomeAssistant before the user has had the chance to switch this setting?

proddy · 2022-12-23T07:07:41Z

Upgrading retains settings. Won't there be the risk that after the upgrade EMS-ESP contacts the MQTT server and spawns the new names to HomeAssistant before the user has had the chance to switch this setting?

good point, forgot about that. I'll add a check in the code that if it's coming from v3.4 to automatically keep the old MQTT setting. thanks!

zibous · 2022-12-23T08:05:09Z

@proddy

In one of the last beta releases of v3.5 you can choose to use the old v3.4 MQTT naming so nothing should change in Home Assistant. If that is not the case let me know please.

Thanks, but how can i find the mapping for the version EMS-ESP-3_5_0b10-ESP32 ?

  - sensor.boiler_burner_stage_2_operating_time
  - binary_sensor.boiler_gas_stage_2
  - sensor.boiler_total_uba_operating_time
  - sensor.boiler_total_burner_operating_time
  - sensor.boiler_outside_temperature
  - sensor.boiler_burner_current_power
  - sensor.boiler_burner_starts
  - sensor.boiler_flame_current
  - binary_sensor.boiler_gas
  - binary_sensor.boiler_heating_active
  - binary_sensor.boiler_heating_pump
  - sensor.boiler_heating_pump_modulation
  - binary_sensor.boiler_fan
  - sensor.boiler_total_heat_operating_time
  - sensor.boiler_last_error_code
  - sensor.boiler_service_code_number
  - sensor.boiler_service_code
  - sensor.boiler_current_flow_temperature
  - sensor.boiler_maintenance_message
  - binary_sensor.boiler_ignition
  - binary_sensor.boiler_tapwater_active
  - sensor.boiler_next_maintenance_date
  - binary_sensor.boiler_dhw_3_way_valve_active
  - binary_sensor.boiler_dhw_active
  - sensor.boiler_dhw_active_time
  - sensor.boiler_dhw_current_extern_temperature
  - sensor.boiler_dhw_current_intern_temperature
  - sensor.boiler_dhw_current_tap_water_flow
  - sensor.boiler_dhw_starts
  - sensor.boiler_dhw_storage_extern_temperature
  - binary_sensor.boiler_dhw_charging
  - binary_sensor.boiler_dhw_recharging
  - sensor.boiler_dhw_set_temperature
  - sensor.boiler_dhw_charging_type
  - binary_sensor.boiler_dhw_temperature_ok
  - sensor.boiler_dhw_type
  - binary_sensor.boiler_oil_preheating
- switch.thermostat_damping_outdoor_temperature
      - select.thermostat_building_type
      - number.thermostat_minimal_external_temperature
      - select.thermostat_dhw_disinfection_day
      - number.thermostat_dhw_disinfection_time
      - switch.thermostat_dhw_disinfecting
      - number.thermostat_dhw_charge_duration
      - switch.thermostat_dhw_charge
      - select.thermostat_dhw_mode
      - number.thermostat_dhw_set_temperature
      - number.thermostat_dhw_set_low_temperature
      - select.thermostat_dhw_circulation_pump_mode
      - select.thermostat_hc1_reduce_mode
      - number.thermostat_hc1_off_reduce_switch_temperature
      - number.thermostat_hc1_design_temperature
      - number.thermostat_hc1_no_reduce_below_temperature
      - number.thermostat_hc1_eco_temperature
      - switch.thermostat_hc1_switch_on_optimization
      - select.thermostat_hc1_set_summer_mode
      - select.thermostat_hc1_nofrost_mode
      - number.thermostat_hc1_nofrost_temperature
      - select.thermostat_hc1_heating_type
      - number.thermostat_hc1_comfort_temperature
      - select.thermostat_hc1_control_mode
      - number.thermostat_hc1_manual_temperature
      - number.thermostat_hc1_max_flow_temperature
      - number.thermostat_hc1_min_flow_temperature
      - select.thermostat_hc1_mode
      - select.thermostat_hc1_program
      - number.thermostat_hc1_room_influence
      - number.thermostat_hc1_room_influence_factor
      - number.thermostat_hc1_fast_heatup
      - number.thermostat_hc1_selected_room_temperature
      - number.thermostat_hc1_summer_temperature
      - number.thermostat_hc1_offset_temperature
      - number.thermostat_hc1_temporary_set_temperature_automode
      - number.thermostat_hc1_temporary_set_temperature_from_remote
 .....

proddy · 2022-12-23T08:16:41Z

Thanks, but how can i find the mapping for the version EMS-ESP-3_5_0b10-ESP32 ?

What do you need exactly? The mapping of the old names to the new names?

I'll create a new GH issue to maintain a table of the shortname, fullname etc. We've done a few passes at this but its not consistent

proddy added the enhancement New feature or request label Dec 14, 2022

proddy added this to the v3.5.0 milestone Dec 16, 2022

proddy added a commit to proddy/EMS-ESP32 that referenced this issue Dec 16, 2022

fixes for emsesp#804

147be12

proddy mentioned this issue Dec 17, 2022

Renaming HA entity IDs #813

Closed

proddy closed this as completed Dec 17, 2022

proddy reopened this Dec 17, 2022

proddy added a commit that referenced this issue Dec 17, 2022

Merge pull request #816 from proddy/dev

575fdcb

support 3 types of MQTT discovery entity id - #804

proddy closed this as completed Dec 17, 2022

proddy reopened this Dec 17, 2022

HansRemmerswaal mentioned this issue Dec 20, 2022

Changed sensor names for Home Assistant since EMS-ESP32 upgrade from v3.5.0b9 to v3.5.0b12 #824

Closed

proddy mentioned this issue Dec 23, 2022

Create table of all EMS-ESP entities, by device, shortname/fullname and characteristics #828

Closed

proddy closed this as completed Dec 24, 2022

add flag to MQTT Settings to use the old v3.4 entity ID names in Home Assistant #804

add flag to MQTT Settings to use the old v3.4 entity ID names in Home Assistant #804

Comments

proddy commented Dec 14, 2022

sciurius commented Dec 14, 2022

bbqkees commented Dec 14, 2022

tp1de commented Dec 14, 2022

proddy commented Dec 14, 2022

tp1de commented Dec 14, 2022

sciurius commented Dec 14, 2022

proddy commented Dec 14, 2022

sciurius commented Dec 14, 2022

sciurius commented Dec 15, 2022

proddy commented Dec 15, 2022 • edited Loading

sciurius commented Dec 16, 2022

proddy commented Dec 16, 2022

sciurius commented Dec 16, 2022

mtc716 commented Dec 17, 2022

proddy commented Dec 17, 2022

sciurius commented Dec 17, 2022

proddy commented Dec 17, 2022

proddy commented Dec 17, 2022 • edited Loading

mtc716 commented Dec 17, 2022

proddy commented Dec 17, 2022

mtc716 commented Dec 17, 2022

proddy commented Dec 17, 2022

mtc716 commented Dec 17, 2022

sciurius commented Dec 17, 2022

proddy commented Dec 17, 2022

HansRemmerswaal commented Dec 20, 2022 • edited Loading

mtc716 commented Dec 20, 2022 • edited Loading

HansRemmerswaal commented Dec 20, 2022

sciurius commented Dec 20, 2022

HansRemmerswaal commented Dec 20, 2022

HansRemmerswaal commented Dec 20, 2022

zibous commented Dec 22, 2022

proddy commented Dec 23, 2022

sciurius commented Dec 23, 2022

proddy commented Dec 23, 2022

zibous commented Dec 23, 2022 • edited Loading

proddy commented Dec 23, 2022

proddy commented Dec 15, 2022 •

edited

Loading

proddy commented Dec 17, 2022 •

edited

Loading

HansRemmerswaal commented Dec 20, 2022 •

edited

Loading

mtc716 commented Dec 20, 2022 •

edited

Loading

zibous commented Dec 23, 2022 •

edited

Loading