Add mysql libraries to the image?

[azhiltsov]

If I am not mistaken DSMR is build on top of django which in a turn support mysql as a backend.

I am currently running a bunch of dockers on my NAS and only DSMR requires Postgres.

I wanted migrate it to mysql in order to simplify management and get rid of postgresql in my setup
but unfortunately there is no mysql libraries in the image, so changing DJANGO_DATABASE_ENGINE to django.db.backends.mysql is not enough

I've tried to add dependencies manually via console of running docker, just to confirm that it will work.
Unfortunately that didn't help

Is this something you can help me with?

/dsmr # apk add mysql-client
(1/5) Installing mariadb-common (10.4.15-r0)
(2/5) Installing libgcc (9.3.0-r2)
(3/5) Installing libstdc++ (9.3.0-r2)
(4/5) Installing mariadb-client (10.4.15-r0)
(5/5) Installing mysql-client (10.4.15-r0)
Executing busybox-1.31.1-r19.trigger
OK: 109 MiB in 65 packages

/dsmr # apk add py3-mysqlclient
(1/2) Installing mariadb-connector-c (3.1.8-r1)
(2/2) Installing py3-mysqlclient (1.4.6-r0)
OK: 110 MiB in 67 packages

/dsmr # apk add python3-dev
(1/2) Installing pkgconf (1.7.2-r0)
(2/2) Installing python3-dev (3.8.5-r0)
Executing busybox-1.31.1-r19.trigger
OK: 156 MiB in 69 packages

/dsmr # /usr/local/bin/python3 /dsmr/manage.py check 
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/django/db/backends/mysql/base.py", line 15, in <module>
    import MySQLdb as Database
ModuleNotFoundError: No module named 'MySQLdb'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/dsmr/manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.9/site-packages/django/core/management/__init__.py", line 401, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.9/site-packages/django/core/management/__init__.py", line 377, in execute
    django.setup()
  File "/usr/local/lib/python3.9/site-packages/django/__init__.py", line 24, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "/usr/local/lib/python3.9/site-packages/django/apps/registry.py", line 114, in populate
    app_config.import_models()
  File "/usr/local/lib/python3.9/site-packages/django/apps/config.py", line 211, in import_models
    self.models_module = import_module(models_module_name)
  File "/usr/local/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 790, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/usr/local/lib/python3.9/site-packages/django/contrib/auth/models.py", line 2, in <module>
    from django.contrib.auth.base_user import AbstractBaseUser, BaseUserManager
  File "/usr/local/lib/python3.9/site-packages/django/contrib/auth/base_user.py", line 48, in <module>
    class AbstractBaseUser(models.Model):
  File "/usr/local/lib/python3.9/site-packages/django/db/models/base.py", line 122, in __new__
    new_class.add_to_class('_meta', Options(meta, app_label))
  File "/usr/local/lib/python3.9/site-packages/django/db/models/base.py", line 326, in add_to_class
    value.contribute_to_class(cls, name)
  File "/usr/local/lib/python3.9/site-packages/django/db/models/options.py", line 206, in contribute_to_class
    self.db_table = truncate_name(self.db_table, connection.ops.max_name_length())
  File "/usr/local/lib/python3.9/site-packages/django/db/__init__.py", line 28, in __getattr__
    return getattr(connections[DEFAULT_DB_ALIAS], item)
  File "/usr/local/lib/python3.9/site-packages/django/db/utils.py", line 214, in __getitem__
    backend = load_backend(db['ENGINE'])
  File "/usr/local/lib/python3.9/site-packages/django/db/utils.py", line 111, in load_backend
    return import_module('%s.base' % backend_name)
  File "/usr/local/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/usr/local/lib/python3.9/site-packages/django/db/backends/mysql/base.py", line 17, in <module>
    raise ImproperlyConfigured(
django.core.exceptions.ImproperlyConfigured: Error loading MySQLdb module.
Did you install mysqlclient?

[xirixiz]

Nice case 😃! However, Im able to help you out. Ill create a development image first so you can verify whether it's working. I`ll let you know when the build process completed.

[xirixiz]

Some info, some build deps are required.... so I've added something like this for the build process of the Docker image.

RUN apk add --no-cache --virtual .build-deps build-base mariadb-dev \
    && python3 -m pip install mysqlclient --no-cache-dir \
    && apk --purge del .build-deps \
    && rm -rf /var/cache/apk/* \
    && rm -rf /tmp/*

[xirixiz]

What is your target architecture? The following images have been created with mysql support:

docker pull xirixiz/dsmr-reader-docker:development-v4.9.0-amd64
docker pull xirixiz/dsmr-reader-docker:development-v4.9.0-arm64v8
docker pull xirixiz/dsmr-reader-docker:development-v4.9.0-arm32v7
docker pull xirixiz/dsmr-reader-docker:development-v4.9.0-arm32v6

Once tested, I can create a PR for the master branch.

[azhiltsov]

Thank you for the swift response.
I am using amd64 arch and already installed development build you provided.
I also migrated the data with the help of stackoverflow: https://stackoverflow.com/a/41045999
Unfortunately container can't start as it missing mysqldump for one of the schedules:

/dsmr # /usr/local/bin/python3 -u /dsmr/manage.py dsmr_backend
Current logging level set to "ERROR". More information can be found here: https://dsmr-reader.readthedocs.io/en/v4/troubleshooting.html#logging
[2020-12-17 22:43:53,273] ERROR    (FileNotFoundError) dsmr_backup.services.backup.run errored: [Errno 2] No such file or directory: 'mysqldump'

I think it needs an mysql-client/mariadb-client
I installed it manually from inside of container and this error was gone.

Another thing to fix will be this function https://github.com/xirixiz/dsmr-reader-docker/blob/master/src/app/run.sh#L136
It DB specific and I created a PR with proposed fix

[xirixiz]

Great! Thanks for helping out.

I just rebuild the development images (same version).

• your PR has been merged
• the mariadb client package has been added

[xirixiz]

Is it working properly?

[azhiltsov]

I installed it and it seems to work properly. I think this issue can be closed.

It worth to be mentioned (might be in docs, but at least here) that there is issue with DSMR running on top of MySQL
It manifests as "Data processing is lagging behind" message on the web-interface
and with only leading information in the logs if they set to DEBUG "Missing consumption data for:"

If you drill down from here then you can eventually find this issue: dsmrreader/dsmr-reader#909
Which can be resolved by installing timezone info tables in MySQL: https://dev.mysql.com/doc/refman/8.0/en/mysql-tzinfo-to-sql.html

Might be I can create another linked issue to implement some checks and warn if someone is trying to run this container on top of MySQL without properly configured timezone info tables. What do you think?

[xirixiz]

Thanks for the additional info. I`ll merge the code and add some additional infor for MySQL.