UnicodeDecodeError during stat generation

[tsabi]

Impacted versions

• Standard Ubuntu 18.04 Server
• Modoboa: 1.14.0 (latest)
• installer used: Yes
• Webserver: Nginx

Steps to reproduce

Today i saw there isn't any stat generated. When i runned the logparsing from command prompt i got this error:

root@h7-mailsrv:/etc/cron.d# /srv/modoboa/env/bin/python /srv/modoboa/instance/manage.py logparser
Traceback (most recent call last):
  File "/srv/modoboa/instance/manage.py", line 22, in <module>
    execute_from_command_line(sys.argv)
  File "/srv/modoboa/env/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line
    utility.execute()
  File "/srv/modoboa/env/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 356, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/srv/modoboa/env/local/lib/python2.7/site-packages/django/core/management/base.py", line 283, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/srv/modoboa/env/local/lib/python2.7/site-packages/django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "/srv/modoboa/env/local/lib/python2.7/site-packages/modoboa_stats/management/commands/logparser.py", line 587, in handle
    p.process()
  File "/srv/modoboa/env/local/lib/python2.7/site-packages/modoboa_stats/management/commands/logparser.py", line 552, in process
    for line in self.f.readlines():
  File "/srv/modoboa/env/lib/python2.7/codecs.py", line 314, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xc1 in position 949: invalid start byte

This modification fixed it:

diff -urN modoboa_stats_/management/commands/logparser.py modoboa_stats/management/commands/logparser.py
--- modoboa_stats_/management/commands/logparser.py     2019-10-31 11:53:10.899853660 +0100
+++ modoboa_stats/management/commands/logparser.py      2019-10-31 11:47:17.264635425 +0100
@@ -53,7 +53,7 @@
         self.debug = options["debug"]
         self.verbose = options["verbose"]
         try:
-            self.f = io.open(self.logfile, encoding="utf-8")
+            self.f = io.open(self.logfile, encoding="latin1")
         except IOError as errno:
             self._dprint("%s" % errno)
             sys.exit(1)

[Salamek]

@tsabi duplicity of modoboa/modoboa-stats#55

[tsabi]

@tonioo Please consider my fix: at the other ticket they tried to play with character replacement, what they needed if they read the log file as utf-8 encoding. But because it is latin1 encoded it is completely wrong way, the real fix is to read the file as it encoded originally.