Skip to content
This repository has been archived by the owner on May 10, 2023. It is now read-only.

FIX: UnicodeDecodeError during logfile parsing #57

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

tsabi
Copy link

@tsabi tsabi commented Dec 1, 2019

If you use postfix internationally, you will hit this error.
I believe this is the right fix, because those log files are NOT utf-8 encoded,
so you should not read it as utf-8, but with the real encoding of that file.

Fixes: modoboa/modoboa#1788, #55, #38,

root@h7-mailsrv:/etc/cron.d# /srv/modoboa/env/bin/python /srv/modoboa/instance/manage.py logparser
Traceback (most recent call last):
  File "/srv/modoboa/instance/manage.py", line 22, in <module>
    execute_from_command_line(sys.argv)
  File "/srv/modoboa/env/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line
    utility.execute()
  File "/srv/modoboa/env/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 356, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/srv/modoboa/env/local/lib/python2.7/site-packages/django/core/management/base.py", line 283, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/srv/modoboa/env/local/lib/python2.7/site-packages/django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "/srv/modoboa/env/local/lib/python2.7/site-packages/modoboa_stats/management/commands/logparser.py", line 587, in handle
    p.process()
  File "/srv/modoboa/env/local/lib/python2.7/site-packages/modoboa_stats/management/commands/logparser.py", line 552, in process
    for line in self.f.readlines():
  File "/srv/modoboa/env/lib/python2.7/codecs.py", line 314, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xc1 in position 949: invalid start byte

If you use postfix internationally, you will hit this error.
I believe this is the right fix, because those log files are NOT utf-8 encoded,
so you should not read it as utf-8, but with the real encoding of that file.

Fixes: modoboa/modoboa#1788, modoboa#55, modoboa#38, 

```
root@h7-mailsrv:/etc/cron.d# /srv/modoboa/env/bin/python /srv/modoboa/instance/manage.py logparser
Traceback (most recent call last):
  File "/srv/modoboa/instance/manage.py", line 22, in <module>
    execute_from_command_line(sys.argv)
  File "/srv/modoboa/env/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line
    utility.execute()
  File "/srv/modoboa/env/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 356, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/srv/modoboa/env/local/lib/python2.7/site-packages/django/core/management/base.py", line 283, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/srv/modoboa/env/local/lib/python2.7/site-packages/django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "/srv/modoboa/env/local/lib/python2.7/site-packages/modoboa_stats/management/commands/logparser.py", line 587, in handle
    p.process()
  File "/srv/modoboa/env/local/lib/python2.7/site-packages/modoboa_stats/management/commands/logparser.py", line 552, in process
    for line in self.f.readlines():
  File "/srv/modoboa/env/lib/python2.7/codecs.py", line 314, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xc1 in position 949: invalid start byte
```
@mundschenk-at
Copy link
Contributor

I got a similar error tonight: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc3 in position 2515: invalid continuation byte. The rought cause in the logfile appears to be an issue with an invalid domain name.

postfix/trivial-rewrite[49231]: warning: midna_domain_to_ascii_create: Problem translating domain [...] to ASCII form: UIDNA_ERROR_DISALLOWED

I don't think opening logfiles as latin1 is the correct fix, but the logparser should handle unicode exceptions.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

UnicodeDecodeError during stat generation
2 participants