Work around ENVELOPE mailboxes that have too many tokens by merging earlier tokens

[ekalchev]

I am getting this exception

MailKit.Net.Imap.ImapProtocolException: 'Syntax error in ENVELOPE. Unexpected qstring token: "facility"'

when trying to parse imap server response

This is the IMAP log

S: * OK [CAPABILITY IMAP4REV1 AUTH=LOGIN MOVE SPECIAL-USE] IMAP4rev1 DavMail 6.0.1-3390 server ready
..........................................
..........................................
C: E00000007 FETCH 18075:18074,15373:15276 (UID FLAGS INTERNALDATE RFC822.SIZE ENVELOPE BODYSTRUCTURE BODY.PEEK[HEADER.FIELDS (IMPORTANCE DELIVERY-DATE X-PRIORITY REFERENCES)])
..........................................
..........................................
S: * 15327 FETCH (UID 16410 FLAGS (\Seen) INTERNALDATE "29-Apr-2021 13:57:16 +0300" RFC822.SIZE 12609 ENVELOPE ("Thu, 29 Apr 2021 10:57:07 +0000" "=?utf-8?B?0J/QsNGA0LrQuNC90LMg0L3QsCDQlNCw0L3QsNC40Lsg0JTQtdGH0LXQsg==?=" (({38}
S: Рецепция Офис сграда "Данаил Дечев" №6 NIL "facility" "xxxxxxxxxxx.com")) NIL NIL (("Team" NIL "team" "xxxxxxxxxxx.com")) NIL NIL NIL "<[email protected]>") BODYSTRUCTURE ("TEXT" "PLAIN" ("CHARSET" "UTF-8") "<240310CE7CC66142A3B8B3DC30DABDF8@1>" NIL "BASE64" 720 9) BODY[HEADER.FIELDS (IMPORTANCE DELIVERY-DATE X-PRIORITY REFERENCES)] {0}

Note I replaced the actual domain with xxxxxxxxxxx.com if this is somehow relevant to the bytes count

I can synchronize this mailbox successfully with Thunderbird

I am using Mailkit version 3.2.0

[jstedfast]

According to LINQPad Encoding.UTF8.GetBytes("Рецепция Офис сграда \"Данаил Дечев\" №6").Length evaluates to 69 which is quite a few more bytes than 38. I'm guessing this must be koi8-u? koi8-u gives me 38 bytes.

That said... if I generate the server response using koi8-u, it "parses" correctly (but the mailbox name is corrupted because the IMAP code doesn't know to use koi8-u when converting raw bytes into a string).

If I generate it using UTF-8, stepping thru in the debugger reveals that it parses:

1. Рецепция Офис сграда
2. "Данаил Дечев"
3. №6
4. NIL

When it gets to "facility", it throws the exact same exception you are seeing because it is expecting a ) to end the address.

What might happening is that the server is calculating the name length based on the unicode string that it uses internally ("Рецепция Офис сграда \"Данаил Дечев\"".Length is 38), but then it converts to UTF-8 when serializing it across the wire.

If you can verify that the length of that string is 69 (instead off 38), that would confirm my suspicion. A hex editor might be a good way to do that.

I wonder what Thunderbird is doing to make this work. Maybe I can keep reading tokens until I get to a ) and merge tokens until I end up with only 4?

[ekalchev]

Those are actual bytes from engine.Stream.input from the process memory

2a 20 31 35 33 32 37 20 46 45 54 43 48 20 28 55 49 44 20 31 36 34 31 30 20 46 4c 41 47 53 20 28 5c 53 65 65 6e 29 20 49 4e 54 45 52 4e 41 4c 44 41 54 45 20 22 32 39 2d 41 70 72 2d 32 30 32 31 20 31 33 3a 35 37 3a 31 36 20 2b 30
33 30 30 22 20 52 46 43 38 32 32 2e 53 49 5a 45 20 31 32 36 30 39 20 45 4e 56 45 4c 4f 50 45 20 28 22 54 68 75 2c 20 32 39 20 41 70 72 20 32 30 32 31 20 31 30 3a 35 37 3a 30 37 20 2b 30 30 30 30 22 20 22 3d 3f 75 74 66 2d 38 3f
42 3f 30 4a 2f 51 73 4e 47 41 30 4c 72 51 75 4e 43 39 30 4c 4d 67 30 4c 33 51 73 43 44 51 6c 4e 43 77 30 4c 33 51 73 4e 43 34 30 4c 73 67 30 4a 54 51 74 64 47 48 30 4c 58 51 73 67 3d 3d 3f 3d 22 20 28 28 7b 33 38 7d 0d 0a d0 a0
d0 b5 d1 86 d0 b5 d0 bf d1 86 d0 b8 d1 8f 20 d0 9e d1 84 d0 b8 d1 81 20 d1 81 d0 b3 d1 80 d0 b0 d0 b4 d0 b0 20 22 d0 94 d0 b0 d0 bd d0 b0 d0 b8 d0 bb 20 d0 94 d0 b5 d1 87 d0 b5 d0 b2 22 20 e2 84 96 36 20 4e 49 4c 20 22 66 61 63
69 6c 69 74 79 22 20 22 6d 6f 62 69 73 79 73 74 65 6d 73 2e 63 6f 6d 22 29 29 20 4e 49 4c 20 4e 49 4c 20 28 28 22 54 65 61 6d 22 20 4e 49 4c 20 22 74 65 61 6d 22 20 22 6d 6f 62 69 73 79 73 74 65 6d 73 2e 63 6f 6d 22 29 29 20 4e
49 4c 20 4e 49 4c 20 4e 49 4c 20 22 3c 64 30 66 36 63 61 36 36 30 38 63 66 62 30 62 36 38 30 62 37 62 39 30 38 32 34 63 37 39 31 31 38 40 6d 6f 62 69 73 79 73 74 65 6d 73 2e 63 6f 6d 3e 22 29 20 42 4f 44 59 53 54 52 55 43 54 55

bytes converted to ANSI by visual studio debugger

* 15327 FETCH (UID 16410 FLAGS (\Seen) INTERNALDATE "29-Apr-2021 13:57:16 +0
300" RFC822.SIZE 12609 ENVELOPE ("Thu, 29 Apr 2021 10:57:07 +0000" "=?utf-8?
B?0J/QsNGA0LrQuNC90LMg0L3QsCDQlNCw0L3QsNC40Lsg0JTQtdGH0LXQsg==?=" (({38}..Р
еÑ.епÑ.иÑ. ОÑ.иÑ. Ñ.Ð.рада "ДаÐ.аил ДеÑ.еÐ." â.–6 NIL "fac
ility" "mobisystems.com")) NIL NIL (("Team" NIL "team" "mobisystems.com")) N
IL NIL NIL "<[email protected]>") BODYSTRUCTU

You are right, the server miscalculated the bytes. I can count 38 bytes in this segment Рецепция Офис сграда ". I guess the server assumed first double quote as end of the string

[jstedfast]

I think what they actually did was "Рецепция Офис сграда \"Данаил Дечев\"".Length which is 38.

Hopefully they don't do that in other places where this would be more difficult to work around (e.g. just combining tokens until we find what we are looking for).

[ekalchev]

I am not convinced that a workaround for this bug should be implemented in the client library.

So far I know Thunderbird was able to deal with that but may be this is just lucky coincidence with their implementation. I just tried Windows Mail and Outlook both don't work. I'll try tomorrow Em client and will let you know.

[jstedfast]

Yea, I'd definitely prefer to see it fixed in the server, but since this work-around (which is mostly just "lucky" to work) wasn't too complicated, I didn't mind writing it up.

[ekalchev]

Thunderbird and Em Client both was able to synchronize the mail account.

Em Client showed this

Thunderbird this

It seems Em client had problem with this email but still manage to complete the synchronization.

Outlook and Windows Mail both failed, but I believe this is not related to this email because they show no emails at all in inbox.

As potential fix, it would be enough to be able to go past this email so the synchronization can be completed. It is not necessary parsed value to be 100% correct

[jstedfast]

I think I can close this now