ImapClient can send commands that are too long

[MartinBrandl]

When i try to download headerdata of e-mails for a specific folder via IMAP this error happens:
MailKit.Net.Imap.ImapCommandException: 'The IMAP server replied to the 'FETCH' command with a 'BAD' response: Command Error. 10'
It does not even matter wich MessageSummaryItems i request.
Here is how i do it:
imapFolder.Fetch(uniqueIds, MessageSummaryItems.UniqueId)

Since i saw a few other issues with similar problems i have also created a log via a ProtocolLogger and attached it.

As you can see from the log a lot of folders worked before the error gets thrown for the "Sent" folder.
imap.log

Can you lead me into a direction what exactly does not work here and what i could do to fix it?

additional information about what my code does
my code basicly synchronizes all folders and mails (only header data) to our database since working live with IMAP is rather slow.
This is used for displaying the whole mailbox of a user in our software.

[MartinBrandl]

I got a second mailbox wich has the same error.
Hope this log helps.
imap.log

[jstedfast]

I'm betting the problem is that the list of UIDs in your request is too long for the server to process.

Try to break apart your uniqueIds list into smaller chunks to see if that solves it. For testing purposes, you could try something like limiting each Fetch call to a list of 100 or so UIDs.

If that is the problem, then I might have to look into breaking apart long commands in MailKit. It's something I figured I might have to do, but was hoping to avoid it because it sucks to have to do :-\

The real problem, if you look at your logs, is not so much the number of UIDs, but the number of characters it takes to serialize the UIDs. So, for example, in your first log, the long string of UIDs starts off with 1:88,.... The 1:88 represents 88 messages which is a lot of messages, but because all of their UIDs are sequential, MailKit can compact them into a very short string. Much shorter than if it did 1,2,3,4,5,6...88. Where the problem arises in serialization of all of your UIDs is that your list is very fragmented in the higher ranges (that's not your fault or anything, just pointing out why serialization isn't as compact as it could be and thus becomes unwieldy in length).

[MartinBrandl]

Ah that makes a lot of sense!
I will try that as soon as possible and tell you if it worked out.

[jstedfast]

Cool, thanks. If you are up for playing around a bit, it would be interesting to know what outlook.com's maximum buffer size allowance is for the list of UIDs. If you played around with changing the number of UIDs in your list to vary the length of the serialized list of UIDs to see how large it can be before it fails, that would give us a pretty good idea of what that limitation is.

[jstedfast]

Another work-around that you could do long-term is that since your query seems like it includes all of the UIDs in the folder, you could use UniqueIdRange.All instead of a UniqueIdSet that contains all of the individual UIDs. That would make the serialized UID list 1:* which is hard to beat for compactness.

Just a thought for you going forward.

[MartinBrandl]

I will try to find out the limitations for you 👍

Thats just because this is the initial synchronization
=> after that it only requests new and moved messages so sadly i cannot fall back to that

[jstedfast]

Ok, gotcha. That makes sense.

[MartinBrandl]

Yep that actually solves the problem!
For me it was exactly 3340 Ids i could fetch at once.
I have attached a log for you wich has that at the end.
Note, that the very last fetch was the one that failes so you have that for comparison.
The third and second from last should be the ones you are looking for to figure out the maximum.
imap.log

[jstedfast]

Awesome, thanks for playing around with this for me. I really appreciate it :-)

[MartinBrandl]

You are very welcome!
Thanks for your work 👍

[jstedfast]

I've committed a fix for this issue now, but it would be fantastic if you could test it out for me.

The nuget version will be 2.1.3.12 on the myget feed linked on the main page of the github repo.

[MartinBrandl]

After testing both mailboxes wich failed i can confirm, that the fix seems to work perfectly!
Thanks for your very fast help!

[jstedfast]

Excellent :-)

There's an RFC that gives recommendations for client and server implementers to follow and their recommendation was to max command-lines at 1000 characters and servers to support at least up to 8000 characters.

My new implementation in MailKit pushes the boundaries a bit (it uses a limit of 8000 characters instead of 1000), but that should be ok and indeed it seems to work for Exchange.

Courier IMAP, for example, allows each individual token in the command to be 16k (there's no limit on the command-line, just on the individual tokens). When MailKit encounters Courier, we use the 16k limit.

I think UW.IMAP is the only IMAP server in wide use that only allows 1000, so I'm working on a way to detect UW.IMAP and limit MailKit to 1000 characters for that server.

Anyway, awesome to hear this is working for you now :-)

I was afraid this was going to be a lot more complicated to fix than it was...