"MailKit.Net.Imap.ImapProtocolException: Syntax error in BODYSTRUCTURE. Unexpected token: NIL" w/ O365

[carmanj]

Hi @jstedfast,

Describe the bug
We have created a scheduled service component in .NET that customers use to fetch emails in bulk from O365 accounts. In this case, we are attempting to fetch 1k email summaries at a time. There seems to be (at least) one particular email grinding things to a hault, and throwing the "Syntax error in BODYSTRUCTURE. Unexpected token: NIL" error that has been reported/fixed many times as a one-off parsing issue. I'm hoping you could point out what is wrong with the parsing from O365 here, and devise a workaround.

Platform (please complete the following information):

• OS: Windows
• .NET Framework: 4.8
• MailKit Version: 3.4.1.513 (as recently referenced from Syntax error in BODY. Unexpected token: ')' #1430, with a similar issue)
• MimeKit Version: 3.4.1.612

Exception

2022-10-05 16:42:14,780 [4] ERROR InfoedServiceLibrary3.Components.MailFetching - MailKit.Net.Imap.ImapProtocolException: Syntax error in BODYSTRUCTURE. Unexpected token: NIL
at MailKit.Net.Imap.ImapUtils.ReadStringToken(ImapEngine engine, String format, CancellationToken cancellationToken) in D:\a\MailKit\MailKit\MailKit\Net\Imap\ImapUtils.cs:line 684
at MailKit.Net.Imap.ImapUtils.ReadStringTokenAsync(ImapEngine engine, String format, Boolean doAsync, CancellationToken cancellationToken) in D:\a\MailKit\MailKit\MailKit\Net\Imap\ImapUtils.cs:line 719
at MailKit.Net.Imap.ImapUtils.d__39.MoveNext() in D:\a\MailKit\MailKit\MailKit\Net\Imap\ImapUtils.cs:line 1118
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Threading.Tasks.ValueTask1.get_Result() at MailKit.Net.Imap.ImapUtils.<ParseBodyAsync>d__40.MoveNext() in D:\a\MailKit\MailKit\MailKit\Net\Imap\ImapUtils.cs:line 1205 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Threading.Tasks.ValueTask1.get_Result()
at MailKit.Net.Imap.ImapUtils.d__39.MoveNext() in D:\a\MailKit\MailKit\MailKit\Net\Imap\ImapUtils.cs:line 1107
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Threading.Tasks.ValueTask1.get_Result() at MailKit.Net.Imap.ImapUtils.<ParseBodyAsync>d__40.MoveNext() in D:\a\MailKit\MailKit\MailKit\Net\Imap\ImapUtils.cs:line 1205 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Threading.Tasks.ValueTask1.get_Result()
at MailKit.Net.Imap.ImapUtils.d__39.MoveNext() in D:\a\MailKit\MailKit\MailKit\Net\Imap\ImapUtils.cs:line 1107
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Threading.Tasks.ValueTask1.get_Result() at MailKit.Net.Imap.ImapUtils.<ParseBodyAsync>d__40.MoveNext() in D:\a\MailKit\MailKit\MailKit\Net\Imap\ImapUtils.cs:line 1205 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Threading.Tasks.ValueTask1.get_Result()
at MailKit.Net.Imap.ImapUtils.d__40.MoveNext() in D:\a\MailKit\MailKit\MailKit\Net\Imap\ImapUtils.cs:line 1244
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Threading.Tasks.ValueTask1.get_Result() at MailKit.Net.Imap.ImapUtils.<ParseMultipartAsync>d__39.MoveNext() in D:\a\MailKit\MailKit\MailKit\Net\Imap\ImapUtils.cs:line 1107 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Threading.Tasks.ValueTask1.get_Result()
at MailKit.Net.Imap.ImapUtils.d__40.MoveNext() in D:\a\MailKit\MailKit\MailKit\Net\Imap\ImapUtils.cs:line 1205
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Threading.Tasks.ValueTask1.get_Result() at MailKit.Net.Imap.ImapUtils.<ParseBodyAsync>d__40.MoveNext() in D:\a\MailKit\MailKit\MailKit\Net\Imap\ImapUtils.cs:line 1244 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Threading.Tasks.ValueTask1.get_Result()
at MailKit.Net.Imap.ImapUtils.d__39.MoveNext() in D:\a\MailKit\MailKit\MailKit\Net\Imap\ImapUtils.cs:line 1107
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Threading.Tasks.ValueTask1.get_Result() at MailKit.Net.Imap.ImapUtils.<ParseBodyAsync>d__40.MoveNext() in D:\a\MailKit\MailKit\MailKit\Net\Imap\ImapUtils.cs:line 1205 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Threading.Tasks.ValueTask1.get_Result()
at MailKit.Net.Imap.ImapFolder.FetchSummaryItems(ImapEngine engine, MessageSummary message, FetchSummaryItemsCompletedCallback completed, CancellationToken cancellationToken) in D:\a\MailKit\MailKit\MailKit\Net\Imap\ImapFolderFetch.cs:line 281
at MailKit.Net.Imap.ImapFolder.FetchSummaryItemsAsync(ImapEngine engine, ImapCommand ic, Int32 index, Boolean doAsync) in D:\a\MailKit\MailKit\MailKit\Net\Imap\ImapFolderFetch.cs:line 706
at MailKit.Net.Imap.ImapEngine.d__189.MoveNext() in D:\a\MailKit\MailKit\MailKit\Net\Imap\ImapEngine.cs:line 2230
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
at MailKit.Net.Imap.ImapCommand.d__84.MoveNext() in D:\a\MailKit\MailKit\MailKit\Net\Imap\ImapCommand.cs:line 915
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at MailKit.Net.Imap.ImapEngine.d__190.MoveNext() in D:\a\MailKit\MailKit\MailKit\Net\Imap\ImapEngine.cs:line 2342
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at MailKit.Net.Imap.ImapEngine.d__191.MoveNext() in D:\a\MailKit\MailKit\MailKit\Net\Imap\ImapEngine.cs:line 2366
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at MailKit.Net.Imap.ImapFolder.d__193.MoveNext() in D:\a\MailKit\MailKit\MailKit\Net\Imap\ImapFolderFetch.cs:line 1042
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at MailKit.Net.Imap.ImapFolder.Fetch(IList1 uids, IFetchRequest request, CancellationToken cancellationToken) in D:\a\MailKit\MailKit\MailKit\Net\Imap\ImapFolderFetch.cs:line 1115 at MailKit.IMailFolderExtensions.Fetch(IMailFolder folder, IList1 uids, MessageSummaryItems items, CancellationToken cancellationToken) in D:\a\MailKit\MailKit\MailKit\IMailFolderFetchExtensions.cs:line 97
at InfoedServiceLibrary3.Components.MailFetching.GetMessageSummaries(FetchedEmailAccount account, ImapClient client, IEnumerable`1 messageUniqueIDs) in C:\InfoEdSVN\Internal\Alpha15\Components\InfoedService3\InfoedServiceLibrary3\Components\MailFetching.cs:line 509
at InfoedServiceLibrary3.Components.MailFetching.fetchMail(FetchedEmailAccount account) in C:\InfoEdSVN\Internal\Alpha15\Components\InfoedService3\InfoedServiceLibrary3\Components\MailFetching.cs:line 320
2022

Code Snippets

private IEnumerable<IMessageSummary> GetMessageSummaries(FetchedEmailAccount account, ImapClient client, IEnumerable<UniqueId> messageUniqueIDs)
{
    log.Debug($"Fetching {messageUniqueIDs.Count()} message summaries for accountName: {account.Name}, accountID: {account.AccountID}.");
    return client.Inbox.Fetch(messageUniqueIDs.ToList(), MessageSummaryItems.UniqueId | MessageSummaryItems.InternalDate | MessageSummaryItems.BodyStructure | MessageSummaryItems.Envelope);
}

Protocol Logs

Connected to imaps://outlook.office365.com:993/
S: * OK The Microsoft Exchange IMAP4 service is ready. [TQBOADIAUABSADEANgBDAEEAMAAwADYAMwAuAG4AYQBtAHAAcgBkADEANgAuAHAAcgBvAGQALgBvAHUAdABsAG8AbwBrAC4AYwBvAG0A]
C: A00000000 CAPABILITY
S: * CAPABILITY IMAP4 IMAP4rev1 AUTH=PLAIN AUTH=XOAUTH2 SASL-IR UIDPLUS ID UNSELECT CHILDREN IDLE NAMESPACE LITERAL+
S: A00000000 OK CAPABILITY completed.
C: A00000001 AUTHENTICATE XOAUTH2 ********
S: A00000001 OK AUTHENTICATE completed.
C: A00000002 CAPABILITY
S: * CAPABILITY IMAP4 IMAP4rev1 AUTH=PLAIN AUTH=XOAUTH2 SASL-IR UIDPLUS MOVE ID UNSELECT CLIENTACCESSRULES CLIENTNETWORKPRESENCELOCATION BACKENDAUTHENTICATE CHILDREN IDLE NAMESPACE LITERAL+
S: A00000002 OK CAPABILITY completed.
C: A00000003 NAMESPACE
S: * NAMESPACE (("" "/")) NIL NIL
S: A00000003 OK NAMESPACE completed.
C: A00000004 LIST "" "INBOX"
S: * LIST (\Marked \HasNoChildren) "/" INBOX
S: A00000004 OK LIST completed.
C: A00000005 EXAMINE INBOX
S: * 6995 EXISTS
S: * 6995 RECENT
S: * FLAGS (\Seen \Answered \Flagged \Deleted \Draft $MDNSent)
S: * OK [PERMANENTFLAGS ()] Permanent flags
S: * OK [UNSEEN 1] Is the first unseen message
S: * OK [UIDVALIDITY 14] UIDVALIDITY value
S: * OK [UIDNEXT 22588] The next unique identifier value
S: A00000005 OK [READ-ONLY] EXAMINE completed.
.......
C: A00000010 UID FETCH 19071 (UID INTERNALDATE ENVELOPE BODYSTRUCTURE)
S: * 5206 FETCH (UID 19071 INTERNALDATE "14-Jul-2022 13:33:43 -0400" ENVELOPE ("Thu, 14 Jul 2022 17:33:18 +0000" "Scrubbed" (("Scrubbed" NIL "Scrubbed" "Scrubbed")) NIL NIL (("Scrubbed" NIL "Scrubbed" "Scrubbed") ("Scrubbed" NIL "Scrubbed" "Scrubbed") ("Scrubbed" NIL "Scrubbed" "Scrubbed") ("Scrubbed" NIL "Scrubbed" "Scrubbed") ("Scrubbed" NIL "Scrubbed" "Scrubbed")) (("Scrubbed" NIL "Scrubbed" "Scrubbed") ("Scrubbed" NIL "Scrubbed" "Scrubbed") ("Scrubbed" NIL "Scrubbed" "Scrubbed")) NIL "[email protected]" "[email protected]") BODYSTRUCTURE (((("text" "plain" ("charset" "us-ascii") NIL NIL "quoted-printable" 44619 793 NIL NIL NIL NIL)("text" "html" ("charset" "us-ascii") NIL NIL "quoted-printable" 143984 2321 NIL NIL NIL NIL) "alternative" ("boundary" "000_BN0P221MB04483769DDD81948BC7C387DC8889BN0P221MB0448NAMP") NIL NIL)("image" "jpeg" ("name" "~WRD0000.jpg") "<~WRD0000.jpg>" "~WRD0000.jpg" "base64" 1130 NIL ("inline" ("filename" "~WRD0000.jpg" "size" "823" "creation-date" "Thu, 14 Jul 2022 17:26:49 GMT" "modification-date" "Thu, 14 Jul 2022 17:33:16 GMT")) NIL NIL)("image" "png" ("name" "image001.png") "[email protected]" "image001.png" "base64" 8174 NIL ("inline" ("filename" "image001.png" "size" "5973" "creation-date" "Thu, 14 Jul 2022 17:33:18 GMT" "modification-date" "Thu, 14 Jul 2022 17:33:18 GMT")) NIL NIL)("image" "png" ("name" "image002.png") "[email protected]" "image002.png" "base64" 3524 NIL ("inline" ("filename" "image002.png" "size" "2572" "creation-date" "Thu, 14 Jul 2022 17:33:18 GMT" "modification-date" "Thu, 14 Jul 2022 17:33:18 GMT")) NIL NIL) "related" ("boundary" "007_BN0P221MB04483769DDD81948BC7C387DC8889BN0P221MB0448NAMP" "type" "multipart/alternative") NIL NIL)("message" "rfc822" NIL NIL NIL "7BIT" 0 ("Thu, 14 Jul 2022 15:12:33 +0000" "Scrubbed" (("Scrubbed" NIL "Scrubbed" "Scrubbed")) NIL NIL (("Scrubbed" NIL "Scrubbed" "Scrubbed")) (("Scrubbed" NIL "Scrubbed" "Scrubbed") ("Scrubbed" NIL "Scrubbed" "Scrubbed")) NIL "[email protected]" "[email protected]") (((("text" "plain" ("charset" "utf-8") NIL NIL "base64" 53608 688 NIL NIL NIL NIL)("text" "html" ("charset" "utf-8") "[email protected]" NIL "base64" 176002 2257 NIL NIL NIL NIL) "alternative" ("boundary" "000_18f52bea798548b88470c3df62d666bcScrubbed") NIL NIL)("image" "png" ("name" "image001.png") "[email protected]" "image001.png" "base64" 8174 NIL ("inline" ("filename" "image001.png" "size" "5973" "creation-date" "Thu, 14 Jul 2022 15:12:32 GMT" "modification-date" "Thu, 14 Jul 2022 17:33:17 GMT")) NIL NIL)("image" "jpeg" ("name" "image002.jpg") "[email protected]" "image002.jpg" "base64" 1130 NIL ("inline" ("filename" "image002.jpg" "size" "823" "creation-date" "Thu, 14 Jul 2022 15:12:32 GMT" "modification-date" "Thu, 14 Jul 2022 17:33:17 GMT")) NIL NIL)("image" "png" ("name" "image003.png") "[email protected]" "image003.png" "base64" 3524 NIL ("inline" ("filename" "image003.png" "size" "2572" "creation-date" "Thu, 14 Jul 2022 15:12:32 GMT" "modification-date" "Thu, 14 Jul 2022 17:33:17 GMT")) NIL NIL) "related" ("boundary" "009_18f52bea798548b88470c3df62d666bcScrubbed" "type" "multipart/alternative") NIL NIL)("application" "pdf" ("name" "Scrubbed.pdf") "[email protected]" "Scrubbed.pdf" "base64" 324012 NIL ("attachment" ("filename" "Scrubbed.pdf" "size" "236776" "creation-date" "Thu, 14 Jul 2022 14:53:00 GMT" "modification-date" "Thu, 14 Jul 2022 17:33:17 GMT")) NIL NIL)("message" "rfc822" NIL "[email protected]" NIL "7BIT" 0 ("Tue, 11 Jan 2022 16:34:33 +0000" "RE: Scrubbed" (("Scrubbed" NIL "Scrubbed" "Scrubbed")) NIL NIL (("Scrubbed" NIL "Scrubbed" "Scrubbed")) (("Scrubbed" NIL "Scrubbed" "Scrubbed") ("Scrubbed" NIL "Scrubbed" "Scrubbed") ("Scrubbed" NIL "Scrubbed" "Scrubbed")) NIL "[email protected]" "[email protected]") (((NIL NIL NIL NIL NIL "7BIT" 0 NIL NIL NIL NIL)(NIL NIL NIL NIL NIL "7BIT" 0 NIL NIL NIL NIL)(NIL NIL NIL NIL NIL "7BIT" 0 NIL NIL NIL NIL)(NIL NIL NIL NIL NIL "7BIT" 0 NIL NIL NIL NIL) "related" ("boundary" "007_YT2PR01MB47524CF92A3AD1F75AFF2D25D9519YT2PR01MB4752CANP" "type" "multipart/alternative") NIL NIL)("application" "pdf" ("name" "Scrubbed.pdf") NIL "Scrubbed.pdf" "base64" 215638 NIL ("attachment" ("filename" "Scrubbed.pdf" "size" "157579" "creation-date" "Wed, 02 Feb 2022 21:33:39 GMT" "modification-date" "Wed, 02 Feb 2022 21:33:39 GMT")) NIL NIL) "mixed" ("boundary" "008_YT2PR01MB47524CF92A3AD1F75AFF2D25D9519YT2PR01MB4752CANP") NIL "en-US") 0 NIL ("attachment" ("creation-date" "Thu, 14 Jul 2022 15:12:31 GMT" "modification-date" "Thu, 14 Jul 2022 17:33:18 GMT")) NIL NIL)("application" "pdf" ("name" "Scrubbed.pdf?=") "[email protected]" "Scrubbed.pdf?=" "base64" 208376 NIL ("attachment" ("filename" "Scrubbed.pdf?=" "size" "152274" "creation-date" "Thu, 14 Jul 2022 15:05:00 GMT" "modification-date" "Thu, 14 Jul 2022 17:33:18 GMT")) NIL NIL) "mixed" ("boundary" "010_18f52bea798548b88470c3df62d666bcScrubbed") NIL "en-US") 0 NIL ("attachment" ("creation-date" "Thu, 14 Jul 2022 17:33:16 GMT" "modification-date" "Thu, 14 Jul 2022 17:33:18 GMT")) NIL NIL) "mixed" ("boundary" "008_BN0P221MB04483769DDD81948BC7C387DC8889BN0P221MB0448NAMP") NIL "en-US"))

If I incorrectly scrubbed something, or you need the original logs/message from the email server, just let me know and I'll be happy to email them to you directly.

Thanks for the help!
Justin

[jstedfast]

Looks like this is a duplicate of this: #1415 (comment)

(I can't blame you for not finding that dupe since the Title of the issue doesn't relate to this)