Poison messages are moved to Error Queue

[johnsimons]

Symptoms

When a transport deems a message poison,

• https://github.com/Particular/NServiceBus.SqlServer/blob/develop/src/NServiceBus.SqlServer/Receiving/ReceiveWithTransactionScope.cs#L61
• https://github.com/Particular/NServiceBus/blob/develop/src/NServiceBus.Core/Transports/Msmq/ReceiveStrategy.cs#L90

The message is forwarded to the error queue, the issue is that this message does not have all the headers a normal error has:

• FailedQ
• Exception details
• ....

As it currently stands ServiceControl fails to import those messages:

The message will be saved in the RavenDb as a FailedErrorImport document and ServiceControl will log an event to Windows event log in the description of which will be the path to a file that includes the actual exception information:

Example Event Log Message:
A message import has failed. A log file has been written to C:\Users\bobla\AppData\Local\Particular\ServiceControl\logs\FailedImports\Error\b045292f-b97a-4b20-ba21-ec98f81b930a.txt

Example Logged File:

The given key was not present in the dictionary.
StackTrace:
   at System.Collections.Generic.Dictionary`2.get_Item(TKey key)
   at ServiceControl.Contracts.Operations.ImportFailedMessage.ParseFailureDetails(Dictionary`2 headers) in C:\Code\NSB\ServiceControl\src\ServiceControl\Contracts\Operations\ImportFailedMessage.cs:line 31
   at ServiceControl.Contracts.Operations.ImportFailedMessage..ctor(TransportMessage message) in C:\Code\NSB\ServiceControl\src\ServiceControl\Contracts\Operations\ImportFailedMessage.cs:line 13
   at ServiceControl.Operations.ErrorQueueImport.InnerHandle(TransportMessage message) in C:\Code\NSB\ServiceControl\src\ServiceControl\Operations\ErrorQueueImport.cs:line 115
   at ServiceControl.Operations.ErrorQueueImport.Handle(TransportMessage message) in C:\Code\NSB\ServiceControl\src\ServiceControl\Operations\ErrorQueueImport.cs:line 75
   at NServiceBus.Satellites.SatelliteLauncher.HandleMessageReceived(Object sender, TransportMessageReceivedEventArgs e, ISatellite satellite) in C:\Build\src\NServiceBus.Core\Satellites\SatelliteLauncher.cs:line 89
   at NServiceBus.Unicast.Transport.TransportReceiver.OnTransportMessageReceived(TransportMessage msg) in C:\Build\src\NServiceBus.Core\Unicast\Transport\TransportReceiver.cs:line 413
   at NServiceBus.Unicast.Transport.TransportReceiver.ProcessMessage(TransportMessage message) in C:\Build\src\NServiceBus.Core\Unicast\Transport\TransportReceiver.cs:line 344
   at NServiceBus.Unicast.Transport.TransportReceiver.TryProcess(TransportMessage message) in C:\Build\src\NServiceBus.Core\Unicast\Transport\TransportReceiver.cs:line 230
   at NServiceBus.Transports.Msmq.MsmqDequeueStrategy.Action(TransactionSettings transactionSettings, TransactionOptions transactionOptions, MsmqUnitOfWork unitOfWork, MessageQueue receiveQueue, MessageQueue errorQueue, CircuitBreaker circuitBreaker, CriticalError criticalError, AutoResetEvent peekResetEvent, TimeSpan receiveTimeout, SemaphoreSlim throttlingSemaphore, Func`2 tryProcessMessage, Action`2 endProcessMessage) in C:\Build\src\NServiceBus.Core\Transports\Msmq\MsmqDequeueStrategy.cs:line 229
Source:
mscorlib
TargetSite:
TValue get_Item(TKey)

Who's Affected

Users that are running ServiceControl v1.25.0 or below and are on NServiceBus v6

[johnsimons]

I have done a bit more analysis and this is not entirely correct.
MSMQ and SqlServer transport redirects this corrupted messages when it cannot deserialize the header for the message, in this scenario, ServiceControl would also fail with exactly the same issue.
Here is the msmq transport code, as you can see we would be forwarding the corrupted message to the configured ServiceControl error queue.
In SqlServer transport a similar strategy occurs, it seems in the Sql trasnport the failure could be more then just headers corruption but the end result is the same, the message will end-up being forwarded to error queue, which ServiceControl would also fail, and the message final destination is the configured ServiceControl error queue.
In Rabbit it seems to be a bit different, it looks like rabbit will just ack the message and throws it away, so message will never end being forwarded to error queue!

[johnsimons]

@boblangley can u check ASB and ASQ, for NSB v5

[Scooletz]

In ASQ (core v6) we do exactly the same. If the wrapper, which contains headers, can't be deserialized it's moved directly to the error queue.