expectLogicOffset is greater than currentLogicOffset in consumeQueue build when the message is illegal

[RongtongJin]

The issue tracker is used for bug reporting purposes ONLY whereas feature request needs to follow the RIP process. To avoid unnecessary duplication, please check whether there is a previous issue before filing a new one.

It is recommended to start a discussion thread in the mailing lists or github discussions in cases of discussing your deployment plan, API clarification, and other non-bug-reporting issues.
We welcome any friendly suggestions, bug fixes, collaboration, and other improvements.

Please ensure that your bug report is clear and self-contained. Otherwise, it would take additional rounds of communication, thus more time, to understand the problem itself.

Generally, fixing an issue goes through the following steps:

1. Understand the issue reported;
2. Reproduce the unexpected behavior locally;
3. Perform root cause analysis to identify the underlying problem;
4. Create test cases to cover the identified problem;
5. Work out a solution to rectify the behavior and make the newly created test cases pass;
6. Make a pull request and go through peer review;

As a result, it would be very helpful yet challenging if you could provide an isolated project reproducing your reported issue. Anyway, please ensure your issue report is informative enough for the community to pick up. At a minimum, include the following hints:

BUG REPORT

1. Please describe the issue you observed:

After running the 5.1.0 cluster for a while, there were cases where the consume queue offset were misplaced, and the error log frequently appeared in the storeerror.log.

expectLogicOffset is greater than currentLogicOffset, which means that the queueOffset in the topicQueueTable in memory is greater than the actual offset of the ConsumeQueue file.

2. Cause analysis

In the store.log, it was found that there were message's properties ​​that were too long, exceeding the maximum value of short.

When message put to commitlog, the offset in the topicQueueTable will be increased first, and then the message will be encoded.

The encode process will verify the length of the propertites. If the verification fails, the message will not be written to the commit log, but the offset in the topicQueueTable has already been increased, resulting in the above phenomenon (the queueOffset in the topicQueueTable in memory is greater than the actual offset in the cq file).