Skip to content

Latest commit

 

History

History
448 lines (313 loc) · 18.1 KB

CHANGELOG.adoc

File metadata and controls

448 lines (313 loc) · 18.1 KB
Raw

Change Log

0.5.3.3

Fixes

  • fix: close parallel consumer on transactional mode when InvalidPidMappingException (#830)

0.5.3.2

Fixes

  • fix: include inflight message count in polling backpressure logic (#836)

  • fix: message loss on closing or partitions revoked (#827) fixes (#826)

  • fix: unbounded retry queue growth preventing polling from being throttled and leading to OOM (#834) fixes (#832, #817)

Note

#836 introduces a change in how buffer size is calculated as now inflight messages are counted as part of buffer size - so behaviour of existing applications may change and pausing of consumer happen sooner.

0.5.3.1

Fixes

  • fix: ConcurrentModificationException Happened while high load and draining (#822) fixes (#821)

  • fix: safely completing doClose() (#818) partially fixes (#809)

  • Improved offset commit retry. Add support for SaslAuthenticationException retry timeout (#819), partially fixes (#809) in Commit_Sync mode

0.5.3.0

Fixes

  • fix: ReactorProcessor - run used-defined function in provided scheduler rather than in pc-pool thread (#798 / #794), fixes (#793)

  • fix: fix issue for cannot close and exit properly when re-balancing storm (#787)

  • fix: Support for PCRetriableException in ReactorProcessor (#733)

  • fix: NullPointerException on partitions revoked (#757)

  • fix: remove lingeringOnCommitWouldBeBeneficial and unused imports (#732)

  • fix: Fix failing auto-commit check for kafka-clients >= v3.7.0 (#721)

  • fix: Fix redundant rebalance callback in LongPollingMockConsumer for Kafka >= 3.6 (#765)

Improvements

  • improvement: stale containers exclusion and handling improvement (#779)

  • improvement: add multiple caches for accelerating available container count calculation （#667）

  • improvement: RecordContext now exposes lastFailureReason (#725)

Dependencies

  • build(deps): Bump Kafka to 3.6.2

  • build(deps): Bump Kafka to 3.7.0

0.5.2.8

Fixes

  • fix: Fix equality and hash code for ShardKey with array key (#638), resolves (#579)

  • fix: Fix target loading computation for inflight records (#662)

  • fix: Fix synchronisation logic for transactional producer commit affecting non-transactional usage (#665), resolves (#637)

  • fix: Fix for race condition in partition state clean/dirty tracking (#666), resolves (#664)

Improvements

  • feature: Make PC message buffer size configurable - two new configuration options for controlling buffer size added (#682)

0.5.2.7

Fixes

  • fix: Return cached pausedPartitionSet (#620), resolves (#618)

  • fix: Parallel consumer stops processing data sometimes (#623), fixes (#606)

  • fix: Add synchronization to ensure proper intializaiton and closing of PCMetrics singleton (#627), fixes (#617)

  • fix: Readme - metrics example correction (#614)

  • fix: Remove micrometer-atlas dependency (#628), fixes (#625)

Improvements

  • Refactored metrics implementation to not use singleton - improves meter separation, allows correct metrics subsystem operation when multiple parallel consumer instances are running in same java process (#630), fixes (#617) improves on (#627)

0.5.2.6

Improvements

  • feature: Micrometer metrics (#594)

  • feature: Adds an option to pass an invalid offset metadata error policy (#537), improves (#326)

  • feature: Lazy intialization of workerThreadPool (#531)

Fixes

  • fix: Don’t drain mode shutdown kills inflight threads (#559)

  • fix: Drain mode shutdown doesn’t pause consumption correctly (#552)

  • fix: RunLength offset decoding returns 0 base offset after no-progress commit - related to (#546)

  • fix: Transactional PConsumer stuck while rebalancing - related to (#541)

Dependencies

  • PL-211: Update dependencies from dependabot, Add mvnw, use mvnw in jenkins (#583)

  • PL-211: Update dependencies from dependabot (#589)

0.5.2.5

Fixes

  • fixes: #195 NoSuchFieldException when using consumer inherited from KafkaConsumer (#469)

  • fix: After new performance fix PR#530 merges - corner case could cause out of order processing (#534)

  • fix: Cleanup WorkManager’s count of in-progress work, when work is stale after partition revocation (#547)

Improvements

  • perf: Adds a caching layer to work management to alleviate O(n) counting (#530)

0.5.2.4

Improvements

  • feature: Simple PCRetriableException to remove error spam from logs (#444)

  • minor: fixes #486: Missing generics in JStreamParallelStreamProcessor #491

  • minor: partially address #459: Moves isClosedOrFailed into top level ParallelConsumer interface (#491)

  • tests: Demonstrates how to use MockConsumer with PC for issue #176

  • other minor improvements

Fixes

  • fixes #409: Adds support for compacted topics and commit offset resetting (#425)

    • Truncate the offset state when bootstrap polled offset higher or lower than committed

    • Prune missing records from the tracked incomplete offset state, when they’re missing from polled batches

  • fix: Improvements to encoding ranges (int vs long) #439

    • Replace integer offset references with long - use Long everywhere we deal with offsets, and where we truncate down, do it exactly, detect and handle truncation issues.

0.5.2.3

Improvements

  • Transactional commit mode system improvements and docs (#355)

    • Clarifies transaction system with much better documentation.

    • Fixes a potential race condition which could cause offset leaks between transactions boundaries.

    • Introduces lock acquisition timeouts.

    • Fixes a potential issue with removing records from the retry queue incorrectly, by having an inconsistency between compareTo and equals in the retry TreeMap.

  • Adds a very simple Dependency Injection system modeled on Dagger (#398)

  • Various refactorings e.g. new ProducerWrap

  • Dependencies

    • build(deps): prod: zstd, reactor, dev: podam, progressbar, postgresql maven-plugins: versions, help (#420)

    • build(deps-dev): bump postgresql from 42.4.1 to 42.5.0

    • bump podam, progressbar, zstd, reactor

    • build(deps): bump versions-maven-plugin from 2.11.0 to 2.12.0

    • build(deps): bump maven-help-plugin from 3.2.0 to 3.3.0

    • build(deps-dev): bump Confluent Platform Kafka Broker to 7.2.2 (#421)

    • build(deps): Upgrade to AK 3.3.0 (#309)

Fixes

  • fixes #419: NoSuchElementException during race condition in PartitionState (#422)

  • Fixes #412: ClassCastException with retryDelayProvider (#417)

  • fixes ShardManager retryQueue ordering and set issues due to poor Comparator implementation (#423)

v0.5.2.2

Fixes

  • Fixes dependency scope for Mockito from compile to test (#376)

v0.5.2.1

Fixes

  • Fixes regression issue with order of state truncation vs commit (#362)

v0.5.2.0

Fixes and Improvements

  • fixes #184: Fix multi topic subscription with KEY order by adding topic to shard key (#315)

  • fixes #329: Committing around transaction markers causes encoder to crash (#328)

  • build: Upgrade Truth-Generator to 0.1.1 for user Subject discovery (#332)

Build

  • build: Allow snapshots locally, fail in CI (#331)

  • build: OSS Index scan change to warn only and exclude Guava CVE-2020-8908 as it’s WONT_FIX (#330)

Dependencies

  • build(deps): bump reactor-core from 3.4.19 to 3.4.21 (#344)

  • build(deps): dependabot bump Mockito, Surefire, Reactor, AssertJ, Release (#342) (#342)

  • build(deps): dependabot bump TestContainers, Vert.x, Enforcer, Versions, JUnit, Postgress (#336)

Linked issues

  • Message with null key lead to continuous failure when using KEY ordering #318

  • Subscribing to two or more topics with KEY ordering, results in messages of the same Key never being processed #184

  • Cannot have negative length BitSet error - committing transaction adjacent offsets #329

v0.5.1.0

Features

  • #193: Pause / Resume PC (circuit breaker) without unsubscribing from topics

Fixes and Improvements

  • #225: Build and runtime support for Java 16+ (#289)

  • #306: Change Truth-Generator dependency from compile to test

  • #298: Improve PollAndProduce performance by first producing all records, and then waiting for the produce results.Previously, this was done for each ProduceRecord individually.

v0.5.0.0

Features

  • feature: Poll Context object for API (#223)

    • PollContext API - provides central access to result set with various convenience methods as well as metadata about records, such as failure count

  • major: Batching feature and Event system improvements

    • Batching - all API methods now support batching. See the Options class set batch size for more information.

Fixes and Improvements

  • Event system - better CPU usage in control thread

  • Concurrency stability improvements

  • Update dependencies

  • #247: Adopt Truth-Generator (#249)

  • Large rewrite of internal architecture for improved maintence and simplicity which fixed some corner case issues

    • refactor: Rename PartitionMonitor to PartitionStateManager (#269)

    • refactor: Queue unification (#219)

    • refactor: Partition state tracking instead of search (#218)

    • refactor: Processing Shard object

  • fix: Concurrency and State improvements (#190)

Build

  • build: Lock TruthGenerator to 0.1 (#272)

  • build: Deploy SNAPSHOTS to maven central snaphots repo (#265)

  • build: Update Kafka to 3.1.0 (#229)

  • build: Crank up Enforcer rules and turn on ossindex audit

  • build: Fix logback dependency back to stable

  • build: Upgrade TestContainer and CP

v0.4.0.1

Improvements

  • Add option to specify timeout for how long to wait offset commits in periodic-consumer-sync commit-mode

  • Add option to specify timeout for how long to wait for blocking Producer#send

Docs

  • docs: Confluent Cloud configuration links

  • docs: Add Confluent’s product page for PC to README

  • docs: Add head of line blocking to README

v0.4.0.0

Features

  • Project Reactor non-blocking threading adapter module

  • Generic Vert.x Future support - i.e. FileSystem, db etc…​

Fixes and Improvements

  • Vert.x concurrency control via WebClient host limits fixed - see #maxCurrency

  • Vert.x API cleanup of invalid usage

  • Out of bounds for empty collections

  • Use ConcurrentSkipListMap instead of TreeMap to prevent concurrency issues under high pressure

  • log: Show record topic in slow-work warning message

v0.3.2.0

Fixes and Improvements

  • Major: Upgrade to Apache Kafka 2.8 (still compatible with 2.6 and 2.7 though)

  • Adds support for managed executor service (Java EE Compatibility feature)

  • #65 support for custom retry delay providers

v0.3.1.0

Fixes and Improvements

  • Major refactor to code base - primarily the two large God classes

    • Partition state now tracked separately

    • Code moved into packages

  • Busy spin in some cases fixed (lower CPU usage)

  • Reduce use of static data for test assertions - remaining identified for later removal

  • Various fixes for parallel testing stability

v0.3.0.3

Fixes and Improvements

Overview

  • Tests now run in parallel

  • License fixing / updating and code formatting

  • License format runs properly now when local, check on CI

  • Fix running on Windows and Linux

  • Fix JAVA_HOME issues

Details:

  • tests: Enable the fail fast feature now that it’s merged upstream

  • tests: Turn on parallel test runs

  • format: Format license, fix placement

  • format: Apply Idea formatting (fix license layout)

  • format: Update mycila license-plugin

  • test: Disable redundant vert.x test - too complicated to fix for little gain

  • test: Fix thread counting test by closing PC @After

  • test: Test bug due to static state overrides when run as a suite

  • format: Apply license format and run every All Idea build

  • format: Organise imports

  • fix: Apply license format when in dev laptops - CI only checks

  • fix: javadoc command for various OS and envs when JAVA_HOME missing

  • fix: By default, correctly run time JVM as jvm.location

v0.3.0.2

Fixes and Improvements

  • ci: Add CODEOWNER

  • fix: #101 Validate GroupId is configured on managed consumer

  • Use 8B1DA6120C2BF624 GPG Key For Signing

  • ci: Bump jdk8 version path

  • fix: #97 Vert.x thread and connection pools setup incorrect

  • Disable Travis and Codecov

  • ci: Apache Kafka and JDK build matrix

  • fix: Set Serdes for MockProducer for AK 2.7 partition fix KAFKA-10503 to fix new NPE

  • Only log slow message warnings periodically, once per sweep

  • Upgrade Kafka container version to 6.0.2

  • Clean up stalled message warning logs

  • Reduce log-level if no results are returned from user-function (warn → debug)

  • Enable java 8 Github

  • Fixes #87 - Upgrade UniJ version for UnsupportedClassVersion error

  • Bump TestContainers to stable release to specifically fix #3574

  • Clarify offset management capabilities

v0.3.0.1

  • fixes #62: Off by one error when restoring offsets when no offsets are encoded in metadata

  • fix: Actually skip work that is found as stale

v0.3.0.0

Features

  • Queueing and pressure system now self tuning, performance over default old tuning values (softMaxNumberMessagesBeyondBaseCommitOffset and maxMessagesToQueue) has doubled.

    • These options have been removed from the system.

  • Offset payload encoding back pressure system

    • If the payload begins to take more than a certain threshold amount of the maximum available, no more messages will be brought in for processing, until the space need beings to reduce back below the threshold. This is to try to prevent the situation where the payload is too large to fit at all, and must be dropped entirely.

    • See Proper offset encoding back pressure system so that offset payloads can’t ever be too large #47

    • Messages that have failed to process, will always be allowed to retry, in order to reduce this pressure.

Improvements

  • Default ordering mode is now KEY ordering (was UNORDERED).

    • This is a better default as it’s the safest mode yet high performing mode. It maintains the partition ordering characteristic that all keys are processed in log order, yet for most use cases will be close to as fast as UNORDERED when the key space is large enough.

  • Support BitSet encoding lengths longer than Short.MAX_VALUE #37 - adds new serialisation formats that supports wider range of offsets - (32,767 vs 2,147,483,647) for both BitSet and run-length encoding.

  • Commit modes have been renamed to make it clearer that they are periodic, not per message.

  • Minor performance improvement, switching away from concurrent collections.

Fixes

  • Maximum offset payload space increased to correctly not be inversely proportional to assigned partition quantity.

  • Run-length encoding now supports compacted topics, plus other bug fixes as well as fixes to Bitset encoding.

v0.2.0.3

Fixes

v0.2.0.2

Fixes

v0.2.0.1 DO NOT USE - has critical bug

Fixes

v0.2.0.0

Features

  • Choice of commit modes: Consumer Asynchronous, Synchronous and Producer Transactions

  • Producer instance is now optional

  • Using a transactional Producer is now optional

  • Use the Kafka Consumer to commit offsets Synchronously or Asynchronously

Improvements

  • Memory performance - garbage collect empty shards when in KEY ordering mode

  • Select tests adapted to non transactional (multiple commit modes) as well

  • Adds supervision to broker poller

  • Fixes a performance issue with the async committer not being woken up

  • Make committer thread revoke partitions and commit

  • Have onPartitionsRevoked be responsible for committing on close, instead of an explicit call to commit by controller

  • Make sure Broker Poller now drains properly, committing any waiting work

Fixes

  • Fixes bug in commit linger, remove genesis offset (0) from testing (avoid races), add ability to request commit

  • Fixes #25 #25:

    • Sometimes a transaction error occurs - Cannot call send in state COMMITTING_TRANSACTION #25

  • ReentrantReadWrite lock protects non-thread safe transactional producer from incorrect multithreaded use

  • Wider lock to prevent transaction’s containing produced messages that they shouldn’t

  • Must start tx in MockProducer as well

  • Fixes example app tests - incorrectly testing wrong thing and MockProducer not configured to auto complete

  • Add missing revoke flow to MockConsumer wrapper

  • Add missing latch timeout check

v0.1

Features:

  • Have massively parallel consumption processing without running hundreds or thousands of

    • Kafka consumer clients

    • topic partitions

      without operational burden or harming the clusters performance

  • Efficient individual message acknowledgement system (without local or third system state) to massively reduce message replay upon failure

  • Per key concurrent processing, per partition and unordered message processing

  • Offsets committed correctly, in order, of only processed messages, regardless of concurrency level or retries

  • Vert.x non-blocking library integration (HTTP currently)

  • Fair partition traversal

  • Zero~ dependencies (Slf4j and Lombok) for the core module

  • Java 8 compatibility

  • Throttle control and broker liveliness management

  • Clean draining shutdown cycle