Prerelease 21.1.0-RC4

This is a prerelease used for internal testing.

Compared to 21.0.0, this contains changes from these PRs:

• #169 – spring-core dependency bump
• #166 – documentation of spring properties
• #171 – batch-delete eventlog entries, implementing #159
• #173 – add QueryStatementBatcher
• #167° – replace SimpleSnapshotEventProducer by static of-Methods
• #176° – Bugfix: no stups token bean if not needed
• #179° – use Zonky embedded postgres instead of opentable one for tests
• #177° – Add DELETE permission to the eventlog table (fixing #101)
• #175° – introduce "switch off all submission" toggle
• #174° – Delete after insert

° – Some of these were not yet merged when this release was created, for those we have the latest state as-of 2023-08-18.

21.0.0 Dependency updates, bulk firing of events, compacted event types

This is the first release of the 21.x.x series, for Spring-Boot 2.5

Relevant changes

• Dependency updates: (#143, #163, #150, #156, #155): We now depend on Spring-Boot 2.5 (Spring-core 5.3), Flyway 7 (but it also should work with newer versions of these), Fahrschein 0.24, and (optionally) tokens 0.14.0. (We also updated the dependency to the postgresql client, and are using a newer version of JUnit internally.)
  • The minimum Java version is now Java 11.
  • This is using the new Flyway callback infrastructure. If you are using this, please be careful.
  • Submission to Nakadi is now using gzip compression by default. You can disable this or pick zstd compression using the nakadi-producer.content-encoding configuration variable.
• Feature (#140, implemented in #142): It is now possible to persist multiple events in batch to the database, with new methods on EventLogWriter.
  This should improve the throughput considerably (and reduce load on the DB).
• Feature (#154, implemented in #164): It is now possible (though only recommended in specific cases) to fire events for compacted event types. This is done by providing Spring-beans of type CompactionKeyExtractor.

Please note that the configuration variable nakadi-producer.lock-size introduced in 20.4.0, with default unlimited, will become mandatory in the next major release (22). We recommend setting it already now, as without it you can run into out-of-memory errors in some cases.

All merged PRs:

• Bump postgresql from 42.2.25 to 42.4.1 in /nakadi-producer-spring-boot-starter by @dependabot in #150
• Bump postgresql from 42.4.1 to 42.4.3 in /nakadi-producer-spring-boot-starter by @dependabot in #156
• Bump postgresql from 42.3.3 to 42.3.8 in /nakadi-producer-loadtest by @dependabot in #155
• #140 Persist multiple events in batch by @fbrns in #141
• use batching for snapshots, too (#140) by @ePaul in #161
• #142 Update flyway and spring boot by @johannesknauft in #143
• Bump spring-core from 5.3.12 to 5.3.20 in /nakadi-producer by @dependabot in #163
• fix boolean operation in flyway migration (&& → ||) by @ePaul in #165
• support compacted event types (#154) via spring beans by @ePaul in #164

New Contributors

• @johannesknauft made their first contribution in #143

Full Changelog: 20.4.0...21.0.0

experimental prerelease 21.0.0-RC3b

This is an experimental release, containing one possible implementation of #154.

It is not clear yet whether this will become part of the final 21.0.0, of whether we go the other route (in 21.0.0-RC3a).

Change compared to 21.0.0-RC2:

• #164: support compacted event types (via spring beans)
  You now can submit events to a compacted event type, by adding a CompactionKeyExtractor spring bean.

experimental prerelease 21.0.0-RC3a

This was an experimental release, containing one possible implementation of #154.

This did not become part of 21.0.0, we decided to go the other route (in 21.0.0-RC3b).

Change compared to 21.0.0-RC2:

• #158: support compacted event types (via register method)
  You now can submit events to a compacted event type, by calling registerCompactionKeyExtractor on the EventLogWriter.

Prerelease 21.0.0-RC2

This is the second release-candidate for the 21.x.x series.

More detailed release notes are still to be written, use at your own peril.

Relevant changes compared to 20.4.0:

• #143 (+ #163): Dependency updates: We now depend on spring-boot 2.5, Flyway 7, and (optionally) tokens 0.14.0.
  This means that the minimum Java version is now Java 11.
  This is using the new Flyway callback infrastructure. If you are using this, please be careful.

• Feature: (#140, implemented in #141) : Possibility to persist multiple events in bulk (using just one request to the DB).
  This should improve the performance if you create many events of the same type in the same transaction.

We might do incompatible changes between this and the final release of 21.0.0 – only use this if you are prepared to work with this.

Relevant changes compared to 21.0.0-RC1 (which was already in 20.4.0):

• Feature (#112 + #111, implemented in #152): Configurable lock size

  When sending events out, we are first locking them. With this change the number of locked events gets configurable in the nakadi-producer.lock-size property. The lock size defines the maximum amount of events which are loaded into memory and published in one run (in one submission per event type). By default, all events are loaded into memory. In future releases, this property will become mandatory.

20.4.0: Make lock-size configurable

This version is for Spring-Boot 2.

Feature (#112): Configurable lock size

When sending events out, we are first locking them. With this change the number of locked events gets configurable in the nakadi-producer.lock-size property. The lock size defines the maximum amount of events which are loaded into memory and published in one run (in one submission per event type). By default, all events are loaded into memory. In future releases, this property will become mandatory.

Prerelease 21.0.0-RC1

This is the first release-candidate for the 21.x.x series.

More detailed release notes are still to be written, use at your own peril.

Relevant changes compared to 20.3.1:

• #143: Dependency updates: We now depend on spring-boot 2.5, Flyway 7, and (optionally) tokens 0.14.0.
  This might mean that this doesn't work with Java 8 anymore (feedback appreciated).
  This is using the new Flyway callback infrastructure. If you are using this, please be careful.

• #141: Possibility to persist multiple events in bulk (using just request to the DB).
  This should improve the performance if you create many events in the same transaction.
  (The API for this feature is not yet finalized, please give feedback on this, specifically
  the way data change events are submitted. Comments in #141, please.)

Note that this Prerelease was built from a merge of the two feature branches, before they were merged to master.
We might do incompatible changes between this and the final release of 21.0.0 – only use this if you are prepared to work with this.

20.3.1: More exception logging

This release is for Spring-Boot 2.

The only change is how errors are logged when not all events can be submitted to Nakadi.
This change was contributed by @riteshpuj2013, who faced an issue which was difficult to analyze.

No special migration considerations needed.

4.4.0: Better Flow-Id handling, configurable lock timeouts

This version is for Spring-Boot 1.

Feature: (#116) Better flow id handling for Snapshots
The current versions of tracer-spring-boot-starter do not setup tracing instrumentation for spring boots management endpoints (at least if they are mapped to a different port). Due to this, snapshots triggered via the management endpoints had no flow ids and a warning was logged for each event before sending.

We now create a new trace iff there is none yet in those cases, so that snapshot events will always have a flow id.

Feature: (#120) Configurable lock timeout

When sending events out, we are first locking them for some time so no other process running the same library is trying to send out the same events. With this change, the time for which those events are locked is configurable, which is helpful for applications with low latency and high volume requirements.
Look for lock-duration and lock-duration-buffer in the README if you want to use this feature.

No migration is needed, the default settings behave as before (but might change in the future).

Minor improvement: Some dependencies received some updates

20.3.0: Make lock-timeout for event submission configurable

This version is for Spring-Boot 2.

Feature (#120): Configurable lock timeout

When sending events out, we are first locking them for some time so no other process running the same library is trying to send out the same events. With this change, the time for which those events are locked is configurable, which is helpful for applications with low latency and high volume requirements.
Look for lock-duration and lock-duration-buffer in the README if you want to use this feature.

No migration is needed, the default settings behave as before (but might change in the future).

Benchmark (#83): We also got a nakadi-producer-load-test subproject, which allows to measure how fast large quantities of events can be submitted (using a locally running Nakadi instance).
See the README in that sub-project folder for how to use it.