Release v2.7.0

Features

042cca4 feat(expression): add new built-in function from_bytes
0623e61 feat(expression): add new built-in function extract_struct_field
4bb15d3 feat(expression): add new built-in function parse_url
8484877 feat(filters): add new built-in CSVFilter (#249)
1bb3c9b feat(filesystem): add support for AmazonS3MoveCleanupPolicy (#238)
a6abb9d feat(filesystem): allow to configure S3 client RetryPolicy with max.retries and back.off delay (#247)

Improvements & Bugfixes

b37628f fix(filters): fix wrong arg type for DelimitedRowFilterConfig
5b5b501 fix(filters): enhance DateFilter to support parsing simple date (#277)
c4998d5 fix(api): fix wrong argurments passed to TypedStruct
e62dc04 fix(api): fix TypedValue should default to string on non standard type-value (#wq108)
689c903 fix(filters): fix DelimitedRowFilter to not discarded trailing empty strings (#257)
2d37f89 refactor(filters): refactor and normalize config names for DelimitedRowFilterConfig
82c3597 fix(api): fix Arrayschema to support merging given empty array
3bef574 Fixed Add Support for AWS AssumeRole #251
ab74f57 fix(api): remove dot from connector-name when storing file states (#231)
7a9a333 last maven version ( > 3.8.1 ) require https repository endpoint

Docs

06a1832 docs: update installation guide
d8b176b docs: update AWS config properties
11450ae docs: fix FileSystem Listing section (#246)
934056c docs: fix getting-started
919b430 Fix documentation for AppendError

Sub-Tasks

6786156 build(deps): bump maven-surefire-plugin from 3.0.0-M3 to 3.0.0-M7
ba06a87 build(deps): bump maven-source-plugin from 3.2.0 to 3.2.1
ec2e1e8 build(deps): bump libraries-bom from 19.2.1 to 25.4.0
fd3d2ce build(deps): bump maven-gpg-plugin from 1.5 to 3.0.1
c416183 build(deps): bump findsecbugs-plugin from 1.10.1 to 1.12.0
4b65855 build(deps-dev): bump json-path from 3.3.0 to 5.1.0
5908d62 build(deps): bump spotbugs from 4.5.3 to 4.7.0
ad72fd0 build(deps): bump jackson-core from 2.13.2 to 2.13.
d3a084a build(deps): bump antlr4.version from 4.7.1 to 4.10.1
d85ce86 test: add junit5 and fix integration-test
b50e247 build: add spotbugs maven plugin
529a44f build(deps-dev): bump testcontainers from 1.11.1 to 1.17.2
41fca73 build(deps): bump json from 20210307 to 20220320
276e0ba build(deps): bump github/codeql-action from 1 to 2
38855e8 build(deps): bump actions/setup-node from 1 to 3
3dc560e build(deps): bump jackson-annotations from 2.13.2 to 2.13.3
4069ffd build(deps): bump maven-jar-plugin from 3.0.2 to 3.2.2
fd07ee3 build(deps): bump avro from 1.10.1 to 1.11.0
90613b5 build(deps): bump actions/checkout from 2 to 3
4a8da8d build(deps): bump org.apache.kafka.version from 2.8.0 to 3.2.0
1e5d48f build(deps): bump kafka-connect-maven-plugin from 0.11.3 to 0.12.0
9453b5d build(deps): bump docker/login-action from 1 to 2
350f49e build(deps): bump actions/setup-java from 2 to 3
04155ac build(deps): bump docker/build-push-action from 2 to 3
265e03e build(deps): bump jackson-databind from 2.12.6.1 to 2.13.3
27f0bfd build(deps): bump Saxon-HE from 9.9.1-5 to 11.3
45952f9 ci: update git actions workflows
364d6b7 build: bump version for next iteration
c836ded build: fix jackson version
adfec26 gh: add issue template
329536d build(deps): bump jackson-databind from 2.12.2 to 2.12.6.1

Docker Images

$ docker pull streamthoughts/kafka-connect-file-pulse:2.7.0

For verifying distirbution zip file

$ curl -L -o streamthoughts-kafka-connect-file-pulse-2.7.0.zip https://github.com/streamthoughts/kafka-connect-file-pulse/releases/download/v2.7.0/streamthoughts-kafka-connect-file-pulse-2.7.0.zip
$ curl -L -o streamthoughts-kafka-connect-file-pulse-2.7.0.zip.asc https://github.com/streamthoughts/kafka-connect-file-pulse/releases/download/v2.7.0/streamthoughts-kafka-connect-file-pulse-2.7.0.zip.asc
$ gpg --keyserver keyserver.ubuntu.com --recv-keys D12696955654914F
$ gpg --verify streamthoughts-kafka-connect-file-pulse-2.7.0.zip.asc streamthoughts-kafka-connect-file-pulse-2.7.0.zip

New Contributors

• @ogawa-takeshi made their first contribution in #252
• @bertrandcedric made their first contribution in #256

Full Changelog: v2.6.0...v2.7.0

Release v2.6.0

Features

10259c5 feat(plugin): add built-in SizeFileListFilter

Improvements & Bugfixes

32d55e4 fix(api): fix typed value parsing may result in a loss of precision
d17298f fix(plugin): add generation-id prop to task-config to force reconfiguraton

Docs

Sub-Tasks

a8ddbf1 script: update debug.sh

Docker Images

$ docker pull streamthoughts/kafka-connect-file-pulse:2.6.0

For verifying distirbution zip file

$ curl -L -o streamthoughts-kafka-connect-file-pulse-2.6.0.zip https://github.com/streamthoughts/kafka-connect-file-pulse/releases/download/v2.6.0/streamthoughts-kafka-connect-file-pulse-2.6.0.zip
$ curl -L -o streamthoughts-kafka-connect-file-pulse-2.6.0.zip.asc https://github.com/streamthoughts/kafka-connect-file-pulse/releases/download/v2.6.0/streamthoughts-kafka-connect-file-pulse-2.6.0.zip.asc
$ gpg --keyserver keyserver.ubuntu.com --recv-keys D12696955654914F
$ gpg --verify streamthoughts-kafka-connect-file-pulse-2.6.0.zip.asc streamthoughts-kafka-connect-file-pulse-2.6.0.zip

Release v2.5.2

Features

Improvements & Bugfixes

e8cbbf2 (origin/master, origin/HEAD) fix(api): fix filter-pipeline to always copy record context between filters (#235)
f639209 fix(filesystem): update LocalFileStorage to fallback to a simple move implements as copy+delete (#234)

Docs

Sub-Tasks

Docker Images

$ docker pull streamthoughts/kafka-connect-file-pulse:2.5.2

For verifying distirbution zip file

$ curl -L -o streamthoughts-kafka-connect-file-pulse-2.5.2.zip https://github.com/streamthoughts/kafka-connect-file-pulse/releases/download/v2.5.2/streamthoughts-kafka-connect-file-pulse-2.5.2.zip
$ curl -L -o streamthoughts-kafka-connect-file-pulse-2.5.2.zip.asc https://github.com/streamthoughts/kafka-connect-file-pulse/releases/download/v2.5.2/streamthoughts-kafka-connect-file-pulse-2.5.2.zip.asc
$ gpg --keyserver keyserver.ubuntu.com --recv-keys D12696955654914F
$ gpg --verify streamthoughts-kafka-connect-file-pulse-2.5.2.zip.asc streamthoughts-kafka-connect-file-pulse-2.5.2.zip

Release v2.5.1

Features

8b1491b feat(plugin): allow to keep leading underscores on field names

Improvements & Bugfixes

d60e574 fix(api): connector should throw an error when merging struct with string
6f6decc fix(plugin): files should be cleanup when an error occurred while converting data to connect record
b308e70 fix(api): fix connect.value.schema is not used when sink topic is computed from record
6aa8123 fix(dataformat): ignore empty XML elements that contain only whitespaces
79deb60 fix(docker): fix docker-compose version

Docs

Sub-Tasks

Docker Images

$ docker pull streamthoughts/kafka-connect-file-pulse:2.5.1

For verifying distirbution zip file

$ curl -L -o streamthoughts-kafka-connect-file-pulse-2.5.1.zip https://github.com/streamthoughts/kafka-connect-file-pulse/releases/download/v2.5.1/streamthoughts-kafka-connect-file-pulse-2.5.1.zip
$ curl -L -o streamthoughts-kafka-connect-file-pulse-2.5.1.zip.asc https://github.com/streamthoughts/kafka-connect-file-pulse/releases/download/v2.5.1/streamthoughts-kafka-connect-file-pulse-2.5.1.zip.asc
$ gpg --keyserver keyserver.ubuntu.com --recv-keys D12696955654914F
$ gpg --verify streamthoughts-kafka-connect-file-pulse-2.5.1.zip.asc streamthoughts-kafka-connect-file-pulse-2.5.1.zip

Release v2.5.0

Features

b8dd10b feat(plugin): allow to configure cache.max.size.capacity for InMemoryFileObjectStateBackingStore
4f892a8 feat: add new config props to define file ordering
f964de0 feat(dataformat): add new XML config to force content field
2a78283 feat: externalize configs for XML filter and reader
b1b8de9 feat: add new config xml.text.node.value.field.name

Improvements & Bugfixes

9163b74 fix(dataformat): fix invalig config and NPE in XMLNodeToStructConverter
a3fe274 fix(api): fix wrong schema for double in TypedStruct (#230)
2884fe0 fix(filters): ignore blank values during DelimitedRowFilter type conversion (#228)
d7a4738 fix(filesystem-s3): fix AmazonS3FileSystemListing get stuck when bucket contains more than max response items (#226)

Docs

5f517f9 docs: fix config props for XML filter and reader
37adaa9 docs: fix typo and config error on AzureBlobStorageFileSystemListing (#229)
a4e28d5 docs: fix yaml
c5cb5a3 docs: fix typos and reformat

Sub-Tasks

Docker Images

$ docker pull streamthoughts/kafka-connect-file-pulse:2.5.0

For verifying distirbution zip file

$ curl -L -o streamthoughts-kafka-connect-file-pulse-2.5.0.zip https://github.com/streamthoughts/kafka-connect-file-pulse/releases/download/v2.5.0/streamthoughts-kafka-connect-file-pulse-2.5.0.zip
$ curl -L -o streamthoughts-kafka-connect-file-pulse-2.5.0.zip.asc https://github.com/streamthoughts/kafka-connect-file-pulse/releases/download/v2.5.0/streamthoughts-kafka-connect-file-pulse-2.5.0.zip.asc
$ gpg --keyserver keyserver.ubuntu.com --recv-keys D12696955654914F
$ gpg --verify streamthoughts-kafka-connect-file-pulse-2.5.0.zip.asc streamthoughts-kafka-connect-file-pulse-2.5.0.zip

Release v2.4.3

Features

fbe8b6f feat(scel-api): add is_empty expression for string/array fields

Improvements & Bugfixes

2232579 fix(plugins): fix task resources should be closed correctly (#224)
8d266f1 fix(amazons3-fs): fix NPE hen configuring credential provider (#219)

Docs

Sub-Tasks

Docker Images

$ docker pull streamthoughts/kafka-connect-file-pulse:2.4.3

For verifying distirbution zip file

$ curl -L -o streamthoughts-kafka-connect-file-pulse-2.4.3.zip https://github.com/streamthoughts/kafka-connect-file-pulse/releases/download/v2.4.3/streamthoughts-kafka-connect-file-pulse-2.4.3.zip
$ curl -L -o streamthoughts-kafka-connect-file-pulse-2.4.3.zip.asc https://github.com/streamthoughts/kafka-connect-file-pulse/releases/download/v2.4.3/streamthoughts-kafka-connect-file-pulse-2.4.3.zip.asc
$ gpg --keyserver keyserver.ubuntu.com --recv-keys D12696955654914F
$ gpg --verify streamthoughts-kafka-connect-file-pulse-2.4.3.zip.asc streamthoughts-kafka-connect-file-pulse-2.4.3.zip

Release v2.4.2

Features

Improvements & Bugfixes

948687d fix(api): fix stackoverflow while merging incompatible schemas (#208)
d7106c5 build(deps): include slf4j-log4j12 only for test (#213)
b89a913 fix(dataformat): ignore XML document node type for converting data (#207)

Docs

0fd88e8 fix filename typo

Sub-Tasks

01e4b4c Bump to actions/setup-java@v2, use Zulu Java 11
4d36de6 test(docker): add akhq to docker-compose-debug.yml
302889e fix(build): fix makefile

Docker Images

$ docker pull streamthoughts/kafka-connect-file-pulse:2.4.2

For verifying distirbution zip file

$ gpg --keyserver keyserver.ubuntu.com --recv-keys D12696955654914F
$ gpg --verify streamthoughts-kafka-connect-file-pulse-2.4.2.zip.asc streamthoughts-kafka-connect-file-pulse-2.4.2.zip

Release v2.4.1

Features

Improvements & Bugfixes

c6c9d3b fix(api): schema value for collection should be inferred from all items
680685e bump azure blob version to 12.14.2
3883456 refactor(filesystems): improve logs and add check for file is readable
917ef7b fix(plugin): ensure that completed files are cleaned before starting task(#194)
05a0ba8 fix(plugins): fix invalid object-file should be reported as invalid (#193)
85368d8 builder(deps): bump kafka-connect base-image version (#192)
333e1ee fix(filesystems): fix GcsStorage should support blob URI with underscore (#190)
1818f00 fix(expression): fix MapAdaptable/Reflective accessors should support dotted-notation (#189)

Docs

9c6ea7e docs(site): add details about configuring connector with SSL/SASL
ae05829 docs(site): add release 2.4 blog post

Sub-Tasks

0967479 build(docker): refactor and cleanup docker build
a9fd0b4 build(scripts): remove build-deploy-site.sh
ff5f8c6 build(github-action): add workflow git docker
413f8f6 build(github): migrate github-pages build to github actions
68aadda build(deps): update kafka and confluent version
372e27e -build(maven): update java version for ossh

Docker Images

$ docker pull streamthoughts/kafka-connect-file-pulse:2.4.1

For verifying distirbution zip file

$ gpg --keyserver keyserver.ubuntu.com --recv-keys D12696955654914F
$ gpg --verify streamthoughts-kafka-connect-file-pulse-2.4.1.zip.asc streamthoughts-kafka-connect-file-pulse-2.4.1.zip

Release v2.4.0

Features

67683d5 feat(expression): add built-in SCeL expression function NOT
7fea775 feat(dataformat): add config to specify a prefix used to prepend XML attributes (#176)
4fc2cb9 feat(expression): add expression function TimestampDiff
9d72e47 feat(expression): add expression function ToTimestamp
0644cb9 feat(expressions): add built-in function 'gt' and 'lt' to ScEL
e4375c8 feat(expressions): add built-in function 'or' and 'and' to ScEL
28a6126 feat(expressions): add built-in function 'if' to ScEL
4fe77fd feat(api): add access to error stacktrace in filter chain
b9c0a40 feat(dataformat): add new config prop to exclude node attributes in namespaces (#175)
8f648c8 feat(dataformat): add new config props to exclude all XML attributes (#174)
355b6e4 feat(expression): add UnixTimestamp expression function
5a62f03 feat(filters): add new XmlToStructFilter
9cad2fa feat(filters): add new simple XmlToJsonFilter
0e29ce2 feat(plugin): add capability to merge schemas deriving from records

Improvements & Bugfixes

165a908 refactor(expressions): allow functions to not evaluate all expression args
4e9f84d fix(expressions): fix equals SCeL expression should support null argument (#187)
e9cd483 fix(build): normalize artefact-ids
2b8d260 refactor(filters): rellocate json packages
4d13731 refactor(filters): cleanup classes
7bdc787 fix(filesystems): fix regression on AmazonS3Client configuration (#184)
d76bac0 fix(plugin): refactor InMemoryFileObjectStateBackingStore to use an LRU cache (#183)
7edf4d7 fix(expression): fix unit tests
f3179a7 refactor(expression): refactor expression function api
bf3fc31 refactor(expression): reorganize packages for built-in functions
50200f7 fix(plugin): fix resources must not be closed while files are not committed
17e9efb fix(plugin): fix regression cleanup object files should not be rescheduled (#178)
e2f74b2 fix(api): fix schemas should be merged per target topic
03bab9a fix(api): enchance mapping to connect schema to handle duplicate schema
760d98b fix(filters): XmlToJson should support bytes input

99c374f fix(api): fix schema behavior on array merge

Docs

643469f site(docs): update documentations
be29aae docs(site): add new function descriptions
2a9a119 docs(site): fix missing config property
7533d2f docs(site): improve installation guide
71a9ebe docs(site): add doc for defining schema

Sub-Tasks

7003141 update debug.sh
71f7198 improve release.sh

Docker Images

$ docker pull streamthoughts/kafka-connect-file-pulse:2.4.0

v2.3.0

Features

3da6a5b feat(filesystems): add the capability to configure alternative AWS S3 endpoint (#172)
608d1c2 feat(plugin): add new prop to cleanup on offset commit
f6b443a feat(api): allow to configure a record-value schema
befbc6f feat(filters): add new NullValueFilter (#169)
d86804b feat(plugin): add new config tasks.empty.poll.wait.ms
39c37d9 feat(plugin): add new prop to configure if task should halt on error (#164)
7c219b8 feat(filesystems/api): enhance XMLFileInputReader to support data type inference (#163)

Improvements & Bugfixes

5314e20 fix(filters): DelimtedRowFileInputFilter should compute schema for each record (#171)
816f48a fix(filters): fix AppendFilter to set record-value to null (#167)
d2e776b fix(api): fix connector should accept nullable record-cord (#170)
2266fc1 fix(expression): fix SCEL expression null

Docs

20983dd docs(site): fix documentation typos on metadata access (#165)
4eef25f docs(site): add missing config props
fdd4124 docs(site): add release note for 2.2.0

Sub-Tasks

Docker Images

Released on Docker hub

docker pull streamthoughts/kafka-connect-file-pulse:2.3.0