Releases: snowplow/snowplow-rdb-loader
Releases · snowplow/snowplow-rdb-loader
0.18.2
R35
Pubilc beta with deprecation of EmrEtlRunner and major changes in architecture.
Architecture/API changes
- RDB Loader: remove EmrEtlRunner support (#251)
- RDB Loader: add discovery through SQS stream (#234)
- RDB Loader: switch to SQS-only loading (#262)
- RDB Loader: swtich to a HOCON config (#250)
- RDB Shredder: swtich to a HOCON config (#256)
- RDB Shredder: add min and max timestamps of events for the batch to the SQS message with shredded types (#275)
- RDB Shredder: add archive discovery (#263)
- RDB Shredder: send shredding info to SQS when it's done (#200)
- Common: get rid of atomic-events folder (#183)
New features
- RDB Loader: integrate Sentry (#235)
Under the hood
- RDB Loader: build Docker image (#247)
- Common: extend copyright notice to 2021 (#287)
- Bump decline from 0.6.2 to 1.3.0 (#223)
- Bump aws-java-sdk from 1.11.319 to 1.11.916 (#244)
- Bump scalacheck from 1.14.0 to 1.14.3 (#243)
- Bump circe-yaml from 0.9.0 to 0.13.1 (#226)
- Bump snowplow-events-manifest from 0.2.0 to 0.3.0 (#224)
- Bump fs2-core from 2.4.4 to 2.4.6 (#222)
- Bump redshift-jdbc42-no-awssdk from 1.2.36.1060 to 1.2.51.1078 (#208)
- Bump kind-projector from 0.9.6 to 0.9.10 (#217)
R34
R33
Bugfix and performance improvements.
Bugfixes
- Common: fix ambigrous empty string (#171)
- Common: create a separate modules root (#195)
- RDB Shredder: optimize DAG (#181)
- RDB Shredder: use single etl_tstamp for cross-batch deduplication (#180)
- Loader: mitigate exhausted input error (#203)
- Loader: fix ignored schema in metadata queries (#178)
Under the hood
- Common: integrate coveralls (#220)
- Common: integrate Snyk (#188)
- Common: integrate Scala Steward (#206)
- Common: switch to Github Actions (#189)
- Common: disable test in assembly (#202)
- Loader: switch from Free monad to cats-effect IO (#184)
Removed features
- Common: remove Processing Manifest (#186)
- RDB Loader: remove Postgres support (#191)
- Loader: drop legacy S3 paths (#198)
Dependency updates
- Common: bump snowplow-scala-analytics-sdk to 2.1.0 (#201)
- Common: bump snowplow-badrows to 2.1.0 (#199)
- Common: bump specs2-core to 4.10.5 (#197)
- Common: bump schema-ddl to 0.12.0 (#192)
- Common: bump sbt to 1.4.4 (#196)
- Common: bump Scala to 2.12 (#127)
- RDB Shredder: bump Spark to 3.0.1 (#101)
- RDB Loader: bump jsch to 0.1.55 (#177)
Release 32
Redshift automigrations.
New features
- Common: allow tabular blacklisting (#156)
- RDB Shredder: add tabular data output (#151)
- RDB Shredder: use snowplow-badrows (#169)
- RDB Loader: add tabular output loading (#152)
Improvements and bugfixes
- RDB Shredder: fix false positive cross-batch deduplication for RT pipelines (#173)
- Common: force storage target ids to be valid UUID (#168)
Under the hood
Release 31
New bad rows format and orphan events bug fix.
Improvements and bugfixes
- RDB Shredder: persist synthetic duplicates on disk (#142)
- RDB Shredder: truncate string values exceeding column's limits (#143)
- RDB Loader: fix non-exhaustive match in data discovery (#117)
- RDB Loader: retry on refused connection (#8)
Under the hood
- Common: add Snowplow Maven resolver (#147)
- Common: extend copyright notice to 2019 (#144)
- Common: bump sbt-assembly to 1.4.9 (#135)
- Common: bump SBT to 1.2.8 (#134)
- Common: bump Travis oraclejdk version to 11 (#146)
- Common: bump Scala to 2.11.12 (#136)
- Common: switch to OpenJDK (#159)
- Common: bump release-manager to 0.4.1 (#132)
- RDB Shredder: bump to 0.15.0 (#139)
- RDB Shredder: factor out scala-common-enrich (#138)
- RDB Shredder: replace real DynamoDB with local version in CrossBatchDeduplicationSpec (#149)
- RDB Shredder: bump Spark to 2.3.2 (#162)
- RDB Loader: bump to 0.16.0 (#140)
- RDB Loader: bump iglu-scala-client to 0.6.0 (#141)
- RDB Loader: bump snowplow-scala-tracker to 0.6.1 (#148)
Release 30
Multiple usability and stability improvements.
Common
- Add VERSION file (#82)
- Bump SBT to 1.1.6 (#89)
- Use processing manifest (#81)
- Fix README links (#109)
- Extend copyright notice to 2018 (#91)
RDB Shredder
- Bump to 0.14.0 (#96)
- Use RDB Loader AWS SDK (#94)
- Remove auto-creation of event manifests table (#62)
RDB Loader
- Bump to 0.15.0 (#100)
- Improve log output (#23)
- Escape input for sanitize function (#87)
- Remove unused configuration properties (#95)
- Add ability to skip all load manifest interactions (#97)
- Make SSL configuration compatible with native JDBC settings (#73)
- Improve "no data discovered" error message (#69)
- Tolerate deleted folder artifacts when consistency_check is skipped (#80)
- Fix manifest population assumes load is most recent (#70)
- Fail load if entry exists in manifest (#14)
- Bump AWS SDK to 1.11.319 (#79)
- Bump snowplow-scala-tracker to 0.5.0 (#57)
- Bump specs2 to 4.0.4 (#90)
- Bump cats to 1.1.0 (#77)
- Fix typo in load_succeeded schema (#108)
Release 29
Release 28
First release with umbrella-version, adding SSH-tunnels and some minor improvements.
New features
- RDB Loader: add support for SSH tunnels (#22)
- RDB Loader: allow JDBC credentials to be stored in EC2 parameter store (#19)
Testing and deployment
- Common: add CI/CD (#55)
- Common: remove AWS Java SDK shading (#54)
- RDB Shredder: add Snowplow and Clojars resolvers (#56)
UX-improvements
- RDB Loader: make logkey optional (#35)
- RDB Loader: make loading shredded data always required (#29)
- RDB Shredder: overwrite output datasets (#41)
Under the hood
- RDB Shredder: bump Spark to 2.2.0 (#52)
- RDB Shredder: bump to 0.13.0 (#49)
- RDB Shredder: bump scala-common-enrich to 0.27.0 (#39)
- RDB Loader: bump sbt-assembly to 0.14.5 (#51)
- RDB Loader: bump SBT to 0.13.16 (#50)
- RDB Loader: bump AWS SDK to 1.11.208 (#48)
- RDB Loader: bump redshift-jdbc to 1.2.8.1005 (#40)