Skip to content

3.8.0
Compare
Choose a tag to compare
@github-actions github-actions released this 03 May 13:14
· 70 commits to master since this release
3.8.0
6a11d15

This version comes with a new Enrich app, enrich-nsq. Also, it has following improvements:

  • Superseding schemas
  • Improvements in API/SQL enrichments
  • Making derived contexts accesible to the JavaScript enrichment

Superseding schemas

Schemas define the structure of the data that you collect. Each schema defines what fields are recorded with each event that is captured, and provides validation criteria for each field. Schemas are also used to describe the structure of entities that are attached to events.

However, there are some cases where we want to replace schema versions in incoming events with another version due to some problem in the tracking code. The new superseding schemas feature makes this possible.

So, how does this work exactly? If we want a schema to be replaced by another one, we state this with $supersededBy field of the schema. Later, when an event with superseded schema arrived, superseded schema version will be replaced by the specified superseding schema version.

Improvements in API/SQL enrichments

The API enrichment lets you perform dimension widening on a Snowplow event via your own or third-party proprietary http(s) API. The SQL enrichment is the relational database counterpart of the API Enrichment. It allows you to use relational database to perform dimension widening.

Enrich caches the results of API requests and SQL queries with the API/SQL enrichments to avoid continuous calls. We've made some improvements in caching the errors. These improvements are:

  • Set TTL for errors to the tenth of TTL for successful results. With this way, API/SQL requests can be retried faster in case of cached error.
  • When we get an error, the error will be cached but we will return last known 'old' good value for further processing. This fallback would allow Enrich to produce fewer bad rows in case of 'getting stuck' with errors in the enrichment cache.

More details about the caching improvements can be found here.

Also, we've made some changes to the way we handle database connections with SQL enrichment. These changes should lead to acquiring database connections in a better way and better usage of existing database connections.

enrich-nsq, new member of 2nd generation enrich apps

In this release, enrich-nsq becomes the newest member of 2nd generation Enrich apps. It allows to read from and write to NSQ topics.

Instructions to setup and configure Enrich can be found on our docs website.

Making derived contexts accesible to the JavaScript enrichment

Previously, the JavaScript enrichment allowed users to call event.getDerived_contexts(), however, it was returning always null. Starting with Enrich 3.8.0, it will be possible to access derived contexts in the JavaScript enrichment.

Changelog

  • Take superseding schema into account during validation (#751)
  • common: Provide derived contexts to JS enrichment (#769)
  • Scan Docker images in Snyk Github action (#772)
  • common: do not validate enrichment names (#767)
  • common: SQL enrichment: get connection only if request not cached (#765)
  • common: SQL enrichment: put getConnection in Blocker (#763)
  • common-fs2: fix env var substitution for JSON files (#753)
  • Add enrich-nsq (#740)
  • fix: add mskAuth to kafka depedencies (#746)
  • common: improve caching in API/SQL enrichments (#747)
Assets 10
Loading