v0.41.0

v0.41.0 (2024-10-02)

Highlights

• It's easier now for SQL tap developers to customize the mapping from SQL column types to JSON schema. See the guide for details.

✨ New

• #2667 Support stream aliasing of BATCH messages via stream maps -- Thanks @ReubenFrankel!
• #2651 SQL taps now emit schemas with maxLength when applicable
• #2618 Developers can now more easily override the mapping from SQL column type to JSON schema

🐛 Fixes

• #2697 All HTTP timeout exceptions are now retried in REST and GraphQL streams
• #2683 A clear error message is now emitted when flattening is enabled but flattening_max_depth is not set
• #2665 Mapped datetime values are now typed as date-time strings in the schema message -- Thanks @gregkoutsimp!
• #2663 Properties dropped using None or __NULL__ in stream maps are now also removed from the schema required array

⚙️ Under the Hood

• #2696 Use tox without installing Poetry explicitly in workflows
• #2654 Added a generic FileStream (still in active development!)
• #2695 Update dependencies in templates
• #2661 Drop support for Python 3.8 in templates
• #2670 Deprecated Faker class in stream maps
• #2666 Remove non-functional record-flattening capability -- Thanks @ReubenFrankel!
• #2652 Renamed SQLConnector.type_mapping to SQLConnector.sql_to_jsonschema
• #2647 Use future warnings.deprecated decorator

📚 Documentation Improvements

• #2658 Added more versions when stuff changed or was added

📦 Packaging changes

• #2694 Removed unused backport importlib_resources dependency in tap template
• #2664 Added a constraint on setuptools <= 70.3.0 to fix incompatibility with some dependencies

v0.40.0

v0.40.0 (2024-09-02)

Highlights

• Targets now emit basic metrics. There's nothing to do on your end, they're enabled by default!
• The Faker class is available in stream maps context. Note that like the faker instance added in v0.35.0, this requires installing the faker library in the tap/target environment. For example, with tap-example[faker].

✨ New

• #2486 Emit target metrics
• #2567 A new schema_is_valid built-in tap test validates stream schemas against the JSON Schema specification
• #2598 Stream map expressions now have access to the Faker class, rather than just a faker instance
• #2549 Added a default user agent for REST and GraphQL taps

🐛 Fixes

• #2613 Mismatch between timezone-aware and naive datetimes in start date and bookmarks is now correctly handled

⚙️ Under the Hood

• #2628 Use context manager to read gzip batch files
• #2619 Default to UTC when parsing dates without a known timezone
• #2603 Backwards-compatible identifier quoting in fully qualified names
• #2601 Improved SQL identifier (de)normalization
• #2599 Remove pytest-durations dependency from testing and use native pytest option --durations
• #2597 Mark pagination classes with @override decorator
• #2596 Made auth_headers and auth_params of APIAuthenticatorBase simple instance attributes instead of decorated properties

📚 Documentation Improvements

• #2639 Documented versions where fake and Faker objects were added to the stream maps context
• #2629 Reference get_starting_timestamp in incremental replication guide
• #2604 Update project sample links
• #2595 Documented examples of stream glob expressions and property aliasing

📦 Packaging changes

• #2640 Remove upper constraint on faker extra

v0.39.1

v0.39.1 (2024-08-07)

• PyPI 🐍

🐛 Fixes

• #2589 Make sink assertion compatible with stream maps -- Thanks @JCotton1123!
• #2592 Fixed typos in --about plain text output
• #2580 Date fields are now properly serialized as ISO dates, i.e. "YYYY-MM-DD"
• #2583 Quote add-column-ddl with column starting with _ (underscore) based on engine -- Thanks @haleemur!
• #2582 DDL for adding a column now uses a SQLAlchemy-compiled clause to apply proper quoting -- Thanks @haleemur!
• #2418 Check replication method instead of key to determine if a SQL stream is sorted

⚙️ Under the Hood

• #2520 Remove unused dependencies pendulum and python-dateutil

📚 Documentation Improvements

• #2579 Documented support for packaging semantic type in contributing guide

v0.39.0

v0.39.0 (2024-07-27)

• PyPI 🐍

Highlights

• The json module can now be used within stream maps

  stream_maps:
    animals:
      id: id
      data: "json.dumps({'description': description, 'verified': verified, 'views': views, 'created_at': created_at})"
      __else__: null

• The crypto dependencies PyJWT and cryptography are now optional and can be installed by requesting the jwt extra, i.e. singer-sdk[jwt]. These are only currently required for JWT authentication, but if your package depends on any of these, make sure the dependency is explicit.

• The pendulum and python-dateutil dependencies are no longer used by this library and will be removed in a future release. If your package relies on any of them, make sure the dependency is explicit.

• We're working on supporting msgspec for more performant IO, so keep an eye out for that 😉.

✨ New

• #2432 Developers can now customize the default logging configuration for their taps/targets by adding default_logging.yml to their package
• #2531 The json module is now avaiable to stream maps -- Thanks @grigi!
• #2529 Stream sync context is now available to all instances methods as a Stream.context attribute

🐛 Fixes

• #2554 Use mapped stream aliases when handling ACTIVATE_VERSION messages in the base target class
• #2526 Moved up the supported Python versions in the Markdown output of --about

⚙️ Under the Hood

• #2564 Make SQLSink a generic with a SQLConnector type parameter
• #2540 Implement abstract serialize_message for Singer writers
• #2259 Centralize JSON SerDe into helper functions -- Thanks @BuzzCutNorman!
• #2525 Make PyJWT and cryptography dependencies optional
• #2528 Moved class-level attributes to the top in REST tap template
• #2132 Limit internal usage of pendulum

📚 Documentation Improvements

• #2557 Document that get_starting_timestamp requires setting a non-null replication_key
• #2556 Reference state partitioning in stream partitioning page
• #2536 Prepare for RTD addons migration
• #2535 Added more intersphinx links to Python and Faker docs
• #2530 Explained how the request URL is generated from url_base, path and the sync context
• #2527 Updated the footer
• #2506 Fixed a typo in the stream maps docs

v0.38.0

v0.38.0 (2024-06-17)

✨ New

• #2433 Tap developers can now disable HTTP redirects
• #2426 Added an optional GitHub workflow to publish to PyPI with trusted publishers

🐛 Fixes

• #2438 Null replication values are now handled when incrementing bookmarks
• #2431 Updated cookiecutter VSCode launch.json to use debugpy
• #2421 An error message is now logged every time schema validation fails for any record

⚙️ Under the Hood

• #2455 Use parent datetime.datetime class in type conforming checks
• #2453 Change to return type of utc_now from pendulum.DateTime to datetime.datetime

📚 Documentation Improvements

• #2449 Add a short guide on defining a configuration schema
• #2436 Documented how context fields are passed to a child stream
• #2435 Using an empty list for __key_properties__ to disable a stream primary keys is now recommended as an alternative to null

v0.37.0

v0.37.0 (2024-04-29)

Highlights

• You can reference your streams with glob patters in stream map configuration:

  stream_maps:
    # This will affect all streams with names starting with 'my_prefix_'
    "my_prefix_*":
      id: id
      first_name: first_name
      email:     # drop the PII field from RECORD and SCHEMA messages
      email_domain: email.split('@')[-1]
      email_hash: md5(config['hash_seed'] + email)
      __else__: null

✨ New

• #2389 JSON schema keyword allOf is now supported
• #1888 Added support for glob patterns in source stream names -- Thanks @DouweM!
• #2345 PropertiesList can now behave as an iterable -- Thanks @ReubenFrankel!

🐛 Fixes

• #2352 Removed unnecessary and problematic column caching -- Thanks @raulbonet!
• #2375 Added sensitive: true to password settings in templates
• #2301 Unmapped sub-fields in object-type fields are now no longer dropped when the field declares additionalProperties
• #2348 Added a condition to the No schema for record field warning -- Thanks @tobiascadee!
• #2342 Avoid failing if VSCode IDE config is not requested for target and mapper cookiecutter templates -- Thanks @ReubenFrankel!
• #2331 Allow importlib-resources >=6.3.2

⚙️ Under the Hood

• #2205 Added a jwt package extra, but the cryptography and jwt dependencies are still installed by default for now

📚 Documentation Improvements

• #2326 Documented BATCH as a default plugin capability -- Thanks @ReubenFrankel!

v0.36.1

v0.36.1 (2024-03-13)

🐛 Fixes

• #2310 Limited importlib-resources to < 6.2 due to breaking changes
• #2288 Bumped min joblib dependency to >=1.3.0 and replaced deprecated parallel_backend with parallel_config -- Thanks @BuzzCutNorman!
• #2281 The state_partition_context dictionary is now correctly interpolated in the error message when duplicate partitions/contexts are detected in the input state
• #2274 Test workflow now fails for any Python matrix job failure in cookiecutter template -- Thanks @ReubenFrankel!

v0.36.0

v0.36.0 (2024-02-26)

✨ New

• #2241 JSON schema keywords such as maxLength are now supported in StringType, IntegerType and NumberType JSON schema helpers
• #2263 Nested settings are now documented in the output of --about --format=markdown
• #2248 Targets now accept a batch_size_rows setting to configure how many rows are loaded in each record batch -- Thanks @BuzzCutNorman!

🐛 Fixes

• #2258 Database disconnects are now handled via SQLAlchemy pool_pre_ping parameter

⚙️ Under the Hood

• #2220 Deprecated singer_sdk.authenticators.BasicAuthenticator in favor of requests.auth.HTTPBasicAuth

v0.35.2

v0.35.2 (2024-02-19)

🐛 Fixes

• #2252 Null type is now correctly appended as {"type": "null"} to oneOf types

v0.35.1

v0.35.1 (2024-02-15)

🐛 Fixes

• #2243 Flattening level of record is now forced to match the provided flattened schema -- Thanks @joaopamaral!
• #2245 Instances of oneOf are now handled by null-appending logic
• #2242 Hard and soft deletes for handling ACTIVATE_VERSION messages now use the same WHERE clause -- Thanks @vitoravancini!
• #2232 Test workflow job now fails for unsupported Python versions in cookiecutter templates -- Thanks @ReubenFrankel!
• #2225 SQL columns that are non-nullable but not required (i.e. not part of a primary key) are now not included in the "required": [...] array of the discovered JSON schema

📚 Documentation Improvements

• #2239 Linked reference docs to source code
• #2231 Added an example implemetation of JSON schema validation that uses fastjsonschema
• #2219 Added reference docs for tap & target testing helpers