Skip to content

Releases: CDCgov/data-exchange-upload

v2.5.0 Direct Upload Delivery

30 Sep 16:11
c10aa3c
Compare
Choose a tag to compare

DEX Upload Release Notes - Version 2.5.0

Release Date: 2024-09-26

Version Number: Version 2.5.0

Release Changelog

API Swagger Documentation

DEX Upload API Github Releases

DEX Upload API Github Tags

Overview

The purpose of Upload API release version 2.5.0 is to facilitate direct delivery of file uploads to program storage destinations. Instead of files being delivered indirectly though a separate routing microservice, the Upload API service will now deliver files directly to CDC program storage. To enable this, we've decoupled source and destination information from the delivery construct. This allows you to register sources and destinations separately, which increases flexibilty in configuring where the file is coming from and going to. In addition, we've increased the flexibilty of where files get dropped in destination storage. Before, files could be configured to be dropped in subfolders by datastream, route, and date. Now, the subfolder path can take on any pattern. This is accomplished through a new path_template field that can be specified in a particular upload config.

Enhancements

  • Direct file delivery to CDC program storage
  • Agnostic subfolder delivery path template

Reporting Team

v2.4.1 Delivery Status and Routing Stability

23 Sep 13:10
Compare
Choose a tag to compare

DEX Upload Release Notes - Version 2.4.1

Release Date: 2024-09-20

Version Number: Version 2.4.1

Release Changelog

API Swagger Documentation

DEX Upload API Github Releases

DEX Upload API Github Tags

Overview

The purpose of Upload API release version 2.4.1 is to expand the /info endpoint to include file delivery status for a given upload ID. End users will be able to use this endpoint to verify whether or not their file got delivered to the intended destination. This release also includes improved stability to the in-memory eventing system, as well as configuration changes for the NDLP use case.

Enhancements

  • Delivery status for uploads provided by the /info endpoint
  • In-memory event system stability improvements

Reporting Team

DEX Upload Release Notes - Version 2.3.0

20 Aug 20:26
5ce6a95
Compare
Choose a tag to compare

Enhancements

  • Improved reporting information
  • Built-in file routing and post processing. Deprecation of function app and event grid
  • Manual routing retry endpoint
  • Improved redis connection pooling resiliency and recovery

Manifest Config Updates and Minor App Observability

30 Jul 20:11
Compare
Choose a tag to compare
  • Update for ABCs
  • Update for ED3N
  • Internal /version endpoint showing git commit short sha and latest release tag

v2.2.0 Redis File Locker

01 Jul 12:50
Compare
Choose a tag to compare

DEX Upload Release Notes - Version 2.2.0

Release Date: 2024-06-24

Version Number: Version 2.2.0

Release Changelog

API Swagger Documentation

DEX Upload API Github Releases

DEX Upload API Github Tags

Overview

The purpose of Enterprise Data Exchange (DEX) Upload release version 2.2.0 is to release an improvement to how the Upload API prevents file corruption when the service horizontally scales. This improvement is implemented in this release via a distributed file locker that uses a Redis cache to keep track of which Tus instance is currently working on a file, and makes sure another Tus instance doesn't work on the same file. Before this improvement, we were relying on Azure's internal blob storage implementation to prevent clusters of tus servers from corrupting files. This implementation is obscured from us and completely out of our control. This custom distributed lock with Redis gives us full control, and has even been adopted by the Tusd maintainers.

Enhancements

  • Improved reliability of horizontal scaling via a custom file locking mechanism that uses an external Redis cache.

Reporting Team

v2.1.1

11 Jun 17:31
ef6349c
Compare
Choose a tag to compare
use websocket for upload processor service bus

v1.6.3

09 May 15:17
c36f4d5
Compare
Choose a tag to compare

Overview

The purpose of Enterprise Data Exchange (DEX) Upload release version 1.6.3 is to apply a hotfix to not include appended tguid values to filenames copied to EDAV storage accounts.

New Features

  • Remove appended tguid values from filenames routed to EDAV storage accounts.

v1.6.2

09 May 15:16
cce648a
Compare
Choose a tag to compare

Overview

The purpose of Enterprise Data Exchange (DEX) Upload release version 1.6.2 is to update upload configurations for data streams, replace filename suffix clock ticks with upload id (tguid), hotfix for Routine Immunization v1 sender manifest config file, and adjust upload filename restrictions.

New Features

  • Update upload filename character restrictions to only disallow forward slashes (/)
  • Update Routine Immunization v1 sender manifest configuration file to remove incorrect metadata fields
  • Add v2 sender manifest configuration files for NRSS and EHDI data streams
  • Replace the filename suffix clock tick value with file upload id (tus guid) value

v1.5.0

09 May 15:15
Compare
Choose a tag to compare

Overview

The purpose of Enterprise Data Exchange (DEX) Upload release version 1.5.0 is to update upload configurations for data streams, add version 1 sender manifest metadata values to version 2 metadata fields, add the version 2 metadata configuration folder structure, add file copy functionality to routing storage, add Processing Status API integration, add Retry functionality, and add Replay functionality.

New Features

  • Version 2 sender manifest configuration folder structure
  • Version 1 sender manifest metadata values to version 2 metadata fields
  • File copy functionality to routing storage accounts
  • Processing Status API integration
  • Retry functionality
  • Replay functionality
  • Upload configurations for the DAART data stream
  • Remove file copy EDAV target from the CELR data stream

v1.4.1

09 May 15:14
f8fb70e
Compare
Choose a tag to compare

Overview

The purpose of Enterprise Data Exchange (DEX) Upload release version 1.4.1 is to enhance telemetry through integration with the Processing Status API, enhance resiliency through retry/replay capabilities for our post-processing component, and publish configurations for onboarding the AIMS CELR use case.

Infrastructure Changes

  • Renaming of Upload Processor function app
  • Infrastructure to support retry/replay functionality
  • Application settings to support connection to Processing Status API
  • Deployment of blue/green slots to the Upload Processor function app