Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Enhance integration tests to ensure accurate data transfer #81

Closed
lorerod opened this issue Apr 10, 2024 · 2 comments · Fixed by #119
Closed

Enhance integration tests to ensure accurate data transfer #81

lorerod opened this issue Apr 10, 2024 · 2 comments · Fixed by #119
Assignees
@lorerod
Labels
Priority: 1 - High Blocks the next release.
Milestone
CHT Sync Production

Comments

@lorerod
Copy link
Contributor

lorerod commented Apr 10, 2024 •
edited
Loading

During integration testing, verify data accuracy from CouchDB to Postgres to ensure the model reflects the source data. Additionally, rename existing End-to-End tests to Integration tests to reflect their purpose.

What else to test?

Initial Data Sync: Verify that data from CouchDB is correctly transferred to PostgreSQL during initial sync (Data in PostgreSQL should exactly match the data in CouchDB).
Continuous Data Sync: Ensure continuous syncing works correctly when data is updated or added in CouchDB. Incremental changes are synced properly without reprocessing the entire dataset.
Conflict Resolution: Ensure that conflicts (e.g., same document updated differently in multiple places) are handled properly during the sync process. (Not sure yet how this could happen)
Large Data Volume: Test the system’s ability to handle large volumes of data.
Error Handling: Validate the application's ability to handle and recover from errors during the sync process.

Pending tests:
Deletion Sync: Confirm that document deletions in CouchDB are correctly propagated to PostgreSQL.
@lorerod lorerod self-assigned this Apr 10, 2024
@lorerod lorerod mentioned this issue Apr 19, 2024
Merged
@lorerod lorerod moved this from Todo to This Week's commitments in Product Team Activities May 20, 2024
@andrablaj andrablaj moved this from This Week's commitments to Todo in Product Team Activities May 28, 2024
@andrablaj andrablaj mentioned this issue May 30, 2024
Closed
@andrablaj andrablaj added this to the CHT Sync Production milestone May 30, 2024
@andrablaj andrablaj added the Priority: 1 - High Blocks the next release. label May 30, 2024
@lorerod lorerod moved this from Todo to This Week's commitments in Product Team Activities Jun 17, 2024
@lorerod lorerod moved this from This Week's commitments to In Progress in Product Team Activities Jun 18, 2024
@lorerod
Copy link
Contributor Author

lorerod commented Jul 4, 2024

Okay, this issue was partially addressed in feat(#112): drop support for multiple copies of every document and 112 handle cascade deletes:

  • Initial Data Sync
  • Continuous Data Sync
    • Updates
    • Deletes
    • Inserts
  • Conflict Resolution
  • Large Data Volume
  • Error Handling
  • Deletion Sync

@lorerod lorerod mentioned this issue Jul 5, 2024
Merged
4 tasks
@andrablaj andrablaj moved this from In Progress to Next Week's Commitments in Product Team Activities Jul 9, 2024
@andrablaj andrablaj moved this from Next Week's Commitments to Todo in Product Team Activities Jul 9, 2024
@lorerod lorerod moved this from Todo to This Week's commitments in Product Team Activities Jul 23, 2024
@lorerod lorerod moved this from This Week's commitments to In Progress in Product Team Activities Jul 24, 2024
lorerod added a commit that referenced this issue Aug 7, 2024
@lorerod @dianabarsan
chore(#81): enhance e2e tests (#119)
f425010 
* Add structure validation

* Add structure validation for reports

* Add incremental insert, conflict resolution and downtime handle scenarios

* avoid logs

* fix lint and docker command

* Add reconnection to postgres in case downtime test fails

* Log error

* Improve readability and extensibility of the code

Co-authored-by: Diana Barsan <[email protected]>

* Update function name to reflect multiple document insertion

Co-authored-by: Diana Barsan <[email protected]>

* Fix typo

* Use existing  function to avoid duplicate filtering

* Use execDockerCommand for executing Docker commands

* Pull branch changes

* Fix eslint

* Fix push documents

* Delete comment lines

* Handle conflicts properly in edit and delete operations

* Change docker compose file handling to avoid redundant -f prefixes

* Fix conflict creation in edit and delete operations using bulkDocs
and getOldRevision function

* Separate functions to create edit and delete conflict

* Remove getOldRevision function

---------

Co-authored-by: Maria Lorena Rodriguez Viruel <[email protected]>
Co-authored-by: Maria Lorena Rodriguez Viruel <[email protected]>
Co-authored-by: Diana Barsan <[email protected]>
@github-project-automation github-project-automation bot moved this from In Progress to Done in Product Team Activities Aug 7, 2024
@lorerod
Copy link
Contributor Author

lorerod commented Aug 7, 2024

Added the remaining scenarios in chore(#81): enhance e2e tests

  • Initial Data Sync
  • Continuous Data Sync
    • Updates
    • Deletes
    • Inserts
  • Conflict Resolution
  • Large Data Volume
  • Error Handling (downtime handling actually)
  • Deletion Sync

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@lorerod lorerod

Labels
Priority: 1 - High Blocks the next release.
Projects
Product Team Activities
Status: Done
Milestone
CHT Sync Production
Development

Successfully merging a pull request may close this issue.

chore(#81): enhance e2e tests medic/cht-sync
2 participants
@lorerod @andrablaj