Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update spec for new staging areas for linking updates and deletes #59

Open
aaclan-ebi opened this issue Mar 17, 2022 · 0 comments
Open
Assignees

Comments

@aaclan-ebi
Copy link

aaclan-ebi commented Mar 17, 2022

Update spec according to what has been agreed on last post DCP2 demo meeting

Related to ebi-ait/dcp-ingest-central#695
Related slack thread https://humancellatlas.slack.com/archives/C01360XN04S/p1646223040031019

Notes:

  • New staging area enum types - normal, delta, updated (or latest tbd)
  • "updated" staging areas will always contain the latest set of metadata, links, descriptor files
  • At some point, we may want to add support for updated data files and in that case, there will be data files present in the staging area
  • Sequence diagram: https://collabuml.com/update-links

A new staging area which we will require Data Import to reset a project and import the staging area like a new project.
Pro(s): The design is simpler and more straightforward and covers all scenarios for metadata updates and deletes.
Con(s): It may be expensive for an extremely large dataset (e.g. Tabula Muris), in which case we could use the delta staging area approach once we really need to.

aaclan-ebi pushed a commit that referenced this issue Mar 17, 2022
aaclan-ebi pushed a commit that referenced this issue Mar 17, 2022
aaclan-ebi pushed a commit that referenced this issue Mar 17, 2022
aaclan-ebi pushed a commit that referenced this issue Mar 18, 2022
aaclan-ebi pushed a commit that referenced this issue Mar 18, 2022
aaclan-ebi pushed a commit that referenced this issue Mar 18, 2022
aaclan-ebi pushed a commit that referenced this issue Mar 18, 2022
aaclan-ebi pushed a commit that referenced this issue Mar 23, 2022
aaclan-ebi pushed a commit that referenced this issue Mar 30, 2022
aaclan-ebi pushed a commit that referenced this issue Mar 31, 2022
aaclan-ebi pushed a commit that referenced this issue Apr 4, 2022
aaclan-ebi pushed a commit that referenced this issue Apr 6, 2022
Co-authored-by: ESapenaVentura <[email protected]>
aaclan-ebi pushed a commit that referenced this issue Apr 7, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant