Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Creation of id-groupid.csv required for validation #253

Closed
nichtich opened this issue Apr 14, 2023 · 3 comments
Closed

Creation of id-groupid.csv required for validation #253

nichtich opened this issue Apr 14, 2023 · 3 comments
Assignees
Labels
Milestone

Comments

@nichtich
Copy link
Collaborator

both validate and sqlite require id-groupid.csv (if grouping is enabled) but this file is only created with completeness. I guess completeness should go first in do_all_analyses in common_script.

@pkiraly
Copy link
Owner

pkiraly commented Apr 14, 2023

Right now I have a serious problem with the postprocessing of CSV files after validation. It works well for some million records, but when I run it on the full data dump, the process runs for long time. I am working on this, and I will fix this issue when the performance problem will be solved. I beg your patience.

@nichtich nichtich added the bug label Apr 28, 2023
@nichtich
Copy link
Collaborator Author

To replicate, run with an empty output directory, otherwise id-groupid.csv from previous analyses will be found.

@pkiraly
Copy link
Owner

pkiraly commented Apr 28, 2023

Yes, it is clear. Completeness creates the is-groupid.csv file. The validation should be modified to create it if it is not yet created.

@pkiraly pkiraly added this to the PICA: 1.2 milestone Apr 28, 2023
@pkiraly pkiraly added this to PICA Apr 28, 2023
@pkiraly pkiraly self-assigned this Apr 28, 2023
@pkiraly pkiraly changed the title id-groupid.csv required before creation Creation of id-groupid.csv required for validation May 4, 2023
@pkiraly pkiraly closed this as completed May 5, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
Status: No status
Development

No branches or pull requests

2 participants