Empty yaml file types (and incorrect formatting) #1424

earlEBI · 2024-09-11T11:32:12Z

Empty yaml file types
Found 785 yamls with empty file types - GCSTs listed in attached .txt file:
yamls-with-empty-filetypes.txt
(Unfortunately these are not all GWAS-SSF so file headers would need to be checked to determine correct file type.)

Incorrect file_type formatting
Also found several yamls with file_type ' GWAS-SSFv1.0' or ' GWAS-SSF v1.0' (with single quotation marks and beginning whitespace (eg. GCST90319314). These should be removed so it reads eg. file_type: GWAS-SSFv1.0.
(There is some variability about usage of 'GWAS-SSFv1.0' and 'GWAS-SSF v1.0' with added space. Could this also be cleaned up?)

ljwh2 · 2024-09-19T09:45:10Z

@karatugo could this please be worked on along with the other yaml issues, thanks

ljwh2 · 2024-09-25T13:22:57Z

@jiyue1214 will check how many of these have been resolved already and also deal with the quotation marks.

jiyue1214 · 2024-09-27T18:08:56Z

Based on the studies status on 27th September 2024:

Number of Studies	Study Type in YAML File
803	''
912	'GWAS-SSFv1.0'
35,823	GWAS-SSFv1.0
16,042	non-GWAS-SSF
1	'Non-GWAS-SSF'
890	Non-GWAS-SSF
56,677	pre-GWAS-SSF

Noticeably, we have not set any limitation on the value of the file_type field.
Harmonisation queue script detects the field type by if the value of the file_type starts with GWAS-SSF or pre-GWAS-SSF. if none of them, the file_type will be set as "not_harm" automatically.

jiyue1214 · 2024-09-27T19:23:11Z

Hi @earlEBI, I extracted the first two rows from each sumstat reformat them into "header: value" and identified their file type based on it. Here are the results. Could you please help me to check if they are correct?

Hi, @karatugo. Since there are more than 800 studies, could you please suggest any best practices for updating both the DB and meta-yaml files?

Thank you for your support and help,

earlEBI · 2024-10-03T09:24:42Z

@jiyue1214 file types look good to me!
Just checking, did you pick up file types starting with quotation mark and space, eg. GCST90319314 ?
Also, for the note column re:reformatting, I would check A1 is effect allele from readme etc. (you might have already done so!)

karatugo · 2024-10-07T14:56:54Z

@jiyue1214 I can update the DB by writing a new script.

jiyue1214 · 2024-10-07T15:27:02Z

@earlEBI, Thank you so much for checking the format. for the data need to be reformatted, GCST90012644, GCST90012743, GCST90012778, GCST90012648, GCST90012784 seems pre-publication data that I cannot find the publication information. It will be super helpful if you can to let me know if A1 is the effect allele.

By the way, could I ask for non-GWAS-SSF format, there are two ways representing it, could I know which one is the correct? non-GWAS-SSF or Non-GWAS-SSF? I found this website, i am still unclear which should be.

jiyue1214 · 2024-10-07T15:30:33Z

@karatugo Thank you sooo much for your help. Here are two issues:

missing the file_type: I list the correct file_type and GCST in the result file.
file types starting with quotation mark and space, like 'GWAS-SSF v1.0', which the quotation need to be removed. For correcting it, could I ask if you need GCST_ids for them or your database can capture it? The correct one should be GWAS-SSF v1.0, with a space in the middle.
Thanks again!

karatugo · 2024-10-07T15:41:00Z

@jiyue1214 Working on a script now that will fix missing file types according to your list.

karatugo · 2024-10-07T15:50:26Z

@jiyue1214 Re: 2 - I'll update them according to #1424 (comment), i.e.,

{''} -> empty string without quotation marks
{'GWAS-SSFv1.0',' GWAS-SSFv1.0', ' GWAS-SSF v1.0'} -> GWAS-SSF v1.0
{'Non-GWAS-SSF', 'Non GWAS-SSF', Non-GWAS-SSF} -> non-GWAS-SSF

earlEBI · 2024-10-07T16:19:28Z

@earlEBI, Thank you so much for checking the format. for the data need to be reformatted, GCST90012644, GCST90012743, GCST90012778, GCST90012648, GCST90012784 seems pre-publication data that I cannot find the publication information. It will be super helpful if you can to let me know if A1 is the effect allele.

By the way, could I ask for non-GWAS-SSF format, there are two ways representing it, could I know which one is the correct? non-GWAS-SSF or Non-GWAS-SSF? I found this website, i am still unclear which should be.

I agree, it's not consistently used in documentation! I think I would go with 'non-GWAS-SSF' as it matches the case of 'pre-GWAS-SSF'

jiyue1214 · 2024-10-07T16:23:35Z

Thank you @earlEBI for the explanation. @karatugo, let's keep using GWAS-SSF v1.0, pre-GWAS-SSF and non-GWAS-SSF. I added constraint to the metadata schema on gwas-sumstat-tools.

karatugo · 2024-10-07T16:54:16Z

@karatugo

For item 1.

Run update script again because the script failed due to Hinxton sw update
- Fix the cases pushed through validation - they are missing GCST IDs in the meta collection, and need a more complex script
Mark GCSTs as pending

karatugo · 2024-10-07T16:56:23Z

@karatugo

For item 2.

Make file types consistent for all documents in the collection as agreed above

karatugo · 2024-10-09T15:42:23Z

Fixed the cases pushed through validation - they are missing GCST IDs in the meta collection, and need a more complex script. Find the script at cron scripts dir for prod.

karatugo · 2024-10-09T16:09:19Z

Marked GCSTs as pending. They should be generated in 2 days.

karatugo · 2024-10-09T18:18:51Z

All file types are in the set

`` 
`GWAS-SSF v1.0`
`non-GWAS-SSF`
`pre-GWAS-SSF`

karatugo · 2024-10-09T18:19:36Z

@earlEBI Do you want me to generate all yaml files to make sure that the file types are consistent?

earlEBI · 2024-10-10T09:08:15Z

All file types are in the set
`` 
`GWAS-SSF v1.0`
`non-GWAS-SSF`
`pre-GWAS-SSF`

@karatugo Does this mean there are some file_types which are empty? I don't think that should be allowed?

earlEBI · 2024-10-10T09:23:30Z

@earlEBI, Thank you so much for checking the format. for the data need to be reformatted, GCST90012644, GCST90012743, GCST90012778, GCST90012648, GCST90012784 seems pre-publication data that I cannot find the publication information. It will be super helpful if you can to let me know if A1 is the effect allele.

By the way, could I ask for non-GWAS-SSF format, there are two ways representing it, could I know which one is the correct? non-GWAS-SSF or Non-GWAS-SSF? I found this website, i am still unclear which should be.

@jiyue1214 Hi again! These GCSTs are all from PMID 33441150 but were permanently unpublished from the Catalog by user request due to unclear traits. I guess those folders should be deleted from public ftp really. Can probably stay on staging but don't need reformatting now : )

karatugo · 2024-10-10T09:48:39Z

Here's the list of GCST IDs with empty file type.

GCST90310293
GCST90315948
GCST90315948
GCST90319473
GCST90319474
GCST90319475
GCST90319476
GCST90319477
GCST90319478
GCST90319479
GCST90319480
GCST90319481
GCST90319482
GCST90319483
GCST90319484
GCST90319485
GCST90319486

earlEBI · 2024-10-10T11:12:38Z

Here's the list of GCST IDs with empty file type.

GCST90310293
GCST90315948
GCST90315948
GCST90319473
GCST90319474
GCST90319475
GCST90319476
GCST90319477
GCST90319478
GCST90319479
GCST90319480
GCST90319481
GCST90319482
GCST90319483
GCST90319484
GCST90319485
GCST90319486

@karatugo I checked the first 2 have file type 'GWAS-SSF v1.0' in their yaml currently. Could you read the file type from the yaml for all of these?

earlEBI · 2024-10-10T11:17:12Z

@earlEBI Do you want me to generate all yaml files to make sure that the file types are consistent?

Could you just regenerate yamls for those in Yue's list PLUS any with file type currently beginning "' " which I don't think are included (eg. GCST90319314)

karatugo · 2024-10-10T14:05:25Z

Here's the list of GCST IDs with empty file type.
GCST90310293
GCST90315948
GCST90315948
GCST90319473
GCST90319474
GCST90319475
GCST90319476
GCST90319477
GCST90319478
GCST90319479
GCST90319480
GCST90319481
GCST90319482
GCST90319483
GCST90319484
GCST90319485
GCST90319486
@karatugo I checked the first 2 have file type 'GWAS-SSF v1.0' in their yaml currently. Could you read the file type from the yaml for all of these?

Sure, I'll fix them too.

karatugo · 2024-10-10T14:12:36Z

@earlEBI Do you want me to generate all yaml files to make sure that the file types are consistent?

Could you just regenerate yamls for those in Yue's list PLUS any with file type currently beginning "' " which I don't think are included (eg. GCST90319314)

The ones beginning "' " are all fixed now but unfortunately I didn't keep a list of them.

karatugo · 2024-10-10T14:32:11Z

File types in yamls

GCST90310293 - GWAS-SSF v1.0
GCST90315948 - yaml not generated yet
GCST90319473 - GWAS-SSF v1.0
GCST90319474 - GWAS-SSF v1.0
GCST90319475 - GWAS-SSF v1.0
GCST90319476 - GWAS-SSF v1.0
GCST90319477 - GWAS-SSF v1.0
GCST90319478 - GWAS-SSF v1.0
GCST90319479 - GWAS-SSF v1.0
GCST90319480 - GWAS-SSF v1.0
GCST90319481 - GWAS-SSF v1.0
GCST90319482 - GWAS-SSF v1.0
GCST90319483 - GWAS-SSF v1.0
GCST90319484 - GWAS-SSF v1.0
GCST90319485 - GWAS-SSF v1.0
GCST90319486 - GWAS-SSF v1.0

karatugo · 2024-10-10T14:38:49Z

GCST90310293 - GWAS-SSF v1.0
GCST90315948 - yaml not generated yet
GCST90319473 - GWAS-SSF v1.0
GCST90319474 - GWAS-SSF v1.0
GCST90319475 - GWAS-SSF v1.0
GCST90319476 - GWAS-SSF v1.0
GCST90319477 - GWAS-SSF v1.0
GCST90319478 - GWAS-SSF v1.0
GCST90319479 - GWAS-SSF v1.0
GCST90319480 - GWAS-SSF v1.0
GCST90319481 - GWAS-SSF v1.0
GCST90319482 - GWAS-SSF v1.0
GCST90319483 - GWAS-SSF v1.0
GCST90319484 - GWAS-SSF v1.0
GCST90319485 - GWAS-SSF v1.0
GCST90319486 - GWAS-SSF v1.0

All file types are GWAS-SSF v1.0 in Mongo DB as well except for GCST90315948 it's empty. This excludes harmonised files though.

@jiyue1214 For harmonised, what should be their file type?

earlEBI · 2024-10-11T08:29:09Z

@karatugo GCST90315948 should be GWAS-SSF

earlEBI · 2024-10-11T15:30:29Z

@karatugo
I think there are 2,482 GCSTs with yaml file_type: ' GWAS-SSF v1.0'
Listed in this .txt file:
' GWAS-SSF file_types.txt

karatugo · 2024-10-14T10:46:09Z

fixed GCST90315948 file type as GWAS-SSF v1.0.

karatugo · 2024-10-14T11:25:20Z

Yamls will be generated in 2 days for:

Yue's list as in Empty yaml file types (and incorrect formatting) #1424 (comment)
Any with file type currently beginning "' " according to Empty yaml file types (and incorrect formatting) #1424 (comment)
GCST90315948

ljwh2 · 2024-10-23T10:43:43Z

@earlEBI @Santhi1901 please confirm

earlEBI · 2024-10-28T10:42:14Z

@karatugo I think these are all fixed except for 40 harmonised yamls still have the leading quotation mark and space:
yaml_harm_file_type_errors.txt

earlEBI · 2024-10-28T11:01:04Z

@karatugo Sorry, there are also 593 yamls with empty file type still. I think a lof of them are strange cases - harmonised yamls, old formatted yamls, etc...
Might need some more headers analysis run to determine file_types?
empty-yamls-28.10.24.txt

karatugo · 2024-10-30T14:07:22Z

@karatugo I think these are all fixed except for 40 harmonised yamls still have the leading quotation mark and space: yaml_harm_file_type_errors.txt

They should be available in the public ftp in 2 days.

jiyue1214 · 2024-10-30T15:12:04Z

HI, @earlEBI. I found most of the files in the list are f.tsv.gz files in the harmonised folder, and their raw file name did not follow the pattern GCST*.tsv. For this situation, we decided to add a yaml file for h.tsv.gz file and keep the rest of them in the way they currently are.

Thanks for reminding me and I will gather them in the ticket Old harmonised data missing yaml files

jiyue1214 · 2024-10-30T16:07:38Z

@karatugo I created a new ticket for new studies @earlEBI reported. Feel free to close the ticket when those 40 studies are done.

karatugo · 2024-10-31T17:01:23Z

They are not done yet, will check again on Monday.

karatugo · 2024-11-05T09:45:40Z

@karatugo I think these are all fixed except for 40 harmonised yamls still have the leading quotation mark and space: yaml_harm_file_type_errors.txt

Checked a few of them and they are present in the public ftp.

earlEBI · 2024-11-05T09:51:31Z

@karatugo Looks good!

karatugo self-assigned this Sep 17, 2024

jiyue1214 self-assigned this Sep 25, 2024

jiyue1214 mentioned this issue Oct 30, 2024

Studies missing file_type in yaml file #1479

Open

Empty yaml file types (and incorrect formatting) #1424

Empty yaml file types (and incorrect formatting) #1424

Comments

earlEBI commented Sep 11, 2024 • edited Loading

ljwh2 commented Sep 19, 2024

ljwh2 commented Sep 25, 2024

jiyue1214 commented Sep 27, 2024

jiyue1214 commented Sep 27, 2024

earlEBI commented Oct 3, 2024

karatugo commented Oct 7, 2024

jiyue1214 commented Oct 7, 2024 • edited Loading

jiyue1214 commented Oct 7, 2024 • edited Loading

karatugo commented Oct 7, 2024

karatugo commented Oct 7, 2024 • edited Loading

earlEBI commented Oct 7, 2024

jiyue1214 commented Oct 7, 2024 • edited Loading

karatugo commented Oct 7, 2024 • edited Loading

karatugo commented Oct 7, 2024 • edited Loading

karatugo commented Oct 9, 2024

karatugo commented Oct 9, 2024

karatugo commented Oct 9, 2024 • edited Loading

karatugo commented Oct 9, 2024

earlEBI commented Oct 10, 2024

earlEBI commented Oct 10, 2024

karatugo commented Oct 10, 2024

earlEBI commented Oct 10, 2024

earlEBI commented Oct 10, 2024

karatugo commented Oct 10, 2024

karatugo commented Oct 10, 2024

karatugo commented Oct 10, 2024 • edited Loading

karatugo commented Oct 10, 2024 • edited Loading

earlEBI commented Oct 11, 2024

earlEBI commented Oct 11, 2024

karatugo commented Oct 14, 2024

karatugo commented Oct 14, 2024

ljwh2 commented Oct 23, 2024

earlEBI commented Oct 28, 2024

earlEBI commented Oct 28, 2024 • edited Loading

karatugo commented Oct 30, 2024

jiyue1214 commented Oct 30, 2024 • edited Loading

jiyue1214 commented Oct 30, 2024

karatugo commented Oct 31, 2024

karatugo commented Nov 5, 2024

earlEBI commented Nov 5, 2024

earlEBI commented Sep 11, 2024 •

edited

Loading

jiyue1214 commented Oct 7, 2024 •

edited

Loading

jiyue1214 commented Oct 7, 2024 •

edited

Loading

karatugo commented Oct 7, 2024 •

edited

Loading

jiyue1214 commented Oct 7, 2024 •

edited

Loading

karatugo commented Oct 7, 2024 •

edited

Loading

karatugo commented Oct 7, 2024 •

edited

Loading

karatugo commented Oct 9, 2024 •

edited

Loading

karatugo commented Oct 10, 2024 •

edited

Loading

karatugo commented Oct 10, 2024 •

edited

Loading

earlEBI commented Oct 28, 2024 •

edited

Loading

jiyue1214 commented Oct 30, 2024 •

edited

Loading