Expand doc-site section "Census data releases" and add data erratum #600

pablo-gar · 2023-07-04T21:49:06Z

In "Census data releases" section:

Adds examples to open different versions of the data releases.
Adds section to describe each LTS data release.
Add data erratum for LTS 2023-05-15. Related to Datasets with incorrect is_primary_data=True single-cell-curation#528

It also adds a file with the soma_joinids of the incorrectly labelled cells.

Notes for reviewers

@bkmartinjr can you confirm that the file you sent over email is for the data release 2023-05-15?
@brianraymor and @jahilton can you make an editorial review of the erratum?

bkmartinjr · 2023-07-04T21:54:29Z

can you confirm that the file you sent over email is for the data release 2023-05-15

confirmed. It was calculated with open_soma(census_version='latest') which resolved to 2023-05-15

bkmartinjr · 2023-07-04T21:56:09Z

docs/cellxgene_census_docsite_data_release_info.md


 To enable data stability and scientific reproducibility, [CZ CELLxGENE Discover](https://cellxgene.cziscience.com/) plans to perform regular LTS Census data releases:

 * Published online every six months for public access, starting on May 15, 2023.
 * Available for public access for at least 5 years upon publication.

+The latest LTS Census data release is the default opened by the APIs and recognized as the `census_version = "stable"`.


You may want to caveat/footnote that "stable" will change every time a new LTS is released, and the perma-name for it is 2023-05-15

bkmartinjr · 2023-07-04T21:57:53Z

docs/cellxgene_census_docsite_data_release_info.md

+### LTS 2023-05-15
+
+Open this data release by specifying `census_version = "2023-06-28"` in future calls to `open_soma()`.


date mismatch - it does not specify the LTS: s/2023-06-28/2023-05-15/

bkmartinjr · 2023-07-04T21:59:32Z

docs/cellxgene_census_docsite_data_release_info.md

+#### 🔴 Erratum 🔴  
+
+There are 243,569 number of cells labelled as `is_primary_data = True` for which a fraction of them the label is incorrect. 


hard to understand wording...

Perhaps: There are 243,569 cells labelled as is_primary_data = True, which have a duplicate cell labelled the same.

If we're being specific about the 243,569 observations labeled, then I would be specific about how many of those are incorrect, instead of saying "a fraction".
And just to be clear, we still have 2 cases that are being reviewed so these numbers may change. And as such, I didn't review the specific counts, etc.

docs/cellxgene_census_docsite_data_release_info.md

bkmartinjr · 2023-07-04T22:04:48Z

docs/cellxgene_census_docsite_data_release_info.md

@@ -1,23 +1,110 @@
 # Census data releases 

-**Last edited**: April, 2023.
+**Last edited**: July, 2023.


recommend full date in case it is edited more than once in a given month

docs/cellxgene_census_docsite_data_release_info.md

bkmartinjr · 2023-07-04T22:06:44Z

docs/cellxgene_census_docsite_data_release_info.md

+Open this data release by specifying `census_version = "2023-06-28"` in future calls to `open_soma()`.
+
+#### 🔴 Erratum 🔴  


Suggest using plural form, i.e., Errata

seems unlikely we will only have one in due time :-)

codecov · 2023-07-04T22:16:15Z

Codecov Report

Merging #600 (47dc864) into main (f06bceb) will decrease coverage by 0.03%.
The diff coverage is n/a.

❗ Current head 47dc864 differs from pull request most recent head 324a21a. Consider uploading reports for the commit 324a21a to get more accurate results

@@            Coverage Diff             @@
##             main     #600      +/-   ##
==========================================
- Coverage   88.10%   88.07%   -0.03%     
==========================================
  Files          62       62              
  Lines        3740     3741       +1     
==========================================
  Hits         3295     3295              
- Misses        445      446       +1

Flag	Coverage Δ
unittests	`88.07% <ø> (-0.03%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

see 2 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

jahilton · 2023-07-05T15:45:14Z

docs/cellxgene_census_docsite_data_release_info.md

+There are 243,569 number of cells labelled as `is_primary_data = True` for which a fraction of them the label is incorrect. 
+
+Such label indicates that a cell is the primary representation of an observation, otherwise a cell is deemed to be a duplicate representation. Based on their count vectors, these 243,569 number of cells are represented at least twice with `is_primary_data = True`.


~~Such label indicates that a cell is the primary representation of an observation~~ - I think this will introduce confusion (What does "primary representation" mean?).
~~Based on their count vectors~~ - I don't think it matters how we found them

Suggest something like...
In order to prevent duplicate data in analyses, each observation should be marked is_primary data = True exactly once in the Census. Since this LTS release, 243,569 observations have been identified that are represented at least twice with is_primary_data = True.

bkmartinjr · 2023-07-06T22:45:43Z

docs/cellxgene_census_docsite_data_release_info.md

+* Available for public access for 1 week or until the next latest release is performed, whichever is the longest.
+
+The weekly release can be opened by the APIs by specifying `census_version = "latest"`.


...The most recent weekly release...

(other weekly releases must be opened by name)

bkmartinjr · 2023-07-06T22:46:03Z

docs/cellxgene_census_docsite_data_release_info.md

they are available for a month by explicit name

bkmartinjr · 2023-07-06T22:48:13Z

docs/cellxgene_census_docsite_data_release_info.md

+**Contents**
+
+1. [What is a Census data release?](#What-is-a-Census-data-release)
+2. [List of LTS Census data releases](#List-of-LTS-Census-data-releases)

 ## What is a Census data release?

 It is a Census build that is publicly hosted online. A Census build is 


Might be worth stating: Any given Census build is named with a unique tag, normally the date of build, e.g., 2020-01-30

bkmartinjr

LGTM. Left a couple of optional nit comments - fix as you see fit.

add erratum

773edab

pablo-gar requested review from brianraymor and bkmartinjr July 4, 2023 21:49

pablo-gar added 2 commits July 4, 2023 14:51

add link to file

84e2760

Add punctuation

f0a0916

pablo-gar marked this pull request as ready for review July 4, 2023 21:54

bkmartinjr reviewed Jul 4, 2023

View reviewed changes

docs/cellxgene_census_docsite_data_release_info.md Show resolved Hide resolved

bkmartinjr reviewed Jul 4, 2023

View reviewed changes

docs/cellxgene_census_docsite_data_release_info.md Show resolved Hide resolved

bkmartinjr reviewed Jul 4, 2023

View reviewed changes

docs/cellxgene_census_docsite_data_release_info.md Show resolved Hide resolved

bkmartinjr reviewed Jul 4, 2023

View reviewed changes

jahilton reviewed Jul 5, 2023

View reviewed changes

pablo-gar added 2 commits July 6, 2023 14:58

address reviewers coments

cacf98b

change header

b160f98

pablo-gar requested review from jahilton and bkmartinjr July 6, 2023 22:42

bkmartinjr reviewed Jul 6, 2023

View reviewed changes

bkmartinjr approved these changes Jul 6, 2023

View reviewed changes

pablo-gar added 2 commits July 7, 2023 09:37

final editorial changes

47dc864

final editorial changes

324a21a

pablo-gar merged commit 5ac97a0 into main Jul 7, 2023

pablo-gar deleted the pablo-gar/update-data-release-page-doc-site branch July 7, 2023 18:18

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Expand doc-site section "Census data releases" and add data erratum #600

Expand doc-site section "Census data releases" and add data erratum #600

pablo-gar commented Jul 4, 2023

bkmartinjr commented Jul 4, 2023

bkmartinjr Jul 4, 2023

bkmartinjr Jul 4, 2023

bkmartinjr Jul 4, 2023

jahilton Jul 5, 2023

bkmartinjr Jul 4, 2023

bkmartinjr Jul 4, 2023

codecov bot commented Jul 4, 2023 •

edited

Loading

jahilton Jul 5, 2023

bkmartinjr Jul 6, 2023

bkmartinjr Jul 6, 2023

bkmartinjr Jul 6, 2023

bkmartinjr left a comment

		### LTS 2023-05-15

		Open this data release by specifying `census_version = "2023-06-28"` in future calls to `open_soma()`.

		#### 🔴 Erratum 🔴

		There are 243,569 number of cells labelled as `is_primary_data = True` for which a fraction of them the label is incorrect.

		There are 243,569 number of cells labelled as `is_primary_data = True` for which a fraction of them the label is incorrect.

		Such label indicates that a cell is the primary representation of an observation, otherwise a cell is deemed to be a duplicate representation. Based on their count vectors, these 243,569 number of cells are represented at least twice with `is_primary_data = True`.

		* Available for public access for 1 week or until the next latest release is performed, whichever is the longest.

		The weekly release can be opened by the APIs by specifying `census_version = "latest"`.

Expand doc-site section "Census data releases" and add data erratum #600

Expand doc-site section "Census data releases" and add data erratum #600

Conversation

pablo-gar commented Jul 4, 2023

bkmartinjr commented Jul 4, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Jul 4, 2023 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bkmartinjr left a comment

Choose a reason for hiding this comment

codecov bot commented Jul 4, 2023 •

edited

Loading