Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update python package and builder to tiledbsoma 1.0.0rc2 #227

Merged
merged 9 commits into from
Feb 28, 2023

Conversation

bkmartinjr
Copy link
Contributor

@bkmartinjr bkmartinjr commented Feb 27, 2023

Update cell census package and builder to utilize tiledbsoma==1.0.0rc2

Other changes:

  • revert the census builder to create the census with default TileDB timestamp behavior. Specifically, the objects comprising the census build will not share a single timestamp.
  • fix small bug in validator which is triggered by test builds containing zero cells for a given organism.

Fixes #208
Fixes #221
Fixes #228

@bkmartinjr bkmartinjr changed the title Update builder to RC1 Update to RC2 Feb 27, 2023
@bkmartinjr bkmartinjr changed the title Update to RC2 Update python package and builder to tiledbsoma 1.0.0rc2 Feb 27, 2023
@codecov
Copy link

codecov bot commented Feb 27, 2023

Codecov Report

Merging #227 (f248d7a) into main (ace0551) will decrease coverage by 0.28%.
The diff coverage is 58.33%.

@@            Coverage Diff             @@
##             main     #227      +/-   ##
==========================================
- Coverage   84.02%   83.75%   -0.28%     
==========================================
  Files          29       29              
  Lines        1628     1619       -9     
==========================================
- Hits         1368     1356      -12     
- Misses        260      263       +3     
Flag Coverage Δ
unittests 83.75% <58.33%> (-0.28%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
tools/cell_census_builder/mp.py 60.00% <ø> (-9.24%) ⬇️
tools/cell_census_builder/validate.py 88.96% <0.00%> (-0.97%) ⬇️
tools/cell_census_builder/consolidate.py 64.44% <50.00%> (ø)
tools/cell_census_builder/globals.py 100.00% <100.00%> (ø)

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@bkmartinjr bkmartinjr marked this pull request as ready for review February 27, 2023 20:31
Copy link
Collaborator

@atolopko-czi atolopko-czi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Some comments re: comments, one unrelated bug fix, and a suggestion to remove support for custom setting of tiledb_ctx.


# Using "end of time" for read_timestamp means that all writes are visible, no matter what write timestamp was used
END_OF_TIME = 0xFFFFFFFFFFFFFFFF


def SOMA_TileDB_Context() -> soma.options.SOMATileDBContext:
global _SOMA_TileDB_Context
if _SOMA_TileDB_Context is None or _SOMA_TileDB_Context != TileDB_Ctx():
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To fix a bug I introduced on an earlier PR:

Suggested change
if _SOMA_TileDB_Context is None or _SOMA_TileDB_Context != TileDB_Ctx():
if _SOMA_TileDB_Context is None or _SOMA_TileDB_Context.tiledb_ctx != TileDB_Ctx():

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But maybe it's time to just eliminate support for custom tiledb_ctx values (remove the setter), since we don't use it, allowing us to remove this check.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, for now, I'll just make it a cached value. Good call.

read_timestamp=END_OF_TIME,
# `None` opts out of explicit timestamp consistency, and uses the underlying
# TileDB per-object write consistency. This is sufficient given that this
# builder explicitly orders / synchronizes all reading/writing.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As "timestamp consistency [is] the default" (per this comment), we could remove this comment. Minimally, update it to be accurate or reword so that it doesn't become outdated if/when TileDB-SOMA semantics change again.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you are right that this is confusing. I meant not using PIT consistency across the entire census collection. I'll delete.

)
return _SOMA_TileDB_Context


def TileDB_Ctx() -> tiledb.Ctx:
# See `DEFAULT_TILEDB_CONFIG` above
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It could be set to something other than DEFAULT_TILEDB_CONFIG. But if we're never going to do that in builder code, I suppose we could remove the ability to set it altogether, in which case this comment is fine.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

resolved by deleting setter

@bkmartinjr bkmartinjr merged commit c67b936 into main Feb 28, 2023
@bkmartinjr bkmartinjr deleted the bkmartinjr/update-builder-to-latest-rc branch February 28, 2023 02:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

update to tiledbsoma RC2 Cell Census builder uses TileDB-SOMA 1.0 RC1 Revert builder PIT hack
2 participants