You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
In principle we could support very large numbers of Assets per DataSource, but in practice at the moment ?include_data_sources=true lists the Assets in one list, not paginated. If we find use cases for very large number of Assets, we'll want a paginated way to list, and perhaps filter and sort them.
For now, should we limit Assets to 10k per DataSource? I think the most likely reason for us to breach that limit would be TIFF sequences, so @Wiebke and @dylanmcreynolds should weigh in on whether this limit will be a problem for present use cases at ALS.
The text was updated successfully, but these errors were encountered:
I do know @taxe10 recently encountered a case where she worked with ~30k tiff images, but I do not recall if these were in a single sequence or spanned across multiple sequences. @taxe, can you share details about this use case?
On that note, we are discussing where the trade-offs and conceptual differences of grouping multiple images into one container are too. There is an argument to be made for only grouping a sequence of 2d-images if it actually conceptually represents a 2D+1 dataset (may that be due to 3d spatial dimensions dimensions, or 2d spatial dimensions+time) and not if it is a grouped collection of 2d-images across several experiments.
At the same time ML algorithms for images may benefit from being able to randomly index into a single container (although there may be performance considerations to investigate in more detail with that too).
I should of course have tagged @taxe10 as well. (I think I opened a new tab to go check my recollection of her GH username and lost my train of thought….) Thanks @Wiebke!
In principle we could support very large numbers of Assets per DataSource, but in practice at the moment
?include_data_sources=true
lists the Assets in one list, not paginated. If we find use cases for very large number of Assets, we'll want a paginated way to list, and perhaps filter and sort them.For now, should we limit Assets to 10k per DataSource? I think the most likely reason for us to breach that limit would be TIFF sequences, so @Wiebke and @dylanmcreynolds should weigh in on whether this limit will be a problem for present use cases at ALS.
The text was updated successfully, but these errors were encountered: