Is there a way to bypass the limit of 10,000 items?

[glaroc]

We have multiple use cases where the only differences between the items in a collection are in a custom property variable. For example, we have models outputs for 5k species for multiple years and multiple models. So, it often happens that we need to filter the catalog for all results for a given species, as specified by a variable in the item's properties with the species name, or for all results from a particular model.

For this reason, the limit of 10,000 items returned by the collection/{}/items or /search endpoints is really problematic for us. Is there any way to bypass this or simply to eliminate the limit in Stac-fastapi ?

[vincentsarago]

you can set a custom Search model with a max limit >1000

https://github.com/stac-utils/stac-fastapi/blob/main/stac_fastapi/types/stac_fastapi/types/search.py#L184-L190

from pydantic import Field
from stac_pydantic.api import Search

class BaseSearchPostRequest(Search):
    """Base arguments for POST Request."""

    limit: Optional[int] = Field(
        10,
        description="Limits the number of results that are included in each page of the response.",  # noqa: E501
    )


api = StacApi(
    search_post_request_model=BaseSearchPostRequest
)

I don't think there is a upper limit for the /items endpoint

[glaroc]

Thank you, we'll give this a try.

I don't think there is a upper limit for the /items endpoint

I believe there is. We can't ever seem to get more than 10,000 items, regardless of the limit= parameter.

[jonhealy1]

Pagination?

[jonhealy1]

Elasticsearch itself has a built-in 10,000 item return limit so switching over would not help ...

[glaroc]

Is pagination available with stac-fastapi ? I haven't seen any docs on this.

[geospatial-jeff]

Item-level pagination is available for most backends (it is a requirement of OGC API Features). Pgstac does implement it, look for a next and previous rel-type in the top-level links returned by the various search endpoints.

You will quickly run into performance bottlenecks using such a large limit, pagination is absolutely the way to go here.

[vincentsarago]

the limit in the /items endpoint comes from

stac-fastapi/stac_fastapi/api/stac_fastapi/api/models.py

Lines 111 to 120 in 4df4947

	@attr.s
	class ItemCollectionUri(APIRequest):
	"""Get item collection."""
	collection_id: Annotated[str, Path(description="Collection ID")] = attr.ib()
	limit: Annotated[int, Query()] = attr.ib(default=10)
	bbox: Optional[BBox] = attr.ib(default=None, converter=_bbox_converter)
	datetime: Optional[DateTimeType] = attr.ib(
	default=None, converter=_datetime_converter
	)

so I don't understand how the limit is max out to 10000 🤷 maybe it's a pgstac thing, cc @bitner 🤷

[vincentsarago]

@glaroc which version of stac-fastapi are you using?

[bitner]

There is no max limit set in pgstac.

The max limit of 10,000 comes from here.

Unconstrained large limits run the risk of causing memory issues or timeouts.

Pagination is definitely the preferred method to get more results than that. If you use pystac-client to interact with a pagination enabled stac api, the library completely takes care of the pagination for you and running an search returns an iterator that will transparently make all the paginated requests for you.

[vincentsarago]

@bitner I don't see this used in the /items endpoint

[vincentsarago]

but it will with #738