Search for a variable across all datasets

[jamclar]

Hello,

we are currently using Erddap at our organization and we'd like to reach out to the wider ERDDAP community to see if what we are trying to do is possible.

We currently have 50 datasets, and 10 of these datasets that have a 'Depth' Variable,

We are able to find these 10 datasets by going to the advanced search and then performing a free text search for 'variableName=depth'. We can then go into these datasets individually and choose the data we'd like that has a depth between a lower and upper bound.

So in our current setup if we want to find datasets that have a depth between x and y, we first have to use the free text search to find out which datasets have a Depth variable, then iterate over each result from this and then query each dataset, which unfortunately takes a fair amount of time to execute.

What we'd like to do, is find a way to perform a single search for datasets that have a depth variable with values between x and y, and we would like to know if this is possible?

Many thanks in advance

[BobSimons]

First, a comment about your search method: Your method to find datasets with variableName=depth is fine (and the free text search method would also work with the regular free text search system). Just so you know, there is an alternative in Advanced Search: there is a built in option to find datasets with specific variable names. So you could also do the search by setting variableName to "depth" in Advanced Search. The results will be the same.

Second: Currently, there is not a way to do a single search for datasets with depth values between x and y. That feature exists for the latitude, longitude and time variables. Since depth/altitude is a similarly special variable, I think it makes sense for Chris to add that option to the Advanced Search system. The programming logic/steps would exactly parallel the logic/steps already in place for lat, lon and time. It seems like a good addition to me. I leave it to Chris to decide whether or not to add that. Or, some other programmer could do it and submit a pull request.

As a variant of that: it might make sense to set up the new feature to allow for you to specify any variable name in the form
variableName operator numericValue
Then users could search for e.g., temperature>30 to find all datasets with a variable named temperature which has values >30. But see my third comment below for an important limitation.

Third, note that there is a quirk to the current Advanced Search for lat/lon/time between x and y: Certain types of datasets don't have known ranges for the lat/lon/time variables, notably datasets based on relational databases. If you do an Advanced Search for lat/lon/time values, those datasets will never be returned because the query is just run against the known ranges of those variables. Your current system (separate requests to each dataset) are the only perfectly correct solution.

I hope that helps. If not, or if you need some refinement, please let me know.

[BobSimons]

Oops. I realize my more generalized solution has a few complications, notably that something=someValue is already used for other purposes. Maybe there is a way to refine the proposal (e.g., just allow >=, <=, > and < as the operator for this type of query, or use some other syntax that looks more like a function call: gt("temperature", 30) to find datasets with temperature values >30) so that it doesn't interfere with existing Advanced Search options.

[BobSimons]

Yet another alternative for the syntax:
find(variableName operator numericValue)
e.g., find(temperature>30)

[ChrisJohnNOAA]

I do think improved advanced search options as described would be a good addition. There are a lot of good additions for the project and my time is limited so I don't know when I'll get to it, but I do welcome contributions.

[jamclar]

Hi @BobSimons and @ChrisJohnNOAA

thanks very much for the swift reply, it's much appreciated.

We discussed your suggestions with the team this morning, and we would like to have a go at contributing to this after the New Year, and we will write back here when we have any news or additional questions regarding this feature.

We would also like to add that if there are additional features that are similar to this that could be added, then we would happy to consider designing & writing them.

Thanks again!

[BobSimons]

Excellent!