Hmm, how this will work with versioning?

What happens when TRS v3 comes out and a new product only implements TRS v3 and doesn't implement v2?

That's assuming TRS v3 will use a new path segment v3, which I suppose it doesn't have to. But I assume it would.

Essentially very primitive content negotiation where the user agent would need to try the API versions it would be familiar with?

I think versioning APIs is an important topic and I think there's most definitely a need for discussion and harmonization across GA4GH APIs (e.g., Stian mentioned HATEOAS in #160), but as this PR does not introduces the issue nor is designed to address it, I don't think it should be blocked by it.

Besides, I don't think that TRS URIs, as identifiers of resources, should be concerned with API versioning at all: IMO, when bumping API versions, care must be taken (both on the spec- and implementation-side) not to introduce ambiguities in the way that the different responses in, e.g., TRS v2 and v3 can be interpreted by a client. And if changes to the internal representation of resources are required that would break backward compatibility, then I think those resources should be cloned and given new identifiers, such that reproducibility is not negatively affected. As long as these considerations/constraints are honored, it shouldn't matter which API version is used to access a given resource.

In any case, I have clarified this/addressed these concerns now in lines 56/57 and 69-72

My question was not about API/schema versioning, which is a very complex subject and I'm no expert in. It was about the example that if you have a URI trs://trs.example.org/tool_ABC/v1.2.3, it says you can fetch the content https://trs.example.org/api/ga4gh/trs/v2/tools/tool_ABC (note the v2 in the path).

If TRS v3 comes along, and a new TRS implementation only supports that v3, then the v2 in the path means they would return a 404. Maybe that's OK, or maybe the requirement is that all TRS implementations need to implement v2 and above. Otherwise a client will need to iterate over paths with all potential versions.

Maybe I'm overthinking this, and it is a real edge case.

I do think this is an API/schema versioning topic because the API version is precisely what v2 refers to. In any case, there anyway needs to be some sort of negotiation between a service and a client, because not every client will implement every version of the API specs, and so there will be cases where clients and servers won't match, regardless of whether the client will iterate or whether we specify the version in the TRS URI. One possibility to facilitate this negotation would be to define the supported API versions in the service info, but of course, the Service Info API is versioned, too...

I agree this is an important issue and needs to be discussed on a larger scale within the Cloud WS in order to come up with a sensible strategy that covers DRS and TRS URIs as well as other artefacts/problems with API versioning. A related issue I faced is also that the path (e.g., ga4gh/trs/v2) defined in the schema may not always be possible or desirable to implement. For example, as far as I know, Dockstore does not. The path info could perhaps also be defined in the service info for any given service (although the same problem applies: how do you get to the service info in the first place; perhaps via the service registry...).

Still, I don't think this PR makes this problem worse than it is already, given the current definitions of DRS and TRS URIs, so I don't think it should block.

I do think this is an API/schema versioning topic

Fair. I was thinking more of different schemas in different versions of the API when I made my comment, but you're right.

Still, I don't think this PR makes this problem worse than it is already, given the current definitions of DRS and TRS URIs

I think the explicitness that you can take a URI and translate it to a specific path is a new wrinkle, at least for TRS. You're right about DRS -- I hadn't looked until now and they do the same thing, with v1 in the path.

so I don't think it should block.

I had already approved, so it is not blocked.

Thanks a lot @coverbeck

So @denis-yuen, others: shall we bring up this important discussion to the Cloud WS?

Also perhaps to see if people agree that it might be useful to specify TRS URIs as inputs to WES (to access workflows) and inside workflows (to access containers through WES/TES implementations)? Because this might require changes on WES and TES specs as well, without which TRS URIs might not be all that useful.

Proposal at Cloud WS was to break this discussion into a separate ticket #221

Essentially very primitive content negotiation where the user agent would need to try the API versions it would be familiar with?

My question was not about API/schema versioning, which is a very complex subject and I'm no expert in. It was about the example that if you have a URI trs://trs.example.org/tool_ABC/v1.2.3, it says you can fetch the content https://trs.example.org/api/ga4gh/trs/v2/tools/tool_ABC (note the v2 in the path).

If TRS v3 comes along, and a new TRS implementation only supports that v3, then the v2 in the path means they would return a 404. Maybe that's OK, or maybe the requirement is that all TRS implementations need to implement v2 and above. Otherwise a client will need to iterate over paths with all potential versions.

Maybe I'm overthinking this, and it is a real edge case.

@uniqueg This example is a bit awkward and likely points to a deficiency in the current approach for defining trs identifiers (and potentially drs identifiers as well). The TES specification defines the root path of the API as /ga4gh/trs/<version>.

Putting aside specific trs versioning issues (from the conversation above), I am now left with another dilema: TRS identifiers as defined only provide the hostname and a toolId and possibly a version id. This approach does not leave ANY room for path values that may come before the /ga4gh/... prefix. Given this, how am I supposed to resolve the actual trs URL for the example?

By convention, any tooling I create would resolve trs://trs.example.org/tool_ABC/v1.2.3 to https://trs.example.org/ga4gh/trs/v2/tools/tool_ABC/versions/v1.2.3, which is not what is presented in this example (note the missing API). I would not be so concerned over this if it was not already present in the wild in the Dockstore api (@denis-yuen). So my question then is how can we possibly resolve the path components without introducing ambiguity into the trs identifier.

One approach could be just adding the prefix paths prior to the toolId trs://trs.example.org/api/tool_ABC. Unfortuantely this is ambiguous since this could be referring to either a tool with the path prefix api and version tool_ABC or a tool tool_ABC with a path api. If we do not allow the path prefix to be in the identifier, any tooling is limited to the trs implementations that It already knows how to resolve, effectively limiting the usefulness of the trs id in general.

We essentially need a reliable way to extract the toolId and versionId without any ambiguity, while still allowing preceding fragments in the path. Some examples are below:

trs://trs.example.org/<prefix-before-ga4gh-api>/<toolId>:<versionId>
trs://trs.example.org/<prefix-before-ga4gh-api>/<toolId>/v/<versionId>
trs://trs.example.org/<prefix-before-ga4gh-api>/<toolId>/version/<versionId>

One could probably calculate the prefixes on the fly from https://github.com/ga4gh/tool-registry-service-schemas/blob/develop/registry.json but it probably does make more sense to change the URI format since we're not using it yet(?)

@denis-yuen I would advise against relying on prefixes in the registry.json file, which automatically enforces that file is the authoratative source of all tool registries. It does not really allow for new registries to dynamically be created (ie in a multitenant system)

I think my personal preference for approaches would likely be: trs://trs.example.org/<prefix-before-ga4gh-api>/<toolId>:<versionId>. The side effect of this is that : is now a restricted character that cannot be used in the identifier itself, but I think that is a reasonable restriction

tl;dr

I agree with @patmagee's suggestion provided that DRS folks agree that it will work for them as well.

Good catch @patmagee - thanks a lot! I had actually realized this issue before when implementing a TRS client that understands TRS URIs and having to introduce an optional base_path parameter so that it would work with Dockstore. It seems I had forgotten about it though. And I agree, the TRS URI loses its utility if this has to be known beforehand.

From what I can see, @patmagee's suggestion should do the trick.

But thinking this through a little more, maybe there are a couple of things that we should discuss outside of this issue before merging:

• To make sure that whatever is decided here has the perspective of aligning with whatever is decided for DRS URIs in the future, I think this discussion should include the DRS community. @patmagee's other worry of being able to include basically any character in TRS Tool/Version IDs could be brought up there as well.
• I am not quite sure whether API versioning could have any effect on a client's ability to properly resolve a TRS URI. I feel that it should not. But without explicit guidelines on API versioning, outlining expected behaviors and discussing their impact on reproducibility, I feel that it is hard to be sure. So perhaps it might be worth to discuss specifying such guidelines somewhere, possibly in a place that includes other work streams as well.

Besides that, it is possibly a good idea to aim for a consensus of how servers are defined in the various Cloud WS APIs in the mid to long term. And whatever is decided here, should not collide with our definitions of TRS and DRS URIs. Currently, the situation is like this:

TRS

servers:
  - url: /ga4gh/trs/v2

DRS

servers:
  - url: https://{serverURL}/ga4gh/drs/v1
    variables:
      serverURL:
        default: drs.example.org
        description: >
          DRS server endpoints MUST be prefixed by the '/ga4gh/drs/v1' endpoint
          path

WES

servers:
- url: https://{defaultHost}/{basePath}/{apiVersion}
  variables:
    defaultHost:
      default: www.example.com
    basePath:
      default: ga4gh/wes
    apiVersion:
      default: v1

TES

servers:
- url: /ga4gh/tes/v1

So, it looks like TRS and TES do not care where the API is deployed, as long as endpoint paths are immediately preceded by /ga4gh/trs/v2 and /ga4gh/tes/v1, respectively. DRS is very similar but a little more prescriptive in that it requires the use of the https schema. WES, on the other hand, requires https as well, but otherwise only seems to require that there be at least two slashes included in the URL before the endpoint path (weird?). In particular, it does not require a specific base path suffix like /ga4gh/wes/v1 (though the default does include it).

Now, there is currently no consensus across those definitions. But perhaps it seems to be going in the direction of the DRS definition, i.e., start with https and end with a specific base path suffix of the form /ga4gh/{api_short_name}/{api_version}. @patmagee's definition would account for that (having the client fill in the appropriate base path suffix). But even with a more relaxed definition that does not prescribed a specific base path for GA4GH Cloud APIs, the suggested solution should still work.

Proposal at Cloud WS was to also break this into a separate ticket, but to prioritize it for discussion before 2.0.1 release and/or the GA4GH plenary
#220

Proposal at Cloud WS was to break this discussion into a separate ticket #221

Proposal at Cloud WS was to also break this into a separate ticket, but to prioritize it for discussion before 2.0.1 release and/or the GA4GH plenary
#220