fix: make rest-explorer aware of openapi server root path

[mgabeler-lee-6rs]

The rest-explorer component needs to be aware of any "external" base paths that are in use. This PR is at least a start to that by making it prefix any paths it sends to clients with the first path from the openApiSpec.servers list (if any).

Fixes: #2285

Checklist

• npm test passes on your machine
• New tests added or existing tests modified to cover all changes
• Code conforms with the style guide
• API Documentation in code was updated
  • N/A?
• Documentation in /docs/site was updated
  • N/A
• Affected artifact templates in packages/cli were updated
  • N/A
• Affected example projects in examples/* were updated
  • N/A

Having some problems with test and lint, mainly related to npm install not having installed prettier for some reason. I have no reason to believe that this is caused by my changes, and I added tests for my changes, which pass, but I've been unable to run the linter or get the full test suite to pass on my system yet.

[raymondfeng]

I'm not sure if inferring from openapi.servers[0].url is a feasible solution. Please note that LoopBack app is not aware how its endpoints are proxies/exposed. The same endpoint can be possibly exposed as many different urls, such as:

• https://a.example.com/api/customer
• https://b.example.com/api/user
• http://internal.example.com:8080/customer

When it comes to redirect via Location header, my understanding is that proxy/load balancer is responsible for rewriting Location to match its clients.

[mgabeler-lee-6rs]

The redirect could be made to depend on the proxy (though that requires config on the proxy). But the same question would then apply to the /openapi.json reference too.

I was concerned about this in my original bug report, but @bajtos had a different perspective.

A brainstorm: I think LB2 dealt with this by having the explorer be the one that serves the openapi.json, at least when "talking to itself". This could be replicated within rest-explorer with a little work I think, though it would require the RestServer to expose _serveOpenApiSpec as a public method. But I think then the explorer could be made to use exclusively relative URLs, avoiding the path rewriting issue in a more general way?

[raymondfeng]

Some references:

• http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_redirect
• Apiserver proxy rewrites URL when service returns absolute path with request's host kubernetes/kubernetes#52556
• https://github.com/kubernetes/ingress-nginx/tree/master/docs/examples/rewrite

[raymondfeng]

Maybe we should set Location to a url based on the request - https://github.com/strongloop/loopback-next/blob/master/packages/rest/src/rest.server.ts#L429.

[raymondfeng]

If we set Location to a relative url, would it prevent reverse proxies from rewriting it correctly?

[mgabeler-lee-6rs]

I was hoping that using a relative Location url would mean the proxy doesn't need to rewrite it :)

[bajtos]

I believe that this.request.url contains only the path portion of the requested URL as seen by the LB4 server, after it has been rewritten by the reverse proxy. IMO, the current implementation is correct, it's redirecting from /explorer to /explorer/index.html.

I am not able to fully understand implications of the change proposed here. More importantly, I don't understand why this particular change is needed.

Can we revert it please?

[mgabeler-lee-6rs]

@raymondfeng @bajtos -- What do you think about having the explorer serve its own copy of the openapi.json so that it can be loaded from a URL relative to the explorer itself, and thus hopefully avoid any issues of computing base paths or the like?

[raymondfeng]

What do you think about having the explorer serve its own copy of the openapi.json so that it can be loaded from a URL relative to the explorer itself, and thus hopefully avoid any issues of computing base paths or the like?

Can you elaborate?

[mgabeler-lee-6rs]

Currently the explorer tries to figure out the URL it should send to the client for it to fetch the openapi.json file that the RestServer is already serving. But trying to figure out that URL, as noted above, can be complex, if even impossible, in more complex proxy & custom config cases.

So, instead of trying to find the URL it's being served on for other purposes, what if it adds its own URL on which it serves the OAS material. I.e. if you have the explorer mounted at .../explorer/, then it could serve the OAS JSON to itself at .../explorer/openapi.json, regardless of what other endpoints you've defined for the OAS content to be served. That way it can safely know that, whatever URL is serving the explorer, ./openapi.json will work as a relative URL to get the API spec, and it doesn't have to figure out what proxy mapping or other configuration is active to find the client-view of the URL for /openapi.json -- i.e. it doesn't have to have any idea what that ... is. It also means the explorer can still work if the user has customized away the default RestServer to e.g. only serve the OAS file in YAML.

This is (roughly) how it seems to have worked in LB2 -- the OAS file was at .../explorer/swagger.json IIRC, so the explorer didn't need to worry about absolute paths to load that file.

[raymondfeng]

I think it's worth exploring. Maybe we should expose an API from @loopback/rest.

[mgabeler-lee-6rs]

I think it's worth exploring.

I see what you did there 😉

Maybe we should expose an API from @loopback/rest.

Indeed ... I was thinking of exposing a version of serveOpenApiSpec, with suitable adjustments for being a public method (if any). Then I could add a openApiSpec method to ExplorerControler, wire it up as explorerPath + '/openapi.json' with registerControllerRoute in RestExplorerComponent, and update index.html.ejs accordingly.

If this sounds sensible, I can update this PR to go down that path.

[bajtos]

+1 to modify our REST API explorer to use a relative URL for the OpenAPI spec if that's going to fix the problem.

I am not convinced that we need a new explorer-specific endpoint for that though. A typical LoopBack 4 application comes with the following endpoints:

• {basePath}/openapi.json serves the spec
• {basePath}/explorer servers Explorer UI

IMO, it should be enough to figure out what's the relative path from /explorer (which can be customized via REST API Explorer config option path) to /openapi.json (which can be customized via restConfig.openApiSpec.endpointMapping.

basePath is / by default, but can be changed by app.basePath API or rewritten by a reverse proxy as described by @mgabeler-lee-6rs.

Can we assume that the base path for openapi and explorer endpoints will be always the same? I believe the answer is true when LB4 is called directly. I am not sure about reverse proxies. How likely it is that somebody will configure the reverse proxy to expose /openapi.json at let's say https://example.com/servicename/api-endpoint/openapi.json and /explorer at https://example.com/servicename/api-explorer?

I think our implementation should:

• Work out of the box for the default setup, where both /openapi.json and /explorer endpoints are exposed at the same base url. This can be fixed either by computing a relative path, or it may be enough to fix the implementation of getOpenApiSpecUrl to take basePath config into account. (/cc @nabdelgadir, I think you run into the problem of basePath support in REST API Explorer too)

• Provide a new REST API Explorer configuration option allowing advanced users to specify a custom URL where to find the spec. For example:

  {
    path: '/explorer',
    specUrl: 'https://example.com/servicename/api-explorer',
  }
  

That way we can keep our implementation simple and easy to maintain and still support advanced use cases.

[bajtos]

@mgabeler-lee-6rs I feel that I may not fully understand what's happening under the hood when a reverse proxy is configured in front of a LB4 app and REST API Explorer is loaded. Could you please create a setup we could use to reproduce the problem you are encountering on our machines? I'd like to investigate this more in depth myself to build a better understanding on how individual parts interact together. A reverse proxy configuration for our todo example would be ideal.

[raymondfeng]

@mgabeler-lee-6rs It would be great to create such scenario based on https://loopback.io/doc/en/lb4/deploying-with-pm2-and-nginx.html to illustrate how the Location header is impacted by nginx reverse proxy.

[raymondfeng]

This seems to be simple way to reproduce the problem too - #2306 (comment)

[bajtos]

Conflicting files

packages/rest-explorer/test/acceptance/rest-explorer.acceptance.ts

In #2330, we have moved test files from test to src/__tests__. Please follow the steps in How to rebase your branch to resolve the merge conflict. Please do not run git merge, it's important to use git rebase master.

Sorry for the inconvenience.

[bajtos]

@mgabeler-lee-6rs ping 👋 what's the status of this pull request? Are you still keen to get it finished?

[mgabeler-lee-6rs]

I am still keen to finish it up, yes. I've just been diverted by some higher priorities temporarily.

[raymondfeng]

I created #2387 to address the issue when rest-server is mounted to express using a path such as expressApp.use('/api', restServer.requestHandler);.

[ludohenin]

@raymondfeng when do you see that one landing ?

[bajtos]

Is this pull request still relevant, considering the work done in #2856 and previous pull requests?

[mgabeler-lee-6rs]

I'd need to do some testing. I think basePath handling doesn't cover my use case, where the reverse proxy is stripping off the path prefix. In that case, setting up a basePath actually breaks everything, at at least that's what I recall from the last time I tried it -- we need the controllers to all be blissfully unaware of the path prefix the reverse proxy is externally applying. Trying to put a basePath into things caused it to want to have two copies of that in the path, which just made things more broken.

@dkrantsberg's reverse-proxy setup described in the comments on #2554 is not dissimilar to mine, though simpler. I'm curious if he has a working setup for comparison.

The workaround I'm still monkey-patching into my apps is to sub-class the explorer and give it an extra config setting for the "external" base path, which is built based on known patterns for our reverse proxy when running in production. This is ... not pretty to say the least, but it gets the job done for me.

I think the simplest way to be able to handle this in a portable fashion is to make the rest explorer exclusively use relative URLs as described earlier -- have the explorer serve its own copy of the openapi spec using the format it needs, at a path relative to the explorer. Then we never need to worry about absolute paths in any of the requests, and most of the path logic should go away?

[bajtos]

@mgabeler-lee-6rs thank you for the response.

I think the simplest way to be able to handle this in a portable fashion is to make the rest explorer exclusively use relative URLs as described earlier -- have the explorer serve its own copy of the openapi spec using the format it needs, at a path relative to the explorer. Then we never need to worry about absolute paths in any of the requests, and most of the path logic should go away?

That sounds like a reasonable solution to me. Would you like to open a new pull request for this proposal?

Another alternative that comes to my mind: modify Explorer to use a relative path to access OpenAPI spec. I.e. when serving /mountPath/explorer/, then use ../openapi.json instead of /mountPath/openapi.json.

[mgabeler-lee-6rs]

Would you like to open a new pull request for this proposal?

Yes, I'll give that a shot 🙂

Another alternative that comes to my mind: modify Explorer to use a relative path to access OpenAPI spec. I.e. when serving /mountPath/explorer/, then use ../openapi.json instead of /mountPath/openapi.json.

I thought about that, but then the problem space of correctly computing the relative path to the OAS for all the possible ways users might modify it starts to feel ... complicated. Much more so than having our own copy that is always at a fixed path relative to the explorer.

[mgabeler-lee-6rs]

I've started a draft of the relative path hosting in #3133. That PR is definitely in a WIP state right now, but there is enough there to take a look at the idea at least. I'll be fleshing it out with tests and whatnot of course.

Closing this PR in favor of that new one.