Apicurio registry security - basic auth support

[christofluethi]

Hi There

Another question related to security with Apicurio 2.0.1.Final. Our ecosystem will contain applications using different client serializer libs (confluent, apicurio, ...).

What our main goals are:

• Read access: Must be compatible with most client libraries (confluent, apicurio, ...).
• Write access: Should be secured (e.g. RBAC like apicurio registry provides with oauth sr-admin, sr-developer)

For read access:

• e.g. confluent libs do not allow to configure oauth. which leads us to leave read access either unprotected or basic auth
• As far as i know apicurio registry does not support to use keycloak and http basic auth at the same time.

Question:

• Documentation "Configuring your Apicurio Registry deployment" only mentions OAuth with Keycloak. Is using Apicurio Registry with Basic Auth possible?
• Is it possible to use OAuth or Basic Auth for admin and developer purpose but leave read access unprotected?
• Are the quarkus.http.auth.permission.* overridable? e.g. removing the read-apis from the quarkus.http.auth.permission.sr-read.paths and add them to the quarkus.http.auth.permission.permit1.paths?

[christofluethi]

i could remove the authentication using the quarkus.http.auth.permission.* properties. they are overridable.

Question remaining is if basic auth is supported in a production env (without using keycloak) and if read access could also be configured to be unprotected.

[EricWittmann]

* Documentation "Configuring your Apicurio Registry deployment" only mentions OAuth with Keycloak. Is using Apicurio Registry with Basic Auth possible?

OK I have some relevant new information for you on these questions. Let's start with the good news.

As of earlier this week, Apicurio Registry supports both OAuth and BASIC auth at the same time. We have implemented a custom authentication handler that uses OAuth client_credentials flow to support BASIC auth. In other words, the client can provide BASIC auth credentials, and our server-side custom handler will extract the username/password and then use OAuth client_credentials flow against the Keycloak server to obtain an access token. Once that is done, everything else works the same as if you had provided an OAuth bearer token to begin with.

That feature can be enabled/disabled, but there is currently no way to configure it to be limited only to Read operations.

* Is it possible to use OAuth or Basic Auth for admin and developer purpose but leave read access unprotected?

Not at present, no. This would require some codebase changes for sure. Such a change would not be difficult. I would recommend opening a separate GH issue for that.

* Are the `quarkus.http.auth.permission.*` overridable? e.g. removing the read-apis from the `quarkus.http.auth.permission.sr-read.paths` and add them to the `quarkus.http.auth.permission.permit1.paths`?

Yes but as of version 2.1 we are no longer using the Quarkus auth permission functionality. Recently we have switched to an annotation based security model implemented by a CDI interceptor here:

https://github.com/Apicurio/apicurio-registry/blob/master/app/src/main/java/io/apicurio/registry/auth/AuthorizedInterceptor.java

[EricWittmann]

Also I should add that we have not tested any authentication providers other than Keycloak. That is on our list of things to do - we want to make sure that users aren't required to install Keycloak. I believe our implementation may be slightly KC specific right now, but actually testing it is required to find that out.

As for BASIC auth I think I answered that above but please request clarification if I didn't. :)

My question to you about read access is this: are you looking for unauthenticated users to have read access? Or are you looking to support authenticated users without any roles in the auth token? I assume it's the former, which is not currently supported and should be requested in a separate GH issue as mentioned above.

Finally, you should know that the recent security updates also introduced the ability to control access within the application (as an alternative) via a new /admin/roleMappings API. This allows roles to be assigned within the application rather than centrally managed in your IDP, if you choose. But you have to pick one approach or the other, they can't currently work at the same time.

[EricWittmann]

Note: now is a good time to request improvements to the auth layer. We can probably react to such requests quickly and then release 2.1.0.Final "soon".

[christofluethi]

first of all thanks to the whole apicurio team for the great work.

indeed, your details are good news.

My question to you about read access is this: are you looking for unauthenticated users to have read access? Or are you looking to support authenticated users without any roles in the auth token? I assume it's the former, which is not currently supported and should be requested in a separate GH issue as mentioned above.

Yes i'm looking for unauthenticated users to have read access. Using the Quarkus auth permission functionality i could easily achieve this using:

quarkus.http.auth.permission.sr-read.policy=permit
quarkus.http.auth.permission.sr-read-post.policy=permit

I'll open a GH issue for this for the new auth layer. I think best would be having a configuration to set if read access must be authenticated or not.

For the Keycloak dependency: As we do not have RBAC on artifact/subject level I usually see a lot of solutions using a proxy (or any other form of pre-authorization) infront of the schema registry handling the access control (critical is usually only write access as i'm mostly working in onprem environments). In such cases we only need one technical user which has write permissions (no further authorization in the schema registry is needed). this user would then be configured in the proxy. in such cases the dependency to keycloak seems to be overkill. This could easily be solved with having one user for writes/admin and one user for read or even unauthenticated read access.

This approach (pre-authorization) would fundamentally change if we could completely authorize at artifacts/subject level directly within the schema registry (which i think is not easy but should probably be the long-term goal).

Currently i've not found any info how to use basic auth or any other form without having to connect to a keycloak. Is that possible?

[christofluethi]

Another Q that arised. Are the changes mentioned above for the auth layer in the image quay.io/apicurio/apicurio-registry-sql:latest-snapshot? If yes, i'll give it a go. Or would you recommend another nightly/snapshot-image or self build?

[EricWittmann]

@carlesarnal Interested in your thoughts on this thread.

I have a few thoughts. If your preferred configuration is an authorization proxy, I wonder if a proxy + mTLS configuration would make sense. It's not something we've explored, but interesting.

I think the configuration you are asking for should be easy enough to implement.

Finally, yes the new security approach is included in the latest-snapshot docker image. Here are some ENV vars you can set:

`ROLE_BASED_AUTHZ_ENABLED` : "true"
`ROLE_BASED_AUTHZ_SOURCE` : "token"

If you want to see all the options (since we haven't documented this yet) go here:

https://github.com/Apicurio/apicurio-registry/blob/master/app/src/main/java/io/apicurio/registry/auth/AuthConfig.java

[EricWittmann]

As for the Keycloak-alternate question - that's something we want to support but haven't tested yet. We continue to make progress on security options and that's an area of interest for sure.

[carlesarnal]

first of all thanks to the whole apicurio team for the great work.

indeed, your details are good news.

My question to you about read access is this: are you looking for unauthenticated users to have read access? Or are you looking to support authenticated users without any roles in the auth token? I assume it's the former, which is not currently supported and should be requested in a separate GH issue as mentioned above.

Yes i'm looking for unauthenticated users to have read access. Using the Quarkus auth permission functionality i could easily achieve this using:

quarkus.http.auth.permission.sr-read.policy=permit
quarkus.http.auth.permission.sr-read-post.policy=permit

I'll open a GH issue for this for the new auth layer. I think best would be having a configuration to set if read access must be authenticated or not.

For the Keycloak dependency: As we do not have RBAC on artifact/subject level I usually see a lot of solutions using a proxy (or any other form of pre-authorization) infront of the schema registry handling the access control (critical is usually only write access as i'm mostly working in onprem environments). In such cases we only need one technical user which has write permissions (no further authorization in the schema registry is needed). this user would then be configured in the proxy. in such cases the dependency to keycloak seems to be overkill. This could easily be solved with having one user for writes/admin and one user for read or even unauthenticated read access.

This approach (pre-authorization) would fundamentally change if we could completely authorize at artifacts/subject level directly within the schema registry (which i think is not easy but should probably be the long-term goal).

Currently i've not found any info how to use basic auth or any other form without having to connect to a keycloak. Is that possible?

That's possible using Quarkus configuration properties though it's only recommended for testing purposes, see this.

In my opinion, for some use cases that are very specific the solution that makes more sense to me is the one that @christofluethi described, having a proxy in from of Apicurio Registry since that will allow users to customize the behaviour of their auth/authz layer. Even if we add the support to disable auth for reading access which, to be honest, I don't really like, security is a complex domain with very specific needs so I would prefer to keep our implementation simple and allow customization where possible.

That said, most of the requirements above can be easily achieved using a custom build, with changes to the configuration properties and using standard Quarkus HTTP permissions like, as you said quarkus.http.auth.permission.sr-read.policy=permit. This will be obviously not supported directly by us but can be seen as a workaround where the existing configuration properties doesn't met user requirements.

[christofluethi]

@carlesarnal Interested in your thoughts on this thread.

I have a few thoughts. If your preferred configuration is an authorization proxy, I wonder if a proxy + mTLS configuration would make sense. It's not something we've explored, but interesting.

I think the configuration you are asking for should be easy enough to implement.

Finally, yes the new security approach is included in the latest-snapshot docker image. Here are some ENV vars you can set:

`ROLE_BASED_AUTHZ_ENABLED` : "true"
`ROLE_BASED_AUTHZ_SOURCE` : "token"

If you want to see all the options (since we haven't documented this yet) go here:

https://github.com/Apicurio/apicurio-registry/blob/master/app/src/main/java/io/apicurio/registry/auth/AuthConfig.java

Sorry for the late reply. I got the setup working with RBAC on the UI level and BasicAuth on the API level using the latest-snapshot. Also the ability to configure the exact role names for admin, developer, readonly made the integration a lot easier.

Two concerns i have about the integration using the client credentials flow. Our KC uses federation to get the users from a central IDM, KC itself does not contain any users. However Service Accounts (client credential flow) cannot be federated. So, we end up in a mix of having the users managed by a central IDM and service accounts managed on KC itself. Thats something i dont really like. Further limiting is that we can only define one single "user" (client_id with its client_secret) which is able to login using BasicAuth. Therefore for BasicAuth-API access I'm not able to distinguish between users having Write-Access and users having Read-Access. BasicAuth is needed as most libraries including confluent do not support OAuth. Thats why I also asked for unauthenticated access for read which would partially solve that issue.

For the mTLS question: yes, mTLS would definitely be an option as a lot of kafka deployments also rely on mTLS.

About custom builds: I dont really like going productive with something which is not officially supported by apicurio. Since custom builds would also not be supported in the enterprise RH Service Registry version I currently do not consider them as an option.

So, whats currently lacking for us is the ability to distinguish between read and write access on the API/BasicAuth level. Either multiple "users" with different roles or unauthenticated read access (workaround) would solve this.

[carlesarnal]

@carlesarnal Interested in your thoughts on this thread.
I have a few thoughts. If your preferred configuration is an authorization proxy, I wonder if a proxy + mTLS configuration would make sense. It's not something we've explored, but interesting.
I think the configuration you are asking for should be easy enough to implement.
Finally, yes the new security approach is included in the latest-snapshot docker image. Here are some ENV vars you can set:

`ROLE_BASED_AUTHZ_ENABLED` : "true"
`ROLE_BASED_AUTHZ_SOURCE` : "token"

If you want to see all the options (since we haven't documented this yet) go here:
https://github.com/Apicurio/apicurio-registry/blob/master/app/src/main/java/io/apicurio/registry/auth/AuthConfig.java

Sorry for the late reply. I got the setup working with RBAC on the UI level and BasicAuth on the API level using the latest-snapshot. Also the ability to configure the exact role names for admin, developer, readonly made the integration a lot easier.

Two concerns i have about the integration using the client credentials flow. Our KC uses federation to get the users from a central IDM, KC itself does not contain any users. However Service Accounts (client credential flow) cannot be federated. So, we end up in a mix of having the users managed by a central IDM and service accounts managed on KC itself. Thats something i dont really like. Further limiting is that we can only define one single "user" (client_id with its client_secret) which is able to login using BasicAuth. Therefore for BasicAuth-API access I'm not able to distinguish between users having Write-Access and users having Read-Access. BasicAuth is needed as most libraries including confluent do not support OAuth. Thats why I also asked for unauthenticated access for read which would partially solve that issue.

I'm a bit lost with the last part about being able to create only one user. Roles should be defined at realm level, then you can create two clients, one for read access and another one for write access with sr-admin and sr-readonly roles (or the name you gave to those roles in your deployment). With that setup, one service account will be able to only read and the other one will be able to write, that's how you can distinguish between roles. I've tested this approach locally and is working fine.

For the mTLS question: yes, mTLS would definitely be an option as a lot of kafka deployments also rely on mTLS.

About custom builds: I dont really like going productive with something which is not officially supported by apicurio. Since custom builds would also not be supported in the enterprise RH Service Registry version I currently do not consider them as an option.

Yes, makes sense, just was suggesting that as a last resource workaround.

So, whats currently lacking for us is the ability to distinguish between read and write access on the API/BasicAuth level. Either multiple "users" with different roles or unauthenticated read access (workaround) would solve this.

As stated above, this can be achieved using realm-level roles and service accounts.

If you have any questions, just let me know and I will try to help.

[christofluethi]

I'm a bit lost with the last part about being able to create only one user. Roles should be defined at realm level, then you can create two clients, one for read access and another one for write access with sr-admin and sr-readonly roles (or the name you gave to those roles in your deployment). With that setup, one service account will be able to only read and the other one will be able to write, that's how you can distinguish between roles. I've tested this approach locally and is working fine.

Maybe that was a misunderstanding on my side. Whats KEYCLOAK_API_CLIENT_ID used for? My understanding was that this property is defining the client id used for the API access and therefore also limits it to one user for API access. We are mapping federated roles to the clients and are not using realm roles - that worked so far. I'll give it a try and let you know if there are any problems.

[carlesarnal]

I'm a bit lost with the last part about being able to create only one user. Roles should be defined at realm level, then you can create two clients, one for read access and another one for write access with sr-admin and sr-readonly roles (or the name you gave to those roles in your deployment). With that setup, one service account will be able to only read and the other one will be able to write, that's how you can distinguish between roles. I've tested this approach locally and is working fine.

Maybe that was a misunderstanding on my side. Whats KEYCLOAK_API_CLIENT_ID used for? My understanding was that this property is defining the client id used for the API access and therefore also limits it to one user for API access. We are mapping federated roles to the clients and are not using realm roles - that worked so far. I'll give it a try and let you know if there are any problems.

It's used by the API but it does not limit the user that you can use for basic authentication, that client id is used in OIDC to identify the application. Even if you're using federated roles, if they're present in the jwt, it should work fine. If you want, share the payload of an example jwt here and I can help to get that working.

[PlugaruT]

I hate doing this, but, is there an ETA for the new stable release with the basic auth? We would like to roll the registry in production and are using an nginx proxy with basic auth, but we would like to avoid this if the registry supports this by default.

[carlesarnal]

Hi @PlugaruT, you can expect an upstream release of 2.1.0 relatively soon, probably this week. @christofluethi my latest comments solved your issues? If so, do you think we can close this?

[PlugaruT]

@carlesarnal out of curiosity, is there a way to use basic auth for the registry but without using keycloak? I am not able to find into the code how to configure the registry to have a user and a password configured when the registry is provisioned.
Since right now I have an nginx in front of the registry that does this, maybe there is a way to have this inside the registry?
Thanks.

[carlesarnal]

Not really, that can be achieved using a custom build but it's not supported out of the box.

[carlesarnal]

That said, and although this is only recommended for testing, it can probably be achieved using regular registry images using this.

[EricWittmann]

We are in the process of testing using a non-Keycloak solution for auth (e.g. Auth0) but don't have results on that yet. The goal is hopefully to support any OpenID Connect solution. But it's unclear how easy that will be. :)

[christofluethi]

Hi @PlugaruT, you can expect an upstream release of 2.1.0 relatively soon, probably this week. @christofluethi my latest comments solved your issues? If so, do you think we can close this?

@carlesarnal I can confirm that i was able to test the basic-auth login with multiple clientids. i did not yet test different role mappings for the clients, but as it was working for one clientId i do not expect any issues here. I agree, we can close the issue.

@EricWittmann is there an issue for testing Apicurio Registry with other OIDC providers? I would like to track this as this would be highly interesting for us.

[EricWittmann]

@carlesarnal Do we have a GH issue for non-Keycloak auth server support? I thought we did but couldn't find one.

[carlesarnal]

No, I think we don't, it's for sure present in my personal Kanban board but I don't think we have a GH issue, I'll create one.

[dlydiard]

Hi, are non-Keycloak auth servers supported yet? Was a GH issue created for this?

[carlesarnal]

Yes, non-Keycloak servers are now supported. Unfortunately, there's no documentation or examples for it yet. If you're curious, I can provide some guidance.

[dlydiard]

we can wait for the docs, thank you for implementing this.

[cthtrifork]

Any news on documentation?

[j-refs]

Question. I am using this locally on my.machines terminal. I do not want any auth at all. Can I disable all auth? If not, can I disable keycloak and just use basic auth?

If I actually am required to install and setup keycloak, it would be a major deal breaker for me

[carlesarnal]

Any news on documentation?

We do have a blog a wrote a while ago about configuring Azure AD -> https://www.apicur.io/blog/2023/07/13/registry-azure-ad.

[carlesarnal]

Question. I am using this locally on my.machines terminal. I do not want any auth at all. Can I disable all auth? If not, can I disable keycloak and just use basic auth?

If I actually am required to install and setup keycloak, it would be a major deal breaker for me

Auth can be disabled (just setting AUTH_ENABLED to false), yes, but if you want basic auth you need something like Keycloak (or any other OIDC provider).