Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CCR exception on retention lease renewal #61308

Open
ywangd opened this issue Aug 19, 2020 · 3 comments
Open

CCR exception on retention lease renewal #61308

ywangd opened this issue Aug 19, 2020 · 3 comments
Labels
>bug :Distributed/CCR Issues around the Cross Cluster State Replication features :Security/Authorization Roles, Privileges, DLS/FLS, RBAC/ABAC Team:Distributed Meta label for distributed team Team:Security Meta label for security team

Comments

@ywangd
Copy link
Member

ywangd commented Aug 19, 2020

Based on the doc, the CCR on the remote/leader cluster can be configured using an user with a role like the follows:

ccr_user:
  cluster:
    - read_ccr
  indices:
    - names: [ 'leader-index' ]
      privileges:
        - monitor
        - read

However, an exception about renewing retention lease will be thrown due to missing of indices:admin/seq_no/renew_retention_lease privilege. This behaviour seems to exist for a long while now (I tested with v7.6.2 to v7.9). I am surprised that no one has reported it. Given the retention lease renewal seems to be an implementation details, the potential solution is probably executing it under system context.

Here is the stacktrace shown in the log of the local/follower cluster (the exception actually happens on the remote/leader cluster, but it gets passed across the wire and only shows up in the local/follower cluster).

[2020-08-19T12:21:38,278][WARN ][o.e.x.c.a.ShardFollowTasksExecutor] [node-9] [follower-index][0] background management of retention lease [local/follower-index/00R3HMB3RYynV2qZcWHIkw-following-leader/leader-index/4IDla51RR724KrVQqcFZiA] failed while following
org.elasticsearch.ElasticsearchSecurityException: action [indices:admin/seq_no/renew_retention_lease] is unauthorized for user [ccr_user]
        at org.elasticsearch.xpack.core.security.support.Exceptions.authorizationError(Exceptions.java:34) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService.denialException(AuthorizationService.java:601) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService.access$300(AuthorizationService.java:95) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.handleFailure(AuthorizationService.java:648) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:634) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:604) ~[?:?]
        at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43) ~[elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.xpack.security.authz.RBACEngine.lambda$authorizeIndexAction$4(RBACEngine.java:326) ~[?:?]
        at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) ~[elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexActionName(RBACEngine.java:339) ~[?:?]
        at org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexAction(RBACEngine.java:310) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeAction(AuthorizationService.java:270) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService.maybeAuthorizeRunAs(AuthorizationService.java:235) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorize$1(AuthorizationService.java:200) ~[?:?]
        at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) ~[elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43) ~[elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.xpack.security.authz.RBACEngine.lambda$resolveAuthorizationInfo$1(RBACEngine.java:123) ~[?:?]
        at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) ~[elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.roles(CompositeRolesStore.java:159) ~[?:?]
        at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRoles(CompositeRolesStore.java:276) ~[?:?]
        at org.elasticsearch.xpack.security.authz.RBACEngine.getRoles(RBACEngine.java:129) ~[?:?]
        at org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizationInfo(RBACEngine.java:117) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:202) ~[?:?]
        at org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.lambda$inbound$1(ServerTransportFilter.java:129) ~[?:?]
        at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) ~[elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:323) ~[?:?]
        at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$6(AuthenticationService.java:384) ~[?:?]
        at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:395) ~[?:?]
        at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:320) ~[?:?]
        at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:261) ~[?:?]
        at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:173) ~[?:?]
        at org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.inbound(ServerTransportFilter.java:120) ~[?:?]
        at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:313) ~[?:?]
        at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) [elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) [elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) [elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) [elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) [elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) [elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) [elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) [transport-netty4-client-7.9.0.jar:7.9.0]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) [netty-handler-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.49.Final.jar:4.1.49.Final]
        at java.lang.Thread.run(Thread.java:832) [?:?]
@ywangd ywangd added >bug :Distributed/CCR Issues around the Cross Cluster State Replication features :Security/Authorization Roles, Privileges, DLS/FLS, RBAC/ABAC labels Aug 19, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-distributed (:Distributed/CCR)

@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-security (:Security/Authorization)

@elasticmachine elasticmachine added Team:Distributed Meta label for distributed team Team:Security Meta label for security team labels Aug 19, 2020
@ywangd ywangd mentioned this issue Aug 19, 2020
Closed
@ywangd ywangd mentioned this issue Sep 13, 2021
Merged
@ywangd ywangd mentioned this issue Feb 16, 2022
Closed
@ywangd ywangd mentioned this issue Feb 28, 2022
Closed
DaveCTurner added a commit to DaveCTurner/elasticsearch that referenced this issue Feb 28, 2022
@DaveCTurner
Drop bogus markAsSystemContext call in CCR
493a066 
Today the `ShardFollowTasksExecutor` enters system context before
renewing a retention lease, but then makes the remote call using a
client which replaces the thread context with a non-system one again.

This commit removes this no-op code to clarify the security model in
this area.

Relates elastic#61308, elastic#84006, elastic#84156
@DaveCTurner DaveCTurner mentioned this issue Feb 28, 2022
Merged
ywangd added a commit to ywangd/elasticsearch that referenced this issue Mar 1, 2022
@ywangd
[Test] Fix FollowIndexSecurityIT by granting needed previleges
76aff0c 
CCR user on the leader cluster needs more privileges than what are
documented (elastic#61308). Specifically it needs to renew the retention lease
at a fixed time interval. This PR fixes it by granting the "manage"
index privilege to the CCR user on the leader cluster.

Note we still want to revisit privileges required CCR or at least fix
our documentation. This will be tracked with elastic#61308.

Resolves: elastic#84156
@ywangd ywangd mentioned this issue Mar 1, 2022
Merged
@ywangd
Copy link
Member Author

ywangd commented Mar 1, 2022

We discussed this issue recently and considered it a bug that should be fixed in phases:

  1. Short term: Fix the documentation to explicitly state that the manage index privilege is required on the leader cluster.
  2. Long term: Revisit all privileges required for CCR and take future direction of CCR/CCS into consideration.

ywangd added a commit that referenced this issue Mar 1, 2022
@ywangd
[Test] Fix FollowIndexSecurityIT by granting needed previleges (#84467)
e1202dd 
CCR user on the leader cluster needs more privileges than what are
documented (#61308). Specifically it needs to renew the retention lease
at a fixed time interval. This PR fixes it by granting the "manage"
index privilege to the CCR user on the leader cluster.

Note we still want to revisit privileges required CCR or at least fix
our documentation. This will be tracked with #61308.

Resolves: #84156
ywangd added a commit to ywangd/elasticsearch that referenced this issue Mar 1, 2022
@ywangd
[Test] Fix FollowIndexSecurityIT by granting needed previleges (elast…
588fd29 
…ic#84467)

CCR user on the leader cluster needs more privileges than what are
documented (elastic#61308). Specifically it needs to renew the retention lease
at a fixed time interval. This PR fixes it by granting the "manage"
index privilege to the CCR user on the leader cluster.

Note we still want to revisit privileges required CCR or at least fix
our documentation. This will be tracked with elastic#61308.

Resolves: elastic#84156
elasticsearchmachine pushed a commit that referenced this issue Mar 1, 2022
@ywangd @elasticmachine
[Test] Fix FollowIndexSecurityIT by granting needed previleges (#84467)…
f37bdc7 
… (#84471)

CCR user on the leader cluster needs more privileges than what are
documented (#61308). Specifically it needs to renew the retention lease
at a fixed time interval. This PR fixes it by granting the "manage"
index privilege to the CCR user on the leader cluster.

Note we still want to revisit privileges required CCR or at least fix
our documentation. This will be tracked with #61308.

Resolves: #84156

Co-authored-by: Elastic Machine <[email protected]>
@ywangd ywangd mentioned this issue Mar 2, 2022
Closed
DaveCTurner added a commit that referenced this issue Mar 3, 2022
@DaveCTurner
Drop bogus markAsSystemContext call in CCR (#84409)
9d57899 
Today the `ShardFollowTasksExecutor` enters system context before
renewing a retention lease, but then makes the remote call using a
client which replaces the thread context with a non-system one again.

This commit removes this no-op code to clarify the security model in
this area.

Relates #61308, #84006, #84156
ywangd added a commit to ywangd/elasticsearch that referenced this issue Mar 31, 2022
@ywangd
[Test] Fix FollowIndexSecurityIT by granting needed previleges (elast…
325ef25 
…ic#84467)

CCR user on the leader cluster needs more privileges than what are
documented (elastic#61308). Specifically it needs to renew the retention lease
at a fixed time interval. This PR fixes it by granting the "manage"
index privilege to the CCR user on the leader cluster.

Note we still want to revisit privileges required CCR or at least fix
our documentation. This will be tracked with elastic#61308.

Resolves: elastic#84156
@ywangd ywangd mentioned this issue Mar 31, 2022
Merged
elasticsearchmachine pushed a commit that referenced this issue Mar 31, 2022
@ywangd
[Test] Fix FollowIndexSecurityIT by granting needed previleges (#84467)…
debaab0 
… (#85514)

CCR user on the leader cluster needs more privileges than what are
documented (#61308). Specifically it needs to renew the retention lease
at a fixed time interval. This PR fixes it by granting the "manage"
index privilege to the CCR user on the leader cluster.

Note we still want to revisit privileges required CCR or at least fix
our documentation. This will be tracked with #61308.

Resolves: #84156
@DaveCTurner DaveCTurner mentioned this issue Aug 3, 2022
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
>bug :Distributed/CCR Issues around the Cross Cluster State Replication features :Security/Authorization Roles, Privileges, DLS/FLS, RBAC/ABAC Team:Distributed Meta label for distributed team Team:Security Meta label for security team
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@ywangd @elasticmachine