Suggestions : Hive Metastore Delegation-Token Authentication Failure

[Sunwoo-Shin]

Situation

I figured out that the delegation-token of the hive metastore server (HMS) is created and the cached token is implemented to continuously access until the time of the hive.metastore.thrift.delegation-token.cache-ttl option among the hive connecter options.
​
As a result, when the hive metastore server (HMS) is restarted, it was confirmed that an authentication error occurred because trino continuously accesses the cached token.

When an error occurs, the following error occurs.

Error Message

error code : https://github.com/trinodb/trino/blob/master/plugin/trino-hive/src/main/java/io/trino/plugin/hive/metastore/thrift/ThriftHiveMetastore.java#L2051

io.trino.spi.TrinoException: Failed connecting to Hive metastore: [a-server:9083, b-server:9083, c-server:9083]
	at io.trino.plugin.hive.metastore.thrift.ThriftHiveMetastore.getTable(ThriftHiveMetastore.java:357)
	at io.trino.plugin.hive.metastore.thrift.BridgingHiveMetastore.getTable(BridgingHiveMetastore.java:96)
	at io.trino.plugin.hive.metastore.cache.CachingHiveMetastore.loadTable(CachingHiveMetastore.java:322)
	at com.google.common.cache.CacheLoader$FunctionToCacheLoader.load(CacheLoader.java:165)
	at com.google.common.cache.CacheLoader$1.load(CacheLoader.java:188)
	at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3529)
	at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2278)
	at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2155)
	at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2045)
	at com.google.common.cache.LocalCache.get(LocalCache.java:3951)
	at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3974)
	at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4935)
	at com.google.common.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache.java:4941)
	at io.trino.plugin.hive.metastore.cache.CachingHiveMetastore.get(CachingHiveMetastore.java:259)
	at io.trino.plugin.hive.metastore.cache.CachingHiveMetastore.getTable(CachingHiveMetastore.java:311)
	at io.trino.plugin.hive.HiveMetastoreClosure.getTable(HiveMetastoreClosure.java:76)
	at io.trino.plugin.hive.metastore.SemiTransactionalHiveMetastore.getTable(SemiTransactionalHiveMetastore.java:206)
	at io.trino.plugin.hive.HiveMetadata.getView(HiveMetadata.java:2040)
	at io.trino.plugin.base.classloader.ClassLoaderSafeConnectorMetadata.getView(ClassLoaderSafeConnectorMetadata.java:535)
	at io.trino.metadata.MetadataManager.getView(MetadataManager.java:1132)
	at io.trino.sql.analyzer.StatementAnalyzer$Visitor.visitTable(StatementAnalyzer.java:1229)
	at io.trino.sql.analyzer.StatementAnalyzer$Visitor.visitTable(StatementAnalyzer.java:343)
	at io.trino.sql.tree.Table.accept(Table.java:53)
	at io.trino.sql.tree.AstVisitor.process(AstVisitor.java:27)
	at io.trino.sql.analyzer.StatementAnalyzer$Visitor.process(StatementAnalyzer.java:360)
	at io.trino.sql.analyzer.StatementAnalyzer$Visitor.analyzeFrom(StatementAnalyzer.java:2770)
	at io.trino.sql.analyzer.StatementAnalyzer$Visitor.visitQuerySpecification(StatementAnalyzer.java:1589)
	at io.trino.sql.analyzer.StatementAnalyzer$Visitor.visitQuerySpecification(StatementAnalyzer.java:343)
	at io.trino.sql.tree.QuerySpecification.accept(QuerySpecification.java:155)
	at io.trino.sql.tree.AstVisitor.process(AstVisitor.java:27)
	at io.trino.sql.analyzer.StatementAnalyzer$Visitor.process(StatementAnalyzer.java:360)
	at io.trino.sql.analyzer.StatementAnalyzer$Visitor.process(StatementAnalyzer.java:370)
	at io.trino.sql.analyzer.StatementAnalyzer$Visitor.visitQuery(StatementAnalyzer.java:1075)
	at io.trino.sql.analyzer.StatementAnalyzer$Visitor.visitQuery(StatementAnalyzer.java:343)
	at io.trino.sql.tree.Query.accept(Query.java:107)
	at io.trino.sql.tree.AstVisitor.process(AstVisitor.java:27)
	at io.trino.sql.analyzer.StatementAnalyzer$Visitor.process(StatementAnalyzer.java:360)
	at io.trino.sql.analyzer.StatementAnalyzer.analyze(StatementAnalyzer.java:323)
	at io.trino.sql.analyzer.Analyzer.analyze(Analyzer.java:91)
	at io.trino.sql.analyzer.Analyzer.analyze(Analyzer.java:83)
	at io.trino.execution.SqlQueryExecution.analyze(SqlQueryExecution.java:263)
	at io.trino.execution.SqlQueryExecution.<init>(SqlQueryExecution.java:186)
	at io.trino.execution.SqlQueryExecution$SqlQueryExecutionFactory.createQueryExecution(SqlQueryExecution.java:768)
	at io.trino.dispatcher.LocalDispatchQueryFactory.lambda$createDispatchQuery$0(LocalDispatchQueryFactory.java:129)
	at io.trino.$gen.Trino_353____20210517_084257_2.call(Unknown Source)
	at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
	at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69)
	at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
	at java.base/java.lang.Thread.run(Thread.java:832)

Suggestions

Since cached tokens may expire, such as restarting Hive Metastore, I think it would be better to retry by renewing the token in case of access failure with the cached token.

So I suggest modifying the logic as below.

    private ThriftMetastoreClient createMetastoreClient(HiveIdentity identity)
            throws TException
    {
        if (!impersonationEnabled) {
            return createMetastoreClient();
        }

        String username = identity.getUsername().orElseThrow(() -> new IllegalStateException("End-user name should exist when metastore impersonation is enabled"));
        if (authenticationEnabled) {
            try {
                return createMetastoreClientWithAuthentication(username, false);
            }
            catch (TException e) {
                log.debug("It may be a delegation token permission failure, so try again with a new delegation token, exception message : %s", e.getMessage());
                return createMetastoreClientWithAuthentication(username, true);
            }
        }

        ThriftMetastoreClient client = createMetastoreClient();
        setMetastoreUserOrClose(client, username);
        return client;
    }

    private ThriftMetastoreClient createMetastoreClientWithAuthentication(String username, boolean refresh)
            throws TException
    {
        if (refresh) {
            delegationTokenCache.refresh(username);
        }
        String delegationToken;
        try {
            delegationToken = delegationTokenCache.getUnchecked(username);
        }
        catch (UncheckedExecutionException e) {
            throwIfInstanceOf(e.getCause(), TrinoException.class);
            throw e;
        }
        return clientProvider.createMetastoreClient(Optional.of(delegationToken));
    }

[Sunwoo-Shin]

Here is my pr of fixing it, #10561