Dynamic Catalogs - Sync Catalog Properties to Worker Node

[yeahyung]

I got below error when using dynamic catalog features(#12709) and this is likely caused by the worker node not knowing the newly added Catalog Properties.

• it works on a single node cluster(or single worker node) but not works on multiple worker nodes.
• it works after desc table command which causes updates catalog properties in worker node
• same issue with Unknown handle id: hive:hive:XXX:io.trino.plugin.hive.HiveTableHandle when using dynamic catalogs #18040

trino> select * from hive_1.default.movie;
Query 20230608_092704_00109_5c4if failed: Unexpected response from http://10.174.65.xx:8080/v1/task/20230608_092704_00109_5c4if.0.0.0?summarize

2023-06-08T18:06:49.700+0900	ERROR	remote-task-callback-2	io.trino.execution.scheduler.PipelinedStageExecution	Pipelined stage execution for stage 20230608_090649_00047_5c4if.0 failed
io.trino.spi.TrinoException: Unexpected response from http://10.174.78.xx:8080/v1/task/20230608_090649_00047_5c4if.0.1.0?summarize
	at io.trino.server.remotetask.SimpleHttpResponseHandler.onSuccess(SimpleHttpResponseHandler.java:70)
	at io.trino.server.remotetask.SimpleHttpResponseHandler.onSuccess(SimpleHttpResponseHandler.java:27)
	at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1138)
	at io.airlift.concurrent.BoundedExecutor.drainQueue(BoundedExecutor.java:80)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	
Caused by: java.lang.IllegalArgumentException: Invalid JSON bytes for [simple type, class io.trino.execution.TaskInfo]
	at io.airlift.json.JsonCodec.fromJson(JsonCodec.java:196)
	at io.airlift.http.client.FullJsonResponseHandler$JsonResponse.<init>(FullJsonResponseHandler.java:105)
	... 29 more
Caused by: com.fasterxml.jackson.core.JsonParseException: Unrecognized token 'io': was expecting (JSON String, Number, Array, Object or token 'null', 'true' or 'false')
 at [Source: (byte[])"io.airlift.jaxrs.JsonMapperParsingException: Invalid json for Java type io.trino.server.TaskUpdateRequest
\u0009at io.airlift.jaxrs.AbstractJacksonMapper.readFrom(AbstractJacksonMapper.java:124)
\u0009at io.airlift.jaxrs.JsonMapper.readFrom(JsonMapper.java:44)
\u0009at org.glassfish.jersey.message.internal.ReaderInterceptorExecutor$TerminalReaderInterceptor.invokeReadFrom(ReaderInterceptorExecutor.java:233)
\u0009at org.glassfish.jersey.message.internal.ReaderInterceptorExecutor$TerminalReaderInterceptor.aroundRead"[truncated 13291 bytes]; line: 1, column: 4]
	at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:2418)
	at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:759)

---

Todo

• Make Worker node to create catalog properties to file system
• Make Worker node to load catalogs when initializing(restart trino)

[xuhaiL]

i need this feature，it can be used？

[dain]

This should be fixed in 435. See #18040 for more details