Load plugins concurrently

[findepi]

This adds an opt-in capability to start server concurrent. When enabled,
plugins are loaded concurrently.

In local testing, this improves DevelopmentServer startup time from
about 88s to about 38s (actual times very much depend on multiple
factors).

As a follow-up work, catalogs can be made to load concurrently.

[kokosing]

In local testing, this changes DevelopmentServer startup time from about 62s to about 33s.

Is it possible that having only 2 threads would get same results, so having Runtime.getAvailableProcessors() is too much. Do you know where where we spend time? Is it IO (class loading)?

[kokosing]

How faster it become in produce tests?

[findepi]

i didn't measure this.

[findepi]

Is it possible that having only 2 threads would get same results,

it is. I would prefer to leave fine-tunings and for the future.
I want to focus on the basic fact that there is quite some work to be done during startup, and the work is parallelizable.
For plugin loading (poking into jars to uncompress set of initially required classes) maximum parallelism possible may depend on CPU vs disk, so my local results may not match exactly yours.

[wendigo]

@findepi I'm worried about logging during startup. Are logs intertwined right now?

[findepi]

Are logs intertwined right now?

absolutely.

[wendigo]

@findepi Maybe we could defer dumping configuration after plugin/catalogs are loaded and server is started? I think that flushing output takes some portion of that 30s. Alternatively we could buffer logs and flush them at once after catalog is loaded. WDYT?

[wendigo]

presto-master       | 2021-10-06T13:50:01.349+0545	ERROR	main	io.trino.server.Server	java.io.FileNotFoundException: etc/catalog/kafka (Is a directory)
presto-master       | java.lang.RuntimeException: java.io.FileNotFoundException: etc/catalog/kafka (Is a directory)
presto-master       | 	at io.airlift.concurrent.MoreFutures.getFutureValue(MoreFutures.java:202)
presto-master       | 	at io.airlift.concurrent.MoreFutures.getFutureValue(MoreFutures.java:176)
presto-master       | 	at io.airlift.concurrent.MoreFutures.getDone(MoreFutures.java:278)
presto-master       | 	at io.trino.util.Executors.executeUntilFailure(Executors.java:40)
presto-master       | 	at io.trino.metadata.StaticCatalogStore.loadCatalogs(StaticCatalogStore.java:74)
presto-master       | 	at io.trino.server.Server.doStart(Server.java:125)
presto-master       | 	at io.trino.server.Server.lambda$start$0(Server.java:78)
presto-master       | 	at io.trino.$gen.Trino_fd69855____20211006_080434_1.run(Unknown Source)
presto-master       | 	at io.trino.server.Server.start(Server.java:78)
presto-master       | 	at io.trino.server.TrinoServer.main(TrinoServer.java:38)
presto-master       | Caused by: java.io.FileNotFoundException: etc/catalog/kafka (Is a directory)

[findepi]

I think that flushing output takes some portion of that 30s. Alternatively we could buffer logs and flush them at once after catalog is loaded. WDYT?

there is not a ton of logs, so logging overhead is negligible.
buffering logs may have the inadvertent effect that logs are not available at the point when one needs them

[findepi]

per conversation with @kokosing i dropped catalog parallel loading.

Also @martint told me he has a more elaborate version of catalog parallel loading.
My version wasn't very concurrent, since io.trino.connector.ConnectorManager#createCatalog(java.lang.String, java.lang.String, java.util.Map<java.lang.String,java.lang.String>) is synchronized.

@kokosing ptal