Multiple firehose providers for firehose metrics when only one Firehose provider is supplied as config

[paymog]

Do you want to request a feature or report a bug?

A bug, I think

What is the current behavior?

When I have a single Firehose provider and then curl the metrics from graph node I see output like the following:

deployment_firehose_blockstream_connect_duration{deployment="QmNntE5hyzhTCoW51UYMvmKHBdFNgXMEYs6hVg5VqFC1gS",provider="firehose-3"} 1.924322659
deployment_firehose_blockstream_connect_duration{deployment="QmNntE5hyzhTCoW51UYMvmKHBdFNgXMEYs6hVg5VqFC1gS",provider="firehose-5"} 1.825530799
deployment_firehose_blockstream_connect_duration{deployment="QmNntE5hyzhTCoW51UYMvmKHBdFNgXMEYs6hVg5VqFC1gS",provider="firehose-8"} 3.675308704
deployment_firehose_blockstream_connect_duration{deployment="QmPwuMJJLzdPfZ6NXX1JNXhFcJyTCVX4Y3uixGp9zSGAwh",provider="firehose-10"} 3.126101749
deployment_firehose_blockstream_connect_duration{deployment="QmPwuMJJLzdPfZ6NXX1JNXhFcJyTCVX4Y3uixGp9zSGAwh",provider="firehose-11"} 3.41320891
deployment_firehose_blockstream_connect_duration{deployment="QmPwuMJJLzdPfZ6NXX1JNXhFcJyTCVX4Y3uixGp9zSGAwh",provider="firehose-13"} 2.987674525
deployment_firehose_blockstream_connect_duration{deployment="QmPwuMJJLzdPfZ6NXX1JNXhFcJyTCVX4Y3uixGp9zSGAwh",provider="firehose-15"} 2.091415605
deployment_firehose_blockstream_connect_duration{deployment="QmPwuMJJLzdPfZ6NXX1JNXhFcJyTCVX4Y3uixGp9zSGAwh",provider="firehose-16"} 2.140956528
deployment_firehose_blockstream_connect_duration{deployment="QmPwuMJJLzdPfZ6NXX1JNXhFcJyTCVX4Y3uixGp9zSGAwh",provider="firehose-3"} 0.816994415
deployment_firehose_blockstream_connect_duration{deployment="QmPwuMJJLzdPfZ6NXX1JNXhFcJyTCVX4Y3uixGp9zSGAwh",provider="firehose-4"} 1.938940324
deployment_firehose_blockstream_connect_duration{deployment="QmPwuMJJLzdPfZ6NXX1JNXhFcJyTCVX4Y3uixGp9zSGAwh",provider="firehose-5"} 1.812108187
deployment_firehose_blockstream_connect_duration{deployment="QmPwuMJJLzdPfZ6NXX1JNXhFcJyTCVX4Y3uixGp9zSGAwh",provider="firehose-8"} 2.500762376
deployment_firehose_blockstream_connect_duration{deployment="QmQ9fGL2DKBYWgFbNVQicw1ADSdq9kR3MYrFRsKVBsYEmu",provider="firehose-1"} 1.657005793
deployment_firehose_blockstream_connect_duration{deployment="QmR2qvm2Vhhjfd65g16CQYbYgLdNHfuSokW7FePj8j7bWq",provider="firehose-14"} 2.163223028
deployment_firehose_blockstream_connect_duration{deployment="QmR2qvm2Vhhjfd65g16CQYbYgLdNHfuSokW7FePj8j7bWq",provider="firehose-15"} 3.48737248
deployment_firehose_blockstream_connect_duration{deployment="QmR2qvm2Vhhjfd65g16CQYbYgLdNHfuSokW7FePj8j7bWq",provider="firehose-17"} 2.954491939
deployment_firehose_blockstream_connect_duration{deployment="QmR2qvm2Vhhjfd65g16CQYbYgLdNHfuSokW7FePj8j7bWq",provider="firehose-3"} 3.078251135
deployment_firehose_blockstream_connect_duration{deployment="QmR2qvm2Vhhjfd65g16CQYbYgLdNHfuSokW7FePj8j7bWq",provider="firehose-4"} 1.7710952180000001
deployment_firehose_blockstream_connect_duration{deployment="QmR2qvm2Vhhjfd65g16CQYbYgLdNHfuSokW7FePj8j7bWq",provider="firehose-8"} 1.9191417720000001
deployment_firehose_blockstream_connect_duration{deployment="QmRzkv1vWnxWaqEZzo5rNr2nTput4mx5y3QqKxWQz7cSGR",provider="firehose-10"} 1.766222365
deployment_firehose_blockstream_connect_duration{deployment="QmRzkv1vWnxWaqEZzo5rNr2nTput4mx5y3QqKxWQz7cSGR",provider="firehose-11"} 2.069972008
deployment_firehose_blockstream_connect_duration{deployment="QmRzkv1vWnxWaqEZzo5rNr2nTput4mx5y3QqKxWQz7cSGR",provider="firehose-16"} 2.951801974
deployment_firehose_blockstream_connect_duration{deployment="QmRzkv1vWnxWaqEZzo5rNr2nTput4mx5y3QqKxWQz7cSGR",provider="firehose-18"} 2.151872044

Note that the same deployment ID appears with many different firehose providers even though I only have a single provider. Why is that?

If the current behavior is a bug, please provide the steps to reproduce and if possible a minimal demo of the problem.

1. Launch a graph node with a single firehose provider
2. deploy a subgraph against this provider
3. wait a bit
4. curl the metrics endpoint

What is the expected behavior?

I expect the firehose related metrics to have only a single provider value when the graph node config only has 1 firehose provider.

[github-actions]

Looks like this issue has been open for 6 months with no activity. Is it still relevant? If not, please remember to close it.

[paymog]

Likely will be resolved by #4778. Is that true @mangas?

[mangas]

This is still going to happen, this happens because these connections have a limit of streams they can handle so we use a connection pool of 20 by default. If this is a problem for some reason and you want to disable the behaviour you can set conn_pool_size of the provide to 1

[mangas]

@paymog you can see more details on #3879

[paymog]

you want to disable the behaviour you can set conn_pool_size of the provide to 1

that would be problematic if there were more than 100 subgraphs deployed into graph node, right?

I personally don't find the numerically suffixed provider tag useful. There's no way for me to act on a misbehaving connection in the pool. I'm not sure how others feel but I personally would find it more useful if the provider tag removed the numeric suffix and just had the name of the provider.

[github-actions]

Looks like this issue has been open for 6 months with no activity. Is it still relevant? If not, please remember to close it.