[BUG] - conda-store-server worker detection does not work when scaling api servers

[soapy1]

Describe the bug

When the conda-store server is starting up, it waits for a worker to become available. It outputs the following message when it's waiting for a worker and then one is found.

2024-10-17 23:07:56   WARNING CondaStoreServer: 379: Waiting for worker... Use --standalone if running outside of docker
2024-10-17 23:07:56      INFO CondaStoreServer: 373: Worker initialized

However, if there is a deployed worker first, then a server is started up, the worker never gets detected and the user is shown the warning message:

...
2024-10-17 23:19:20   WARNING CondaStoreServer: 379: Waiting for worker... Use --standalone if running outside of docker
2024-10-17 23:19:35   WARNING CondaStoreServer: 379: Waiting for worker... Use --standalone if running outside of docker
2024-10-17 23:19:40   WARNING CondaStoreServer: 379: Waiting for worker... Use --standalone if running outside of docker
...

Note, that when this happens, the server is still functional and can submit tasks to workers, and the available workers can pickup tasks.

Expected behavior

Users should not receive a message WARNING CondaStoreServer: 379: Waiting for worker... Use --standalone if running outside of docker when workers are available.

How to Reproduce the problem?

1. launch a conda-store worker
2. launch a conda-store server
3. view the logs for conda-store server, noting the warning message

For example, if using kubernetes:

1. confirm all pods are up after setting up the cluster (kubectl get po)
2. scale down the replicas for the server deployment

$ kubectl scale  deployment/conda-store-server --replicas=0

3. scale up the replicas

$ kubectl scale  deployment/conda-store-server --replicas=1

5. view the logs

$ kubectl logs deployment/conda-store-server

...
2024-10-17 23:19:30   WARNING CondaStoreServer: 379: Waiting for worker... Use --standalone if running outside of docker
2024-10-17 23:19:35   WARNING CondaStoreServer: 379: Waiting for worker... Use --standalone if running outside of docker
2024-10-17 23:19:40   WARNING CondaStoreServer: 379: Waiting for worker... Use --standalone if running outside of docker

Output

No response

Versions and dependencies used.

conda-store - main branch

Anything else?

Looks like this functionality was introduced #705, to resolve #605.

[peytondmurray]

Will investigate to figure out what the best path forward is.

[soapy1]

This happens because as part of the server determines if a worker is available by querying the workers table in the database. But the server start up process involves clearing all entries in the workers table. So, if a worker comes up before the server, the server will never "know" that a worker is available.

Here is the snippet that clears the table https://github.com/conda-incubator/conda-store/blob/main/conda-store-server/conda_store_server/_internal/server/app.py#L394-L413

I think this isn't the best pattern. Like, the server should not need to know if a worker is available. I think the best step forward is to remove this functionality and remove the workers table in the database.