"Error response from daemon: Error response from daemon: plugin "weave" not found"

[bklau]

Symptoms:

Currently, Weave must be installed on Swarm nodes(workers and masters) to avoid getting sporadic error
"Error response from daemon: Error response from daemon: plugin "weave" not found"

Note: I'm using Swarm classic(Standalone) with external KV store(Consul).

Cause:

It appears that Swarm is using some internal criteria to select a Docker engine to create an overlay network before replicating the info to rest of remaining Docker engines(Swarm nodes). This node selection is non-deterministic. If a selected Swarm node(Docker engine) do not have Weave installed, then we'll get this error.

Code:

See cluster.CreateNetwork method:
https://sourcegraph.com/github.com/docker/swarm@f5150c7f1d398a4fe5530cc42eb77a45887efd81/-/blob/cluster/swarm/cluster.go#L72

Issue:

The current foolproof way is to install Weave on all Swarm nodes(workers and masters) but --conn-limit cannot take large values.
(1). How can we influenced Swarm to create an initial overlay only on Swarm nodes that have Weave installed on them?
(2) If we cannot do #1 and need Weave on all Swarm nodes, For example, if I have 1000 Swarm nodes, it might be smart for weave connect able to prune down network overlay mesh connections since Weave can use transitive "hopping" to route the traffics and due to --conn-limit