after 'swarm init', unable to create network: API error (500): invalid ipam driver: "weavemesh"

[nkratzke]

I tried to start weave alongside docker swarm (1.12 rc2).
My weave version:

• weave script 1.6.0
• weave router 1.6.0
• weave proxy 1.6.0
• weave plugin 1.6.0

So after

sudo docker swarm init
sudo weave launch

I got the following error:

unable to create network: API error (500): invalid ipam driver: "weavemesh"

Network creation is working (partly), however there is no driver associated with the created network

sudo docker network create mynet -d weave
sudo docker network ls

is returning

NETWORK ID          NAME                DRIVER              SCOPE
5e4569ae5444        bridge              bridge              local               
163fffb4ac99        docker_gwbridge     bridge              local               
8a14ac628cd9        host                host                local               
6wpe2ly19hd1        ingress             overlay             swarm               
9pvzrk2nwlgf        mynet                                   swarm  <--- STRANGE ??? !!!             
3cb9c715bf07        none                null                local  

[rade]

Are you using docker-for-mac?

[nkratzke]

No, this happened on a Ubuntu 14.04 machine.

[rade]

Reproduced.

Docker have said that plugins aren't supported in swarm mode.

[rade]

btw, I've not seen the error on a few occasions, so there may well be a race condition somewhere too.

[rade]

This situation persists even after a docker swarm leave --force, though subsequently restarting docker clears that.

[nkratzke]

So, because Docker is not supporting plugins in swarm mode, weave can not be used with Docker swarm after Docker 1.12? Is that correct?

[rade]

That is my understanding.

[rade]

This is bizarre:

$ docker swarm init
Swarm initialized: current node (2d8gdiju4rxeztsr63y4qhfcf) is now a manager.
$ weave launch
unable to create network: API error (500): invalid ipam driver: "weavemesh"
$ weave reset
$ docker swarm leave --force
Node left the default swarm.

$ sudo service docker restart

$ weave launch
$ weave reset
$ docker swarm init
Swarm initialized: current node (dbym852em3zul1tgcq2ul0qdn) is now a manager.
$ weave launch

I've run through the above sequence a few times, with the same result.

So for some reason launching weave first, resetting it, then initialising swarm, and then re-launching weave makes the error go away. And following that...

$ docker run --rm --net=weave alpine /bin/true
$ docker network create mynet -d weave
c3yxfjg0w3dht86gbyhg6fqxr
$ docker run --rm --net=mynet alpine /bin/true
docker: Error response from daemon: network mynet not found.
See 'docker run --help'.

So the default 'weave' network can be used, and new 'weave' driver networks can be created, but the latter cannont be used.

[nkratzke]

I forwarded this issue as a Docker ticket.
moby/moby#23990

Maybe that helps ...

[mrjana]

Docker have said that plugins aren't supported in swarm mode.

@rade, small clarification here. Only global scope plugins are not supported at this point in time when swarm mode is enabled. If the plugin is local scope they are expected to continue to work even if swarm mode is enabled. If not, then that is a release gating bug which should be fixed before 1.12 release

[mavenugo]

@rade @mrjana looking at the weave launch code, it seems the weave driver is launched as a global driver : https://github.com/weaveworks/weave/blob/master/prog/plugin/main.go#L91 . Is that correct ?

As per this issue docker network create mynet -d weave the created network mynet uses the weave driver (not weavemesh). Hence, the comment above (moby/libnetwork#1296 (comment)) is applicable in this case.

If this infact is the issue, then a recent fix (which will be available in docker 1.12 RC3), will explicitly block such network creation at swarmkit via moby/swarmkit#1078.

Can you please confirm this and maybe it is NOT exactly the same as represented by @bboreham 's comment in moby/moby#23990 (comment)

[bboreham]

@mavenugo it's true, we can run the Weave Network plugin as either a global or a local plugin.

However the first error @nkratzke reports:

unable to create network: API error (500): invalid ipam driver: "weavemesh"

is using the local plugin, and my moby/moby#23990 (comment) is recreating the specific command run inside the weave script to generate that message.

[mavenugo]

@bboreham I think there is a confusion on the bug report.

1. sudo docker network create mynet -d weave is clearly using weave global driver.
2. 9pvzrk2nwlgf mynet swarm <--- STRANGE ??? !!! clearly indicates that it is creating a global-scoped network using the weave global driver.

This is absolutely related to the swarm init case and it is not currently supported.

But the error unable to create network: API error (500): invalid ipam driver: "weavemesh" is not related to swarm-mode at all. It should be seen in non-swarm mode as well. Can you please confirm that ?

[mavenugo]

@bboreham I think there is infact an issue with swarmkit and local ipam remote driver. I will see what we can do about that.

[bboreham]

@mavenugo I can confirm that the command to recreate the original error from @nkratzke is:

docker network create --driver=weavemesh --ipam-driver=weavemesh foo

and this is using a local-scope driver. @nkratzke and @rade went on to run some other commands, which generated other errors.

It should be seen in non-swarm mode as well. Can you please confirm that ?

No, we do not see that.

$ docker info
Containers: 8
 Running: 3
 Paused: 0
 Stopped: 5
Images: 53
Server Version: 1.12.0-rc2
Storage Driver: overlay
 Backing Filesystem: extfs
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: overlay bridge weavemesh host null
Swarm: inactive
Runtimes: default
Default Runtime: default
Security Options: apparmor seccomp
Kernel Version: 4.2.0-35-generic
Operating System: Ubuntu 15.10
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 362.9 MiB
Name: host1
ID: WSO5:2WPX:NINR:DJJJ:U5VI:VZRJ:KBFJ:JCSE:VQGB:Y5WX:AOMI:24L6
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Insecure Registries:
 127.0.0.0/8
$ docker network create --driver=weavemesh --ipam-driver=weavemesh foo2
b0d01a7fc1ddbbd34ee16fa52b9a9d458b5c720f1663d75a590f3d995d0bcf2b

[mavenugo]

@bboreham can you please give moby/moby#24172 a try ?

[bboreham]

@mavenugo I pulled your branch, compiled Docker, installed on a VM and re-tried the docker swarm init ... weave launch as at the top of this issue; all seemed to go fine. Thanks!

[rade]

I suggest we close this once the fix has made it into a Docker RC or actual release.

[rade]

I have just tested the Docker 1.12.0-rc4 and it all seems to go fine. -> closing