Functions with same name in different stacks collide

[dschulten]

Expected Behaviour

It should be possible to deploy functions independently into two separate openfaas stacks in the same swarm while avoiding name collisions.

Current Behaviour

I have two openfaas gateways in two different docker swarm openfaas stacks in the same swarm. When I deploy a function into the gateway of the first stack, a function service gets created for that gateway and works fine. But when I deploy the same function (same name) into the other gateway, then the first gateway loses the connection to that function. Redeploying the function into the first gateway does not help. In the gateway logs I see:

2019/04/16 09:20:10 error with upstream request to: , Post http://my-function:8080: dial tcp: lookup my-function on 127.0.0.11:53: no such host
bemk_gateway.1.2jy84fkyzxz9@m1rrzsbem01t    | 2019/04/16 09:20:10 Forwarded [POST] to /function/my-function - [502] - 0.004523 seconds

Ultimately I had to remove all functions and all openfaas stacks and recreate everything from scratch to get it working again.

My understanding is that function services do not become part of a stack. The gateway finds its function services by means of a label. Here is my function in faas-cli describe:

Name:                my-function                                            
Status:              Ready                                                     
Replicas:            1                                                         
Available replicas:  1                                                         
Invocations:         0                                                         
Image:               bem-docker-registry.kvnurs.intra:443/my-function:latest
Function process:                                                              
URL:                 http://bemk.kvnurs.intra/function/my-function          
Async URL:           http://bemk.kvnurs.intra/async-function/my-function    
Labels:              function : true                                           
                     com.openfaas.function : my-function

I assume the gateway finds its functions using the com.openfaas.function label. On deployment the gateway apparently attaches the function to its own network.
Now, when the gateway in the second stack deploys the same function, it seems to disconnect the function from its current network and attaches it to its own network (and probably does other things which cause more confusion).

Possible Solution

A solution must allow the function to be a separate entity for docker if it is deployed into a different stack. The function must become a separate service for each stack, which scales independently etc. The only way to achieve that is a different service name, I am not aware that service labeling alone can help us here.

As long as the service has the same name in both stacks, it is probably not possible to achieve that.

Suggestion:
The openfaas yml format could introduce a name field below each function key whose value supports env variable substitution. The gateway uses the service key to build the service url and the name field to create the service name in docker.
Docker has done similar things with name fields for configs and volumes, so maybe that is not so far-fetched after all.

Workaround:
Create separate stack.yml files with separate service names prefixed for each stack and let callers use the appropriate url for each stack, i.e. http://localhost:8080/functions/develop_my-function.

Steps to Reproduce

1. deploy the openfaas stack twice under two different stack names. (To make this possible, let the gateway of the second stack run under a different port)
2. use faas-cli to deploy a function into the gateway of the first stack. Check that it works properly
3. deploy the same function into the gateway of the second stack

Result:

• faas-cli list shows that both stacks know the function, fass-cli describe even claims that the function is ready in both stacks
• the function does not work, rather you get a 502
• redeploying the function into the gateway of the first stack does not fix it. I had to delete all functions and all stacks and deploy a new stack from scratch to get it working again with a single stack.

Context

In my scenario we have used swarm stacks for staging. Each swarm stack is a different stage (stages as in development - acceptance test etc. where placement constraints are used to distribute the containers to separate nodes depending on the stage).

Your Environment

• FaaS-CLI version ( Full output from: faas-cli version ):

 commit:  87ca614cfe27c4cf2975f7629992c1351b18c2bc
 version: 0.8.8

• Docker version docker version (e.g. Docker 17.0.05 ):

Client:
 Version:           18.06.1-ce
 API version:       1.38
 Go version:        go1.10.7
 Git commit:        e68fc7a215d7
 Built:             Wed Dec 19 10:23:04 2018
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.06.1-ce
  API version:      1.38 (minimum version 1.12)
  Go version:       go1.10.7
  Git commit:       e68fc7a215d7
  Built:            Tue Aug 21 17:16:31 2018
  OS/Arch:          linux/amd64
  Experimental:     false

• Are you using Docker Swarm or Kubernetes (FaaS-netes)?
  Swarm

• Operating System and version (e.g. Linux, Windows, MacOS):
  Linux Suse SLE12 sp 4

• Link to your project or a code example to reproduce issue:
  n/a yet

[LucasRoesler]

I think you are right about needing to add a prefix to the service name, but I think i would rather see tis as something that is configured in the gateway and applied to all functions that are deployed through it instead of extending the yaml. This seems like something that the function author should not really need to care about and that the gateway can solve. It would be even better if the gateway could infer this from the Docker stack in someway so that it can automatically be created without any additional configuration flags

[alexellis]

Hi @dschulten thanks for your interest in faas-swarm.

Can I ask if you are using exactly the same Docker network and network label for both stacks?

Can you give step by step bash commands so that one of us can reproduce what you're seeing?

Thanks

Alex

[alexellis]

You may also be interested in OpenFaaS Cloud which scopes or namespaces functions to a user-account or organization/project. This may be more suitable for your needs than deploying OpenFaaS to the same set of nodes or same Swarm cluster multiple times.

[dschulten]

I am not just interested in openfaas, I am wholly committed ;-)

I needed to customize the vanilla openfaas docker-compose.yml to put the stack behind traefik, and I have added more services into the same file; also I let them use my own common network (not functions). The network name in the docker-compose.yml is the same, but docker swarm creates two separate networks when it creates the stacks. Both are attachable and both are labeled the same, as in the following contrived example:

networks:
  myapps:
    driver: overlay
    attachable: true
    labels:
      - "openfaas=true"

$ docker network ls

NETWORK ID          NAME                          DRIVER              SCOPE
l5zw89zgl2de        development_myapps            overlay             swarm
jfqnphbw3iku        acceptance_myapps             overlay             swarm

If it helps, I can try to create a cleanroom example which shows the setup.

[dschulten]

@LucasRoesler that was my first thought, too, but I wasn't sure if a service should know in which swarm stack it is running from an architectural point of view. OTOH if an openfaas stack had a name - which could be passed with faas-cli deploy or a top-level name key in stack.yml or some calculated value, then there would be no dependency on swarm apis.

[LucasRoesler]

@dschulten i was thinking that the gateway would simply mutate the name internally, adding the required prefix. The gateway should have a way to determine what it's "name" is, either by asking the swarm api or by a configuration flag at deployment. I would like to avoid changing the stack schema and the cli if we can.

We already have a dependency in the swarm provider on the swarm apis, by definition, so that doesn't concern me. This is a very swarm specific issue that would only be changed in the swarm-provider anyway.

[dschulten]

@LucasRoesler If we depend on the swarm api anyway, it's fine. Roughly where in the code would that happen?

[LucasRoesler]

The handling of the service name would need to be updated in each of the handlers, which are conveniently located in handlers/. We would need to update to add/remove the name prefix as appropriate

[LucasRoesler]

@alexellis what do you think? We could update the the swarm provider so that the service name in docker has a prefix/namespace per gateway. This prefix would be completely internal and would not be exposed to the user in urls or api responses.

[davidecavestro]

@LucasRoesler I just hit this issue and checked out faas-swarm handlers. The solution you proposed seems reasonable to me.
It would avoid polluting function names with prefixes at the caller side.