Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Port to CNI networking #21

Merged
merged 1 commit into from
Jan 10, 2020
Merged

Port to CNI networking #21

merged 1 commit into from
Jan 10, 2020

Conversation

carlosedp
Copy link
Contributor

@carlosedp carlosedp commented Jan 9, 2020 •
edited
Loading

Description

Replaced netns utility with CNI plugins to create a bridge network
and allow communication between containers with firewall plugin.

Works together with faas-containerd CNI configuration implemented recently.

Motivation and Context

  • I have raised an issue to propose this change this is required

Addresses issue #19

How Has This Been Tested?

Deployed into linux/amd64 host with test functions:

Jan 09 13:48:27 debian10 faasd[13795]: 2020/01/09 13:48:27 File exists: "/run/faasd/secrets/basic-auth-password"
Jan 09 13:48:27 debian10 faasd[13795]: 2020/01/09 13:48:27 File exists: "/run/faasd/secrets/basic-auth-user"
Jan 09 13:48:27 debian10 faasd[13795]: 2020/01/09 13:48:27 Writing network config...
Jan 09 13:48:27 debian10 faasd[13795]: 2020/01/09 13:48:27 Supervisor created in: 903.293µs
Jan 09 13:48:27 debian10 faasd[13795]: Preparing: basic-auth-plugin with image: docker.io/openfaas/basic-auth-plugin:0.18.10
Jan 09 13:48:27 debian10 faasd[13795]: Prepare done for: docker.io/openfaas/basic-auth-plugin:0.18.10, 7171180 bytes
Jan 09 13:48:27 debian10 faasd[13795]: Preparing: nats with image: docker.io/library/nats-streaming:0.11.2
Jan 09 13:48:27 debian10 faasd[13795]: Prepare done for: docker.io/library/nats-streaming:0.11.2, 4647125 bytes
Jan 09 13:48:27 debian10 faasd[13795]: Preparing: prometheus with image: docker.io/prom/prometheus:v2.14.0
Jan 09 13:48:27 debian10 faasd[13795]: Prepare done for: docker.io/prom/prometheus:v2.14.0, 53527559 bytes
Jan 09 13:48:27 debian10 faasd[13795]: Preparing: gateway with image: docker.io/openfaas/gateway:0.18.8
Jan 09 13:48:27 debian10 faasd[13795]: Prepare done for: docker.io/openfaas/gateway:0.18.8, 11117867 bytes
Jan 09 13:48:27 debian10 faasd[13795]: Preparing: queue-worker with image: docker.io/openfaas/queue-worker:0.9.0
Jan 09 13:48:27 debian10 faasd[13795]: Prepare done for: docker.io/openfaas/queue-worker:0.9.0, 4293659 bytes
Jan 09 13:48:27 debian10 faasd[13795]: Reconciling: basic-auth-plugin
Jan 09 13:48:27 debian10 faasd[13795]: Status of basic-auth-plugin is: created
Jan 09 13:48:27 debian10 faasd[13795]: 2020/01/09 13:48:27 Need to kill basic-auth-plugin
Jan 09 13:48:27 debian10 faasd[13795]: 2020/01/09 13:48:27 Created container basic-auth-plugin
Jan 09 13:48:27 debian10 faasd[13795]: 2020/01/09 13:48:27 basic-auth-plugin has IP: 10.62.0.115
Jan 09 13:48:27 debian10 faasd[13795]: 2020/01/09 13:48:27 Task: basic-auth-plugin        Container: basic-auth-plugin
Jan 09 13:48:27 debian10 faasd[13795]: 2020/01/09 18:48:27 Listening on: 8080
Jan 09 13:48:27 debian10 faasd[13795]: Reconciling: nats
Jan 09 13:48:27 debian10 faasd[13795]: 2020/01/09 13:48:27 Created container nats
Jan 09 13:48:28 debian10 faasd[13795]: 2020/01/09 13:48:28 nats has IP: 10.62.0.116
Jan 09 13:48:28 debian10 faasd[13795]: 2020/01/09 13:48:28 Task: nats        Container: nats
Jan 09 13:48:28 debian10 faasd[13795]: Reconciling: prometheus
Jan 09 13:48:28 debian10 faasd[13795]: [1] 2020/01/09 18:48:28.210070 [INF] STREAM: Starting nats-streaming-server[faas-cluster] version 0.11.2
Jan 09 13:48:28 debian10 faasd[13795]: [1] 2020/01/09 18:48:28.210193 [INF] STREAM: ServerID: OLR6OkVQxr8xgiHJIva02P
Jan 09 13:48:28 debian10 faasd[13795]: [1] 2020/01/09 18:48:28.210198 [INF] STREAM: Go version: go1.11.1
Jan 09 13:48:28 debian10 faasd[13795]: [1] 2020/01/09 18:48:28.211044 [INF] Starting nats-server version 1.3.0
Jan 09 13:48:28 debian10 faasd[13795]: [1] 2020/01/09 18:48:28.211171 [INF] Git commit [not set]
Jan 09 13:48:28 debian10 faasd[13795]: [1] 2020/01/09 18:48:28.211573 [INF] Starting http monitor on 0.0.0.0:8222
Jan 09 13:48:28 debian10 faasd[13795]: [1] 2020/01/09 18:48:28.211638 [INF] Listening for client connections on 0.0.0.0:4222
Jan 09 13:48:28 debian10 faasd[13795]: [1] 2020/01/09 18:48:28.211642 [INF] Server is ready
Jan 09 13:48:28 debian10 faasd[13795]: [1] 2020/01/09 18:48:28.238284 [INF] STREAM: Recovering the state...
Jan 09 13:48:28 debian10 faasd[13795]: [1] 2020/01/09 18:48:28.238297 [INF] STREAM: No recovered state
Jan 09 13:48:28 debian10 faasd[13795]: 2020/01/09 13:48:28 Created container prometheus
Jan 09 13:48:28 debian10 faasd[13795]: [1] 2020/01/09 18:48:28.489413 [INF] STREAM: Message store is MEMORY
Jan 09 13:48:28 debian10 faasd[13795]: [1] 2020/01/09 18:48:28.489441 [INF] STREAM: ---------- Store Limits ----------
Jan 09 13:48:28 debian10 faasd[13795]: [1] 2020/01/09 18:48:28.489444 [INF] STREAM: Channels:                  100 *
Jan 09 13:48:28 debian10 faasd[13795]: [1] 2020/01/09 18:48:28.489446 [INF] STREAM: --------- Channels Limits --------
Jan 09 13:48:28 debian10 faasd[13795]: [1] 2020/01/09 18:48:28.489447 [INF] STREAM:   Subscriptions:          1000 *
Jan 09 13:48:28 debian10 faasd[13795]: [1] 2020/01/09 18:48:28.489449 [INF] STREAM:   Messages     :       1000000 *
Jan 09 13:48:28 debian10 faasd[13795]: [1] 2020/01/09 18:48:28.489450 [INF] STREAM:   Bytes        :     976.56 MB *
Jan 09 13:48:28 debian10 faasd[13795]: [1] 2020/01/09 18:48:28.489452 [INF] STREAM:   Age          :     unlimited *
Jan 09 13:48:28 debian10 faasd[13795]: [1] 2020/01/09 18:48:28.489454 [INF] STREAM:   Inactivity   :     unlimited *
Jan 09 13:48:28 debian10 faasd[13795]: [1] 2020/01/09 18:48:28.489456 [INF] STREAM: ----------------------------------
Jan 09 13:48:28 debian10 faasd[13795]: 2020/01/09 13:48:28 prometheus has IP: 10.62.0.117
Jan 09 13:48:28 debian10 faasd[13795]: 2020/01/09 13:48:28 Task: prometheus        Container: prometheus
Jan 09 13:48:28 debian10 faasd[13795]: Reconciling: gateway
Jan 09 13:48:28 debian10 faasd[13795]: 2020/01/09 13:48:28 Created container gateway
Jan 09 13:48:28 debian10 faasd[13795]: level=info ts=2020-01-09T18:48:28.628Z caller=main.go:296 msg="no time or size retention was set so using the default time retention" duration=15d
Jan 09 13:48:28 debian10 faasd[13795]: level=info ts=2020-01-09T18:48:28.628Z caller=main.go:332 msg="Starting Prometheus" version="(version=2.14.0, branch=HEAD, revision=edeb7a44cbf745f1d8be4ea6f215e79e651bfe19)"
Jan 09 13:48:28 debian10 faasd[13795]: level=info ts=2020-01-09T18:48:28.630Z caller=main.go:333 build_context="(go=go1.13.4, user=root@df2327081015, date=20191111-14:27:12)"
Jan 09 13:48:28 debian10 faasd[13795]: level=info ts=2020-01-09T18:48:28.630Z caller=main.go:334 host_details="(Linux 4.19.0-6-amd64 #1 SMP Debian 4.19.67-2+deb10u2 (2019-11-11) x86_64 debian10 (none))"
Jan 09 13:48:28 debian10 faasd[13795]: level=info ts=2020-01-09T18:48:28.630Z caller=main.go:335 fd_limits="(soft=1024, hard=1024)"
Jan 09 13:48:28 debian10 faasd[13795]: level=info ts=2020-01-09T18:48:28.630Z caller=main.go:336 vm_limits="(soft=unlimited, hard=unlimited)"
Jan 09 13:48:28 debian10 faasd[13795]: level=info ts=2020-01-09T18:48:28.631Z caller=main.go:657 msg="Starting TSDB ..."
Jan 09 13:48:28 debian10 faasd[13795]: level=info ts=2020-01-09T18:48:28.631Z caller=web.go:496 component=web msg="Start listening for connections" address=0.0.0.0:9090
Jan 09 13:48:28 debian10 faasd[13795]: level=info ts=2020-01-09T18:48:28.633Z caller=head.go:535 component=tsdb msg="replaying WAL, this may take awhile"
Jan 09 13:48:28 debian10 faasd[13795]: level=info ts=2020-01-09T18:48:28.638Z caller=head.go:583 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0
Jan 09 13:48:28 debian10 faasd[13795]: level=info ts=2020-01-09T18:48:28.639Z caller=main.go:672 fs_type=794c7630
Jan 09 13:48:28 debian10 faasd[13795]: level=info ts=2020-01-09T18:48:28.639Z caller=main.go:673 msg="TSDB started"
Jan 09 13:48:28 debian10 faasd[13795]: level=info ts=2020-01-09T18:48:28.639Z caller=main.go:743 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
Jan 09 13:48:28 debian10 faasd[13795]: level=info ts=2020-01-09T18:48:28.651Z caller=main.go:771 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml
Jan 09 13:48:28 debian10 faasd[13795]: level=info ts=2020-01-09T18:48:28.651Z caller=main.go:626 msg="Server is ready to receive web requests."
Jan 09 13:48:28 debian10 faasd[13795]: 2020/01/09 13:48:28 gateway has IP: 10.62.0.118
Jan 09 13:48:28 debian10 faasd[13795]: 2020/01/09 13:48:28 Task: gateway        Container: gateway
Jan 09 13:48:28 debian10 faasd[13795]: Reconciling: queue-worker
Jan 09 13:48:28 debian10 faasd[13795]: 2020/01/09 18:48:28 HTTP Read Timeout: 1m0s
Jan 09 13:48:28 debian10 faasd[13795]: 2020/01/09 18:48:28 HTTP Write Timeout: 1m0s
Jan 09 13:48:28 debian10 faasd[13795]: 2020/01/09 18:48:28 Binding to external function provider: http://faas-containerd:8081/
Jan 09 13:48:28 debian10 faasd[13795]: 2020/01/09 18:48:28 Async enabled: Using NATS Streaming.
Jan 09 13:48:28 debian10 faasd[13795]: 2020/01/09 18:48:28 Opening connection to nats://nats:4222
Jan 09 13:48:28 debian10 faasd[13795]: 2020/01/09 18:48:28 Connect: nats://nats:4222
Jan 09 13:48:28 debian10 faasd[13795]: 2020/01/09 13:48:28 Created container queue-worker
Jan 09 13:48:29 debian10 faasd[13795]: 2020/01/09 13:48:29 queue-worker has IP: 10.62.0.119
Jan 09 13:48:29 debian10 faasd[13795]: 2020/01/09 13:48:29 Task: queue-worker        Container: queue-worker
Jan 09 13:48:29 debian10 faasd[13795]: Loading basic authentication credentials
Jan 09 13:48:29 debian10 faasd[13795]: Connect: nats://nats:4222
Jan 09 13:48:29 debian10 faasd[13795]: 2020/01/09 13:48:29 Supervisor init done in: 1.866779447s
Jan 09 13:48:29 debian10 faasd[13795]: 2020/01/09 13:48:29 faasd: waiting for SIGTERM or SIGINT
Jan 09 13:48:31 debian10 faasd[13795]: can't connect to nats://nats:4222: nats: no servers available for connection
Jan 09 13:48:31 debian10 faasd[13795]: panic: can't connect to nats://nats:4222: nats: no servers available for connection
Jan 09 13:48:31 debian10 faasd[13795]: goroutine 1 [running]:
Jan 09 13:48:31 debian10 faasd[13795]: log.Panic(0xc00003dda0, 0x1, 0x1)
Jan 09 13:48:31 debian10 faasd[13795]:         /usr/local/go/src/log/log.go:326 +0xc0
Jan 09 13:48:31 debian10 faasd[13795]: main.main()
Jan 09 13:48:31 debian10 faasd[13795]:         /go/src/github.com/openfaas/nats-queue-worker/main.go:211 +0x609
Jan 09 13:48:32 debian10 faasd[13795]: 2020/01/09 13:48:32 [up] Sending 10.62.0.118 to proxy
Jan 09 13:48:32 debian10 faasd[13795]: 2020/01/09 13:48:32 Starting faasd proxy on 8080
Jan 09 13:48:32 debian10 faasd[13795]: Gateway: 10.62.0.118:8080
Jan 09 13:48:32 debian10 faasd[13795]: 2020/01/09 13:48:32 [proxy] Wait for done
Jan 09 13:48:32 debian10 faasd[13795]: 2020/01/09 13:48:32 [proxy] Begin listen on 8080
Jan 09 13:48:51 debian10 faasd[13795]: 2020/01/09 18:48:51 Get http://faas-containerd:8081/system/functions: dial tcp: i/o timeout
Jan 09 13:53:18 debian10 faasd[13795]: [faasd] proxy: http://10.62.0.118:8080/system/functions
Jan 09 13:53:18 debian10 faasd[13795]: 2020/01/09 18:53:18 Validated request 200.
Jan 09 13:53:18 debian10 faasd[13795]: 2020/01/09 18:53:18 Forwarded [GET] to /system/functions - [200] - 0.008251 seconds
Jan 09 13:53:47 debian10 faasd[13795]: [faasd] proxy: http://10.62.0.118:8080/system/functions
Jan 09 13:53:47 debian10 faasd[13795]: 2020/01/09 18:53:47 Validated request 200.
Jan 09 13:53:47 debian10 faasd[13795]: 2020/01/09 18:53:47 Forwarded [GET] to /system/functions - [200] - 0.000540 seconds
Jan 09 13:53:57 debian10 faasd[13795]: [faasd] proxy: http://10.62.0.118:8080/system/functions
Jan 09 13:53:57 debian10 faasd[13795]: 2020/01/09 18:53:57 Validated request 200.
Jan 09 13:53:57 debian10 faasd[13795]: 2020/01/09 18:53:57 Forwarded [PUT] to /system/functions - [404] - 0.000578 seconds
Jan 09 13:53:57 debian10 faasd[13795]: [faasd] proxy: http://10.62.0.118:8080/system/functions
Jan 09 13:53:57 debian10 faasd[13795]: 2020/01/09 18:53:57 Validated request 200.
Jan 09 13:53:57 debian10 faasd[13795]: 2020/01/09 18:53:57 Forwarded [POST] to /system/functions - [400] - 0.010569 seconds
Jan 09 13:54:03 debian10 faasd[13795]: [faasd] proxy: http://10.62.0.118:8080/system/functions
Jan 09 13:54:03 debian10 faasd[13795]: 2020/01/09 18:54:03 Validated request 200.
Jan 09 13:54:03 debian10 faasd[13795]: 2020/01/09 18:54:03 Forwarded [PUT] to /system/functions - [404] - 0.001438 seconds
Jan 09 13:54:03 debian10 faasd[13795]: [faasd] proxy: http://10.62.0.118:8080/system/functions
Jan 09 13:54:03 debian10 faasd[13795]: 2020/01/09 18:54:03 Validated request 200.
Jan 09 13:54:03 debian10 faasd[13795]: 2020/01/09 18:54:03 Forwarded [POST] to /system/functions - [200] - 0.321567 seconds
Jan 09 13:54:06 debian10 faasd[13795]: 2020/01/09 18:54:06 Get http://faas-containerd:8081/system/functions: dial tcp: i/o timeout
Jan 09 13:54:07 debian10 faasd[13795]: [faasd] proxy: http://10.62.0.118:8080/system/functions
Jan 09 13:54:07 debian10 faasd[13795]: 2020/01/09 18:54:07 Validated request 200.
Jan 09 13:54:07 debian10 faasd[13795]: 2020/01/09 18:54:07 Forwarded [GET] to /system/functions - [200] - 0.000497 seconds

Validated that the containers and their tasks are running:

❯ sudo ctr container ls
CONTAINER            IMAGE                                           RUNTIME
basic-auth-plugin    docker.io/openfaas/basic-auth-plugin:0.18.10    io.containerd.runc.v2
gateway              docker.io/openfaas/gateway:0.18.8               io.containerd.runc.v2
nats                 docker.io/library/nats-streaming:0.11.2         io.containerd.runc.v2
prometheus           docker.io/prom/prometheus:v2.14.0               io.containerd.runc.v2
queue-worker         docker.io/openfaas/queue-worker:0.9.0           io.containerd.runc.v2


❯ sudo ctr task ls
TASK                 PID      STATUS
nats                 15149    RUNNING
prometheus           15257    RUNNING
gateway              15363    RUNNING
queue-worker         15465    RUNNING
basic-auth-plugin    15051    RUNNING

Deploy and test function:

❯ faas store deploy figlet --name=figlet3 --update=true --replace=false
WARNING! Communication is not secure, please consider using HTTPS. Letsencrypt.org offers free SSL/TLS certificates.

Deployed. 200 OK.
URL: http://localhost:8080/function/figlet3


❯ faas-cli list -g 127.0.0.1:8081 -v
Function                      	Image                                   	Invocations    	Replicas
figlet2                       	                                        	0              	1
figlet3                       	                                        	0              	1


❯ curl -d Test http://127.0.0.1:8081/function/figlet3
 _____         _
|_   _|__  ___| |_
  | |/ _ \/ __| __|
  | |  __/\__ \ |_
  |_|\___||___/\__|

Invocations are counted:

❯ faas-cli list
Function                      	Invocations    	Replicas
figlet3                       	2              	1

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

Commits:

  • I've read the CONTRIBUTION guide
  • My commit message has a body and describe how this was tested and why it is required.
  • I have signed-off my commits with git commit -s for the Developer Certificate of Origin (DCO)

Code:

  • My code follows the code style of this project.
  • I have added tests to cover my changes.

Docs:

  • My change requires a change to the documentation.
  • I have updated the documentation accordingly.

@derek derek bot added the new-contributor label Jan 9, 2020
@derek
Copy link

derek bot commented Jan 9, 2020

Thank you for your contribution. I've just checked and your commit doesn't appear to be signed-off. That's something we need before your Pull Request can be merged. Please see our contributing guide.
Tip: if you only have one commit so far then run: git commit --amend --signoff and then git push --force.

@derek derek bot added no-dco and removed no-dco labels Jan 9, 2020
@carlosedp
Port networking to CNI
82ed7c8 
Replaced netns utility with CNI plugins to create a bridge network
and allow communication between containers with firewall plugin.

Overwrite the CNI config in case it exists. Allow updating the config
on new versions.

Signed-off-by: Carlos de Paula <[email protected]>
alexellis
alexellis approved these changes Jan 10, 2020
View reviewed changes
Copy link
Member

@alexellis alexellis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@alexellis alexellis merged commit 95e278b into openfaas:master Jan 10, 2020
@carlosedp carlosedp deleted the cni branch January 10, 2020 18:46
@alexellis alexellis mentioned this pull request Jan 10, 2020
Merged
11 tasks
@alexellis
Copy link
Member

Was this as expected?

Jan 09 13:48:51 debian10 faasd[13795]: 2020/01/09 18:48:51 Get http://faas-containerd:8081/system/functions: dial tcp: i/o timeout

@alexellis
Copy link
Member

After merging and using PR alexellis/faas-containerd#21, I can no longer get an operational system on Ubuntu 16.04 and a x86_64 machine:

Jan 10 19:00:09 alexx faasd[13600]: [faasd] proxy: http://10.62.0.16:8080/system/functions
Jan 10 19:00:09 alexx faasd[13600]: 2020/01/10 19:00:09 Validated request 200.
Jan 10 19:00:09 alexx faasd[13600]: 2020/01/10 19:00:09 error with upstream request to: /system/functions, Post http://faas-containerd:8081/system/functions: EOF
Jan 10 19:00:09 alexx faasd[13600]: 2020/01/10 19:00:09 Forwarded [POST] to /system/functions - [502] - 0.223180 seconds
Jan 10 19:00:09 alexx faasd[13600]: 2020/01/10 19:00:09 Get http://faas-containerd:8081/system/functions: dial tcp 10.62.0.1:8081: connect: connection refused

I can reach faas-containerd using the bridge IP via the host however.

@carlosedp carlosedp mentioned this pull request Jan 21, 2020
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@alexellis alexellis alexellis approved these changes

Assignees
No one assigned
Labels
new-contributor
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@carlosedp @alexellis