Skip to content
This repository has been archived by the owner on Jun 6, 2024. It is now read-only.

Deployment on Amazon EKS

Jump to bottom
Aleksey Zalesov edited this page Jan 29, 2019 · 28 revisions

You will need to follow the guide here to deploy a cluster with EKS.

Make sure to use node instance sizes that are at least a t2.large. Anything smaller won't fit a diego cell properly.

Once you have a running cluster, and you can do kubectl get nodes and you can see Ready nodes, please continue.

See also

Disk space

Usually EC2 nodes come with 20GB of disk space. This is insufficient for a cap deployment. Make sure you increase the disk space for each node after creating the kubernetes cluster:

  • Enable SSH to the nodes
    • go to Services -> EC2
    • select Security Groups
    • Click on the Node Security Group for your nodes
    • click Actions -> Edit inbound rules
    • click Add Rule
    • choose "SSH" in the Type column and "Anywhere" in the Source column
    • click "Save"
  • Grow the volumes in AWS
    • go to Services -> EC2
    • select Volumes
    • increase the size for all volumes attached to your nodes from 20GB to a larger size (at least 60GB)
  • Make the growth available to the actual filesystems

The Helm CLI and tiller

Use this version of helm (or newer): https://storage.googleapis.com/kubernetes-helm/helm-v2.9.0-rc4-linux-amd64.tar.gz

In rbac-config.yaml:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: tiller
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: tiller
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
  - kind: ServiceAccount
    name: tiller
    namespace: kube-system

Then:

kubectl create -f rbac-config.yaml
helm init --service-account tiller

A default storage class

Create the following storage-class.yaml:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: gp2
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
  labels:
    kubernetes.io/cluster-service: "true"
provisioner: kubernetes.io/aws-ebs
parameters:
  type: gp2

Run:

kubectl create -f storage-class.yaml

Deploy Cloud Foundry

Security Group rules

In your EC2 VM List, find one of the EKS nodes you've deployed.

Find its security group, then add the following rules to it:

Type Protocol Port Range Source Description
HTTP TCP 80 0.0.0.0/0 CAP HTTP
Custom TCP Rule TCP 2793 0.0.0.0/0 CAP UAA
Custom TCP Rule TCP 2222 0.0.0.0/0 CAP SSH
Custom TCP Rule TCP 4443 0.0.0.0/0 CAP WSS
Custom TCP Rule TCP 443 0.0.0.0/0 CAP HTTPS
Custom TCP Rule TCP 20000-20009 0.0.0.0/0 CAP Brains

Load Balancing

Specify in values.yaml that deployment uses LoadBalancer services.

services:
  loadbalanced: true

After deploying UAA create two DNS A records for uaa.domain and *.uaa.domain pointing to the created load balancer.

During Cloud Foundry deployment Helm creates 3 load balancer. Map DNS records to the load balancers this way:

*.cf.domain -> router-gorouter-public
tcp.cf.domain -> tcp-router-tcp-router-public
ssh.cf.domain -> diego-ssh-ssh-proxy-public

The same way create DNS record for the Stratos UI after deploying it.

console.cf.domain -> console-ui-ext

Deployment

You'll deploy CAP using the usual procedure described here. Make the following changes in your values.yaml:

  • use overlay-xfs for env.GARDEN_ROOTFS_DRIVER
  • use "" for env.GARDEN_APPARMOR_PROFILE
  • the following roles need to have SYS_RESOURCE capabilities:
    • cc_uploader
    • diego_ssh
    • diego_brain
    • diego_api
    • nats
    • routing_api
  • set kube.storage_class.persistent and kube.storage_class.shared to gp2

Example values.yaml:

env:
    # Domain for SCF. DNS for *.DOMAIN must point to a kube node's (not master)
    # external ip address.
    DOMAIN: <PUBLIC IP OF A NODE VM>.nip.io
    #### The UAA hostname is hardcoded to uaa.$DOMAIN, so shouldn't be
    #### specified when deploying
    # UAA host/port that SCF will talk to. If you have a custom UAA
    # provide its host and port here. If you are using the UAA that comes
    # with the SCF distribution, simply use the two values below and
    # substitute the cf-dev.io for your DOMAIN used above.
    UAA_HOST: uaa.<PUBLIC IP OF A NODE VM>.nip.io
    UAA_PORT: 2793
    GARDEN_ROOTFS_DRIVER: overlay-xfs
    GARDEN_APPARMOR_PROFILE: ""
sizing:
  cc_uploader:
    capabilities: ["SYS_RESOURCE"]
  nats:
    capabilities: ["SYS_RESOURCE"]
  routing_api:
    capabilities: ["SYS_RESOURCE"]
  router:
    capabilities: ["SYS_RESOURCE"]
  diego_ssh:
    capabilities: ["SYS_RESOURCE"]
  diego_brain:
    capabilities: ["SYS_RESOURCE"]
  diego_api:
    capabilities: ["SYS_RESOURCE"]
kube:
    storage_class:
        # Make sure to change the value in here to whatever storage class you use
        persistent: "gp2"
        shared: "gp2"
    # The registry the images will be fetched from. No values below should work for
    # a default installation of opensuse based scf containers from dockerhub. If you
    # are going to deploy sle based cap containers, comment out the next five lines.
#    registry:
#      hostname: "registry.suse.com"
#      username: ""
#      password: ""
#    organization: "cap"
    auth: rbac
secrets:
    # Password for user 'admin' in the cluster
    CLUSTER_ADMIN_PASSWORD: changeme
    # Password for SCF to authenticate with UAA
    UAA_ADMIN_CLIENT_SECRET: uaa-admin-client-secret

Delete EKS cluster after use

Clone this wiki locally