[Explore]: Spike out User Provided Service Instances Support Using Kubernetes Service Bindings for Runtime

[tcdowney]

Background

Developers using CF can bind their apps to two types of services: user provided and managed. Managed Services on CF are those whose lifecycle and credentials are controlled by a separate service broker. Cloud Foundry interfaces with these brokers via the Open Service Broker API and exposes them to developers through its Service Marketplace. Developers can ask brokers to create new Service Instances for them and then bind their apps (via a Service Binding) to these instances. As part of that binding, the broker will supply credentials and other important information (such as connection strings, certs, port numbers, etc.) to the app via the VCAP_SERVICES environment variable.

User Provided Services are managed directly by CF users. Users can create what are known as “User Provided Service Instances” (UPSIs) with an arbitrary set of configuration. When a Service Binding is created to one of these UPSIs, the configuration is shared with the bound app under the same VCAP_SERVICES environment variable. For example, a developer may be given credentials to an already existing database instance and use a UPSI to expose it to their app running on TAS.

As a MVP, we are looking to support basic (just credentials -- no route services, no syslog drains) User Provided Service Instances first before moving on to exploring the more complicated world of managed services.

There are existing projects in the Kubernetes ecosystem such as the Kubernetes Service Bindings Specification that provide custom resources and "duck types" (ProvisionedService) that we believe we can leverage to achieve our immediate compatibility goals while providing flexibility for the evolving services landscape in the Kubernetes community.

---

Acceptance Criteria

We want to explore how we might implement support for User Provided Service Instances in CF on K8s.

As part of this exploration let's spike out an implementation using a new CFServiceInstance CR that implements the ProvisionedService duck type from the K8s Service Binding Spec as well as the ServiceBinding custom resource.

Example CFServiceInstance

---
kind: CFServiceInstance
…
metadata:
  …
  name: my-upsi
  namespace: my-cf-space-guid
spec:
  …
  type: user-provided
…
status:
  …
  binding:
    name: my-upsi-credentials

At a minimum we want credentials from multiple bindings to be aggregated under a VCAP_SERVICES env var so that existing CF apps can continue to work. This looks something like this in a CF for VMs app runtime and staging containers:

vcap@ea6b07a8-bb91-4b16-43d3-9b4e:~$ env | grep VCAP_SERVICES -A 18
VCAP_SERVICES={"user-provided":[{
  "label": "user-provided",
  "name": "test-upsi",
  "tags": [

  ],
  "instance_guid": "f4a3a20f-8fd5-4233-afd3-cfcd863986e1",
  "instance_name": "test-upsi",
  "binding_guid": "13bbc5ee-14a5-400b-a354-a68dd52a1807",
  "binding_name": null,
  "credentials": {
    "user-provided-key-1": "user-provided-value-1",
    "user-provided-key-2": "user-provided-value-2"
  },
  "syslog_drain_url": null,
  "volume_mounts": [

  ]
}]}

We should also explore what it looks like to actually implement the Kubernetes Service Bindings spec and project these credentials as volumes on the app container. See the Workload Projection portion of the specification for more details on what that looks like.

Questions to answer

• What are some ways to project credentials from the service binding into the app runtime container?
  • In our CFProcess reconciler?
  • Mutating webhook on Pods?
  • Something else?
• How do we aggregate multiple credentials under VCAP_SERVICES? How do we make them volume mounts?
• What is missing from the example CR that's necessary for this to work and power the V3 Services APIs? We ultimately want to support the full set of CLI commands here, such as: cf create-user-provided-service, cf bind-service, cf services, cf update-user-provided-service, cf unbind-service, etc.
  • Read through, understand, and explore how these APIs behave on CF for VMs: https://v3-apidocs.cloudfoundry.org/version/3.112.0/index.html#service-instances

TODO write a separate story for staging:

• What are some ways to project credentials from the service binding into the app staging container?
  • In our CFBuild reconciler?
  • Mutating webhook on Pods?
  • Is there a point of projecting them as VCAP_SERVICES for staging or do CNBs not even look at that.

Output

A proposal to share with the CF on K8s SIG group about how we're going to tackle this.

Number of days this exploration is expected to take. This is what is “pointed”.

3

Dev Notes

---

Resources:

• CF Services Overview docs
• CF User-Provided Service Instance docs
• Kubernetes Service Bindings Spec

[akrishna90]

@julian-hj and I investigated the Kubernetes service binding spec using a concrete implementation here - https://github.com/vmware-labs/service-bindings.

We were able to bind a service to an app (pushed using cf push) by using the service-binding and secret below. We verified the service-binding by ssh-ing into the app and listing the volume mounts. This implementation of service binding spec doesn't solve the problem of exposing the service credentials as a VCAP_SERVICES env variable.

Secret

apiVersion: v1
kind: Secret
metadata:
  name: node-db
type: servicebinding.io/mysql
stringData:
  type: mysql
  provider: mariadb
  host: node-db
  port: "3306"
  database: default
  # demo credentials
  username: user
  password: pass

Service Binding

---
apiVersion: servicebinding.io/v1alpha3
kind: ServiceBinding
metadata:
  name: node-db
spec:
  # direct Secret reference is used for compatibility, but not recommended for dynamically provisioned services
  service:
    apiVersion: v1
    kind: Secret
    name: node-db
  workload:
    apiVersion: apps/v1
    kind: Statefulset
    name: cfeecdaf-47e4-44a0-8b53-ce3b391b7530-bef260a699

[julian-hj]

Various work items for an initial implementation of the VCAP_SERVICES/cf cups feature

API Shim

• implements handlers for the service endpoints (/v3/service_instances /v3/service_credential_bindings)
• On service creation, creates the corresponding CFServiceInstance object and k8s Secret in the app namespace
• On service binding, creates the ServiceBinding object

CFServiceInstance controller

• We need to create this.
• This controller will do almost nothing--it needs to copy the secret name from the spec into the status which will make the object conform to the ProvisionedService duck type.

CFProcess controller

• At process start time, the controller will look for bindings and corresponding secrets for the app, and generate a VCAP_SERVICES json string that we pass into the LRP in the environment hash that we are already setting. (we have some spiked implementation for this in the works)

[julian-hj]

Notes from pairing Jan-5-2022 with @akrishna90

• We got our spike working end-to-end with CFServiceInstances bound to a cf-pushed app. Spike code is in the spike-servince-bindings branch
• We did not do any of the API work yet, only the controllers
• We took a bunch of short-cuts.
  • We don't filter on app guid, so we bind any service binding in the same space as the app. This obviously needs to be refined.
  • We also wrote 0 lines of test code.
  • We only handle the first service binding. We need a bit of code to aggregate service bindings into the JSON for VCAP_SERVICES, but we think that should be simple enough.
• We used the lightweight implementation for service binding controller here This implementation does literally nothing in its reconcile loop. Ideally we should have one that copies the service instance details from the spec to the status of the service binding so that we can pick it up from status which is theoretically more correct.
• The spike branch adds 3 new samples for the secret, service instance, and service binding. Something to note is that the service binding refers to the CFApp object, not the statefulset. This will be easier for us to consume from the process controller where we are mixing in the VCAP_SERVICES environment variable. But, it does mean that we won't be able to just use the VMware ootb service binding implementation if we want to mount credentials as a volume mount. We might have to write our own implementation to add volume mounts to our app workloads, but it doesn't seem like it would be very difficult.
• We believe that the CFProcess controller is the better place to add the VCAP_SERVICES injection logic because it affords us more opportunity to recover from failure--if a secret or service instance is unavailable, we can just fail and retry on the next iteration. Whereas in a MutatingWebhook, we only get one opportunity to mutate the pod, and if it fails then we don't have much recourse.

@tcdowney PTAL, and let us know if you think this story can be considered "Done".

[tcdowney]

Thanks @akrishna90 and @julian-hj for the detailed write up! I've synthesized some of your findings under the "Design Details" section of the Proposal we've been working on.

[tcdowney]

Diagram from our proposal:

[tcdowney]

Given some feedback from the proposal, we now think it's valuable to have an abstraction on top of the K8s ServiceBinding that we'll call the CFServiceBinding. This will give us the following:

1. Closely match the existing CF v3 service API modeling and allow us to more easily implement managed service usecases in the future (things like unique credentials per binding, async service bindings, etc.)
2. We'll be able to consume changes to the K8s ServiceBinding CR without having to change the API shim's implementation
3. Avoid having to rely heavily on labels/annotations on the K8s ServiceBinding CR for bookkeeping

An updated diagram is attached below:

We also sketched out various options on this Miro:
https://miro.com/app/board/uXjVOULKK0M=/?invite_link_id=592014892164

[tcdowney]

Closing this exploration. Feature development is well underway.