feat: add support for traffic router plugins

Makefile

@@ -286,3 +286,7 @@ checksums:
 build-sample-metric-plugin-debug:
 	go build -gcflags="all=-N -l" -o metric-plugin test/cmd/sample-metrics-plugin/main.go
 
+.PHONY: build-sample-traffic-plugin-debug
+build-sample-traffic-plugin-debug:
+	go build -gcflags="all=-N -l" -o traffic-plugin test/cmd/sample-trafficrouter-plugin/main.go
+

docs/analysis/plugins.md

@@ -14,17 +14,14 @@ into the rollouts controller container. The second method is to use a HTTP(S) se
 
 ### Mounting the plugin executable into the rollouts controller container
 
-To use this method, you will need to build or download the plugin executable and then mount it into the rollouts controller container.
-The plugin executable must be mounted into the rollouts controller container at the path specified by the `--metric-plugin-location` flag.
-
 There are a few ways to mount the plugin executable into the rollouts controller container. Some of these will depend on your
 particular infrastructure. Here are a few methods:
 
 * Using an init container to download the plugin executable
 * Using a Kubernetes volume mount with a shared volume such as NFS, EBS, etc.
 * Building the plugin into the rollouts controller container
 
-Then you can use the configmap to point to the plugin executable. Example:
+Then you can use the configmap to point to the plugin executable file location. Example:
 
 ```yaml
 apiVersion: v1
@@ -34,7 +31,7 @@ metadata:
 data:
   plugins: |-
     metrics:
-    - name: "prometheus" # name the plugin uses to find this configuration, it must match the name required by the plugin
+    - plugin: "argoproj-labs/sample-rollouts-metric-plugin" # name of the plugin, it must match the name required by the plugin so it can find it's configuration
       pluginLocation: "file://./my-custom-plugin" # supports http(s):// urls and file://
 ```
 
@@ -51,7 +48,7 @@ metadata:
 data:
   plugins: |-
     metrics:
-    - name: "prometheus" # name the plugin uses to find this configuration, it must match the name required by the plugin
+    - plugin: "argoproj-labs/sample-rollouts-metric-plugin" # name of the plugin, it must match the name required by the plugin so it can find it's configuration
       pluginLocation: "https://github.com/argoproj-labs/sample-rollouts-metric-plugin/releases/download/v0.0.3/metric-plugin-linux-amd64" # supports http(s):// urls and file://
       pluginSha256: "08f588b1c799a37bbe8d0fc74cc1b1492dd70b2c" #optional sha256 checksum of the plugin executable
 ```

docs/features/traffic-management/plugins.md

@@ -0,0 +1,75 @@
+# Traffic Router Plugins
+
+!!! important Available since v1.5
+
+Argo Rollouts supports getting analysis metrics via 3rd party plugin system. This allows users to extend the capabilities of Rollouts
+to support metric providers that are not natively supported. Rollout's uses a plugin library called
+[go-plugin](https://github.com/hashicorp/go-plugin) to do this. You can find a sample plugin
+here: [sample-rollouts-trafficrouter-plugin](https://github.com/argoproj-labs/sample-rollouts-trafficrouter-plugin)
+
+## Using a Traffic Router Plugin
+
+There are two methods of installing and using an argo rollouts plugin. The first method is to mount up the plugin executable
+into the rollouts controller container. The second method is to use a HTTP(S) server to host the plugin executable.
+
+### Mounting the plugin executable into the rollouts controller container
+
+There are a few ways to mount the plugin executable into the rollouts controller container. Some of these will depend on your
+particular infrastructure. Here are a few methods:
+
+* Using an init container to download the plugin executable
+* Using a Kubernetes volume mount with a shared volume such as NFS, EBS, etc.
+* Building the plugin into the rollouts controller container
+
+Then you can use the configmap to point to the plugin executable file location. Example:
+
+```yaml
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: argo-rollouts-config
+data:
+  plugins: |-
+    trafficrouters:
+    - plugin: "argoproj-labs/sample-rollouts-trafficrouter-plugin" # name of the plugin, it must match the name required by the plugin so it can find it's configuration
+      pluginLocation: "file://./my-custom-plugin" # supports http(s):// urls and file://
+```
+
+### Using a HTTP(S) server to host the plugin executable
+
+Argo Rollouts supports downloading the plugin executable from a HTTP(S) server. To use this method, you will need to
+configure the controller via the `argo-rollouts-config` configmap and set `pluginLocation` to a http(s) url. Example:
+
+```yaml
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: argo-rollouts-config
+data:
+  plugins: |-
+    trafficrouters:
+    - plugin: "argoproj-labs/sample-rollouts-trafficrouter-plugin" # name of the plugin, it must match the name required by the plugin so it can find it's configuration
+      pluginLocation: "https://github.com/argoproj-labs/sample-rollouts-trafficrouter-plugin/releases/download/v0.0.3/metric-plugin-linux-amd64" # supports http(s):// urls and file://
+      pluginSha256: "08f588b1c799a37bbe8d0fc74cc1b1492dd70b2c" #optional sha256 checksum of the plugin executable
+```
+
+## Some words of caution
+
+Depending on which method you use to install and the plugin, there are some things to be aware of.
+The rollouts controller will not start if it can not download or find the plugin executable. This means that if you are using
+a method of installation that requires a download of the plugin and the server hosting the plugin for some reason is not available and the rollouts
+controllers pod got deleted while the server was down or is coming up for the first time, it will not be able to start until
+the server hosting the plugin is available again.
+
+Argo Rollouts will download the plugin at startup only once but if the pod is deleted it will need to download the plugin again on next startup. Running
+Argo Rollouts in HA mode can help a little with this situation because each pod will download the plugin at startup. So if a single pod gets
+deleted during a server outage, the other pods will still be able to take over because there will already be a plugin executable available to it. However,
+it is up to you to define your risk for and decide how you want to install the plugin executable.
+
+## List of Available Plugins (alphabetical order)
+
+#### Add Your Plugin Here
+* If you have created a plugin, please submit a PR to add it to this list.
+#### [sample-rollouts-trafficrouter-plugin](https://github.com/argoproj-labs/sample-rollouts-trafficrouter-plugin)
+* This is just a sample plugin that can be used as a starting point for creating your own plugin.
+  It is not meant to be used in production. It is based on the built-in prometheus provider.

docs/plugins.md

@@ -0,0 +1,138 @@
+# Creating an Argo Rollouts Plugin
+
+## High Level Overview
+
+Argo Rollouts plugins depend on hashicorp's [go-plugin](https://github.com/hashicorp/go-plugin) library. This library 
+provides a way for a plugin to be compiled as a standalone executable and then loaded by the rollouts controller at runtime.
+This works by having the plugin executable act as a rpc server and the rollouts controller act as a client. The plugin executable
+is started by the rollouts controller and is a long-lived process and that the rollouts controller connects to over a unix socket.
+The communication protocol uses golang build in net/rpc library so plugins have to be written in golang.
+
+## Plugin Repository
+
+In order to get plugins listed in the main argo rollouts documentation we ask that the plugin repository be created under
+the [argoproj-labs](https://github.com/argoproj-labs) organization. Please open an issue under argo-rollouts requesting a 
+repo which you would be granted admin access on. 
+
+There is also a slight standardization of naming convention for plugin names which are used in the configmap registration,
+as well as what the plugin uses for locating its specific configuration on rollout or analysis resources. The name
+needs to be in the form of `<namespace>/<pluginname>` and both <namespace> and <pluginname> have a regular expression check
+that matches Github's requirements for `username/org` and `repository name`. This requirement is in place to help allow multiple creators
+of the same plugin type to exists such as `<org1>/nginx` and `<org2>/nginx`. These names could be based of the repo name 
+such as `argoproj-labs/sample-rollouts-metric-plugin` but it is not a requirement. 
+
+There will also be a standard for naming repositories under argoproj-labs in the form of `rollouts-<type>-<tool>-plugin`
+where `<type>` is say `metric`, or `trafficrouter` and `<tool>` is the software the plugin is for say nginx.
+
+## Plugin Name
+
+So now that we have an idea on plugin naming and repository standards let's pick a name to use for the rest of this 
+documentation and call our plugin `argoproj-labs/nginx`.
+
+This name will be used in a few different spots the first is the config map that your plugin users will need to configure.
+It looks like this below.
+
+```yaml
+kind: ConfigMap
+metadata:
+  name: argo-rollouts-config
+data:
+  plugins: |-
+    metrics:
+    - plugin: "argoproj-labs/metrics"
+      pluginLocation: "file:///Users/zaller/Development/argo-rollouts/metric-plugin"
+    trafficrouters:
+    - plugin: "argoproj-labs/nginx"
+      pluginLocation: "file:///tmp/argo-rollouts/traffic-plugin"
+```
+
+As you can see there is a field called `plugin:` under both `metrics` or `trafficrouters` this is the first place where your
+end users will need to configure the name of the plugin. The second location is either in the rollout object or the analysis
+template which you can see the examples below.
+
+#### AnalysisTemplate Example
+```yaml
+apiVersion: argoproj.io/v1alpha1
+kind: AnalysisTemplate
+metadata:
+  name: success-rate
+spec:
+  metrics:
+    - name: success-rate
+      ...
+      provider:
+        plugin:
+          argoproj-labs/metrics:
+            address: http://prometheus.local
+```
+
+#### Traffic Router Example
+```yaml
+apiVersion: argoproj.io/v1alpha1
+kind: Rollout
+metadata:
+  name: example-plugin-ro
+spec:
+  strategy:
+    canary:
+      canaryService: example-plugin-ro-canary-analysis
+      stableService: example-plugin-ro-stable-analysis
+      trafficRouting:
+        plugin:
+          argoproj-labs/nginx:
+            stableIngress: canary-demo
+```
+
+You can see that we use the plugin name under `spec.metrics[].provider.plugin` for analysis template and `spec.strategy.canary.trafficRouting.plugin`
+for traffic routers. You as a plugin author can then put any configuration you need under that object and you will be able to
+look up that config in your plugin via the plugin name key.
+
+## Plugin Interfaces
+
+Argo Rollouts currently supports two plugin systems as a plugin author your end goal is to implement these interfaces as
+a hashicorp go-plugin. The two interfaces are `MetricsPlugin` and `TrafficRouterPlugin` for each of the respective plugins:
+
+```go
+type MetricsPlugin interface {
+	NewMetricsPlugin(v1alpha1.Metric) types.RpcError
+	Run(*v1alpha1.AnalysisRun, v1alpha1.Metric) v1alpha1.Measurement
+	Resume(*v1alpha1.AnalysisRun, v1alpha1.Metric, v1alpha1.Measurement) v1alpha1.Measurement
+	Terminate(*v1alpha1.AnalysisRun, v1alpha1.Metric, v1alpha1.Measurement) v1alpha1.Measurement
+	GarbageCollect(*v1alpha1.AnalysisRun, v1alpha1.Metric, int) RpcError
+	Type() string
+	GetMetadata(metric v1alpha1.Metric) map[string]string
+}
+
+type TrafficRouterPlugin interface {
+	NewTrafficRouterPlugin() RpcError
+	UpdateHash(rollout *v1alpha1.Rollout, canaryHash, stableHash string, additionalDestinations []v1alpha1.WeightDestination) RpcError
+	SetWeight(rollout *v1alpha1.Rollout, desiredWeight int32, additionalDestinations []v1alpha1.WeightDestination) RpcError
+	SetHeaderRoute(rollout *v1alpha1.Rollout, setHeaderRoute *v1alpha1.SetHeaderRoute) RpcError
+	SetMirrorRoute(rollout *v1alpha1.Rollout, setMirrorRoute *v1alpha1.SetMirrorRoute) RpcError
+	VerifyWeight(rollout *v1alpha1.Rollout, desiredWeight int32, additionalDestinations []v1alpha1.WeightDestination) (*bool, RpcError)
+	RemoveManagedRoutes(ro *v1alpha1.Rollout) RpcError
+	Type() string
+}
+```
+
+## Plugin New Function
+
+Each plugin interface has a `New` function, this function is called when the plugin is first started up and is only called 
+once per startup. The `New` function is used as a means to initialize the plugin it gives you the plugin author the ability 
+to either set up a client for a specific metrics provider or in the case of a traffic router construct a client or informer 
+for kubernetes api.
+
+## Kubernetes RBAC
+
+The plugin runs as a child process of the rollouts controller and as such it will inherit the same RBAC permissions as the
+controller. This means that if you are using a service account for the rollouts controller you will need to make sure that
+the service account has the correct permissions for the plugin to function. This might mean instructing users to create a role
+and role binding to the standard rollouts service account for the plugin to use. This will probably affect traffic router 
+plugins more than metrics plugins.
+
+## Sample Plugins
+
+There are two sample plugins within the argo-rollouts repo that you can use as a reference for creating your own plugin.
+
+* [Sample Metrics Plugin](https://github.com/argoproj/argo-rollouts/tree/master/test/cmd/sample-metrics-plugin)
+* [Sample Traffic Router Plugin](https://github.com/argoproj/argo-rollouts/tree/master/test/cmd/sample-trafficrouter-plugin)

manifests/crds/rollout-crd.yaml

@@ -862,6 +862,9 @@ spec:
                             required:
                             - stableIngress
                             type: object
+                          plugin:
+                            type: object
+                            x-kubernetes-preserve-unknown-fields: true
                           smi:
                             properties:
                               rootService:

manifests/install.yaml

@@ -11881,6 +11881,9 @@ spec:
                             required:
                             - stableIngress
                             type: object
+                          plugin:
+                            type: object
+                            x-kubernetes-preserve-unknown-fields: true
                           smi:
                             properties:
                               rootService:

metricproviders/plugin/rpc/rpc.go

@@ -39,7 +39,6 @@ func init() {
 	gob.RegisterName("TerminateAndResumeArgs", new(TerminateAndResumeArgs))
 	gob.RegisterName("GarbageCollectArgs", new(GarbageCollectArgs))
 	gob.RegisterName("InitMetricsPluginAndGetMetadataArgs", new(InitMetricsPluginAndGetMetadataArgs))
-	gob.RegisterName("RpcError", new(types.RpcError))
 }
 
 // MetricsPlugin is the interface that we're exposing as a plugin. It needs to match metricproviders.Providers but we can
@@ -52,7 +51,7 @@ type MetricsPlugin interface {
 // MetricsPluginRPC Here is an implementation that talks over RPC
 type MetricsPluginRPC struct{ client *rpc.Client }
 
-// NewMetricsPlugin is the client side function that is wrapped by a local provider this makes an rpc call to the
+// NewMetricsPlugin is the client side function that is wrapped by a local provider this makes a rpc call to the
 // server side function.
 func (g *MetricsPluginRPC) NewMetricsPlugin(metric v1alpha1.Metric) types.RpcError {
 	var resp types.RpcError

mkdocs.yml

@@ -51,6 +51,7 @@ nav:
   - AWS ALB: features/traffic-management/alb.md
   - Istio: features/traffic-management/istio.md
   - NGINX: features/traffic-management/nginx.md
+  - Plugins: traffic-management/plugins.md
   - SMI: features/traffic-management/smi.md
   - Traefik: features/traffic-management/traefik.md
 - Analysis:
@@ -133,6 +134,7 @@ nav:
   - Contribution Guide: CONTRIBUTING.md
   - Environment Setup: getting-started/setup/index.md
   - Releasing: releasing.md
+  - Plugins: plugins.md
 - Releases ⧉: https://github.com/argoproj/argo-rollouts/releases
 - Roadmap ⧉: https://github.com/argoproj/argo-rollouts/milestones
 - Blog ⧉: https://blog.argoproj.io/

pkg/apiclient/rollout/rollout.swagger.json

@@ -1620,6 +1620,14 @@
         "apisix": {
           "$ref": "#/definitions/github.com.argoproj.argo_rollouts.pkg.apis.rollouts.v1alpha1.ApisixTrafficRouting",
           "title": "Apisix holds specific configuration to use Apisix to route traffic"
+        },
+        "plugin": {
+          "type": "object",
+          "additionalProperties": {
+            "type": "string",
+            "format": "byte"
+          },
+          "title": "+kubebuilder:validation:Schemaless\n+kubebuilder:pruning:PreserveUnknownFields\n+kubebuilder:validation:Type=object\nPlugin holds specific configuration to use Apisix to route traffic"
         }
       },
       "title": "RolloutTrafficRouting hosts all the different configuration for supported service meshes to enable more fine-grained traffic routing"