Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

sketch of an extension manifest #1221

Closed
wants to merge 3 commits into from
Closed
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
213 changes: 213 additions & 0 deletions extension/v1alpha1/manifest.proto
Original file line number Diff line number Diff line change
@@ -0,0 +1,213 @@
// Copyright 2019 Istio Authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

syntax = "proto3";

// $schema: istio.extension.v1alpha1.ExtensionManifest
// $title: Extension Manifest

// `ExtensionManifest` specifies additional extensions to be
// applied in a service mesh. Extensions are injected into the
// data plane processing based on a set of criteria, such as
// the workload labels, service ports, protocols, etc.
//
// ```yaml
// apiVersion: extension.istio.io/v1alpha1
// kind: ExtensionManifest
kyessenov marked this conversation as resolved.
Show resolved Hide resolved
// metadata:
// name: zip
// namespace: istio-config
// spec:
// filters:
// - match:
// proxyVersion: '1\.4.*'
// context: SIDECAR_OUTBOUND
// portNumber: 8080
// type: HTTP_FILTER
// name: compress-outbound
// config:
// typeUrl: config.filter.http.gzip.v2.Gzip
// value:
// compression_level: BEST
// function: POST_PROCESSING
//```
//
package istio.extension.v1alpha1;

option go_package = "istio.io/api/extension/v1alpha1";

import "google/protobuf/struct.proto";

// Opaque configuration for an extension (wire and JSON compatible with UDPA).
// The extension resource allows white-listing pre-defined configuration for
// extensions.
message Extension {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Naming nit: maybe "ExtensionConfig"?

// A URL that uniquely identifies the type of the serialize protocol buffer
// message. This has same semantics and format described in
// google.protobuf.Any:
// https://github.com/protocolbuffers/protobuf/blob/master/src/google/protobuf/any.proto
string type_url = 1;

// A JSON representation of the above specified type.
google.protobuf.Struct value = 2;
kyessenov marked this conversation as resolved.
Show resolved Hide resolved
}

// Reference to extension resource.
message ExtensionRef {
string name = 1;
string namespace = 2;
}

// A condition limiting the extent of the extension application.
message MatchContext {
// A regular expression in golang regex format (RE2) that can be
// used to select proxies using a specific version of istio
// proxy. The Istio version for a given proxy is obtained from the
// node metadata field ISTIO_VERSION supplied by the proxy when
// connecting to Pilot. This value is embedded as an environment
// variable (ISTIO_META_ISTIO_VERSION) in the Istio proxy docker
// image. Custom proxy implementations should provide this metadata
// variable to take advantage of the Istio version check option.
string proxy_version = 1;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should we consider making this semver instead of regex?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good to me.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

some more context - back when we made this in EnvoyFilter we had versions like master-latest-daily. We now call these 1.5-dev.SHA for example (technically its not semver without 3 numbers but close enough to work) so it should work

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was a requests in the original envoyfilter API as well, but we did not have correct proxy version that time.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Semver in UDPA is just a numeric version. Don't we need to handle ASM qualifier in addition to semver?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess that may be an issue, may want to select on 1.4.2-custom vs 1.4.2. Or we could say that is overloading the version and should just set another metadata field and use that?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Envoy BuildVersion in envoyproxy/envoy#9301 includes metadata capable of expressing these labels.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ISTIO_VERSION is part of node metadata, so we would be regex matching on a field in metadata. We could express it as a map from a field to a string matcher, but that feels too level in istio, since it does not convey the purpose of using the version.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Essentially you need the version to check capabilities of the proxy itself (will it accept the extension or not). What is the reason not to rely on BuildVersion, which would make Istio more UDPA compliant, vs Istio-specific API such as using ISTIO_VERSION metadata?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we have any reason other than that build version doesn't exist yet as far as I can tell? We would likely need to fallback to istio_version for old proxies even when it does though


// Match on the node metadata supplied by a proxy when connecting
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How does this interact with the match criteria enabled by envoyproxy/envoy#9301? Specifically the ability to match on the presence of a given filter?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure we need that match criterion in istio. We generally know what filters are packaged, and don't select on presence or absence of compiled filters. In the future, we expect filters to be dynamically loadable.

// to Istio Pilot. Note that while Envoy's node metadata is of
// type Struct, only string key-value pairs are processed by
kyessenov marked this conversation as resolved.
Show resolved Hide resolved
// Pilot. All keys specified in the metadata must match with exact
// values. The match will fail if any of the specified keys are
// absent or the values fail to match.
map<string, string> metadata = 2;

// Configuration type.
enum ProxyContext {
UNSPECIFIED = 0;
SIDECAR_INBOUND = 1;
SIDECAR_OUTBOUND = 2;
GATEWAY = 3;
};

// Match on the proxy type (applicable for network and HTTP filters).
ProxyContext context = 3;

// The service port of the upstream or downstream configuration resource. If
// omitted, applies to clusters and listeners for any port.
uint32 port_number = 4;

// The Istio gateway config's namespace/name for which this route
// configuration was generated. Applies only if the context is GATEWAY. Should
// be in the namespace/name format. Use this field in conjunction with the
// portNumber to accurately select the Envoy route configuration for a
// specific HTTPS server within a gateway config object.
string gateway = 5;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a gateway can have many Servers with its own hosts. Each of which ends up in its own filter chain if using TLS. So you need a way to specify the hostnames here as well, so that you can match a filter chain based on the sni.

The match as it stands is insufficient as people have most commonly wanted to add a lua filter only for certain paths and not for all routes (like /user/.. and not for /index.html). I have seen this way too many times to keep track. This then requires route.PerFilterconfig in envoy. You cant do that with the way you have expressed the matches as it stand today. You would also now need to match on a specific route by name so that you can add this config to the perFilterConfig.

Finally, since we are in the land of designing for future, we also need to take into account the passthrough listeners. People will want to write loggers/trackers/enforces for the traffic that is passthrough while using standard rbac for the rest.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't per filter config be part of virtual service or other service/route-specific resource? This API is written from an administrator point of view (MUST enable telemetry and authorization) so opting out should be an override.


// The fully qualified service name for this cluster or listener. For
// services defined through service entries, the service name is same as the
// hosts defined in the service entry.
string service = 6;

// The subset associated with the service. If omitted, applies to
// clusters for any subset of a service.
string subset = 7;

// Criteria used to select the specific set of pods/VMs on which
// this patch configuration should be applied. If omitted, the set
// of patches in this configuration will be applied to all workload
// instances in the same namespace. If omitted, the EnvoyFilter
// patches will be applied to all workloads in the same
// namespace. If the EnvoyFilter is present in the config root
// namespace, it will be applied to all applicable workloads in any
// namespace.
map<string, string> workload_labels = 8;

// Namespace of the proxy.
string namespace = 9;
}

// ExtensionManifest specifies a set of custom extensions.
// Manifests can be applied in any namespace. Per-namespace manifests
// may override the root manifests for the overlapping filter names if
// both match conditions are satisfied.
// Beyond that, the order of application of the manifests is unspecified.
// Additional control over the delegation of the extension injection should
// be enforced with either resource-level RBAC or an admission controller.
message ExtensionManifest {
// Extension filter type.
enum FilterType {
UNSPECIFIED_FILTER_TYPE = 0;
HTTP_FILTER = 1;
NETWORK_FILTER = 2;
UPSTREAM_NETWORK_FILTER = 3;

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the difference between NETWORK_FILTER and UPSTREAM_NETWORK_FILTER?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NETWORK_FILTER is a downstream network filter, UPSTREAM_NETWORK_FILTER is an upstream network filter a.k.a cluster filter.

LISTENER_FILTER = 4;
};

// Operation performed with the supplied config when the match condition is
// satisfied. By default, the operation is to displace the filter with the
// same name. If no filter with the same name is present, then the filter is
// added to chain using the provided function hint to locate the position of
// insertion. If the operation is to remove, then the filter with the same
// name is removed (config is optional). If no filter with the same name is
// found then, then it is a no-op.
enum Operation {
ADD = 0;
REMOVE = 2;
};

// Extension purpose serving as a hint to determine the order of the
// application of an extension in a chain of extensions. Istio recognizes the
// following categories of extensions and applies them in the order specified
// here.
enum Function {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What would be the mechanism for two independent providers of filters to coordinate relative order?

I.e. I have filters A and B, is there a way to specify that B should go before A?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't want to allow specific total order of filters. A partial order constraint like this enum is all we want to support. That is we can express that authorization filters should come before post-processing filters, but no specific order between authorization filters.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure about that. Having at least a relative ordering within each category might help with inter-dependant filters. I know it sounds like a long shot at this point but limitations like this also cause problems with more complex setups of e.g. MutatingWebhooks or CNI plugins - as for CNI, we've hit this problem in Istio even.

Could be solved with a simple priority integer, where if you have filters 1,2 with type==Authorization and priority(1) > priority(2), filter 1 is guaranteed to be injected earlier in the filter chain. priority(1) == priority(2) could be undefined order, caught using our ValidatingWebhook.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is no impl under way yet. We are still in the process of getting agreement.

Re: ordering, integers priority helps, but what is easier is to preserve the manifest order in the actual output within a filter type.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks, yeah I remember now that this was talked about in the last meeting. I'm not sure that's very user-friendly- it's non-obvious that this implicit order matters, as the filters could be matching on completely separate sidecars. It also leaves out how this would behave if different ExtensionManifests add filters to the same Envoy config. Which one wins?

UNSPECIFIED_FUNCTION = 0;
TELEMETRY = 1;
PRE_PROCESSING = 2;
AUTHORIZATION = 3;
POST_PROCESSING = 4;
};

// Extension description for a data plane filter.
message FilterExtension {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@htuch : I'm curious to hear your thoughts on this API sketch to globally apply a filter across xDS configs.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it would also be valuable to hear from @alexburnos here, as he has experience with similar systems.

// Optional conditions to limit the scope of the extension.
// At least one of the conditions must be satisfied for the filter to be
// applied.
repeated MatchContext match = 1;

// Specifies the type of the filter.
FilterType type = 2;

// The required name of the filter instance. The name should generally be
// unique in a filter chain.
string name = 3;

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I assume it is not a UDPA Filter name field (as it has uniqueness requirement)? In this case, where is UDPA filter name will be specified, in the Extension?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm working to change UDPA filter name to be free-form in envoy. UDPA filter type URL is sufficient to determine an extension.


// Opaque configuration for the filter.
oneof configuration {
// Inlined extension configuration.
Extension config = 4;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Supporting both ref and inline means that k8s rbac cannot be used. We need to use a more sophisitcated admission control.

// Configuration reference.
ExtensionRef config_ref = 5;
}

// Modification operation (function is optional in case of replacement or
// removal).
Operation operation = 6;
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not an Istio API expert, but this strikes me as providing an imperative vs. a declarative API. I was under the impression we tend to describe what the system should be like, rather than how to get there.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This part is debatable. We have a choice here.
We can forgo these operations, and then leave “how to get there” to CI/CD
In that case you install a whole new named manifest in your namespace, so someone else must do copy and update operation outside the istio API.
Please take a look at the linked doc where we have listed the use cases this needs to support.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@htuch We really only need the removal operation since there is no other way to provide an override for a filter that disables it. Think of a per-namespace declaration that disables a root default. .

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it would be cleaner (and very simple) to add a per-filter disable override in Envoy. We've discussed this before and it just needs O(1 day) of effort to add if someone is inclined.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If this is a bool inside the filter proto, it would mean merging the default proto with the override. We want to avoid proto merges.

a “conditional” dispatch of filters is a useful feature, and we have an open feature request in envoy for that.

However wouldn’t it be better to not send a filter configuration at all instead of sending it with a disabled flag?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@htuch Are you thinking of adding a bool field to filter config protos to disable it? I think that might be useful with filter chain discovery, to be able to dynamically disable filters. If so, we could mirror it here with a bool field instead of operation enum.


// Purpose of the extension used to place the filter in a chain.
Function function = 7;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

name: filter_phase

};

// A list of extensions to be applied. The filter extensions are processed in
// the order specified here. If multiple extension conditions are satisfied
// for the same filter name within a manifest, then the last filter config
// applies.
repeated FilterExtension filters = 1;
}