Skip to content

Latest commit

 

History

History
315 lines (254 loc) · 22.6 KB

annotations.md

File metadata and controls

315 lines (254 loc) · 22.6 KB
Raw

Annotations

Ingress annotations

The following are the Ingress annotations supported by Citrix:

Annotations Possible value Description Default
ingress.citrix.com/frontend-ip IP address Use this annotation to customize the virtual IP address (VIP). This IP address is configured in Netscaler as VIP. The annotation is mandatory if you are using Netscaler VPX or MPX.
Note: Do not use the annotation if you want to use the Netscaler IP address as VIP.		 Netscaler IP address is used as VIP.
Ingress.citrix.com/frontend-ipset-name Name of the IPSET Use this annotation to specify the IPSET name for the front-end configuration. The IPSET is bound to the CS virtual server. Use this annotation with the ingress.citrix.com/frontend-ip.
Note: The IPSET name that you specify in the annotation should already be configured in Netscaler.		 NA
ingress.citrix.com/secure-port Port number Use this annotation to configure the port for HTTPS traffic. This port is configured in Netscaler as a port value for the corresponding Content Switching (CS) virtual server. 443
ingress.citrix.com/insecure-port Port number Use this annotation to configure the port for HTTP, TCP, or UDP traffic. This port is configured in Netscaler as a port value for the corresponding CS virtual server. 80
ingress.citrix.com/insecure-termination allow, redirect, or disallow Use allow to allow HTTP traffic, Use redirect to redirect the HTTP request to HTTPS, or Use disallow if you want to drop the HTTP traffic.

For example: ingress.citrix.com/insecure-termination: "redirect"		 disallow
ingress.citrix.com/secure-backend In JSON form, list of services for secure-backend Use True, if you want to establish secure HTTPS between Netscaler and the application, Use False, if you want to establish insecure HTTP connection Netscaler to the application.

For example: ingress.citrix.com/secure-backend: {"app1":"True", "app2":"False", "app3":"True"}		 False
kubernetes.io/ingress.class ingress class name It is a way to associate a particular ingress resource with an ingress controller.

For example: kubernetes.io/ingress.class:"Citrix"		 Configures all ingresses
ingress.citrix.com/secure-service-type ssl or ssl_tcp The annotation allows L4 load balancing with SSL over TCP as protocol. Use ssl_tcp, if you want to use SSL over TCP. ssl
ingress.citrix.com/insecure-service-type http, tcp, udp, sip_udp, or any The annotation allows L4 load balancing with tcp/udp/sip_udp any as protocol. Use tcp, if you want TCP as the protocol. Use udp, if you want UDP as the protocol. http
ingress.citrix.com/path-match-method prefix or exact Use this annotation for ingress path matching.
- Use prefix for Netscaler ingress controller to consider any path string as a prefix expression.
- Use exact for the Netscaler ingress controller to consider the path as an exact match.

For example, the ingress.citrix.com/path-match-method: "prefix" annotation defines the Netscaler ingress controller to consider any path string as a prefix expression.		 prefix
ingress.citrix.com/deployment dsr Use this annotation to create Direct Server Return (DSR) configuration on Netscaler. For example, the ingress.citrix.com/deployment: "dsr" annotation creates DSR configuration on the Netscaler.
ingress.citrix.com/preconfigured-certkey Pre-configured certificate keys and type. Certificate type can be default, SNI, or CA. Use this annotation to specify the preconfigured certificate key or keys in the Netscaler that you want to reuse and bind to applications. If the type parameter is not provided with the name of a certificate, then the certificate is considered as the default (non-SNI) type.
For example in the ingress.citrix.com/preconfigured-certkey : '{"certs": [ {"name": "certkey1", "type": "default"}, {"name": "certkey2", "type": "sni"} ] } annotation, certkey1 is used as a non-SNI certificate and certkey2 is used as a SNI certificate.

Smart annotations for Ingress

Smart annotation is an option provided by the Netscaler ingress controller to efficiently enable Netscaler features using the Netscaler entity name. The Netscaler ingress controller converts the Ingress in Kubernetes to a set of Netscaler objects. You can efficiently control these objects using smart annotations.

!!! Info "Important" To use smart annotations, you must have good understanding of Netscaler features and their respective entity names. For more information on Netscaler features and entity names, see Netscaler Documentation.

Smart annotation takes JSON format as input. The key and value that you pass in the JSON format must match the Netscaler NITRO format. For more information on the Netscaler NITRO API, see Netscaler 12.1 REST APIs - NITRO Documentation.

For example, if you want to enable the SRCIPDESTIPHASH based lb method, you must use the corresponding NITRO key and value format lbmethod, SRCIPDESTIPHASH respectively.

The following table details the smart annotations provided by the Netscaler ingress controller:

Netscaler Entity Name Smart Annotation Example
lbvserver ingress.citrix.com/lbvserver ingress.citrix.com/lbvserver: '{"citrix-svc":{"lbmethod":"SRCIPDESTIPHASH"}}'
servicegroup ingress.citrix.com/servicegroup ingress.citrix.com/servicegroup: '{"appname":{"cip": "Enabled","cipHeader":"X-Forwarded-For"}}'
monitor ingress.citrix.com/monitor ingress.citrix.com/monitor: '{"appname":{"type":"http"}}'
csvserver ingress.citrix.com/csvserver ingress.citrix.com/csvserver: '{"stateupdate": "ENABLED"}'

For information on smart annotations for HTTP, TCP, and SSL profiles, see Configure HTTP, TCP, or SSL profiles on Netscaler.

Sample ingress YAML with smart annotations

The following is a sample Ingress YAML. It includes smart annotations to enable Netscaler features using the entities such as, lbvserver, servicegroup, and monitor:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    ingress.citrix.com/frontend-ip: 192.168.1.1
    ingress.citrix.com/insecure-port: "80"
    ingress.citrix.com/lbvserver: '{"citrix-svc":{"lbmethod":"LEASTCONNECTION", "persistenceType":"SOURCEIP"}}'
    ingress.citrix.com/monitor: '{"citrix-svc":{"type":"http"}}'
    ingress.citrix.com/servicegroup: '{"citrix-svc":{"usip":"yes"}}'
  name: citrix
spec:
  rules:
  - host: citrix.org
    http:
      paths:
      - backend:
          service:
            name: citrix-svc
            port:
              number: 80
        path: /
        pathType: Prefix

The sample Ingress YAML includes use cases related to the service, citrix-svc, and the following table explains the smart annotations used in the sample:

Smart Annotation Description
ingress.citrix.com/lbvserver: '{"citrix-svc":{"lbmethod":"LEASTCONNECTION", "persistenceType":"SOURCEIP"}}' Sets the load balancing method as Least Connection and also configures Source IP address persistence.
ingress.citrix.com/servicegroup: '{"citrix-svc":{"usip":"yes"}}' Enables Use Source IP Mode (USIP) on the Ingress Netscaler device. When you enable USIP on the Netscaler, it uses the client's IP address for communication with the back-end pods.
ingress.citrix.com/monitor: '{"citrix-svc":{"type":"http"}}' Creates a custom HTTP monitor for the servicegroup.

Note: When multiple ingresses are sharing the same front-end IP address and port, you cannot have conflicting configurations provided through multiple ingress configurations.

By default, the content switching virtual server does not depend on the state of the target load balancing virtual servers bound to it. The annotation ingress.citrix.com/csvserver: '{"stateupdate": "ENABLED"} sets the content switching virtual server to consider its state based on the state of the load balancing virtual server bound to it via the content switching policies.

Smart annotations for routes

Similar to Ingress, you can also use smart annotations with OpenShift routes. The Netscaler ingress controller converts the routes in OpenShift to a set of Netscaler objects.

The following table details the smart annotations provided by the Netscaler ingress controller:

Netscaler entity name Smart annotation Example
lbvserver route.citrix.com/lbvserver route.citrix.com/lbvserver: '{"citrix-svc":{"lbmethod":"SRCIPDESTIPHASH"}}'
servicegroup route.citrix.com/servicegroup route.citrix.com/servicegroup: '{"appname":{"cip": "Enabled","cipHeader":"X-Forwarded-For"}}'
monitor route.citrix.com/monitor route.citrix.com/monitor: '{"appname":{"type":"http"}}'

Sample route manifest with smart annotations

The following is a sample route YAML file.

apiVersion: route.openshift.io/v1
kind: Route
metadata:
  name: citrix
  annotations:
    route.citrix.com/lbvserver: '{"citrix-svc":{"lbmethod":"LEASTCONNECTION", "persistenceType":"SOURCEIP"}}'
    route.citrix.com/servicegroup: '{"citrix-svc":{"usip":"yes"}}'
    route.citrix.com/monitor: '{"citrix-svc":{"type":"http"}}'
spec:
  host:  citrix.org
  port:
    targetPort: 80
  to:
    kind: Service
    name: citrix-svc
    weight: 100
  wildcardPolicy: None

The sample route manifest includes use cases related to the service citrix-svc and the following table explains the smart annotations used in the sample Route:

Smart annotation Description
route.citrix.com/lbvserver: '{"citrix-svc":{"lbmethod":"LEASTCONNECTION", "persistenceType":"SOURCEIP"}}' Sets the load balancing method as Least Connection and also configures Source IP address persistence.
route.citrix.com/servicegroup: '{"citrix-svc":{"usip":"yes"}}' Enables Use Source IP Mode (USIP) on the Netscaler device. When you enable USIP on the Netscaler, it uses the IP address of the client for communication with the back-end pods.
route.citrix.com/monitor: '{"citrix-svc":{"type":"http"}}' Creates a custom HTTP monitor for the servicegroup.

Service annotations

The following are the service annotations supported by Citrix.

Note: In service annotations, index is the ordered index of the ports in a service specification file. For example, if there are two ports in the service specification, then the index for the first port is zero and the second one is one.

Annotations Description Example
service.citrix.com/service-type-<index> Use this annotation to specify the service type for the Netscaler entities created. The acceptable values are TCP, HTTP, SSL,UDP,ANY, SSL_TCP, and SIP_UDP. service.citrix.com/service-type-0: ‘SSL’
service.citrix.com/ssl-certificate-data-<index> Use this annotation to specify the server certificate value in the PEM format. service.citrix.com/ssl-certificate-data-0: | <certificate>
service.citrix.com/ssl-key-data-<index> Use this annotation to specify the server key value in the PEM format. service.citrix.com/ssl-key-data-0: | <key>
service.citrix.com/ssl-ca-certificate-data-<index> Use this annotation to specify the server CA certificate value to verify the client certificate in PEM format. service.citrix.com/ssl-ca-certificate-data-0: | <certificate>
service.citrix.com/ssl-backend-ca-certificate-data-<index> Use this annotation to specify the CA certificate value to verify the server certificate of the back-end in PEM format. service.citrix.com/ssl-backend-ca-certificate-data-0: | <certificate>
service.citrix.com/ssl-termination-<index> Use this annotation to specify the SSL termination. The accepted values are EDGE and REENCRYPT. service.citrix.com/ssl-termination-0: 'EDGE'
service.citrix.com/insecure-redirect Use this annotation to redirect insecure traffic to a secure port. You can either specify the secure port using {secure-portname : port-number} or {secure-portnumber- secure-port-protocol : insecure-portnumber } to redirect traffic from an insecure port. service.citrix.com/insecure-redirect: '{"port-443": 80 }'
or
service.citrix.com/insecure-redirect: '{"443-tcp": 80 }'
service.citrix.com/frontend-ip Use this annotation to pass the VIP for services of type LoadBalancer. service.citrix.com/frontend-ip: '192.168.1.1'
service.citrix.com/ipam-range Use this annotation to select a particular IP address range from a set of ranges specified to the Citrix IPAM controller. This annotation is used for services of type LoadBalancer. service.citrix.com/ipam-range: 'Dev'
service.citrix.com/secret Use this annotation to specify the name of the secret resource for the front-end server certificate. For more information and example, see SSL certificate for services of type LoadBalancer. service.citrix.com/secret: 'hotdrink-secret'
service.citrix.com/ca-secret Use this annotation to provide a CA certificate for client certificate authentication. This certificate is bound to the front-end SSL virtual server in Netscaler. For more information and example, see SSL certificate for services of type LoadBalancer. service.citrix.com/ca-secret: 'hotdrink-ca-secret'
service.citrix.com/backend-secret Use this annotation if the back-end communication between Netscaler and your workload is on an encrypted channel, and you need the client authentication in your workload. This certificate is sent to the server during the SSL handshake and it is bound to the back end SSL service group. For more information and example, see SSL certificate for services of type LoadBalancer. service.citrix.com/backend-secret: 'hotdrink-secret'
service.citrix.com/backend-ca-secret Use this annotation to enable server authentication which authenticates the back-end server certificate. This configuration binds the CA certificate of the server to the SSL service on the Netscaler. For more information and example, see SSL certificate for services of type LoadBalancer. service.citrix.com/backend-ca-secret: 'hotdrink-ca-secret'
service.citrix.com/preconfigured-certkey Use this annotation to specify the name of the preconfigured certificate key in the Netscaler to be used as a front-end server certificate. service.citrix.com/preconfigured-certkey: 'coffee-cert'
service.citrix.com/preconfigured-ca-certkey Use this annotation to specify the name of the preconfigured certificate key in the Netscaler to be used as a CA certificate for client certificate authentication. This certificate is bound to the front-end SSL virtual server in Netscaler. service.citrix.com/preconfigured-backend-certkey: 'coffee-cert'
service.citrix.com/preconfigured-backend-certkey Use this annotation to specify the name of the preconfigured certificate key in the Netscaler to be bound to the back-end SSL service group. This certificate is sent to the server during the SSL handshake for server authentication. service.citrix.com/preconfigured-ca-certkey: 'coffee-ca-cert'
service.citrix.com/preconfigured-backend-ca-certkey Use this annotation to specify the name of the preconfigured CA certificate key in the Netscaler to bound to the back-end SSL service group for server authentication. service.citrix.com/preconfigured-backend-ca-certkey: 'coffee-ca-cert'

Sample YAML with the service annotation to redirect insecure traffic

This example shows how to redirect traffic from clients making requests on an insecure port 80 to the secure port 443.

The following annotation is specified in the service YAML file to redirect traffic:

service.citrix.com/insecure-redirect: '{"port-443": 80}'

Following is a sample service definition:

apiVersion: v1
kind: Service
metadata:
  name: frontend-service
  annotations:
    service.citrix.com/service-type-0: SSL
    service.citrix.com/frontend-ip: '192.2.170.26'
    service.citrix.com/secret: '{"port-443": "web-ingress-secret"}'
    service.citrix.com/ssl-termination-0: 'EDGE'
    service.citrix.com/insecure-redirect: '{"port-443": 80}'
spec:
  type: LoadBalancer
  selector:
    app: frontend
  ports:
  - port: 443
    targetPort: 80
    name: port-443

Smart annotations for services

Smart annotations for services are used to configure the Netscaler with custom values for Netscaler configuration parameters. The annotations are used for services of type LoadBalancer and for the services in Netscaler CPX used for East-West traffic.

Note: If you have configured a service with NodePort or ClusterIP for the North-South traffic, then the Netscaler is configured using the applicable ingress smart annotations rather than then service annotations.

Smart annotations for services take JSON format as input. The key and value that you pass in the JSON format must match the Netscaler NITRO format. For more information on the Netscaler NITRO API, see Netscaler 12.1 REST APIs - NITRO Documentation.

The following is a sample smart annotation for services:

service.citrix.com/lbvserver: '{"80-tcp":{"lbmethod":"SRCIPDESTIPHASH"}}'

This annotation sets the load balancing method as SRCIPDESTIPHASH in the load balancing virtual server for the 80-tcp port of the given service.

The following table details the smart annotations for services:

Netscaler Entity Name Smart Annotation for Service Example
lbvserver service.citrix.com/lbvserver service.citrix.com/lbvserver: '{"80-tcp":{"lbmethod":"SRCIPDESTIPHASH"}}'
csvserver service.citrix.com/csvserver service.citrix.com/csvserver: '{"l2conn":"on"}'
servicegroup service.citrix.com/servicegroup service.citrix.com/servicegroup: '{"80-tcp":{"usip":"yes"}}'
monitor service.citrix.com/monitor service.citrix.com/monitor: '{"80-tcp":{"type":"http"}}'
analyticsprofile service.citrix.com/analyticsprofile service.citrix.com/analyticsprofile: '{"80-tcp":{"webinsight": {"httpurl":"ENABLED", "httpuseragent":"ENABLED"}}}'

You can use the smart annotations for services as follows:

  • By providing the port-protocol value in the annotation. In the service definition, if you provide the port-protocol value in the annotation then the annotation is restricted to the particular port of that service.
  • By not providing the port-protocol value in the annotation. If you do not provide the port-protocol value in the annotation, then the annotation is applicable to all the ports used by the service.

Sample ingress YAML with smart annotations for services

The following is a sample deployment and service definition for a basic apache web server based application. It includes smart annotations for services to enable Netscaler features using the entities such as, lbvserver, csvserver, servicegroup, monitor, and analyticsprofile:

# If using this on GKE, eusure sure you have cluster-admin role for your account
#The sample is a basic apache web server as application for illustration
apiVersion: apps/v1beta2
kind: Deployment
metadata:
  name: apache
  labels:
      name: apache
spec:
  selector:
    matchLabels:
      app: apache
  replicas: 8
  template:
    metadata:
      labels:
        app: apache
    spec:
      containers:
      - name: apache
        image: httpd:latest
        ports:
        - name: http
          containerPort: 80
        imagePullPolicy: IfNotPresent

---
#Expose the apache web server as a service
apiVersion: v1
kind: Service
metadata:
  name: apache
  annotations:
    service.citrix.com/csvserver: '{"l2conn":"on"}'
    service.citrix.com/lbvserver: '{"80-tcp":{"lbmethod":"SRCIPDESTIPHASH"}}'
    service.citrix.com/servicegroup: '{"80-tcp":{"usip":"yes"}}'
    service.citrix.com/monitor: '{"80-tcp":{"type":"http"}}'
    service.citrix.com/frontend-ip: '10.217.212.16'
    service.citrix.com/analyticsprofile: '{"80-tcp":{"webinsight": {"httpurl":"ENABLED", "httpuseragent":"ENABLED"}}}'
    NETSCALER_VPORT: '80'
  labels:
    name: apache
spec:
  externalTrafficPolicy: Local
  type: LoadBalancer
  selector:
    name: apache
  ports:
  - name: http
    port: 80
    targetPort: http
  selector:
    app: apache
---

Examples

Sample Ingress YAML for SIP_UDP support in insecure service type annotation

The following is a sample Ingress YAML which includes the configuration for enabling SIP over UDP support using the ingress.citrix.com/insecure-service-type annotation.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    ingress.citrix.com/frontend-ip: 1.1.1.1
    ingress.citrix.com/insecure-port: '5060'
    ingress.citrix.com/insecure-service-type: sip_udp
    ingress.citrix.com/lbvserver: '{"asterisk17":{"lbmethod":"CALLIDHASH","persistenceType":"CALLID"}}'
  name: sip-ingress
spec:
  defaultBackend:
    service:
      name: asterisk17
      port:
        number: 5060
  ingressClassName: cic-vpx
---
apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
  name: cic-vpx
spec:
  controller: citrix.com/ingress-controller
---