Skip to content

Latest commit

 

History

History
126 lines (103 loc) · 4.43 KB

custom_processors.adoc

File metadata and controls

126 lines (103 loc) · 4.43 KB
Raw

Loading custom components

It is possible to develop custom components for Apache NiFi to extend the available functionality - most often this will probably be custom processors.

For these to be available in the NiFi UI, they need to be present in the classpath at startup, so the nar bundles have to be injected into your NiFi instance.

The Stackable Data Platform supports two ways to achive this goal, both of which are described below.

Please note that building your own Docker image is the recommended solution, as giving multiple pods access to a shared PVC can often be tricky (see comment on access mode in the second section).

Custom Docker image

You can extend the official Stackable NiFi image and copy all required nar files into the classpath of Nifi. The benefit of this method is that there is no need for any config overrides or extra mounts, you can use any ouf our NiFi examples, swap the image and your components will be available. But this means you will need to have access to a registry to push your custom image to.

The basic Dockerfile below shows how to achieve this:

FROM docker.stackable.tech/stackable/nifi:1.27.0-stackable0.0.0-dev
COPY /path/to/your/nar.file /stackable/nifi/lib/

You then need to make this image available to your Kubernetes cluster and specify it in your NiFi resource as described in concepts:product_image_selection.adoc.

spec:
  image:
    productVersion: 1.27.0
    custom: "docker.company.org/nifi:1.27.0-customprocessor"

Also read the Using customized product images guide for additional information.

Using the official image

If you don’t want to create a custom image or don’t have access to an image registry, you can use the extra volume mount functionality to mount a volume containing your custom components and configure NiFi to read these from the mounted volumes.

For this to work you’ll need to prepare a PersistentVolumeClaim (PVC) containing your components. Usually the best way to do this will be to mount the PVC into a temporary container and then kubectl cp the nar files into that volume.

The following listing shows an example of creating the PVC and the Pod:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: nifi-processor
spec:
  accessModes:
    - ReadWriteOnce  # (1)
  resources:
    requests:
      storage: 1Gi
---
apiVersion: v1
kind: Pod
metadata:
  name: processorcopy
spec:
  volumes:
    - name: nifi-processor
      persistentVolumeClaim:
        claimName: nifi-processor
  containers:
    - name: copycontainer
      image: alpine
      command:
        - tail
        - "-f"
        - "/dev/null"
      volumeMounts:
        - mountPath: "/volume"
          name: nifi-processor

  1. Please note that this access mode will mean that you can only use this volume with a single NiFi Pod. For a distributed scenario, an additional list value of ReadOnlyMany could be specified here if this is supported.

The command to then copy the nar bundle into the PVC is:

kubectl cp /path/to/component.nar processorcopy:/volume/

Now you can mount the extra volume into your NiFi instance as described in nifi:usage_guide/extra-volumes.adoc.

After this is done your components will be available within the NiFi Pods and NiFi can be configured to load these at startup by adding an extra directory to the classpath:

apiVersion: nifi.stackable.tech/v1alpha1
kind: NifiCluster
metadata:
  name: simple-nifi
spec:
  image:
    productVersion: 1.27.0
  clusterConfig:
    authentication:
      - authenticationClass: simple-nifi-admin-user
    extraVolumes: # (1)
      - name: nifi-processor
        persistentVolumeClaim:
          claimName: nifi-processor
    listenerClass: external-unstable
    sensitiveProperties:
      keySecret: nifi-sensitive-property-key
      autoGenerate: true
    zookeeperConfigMapName: simple-nifi-znode
  nodes:
    config:
    configOverrides:
      nifi.properties:
        nifi.nar.library.directory.myCustomLibs: "../userdata/nifi-processor/" # (2)
    roleGroups:
      default:
        replicas: 1

  1. Specify your prepared PVC here.

  2. The directory name after userdata has to match the name of the volume, while myCustomLibs is a name you can freely change.