NiFi upgrade doesn't work

[soenkeliebau]

Affected version

0.5.0

Current and expected behavior

Scenario
A NifiCluster with three nodes was deployed with version 1.13.2 and is up and running.

The NifiCluster CRD is now changed to version 1.15.0.

Current Behavior
The StatefulSet is updated with the new image and triggers a rolling restart of the NiFi Pods with the new container image set to NiFi 1.15.0.

However NiFi does not support running a cluster with mixed versions, instead a full stop and restart with the new version is required.

Reference ticket:
https://issues.apache.org/jira/browse/NIFI-4068?jql=project%20%3D%20NIFI%20AND%20text%20~%20%22rolling%20upgrade%22

Due to this, the new pod never successfully starts and the restart hangs indefinitely, or until the user deletes all pods and they are rewritten with the same version by the StatefulSet.

Expected Behavior
The operator should notice that a version changes is happening and trigger a full restart of NiFi.

This is done when

• I can update my NiFi CR from one NiFi version to another and have the operator automatically run a full-restart of NiFi to have a fully running new NiFi (new = new version)
• The solution to implement this can be reused by other operators for similar scenarios
• The proposed solution has been discussed in the architecture meeting before it's fully implemented
• The implementation solution has been documented in the Contributor's Guide

Possible solution

The operator needs to be able to recognize a version change during reconciliation and then act accordingly to perform a full cluster restart.

Something along the lines of this code (suggested by @teozkr ) might work:

if current_sts.spec.template.spec.image != new_image:
  if current_sts.status.replicas > 0:
    # Wait for all current replicas to die
    new_sts.spec.replicas = 0
  else:
    # All old replicas are dead, do the upgrade
    new_sts.spec.replicas = rolegroup.replicas
    new_sts.spec.template.spec.image = new_image

Environment

This should be reproducable independently of the K8s environment.

[lfrancke]

None of the boxes have been checked.
Do they still make sense and if so could you check what's been done?

@razvan @soenkeliebau

[soenkeliebau]

I'm afraid the boxes are not really checkable at the moment, as we did not implement a generic solution for this. Not sure if I wrote those checkboxes back in the day, but I'd say we don't need something abstract at the moment, as this only affects NiFi at the moment.
Opinions?

[lfrancke]

No, I'm fine with that and I'm fine with not checking the boxes.
Do we need a follow-up ticket then or can we create one if something comes up?

[soenkeliebau]

We can create that as and when needed I think.