[operator] Readiness for Paladins and SCD Mappings

[onelapahead]

Description

Adds a readiness check to the PaladinReconciler based on if the underlying StatefulSet is ready or not, and a controller ref to allow for watching relevant StatefulSet updates.

The SmartContractDeploymentReconciler is then updated to only reconcile a Paladin if the node is Ready and if the contracts refer to it.

Renames the Completed phase to Ready, as a Paladin can now go back to Pending if the StatefulSet every re-reconciles due to config changes.

Results

One set of enhancements which seemed to help SCD reconcile speed decently in efforts to address to #397, but contracts still get stuck even after the Paladin goes back to Ready after a restart, so missing a retry or a readiness check in the right place. And ideally we avoid too many pod restarts at once, but will address that in a follow up PR.

I did have other contracts stuck due to encoding issues I believe will be addressed by #401.

[peterbroadhurst]

This seems like a good step forwards on all fronts, but as we've discussed it's probably not quite the end of the journey to faster deploy times.

I note we don't have service-level readiness checks on our pod defs right now, which might be good to add (although I don't know if we can make them part of the checking process for the SCD/TXInvoke paths)

[peterbroadhurst]

Think this is a lot clearer 👍

[peterbroadhurst]

We have some helpers to allow generics. Found it surprisingly hard to support them in k8s data structures, I had to introduce a thing like this to every file:

paladin/operator/internal/controller/transactioninvoke_controller.go

Lines 49 to 56 in ef6a417

	// allows generic functions by giving a mapping between the types and interfaces for the CR
	var TransactionInvokeCRMap = CRMap[corev1alpha1.TransactionInvoke, *corev1alpha1.TransactionInvoke, *corev1alpha1.TransactionInvokeList]{
	NewList: func() *corev1alpha1.TransactionInvokeList { return new(corev1alpha1.TransactionInvokeList) },
	ItemsFor: func(list *corev1alpha1.TransactionInvokeList) []corev1alpha1.TransactionInvoke {
	return list.Items
	},
	AsObject: func(item *corev1alpha1.TransactionInvoke) *corev1alpha1.TransactionInvoke { return item },
	}