design-proposal: VirtualMachineInstanceMigration - Live migration to a named node

[tiraboschi]

What this PR does / why we need it:
Adding a design proposal to extend VirtualMachineInstanceMigration
object with an additional API to let a cluster admin
try to trigger a live migration of a VM injecting
on the fly and additional NodeSelector constraint.
The additional NodeSelector can only restrict the set
of Nodes that are valid target for the migration
(eventually down to a single host).
All the affinity rules defined on the VM spec are still
going to be satisfied.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes https://issues.redhat.com/browse/CNV-7075

Special notes for your reviewer:
Something like this was directly proposed/implemented with kubevirt/kubevirt#10712 getting already discussed there.

Checklist

This checklist is not enforcing, but it's a reminder of items that could be relevant to every PR.
Approvers are expected to review this list.

• Design: A design document was considered and is present (link) or not required
• PR: The PR description is expressive enough and will help future contributors
• Code: Write code that humans can understand and Keep it simple
• Refactor: You have left the code cleaner than you found it (Boy Scout Rule)
• Upgrade: Impact of this change on upgrade flows was considered and addressed if required
• Testing: New code requires new unit tests. New features and bug fixes require at least on e2e test
• Documentation: A user-guide update was considered and is present (link) or not required. You want a user-guide update if it's a user facing feature / API change.
• Community: Announcement to kubevirt-dev was considered

Release note:

design-proposal: VirtualMachineInstanceMigration - Live migration to a named node

[kubevirt-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign davidvossel for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[dankenigsberg]

Lovely to see this clear design proposal (even if I don't like anything that assumes a specific node is long-living). I have two questions, though.

[dankenigsberg]

I find this odd, as the VM and the application in it may not function well (or at all) if affinity is ignored. Can you share more about the origins of this goal? I'd expect the target node to be ANDed with existing anti/affinity rules.

[tiraboschi]

I tend to think that as a cluster admin that is trying to force a VM to migrate to named node this is the natural and expected behaviour:
if I explicitly select a named node, I'm expecting that my VM will be eventually migrated there and nowhere else (such as on a different node selected by the scheduler according to a weighted combination of affinity criteria and resource availability and so on); then I can tolerate that the live migration will fail since I chose a wrong node, but the controller should only try to live-migrate it according to what I'm explicitly asking for.
And by the way this is absolutely consistent with the native k8s behaviour for pods.
spec.nodeName for pods is under spec for historical reasons but it's basically controlled by the scheduler:
when a pod is going to be executed, the scheduler is going to check it and, according to available cluster resources, nodeselectors, weighted affinity and anti-affinity rules and so on, it's going to select a node and write it on spec.nodeName on the pod objects. At this point the kubelet on the named node will try to execute the Pod on that node.
If the user explicitly sets spec.nodeName on a pod (or in the template in a deployment and so on), the scheduler is not going to be involved in the process since the pod is basically already scheduled for that node and nothing else and so the kubelet on that node will directly try to execute it there eventually failing.
https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#nodename explictly state:

If the nodeName field is not empty, the scheduler ignores the Pod and the kubelet on the named node tries to place the Pod on that node.
Using nodeName overrules using nodeSelector or affinity and anti-affinity rules.

And this in my opinion is exactly how we should treat a Live migration attempt to a named node.

[fabiand]

Let's take the following example (this is a real world use-case):

1. An admin is adding a new node to a cluster to take it into prod. This node has a taint to prevent workloads to immediately land there.
2. The admin wants to migrate a VM to this now to validted it is working properly.

If we AND a new selector for this node, then the migration will not take place, because there is the taint. We'd also need to add a toleration to get the vm scheduled to that node.

With spec.nodeName it would be no issue - initially - it could become one if Require*atRuntime effects are used.
However, with spec.nodeName all other validations - CPU caps, extended, storage, and local resources etc will be ignored. We are asking a VM to not start.
Worse: It would be really hard now to understand WHY the vm is not launching.

Thus I think we have to AND to the node selector, but need code to understand taints specifically (because taints keep workloads away).
Then we still need to think about a generic mechanism to deal with reasons of why a pod can not be placed on the selected node.

[dankenigsberg]

I do not like taking examples from the historically-understandable Pod.spec.nodeName. Node identity is not something that should have typically been exposed to workload owners.

Can you summarize your reasoning into the proposal? I think I understand it now, but I am not at ease with it. For example, a cluster admin may easily violate anti/affinity rules that are important for app availability.

[tiraboschi]

@fabiand with taints is a bit more complex: the valid effects for a taint are NoExecute, NoSchedule and PreferNoSchedule.
Bypassing the scheduler directly setting spec.nodeName will allow us to bypass taints with NoSchedule and PreferNoSchedule effect but, AFAIK, it will be still blocked by a NoExecute that is also enforced by the Kubelect with eviction.

[tiraboschi]

@dankenigsberg yes, this is a critical aspect of this design proposal so we should carefully explore and weight the different alternatives tracking them down in the design proposal itself as a future reference.

In my opinion the choice strictly depends on the use case and the power we want to offer to the cluster admin when creating a live migration request to a named node.

Directly setting spec.nodeName on the target pod will completely bypass all the scheduling hints (spec.affinity.nodeAffinity.preferredDuringSchedulingIgnoredDuringExecution) and constraints (spec.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution) meaning that the target pod will be started on the named nome regardless how the VM is actually configured.

Another option is trying to append/merge (this sub-topic deserves by itself another discussion) something like

spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchFields:
            - key: metadata.name
              operator: In
              values:
              - <nodeName>

to the affinity rules already defined on the VM.
My concern with this choice is that affinity/anti-affinity grammar is pretty complex so, if the VM owner already defined some affinity/anti-affinity rules, we can easily end up with a set of conflicting rules so that the target pod cannot be scheduled on the named node as on any other node.

If the use case that we want to address is giving to the cluster admin the right to try migrating a generic VM to a named node (for instance for maintenance/emergency reasons), this is approach is not fully addressing it with many possible cases where the only viable option is still about manually overriding affinity/anti-affinity rules set by the VM owner.

I still tend to think that the always bypass the scheduler with a spec.nodeName is the K.I.S.S. approach here if try to forcing a live migration to a named node is exactly what the cluster admin is trying to do.

[tiraboschi]

I summarized this considerations into the proposal itself, let's continue from there.

[dankenigsberg]

Would you outline the nature of the pushback? Do we currently have good answers to the issues raised back then?

[tiraboschi]

Trying to summarize (@EdDev please keep me honest on this), it was somehow considered a semi-imperative approach and it was pointed out that a similar behavior could already indirectly be achieved modifying on the fly and then reverting affinity rules on the VM object.
see: kubevirt/kubevirt#10712 (comment)
and: kubevirt/kubevirt#10712 (comment)

How much this is imperative is questionable: at the end we already have a VirtualMachineInstanceMigration object that you can use to declare that you want to trigger a live migration, this is only about letting you also declare that you want to have a live migration to a named host.

The alternative approach based on amending the affinity rules on the VM object and waiting for the LiveUpdate rollout strategy to propagate it to the VMI before trying a live migration is described, pointing out its main drawback, in the Alternative design section in this proposal.

[dankenigsberg]

Can you inline this succinctly? E.g, that Pr got some pushbacks because it was not clear why a new API for one-off migration is needed. We give here a better explanation why this one-off migration destination request is necessary.

[tiraboschi]

done

[EdDev]

• The "one-time" operation convinced me.
• The reasoning for the real need is hard for me, but I did feedback on this proposal what is convincing me.

[iholder101]

/cc

[davidvossel]

I'd like to see more fleshed out user stories. It's unclear to me based on these user stories why the existing methods wouldn't suffice.

As a cluster admin I want to be able to try to live-migrate a VM to specific node for maintenance reasons eventually overriding what the VM owner set

For example, why wouldn't the cluster admin taint the source node and live migrate the vms away using the existing methods? Why would the admin need direct control over the exact node the VM goes to? I'd like to see a solid answer for why this is necessary over existing methods.

That's where this discussion usually falls apart and why it hasn't seen progress through the years. I'm not opposed to this feature, but I do think we need to articulate clearly why the feature is necessary

[tiraboschi]

I expanded this section

[davidvossel]

nice! these are good reasons that hadn't been explored during previous discussions, thanks

[davidvossel]

I think using pod.spec.nodeName is likely the most straightforward approach. This does introduce some new failure modes that might not be obvious to admins.

For example, today if a target pod is unschedulable due to lack of resources, the migration object will time out due to the pod being stuck in "pending". This information is feed back to admin as an k8s event associated with the migration object.

However, by setting the pod.spec.NodeName directly, we'd be bypassing the checks that ensure the required resources are available on the node (like the node having the "kvm" device available for instance), and the pod would likely get scheduled and immediately fail. I don't think we are currently bubbling up these types of errors to the migration object, so this could leave admins wondering why their migration failed.

I guess what I'm trying to get at here is, I like this approach, let's make sure the new failure modes get reported back on the migration object so the Admin has some sort of clue as to why a migration has failed.

[vladikr]

@davidvossel We already report the failure reason on the VMIM. This is part of the VMIM status.

pod.spec.nodeName entirely bypassed the scheduler making AAQ unusable as it relies on "pod scheduling readiness".

From my pov, bypassing the scheduler is a no go.

[tiraboschi]

From my pov, bypassing the scheduler is a no go.

luckily we have also another option as described on:
### B. appending/merging an additional nodeAffinity rule on the target virt-launcher pod (merging it with VM owner set affinity/anti-affinity rules)

This will add an additional constraint for the scheduler summing it up with existing constraints/hints.
In case of mismatching/oppositing rules, the destination pod will not be scheduled and the migration will fail.

[iholder101]

@vladikr @davidvossel +1.

spec.nodeName is a horrible field that is not being removed from Kubernetes only due to backward compatibility and causes a lot of trouble. I agree that it should be considered as a no-go.

[vladikr]

I understand the intention behind introducing the nodeName field, but I fail to see how something like this may work at scale. It seems to me that most, if not all, of the user stories listed in the proposal can already be achieved through existing methods. Adding this field could potentially cause confusion for admins and lead to unnecessary friction with the Kubernetes scheduler and descheduler flows. I'd prefer to see solutions to the user stories to be aligned closely with established patterns. (descheduler policies or scheduler plugins )

[vladikr]

I wonder what would be so special about these VMs that cannot be handled by a descheduled?
Also, how would the admin know that the said descheduler did not remove these VMs at a later time?

[tiraboschi]

The descheduler it's going to decide according to its internal policy.
In the more general use case it will be a cluster admin who can decide to live migrate a VM just because he thinks it's the right thing to do.

[vladikr]

This could be achieved today by modifying the VM's node selector or creating a new VM. New nodes will be the schedulers' very likely target for new pods already.

[tiraboschi]

Right,
from a pure technical perspective this feature can be already simply achieved directly manipulating the node affinity rules on the VM object. Now we have LiveUpdate rollout strategy and so the new affinity rules will be quickly propagated to the VMI and so consumed on the target pod of the live-migration.
No doubt, on the technical side it will work.

But the central idea of this proposal is about allowing a cluster admin doing that without touching the VM object.
This for two maina reasons:

• separation of personas: the VM owner can set rules on his VM, a cluster admin could be still interested in migrating a VM without messing up or altering the configuration set by the owner on the VM object.
• separating what it a one-off configuration for the single migration attempt (so set on the VirtualMachineInstanceMigration object) that is relevant only for this single migration attempt but it should not produce any side effect in the future from what is a long-term configuration that is going to stay there and be applied also later on (future live migrations, restarts).

This comment applies to all the user stories here.

[vladikr]

This is also doable today as the default scheduler will try to choose the least busy node to schedule the target pod.

[vladikr]

Could you please elaborate on this?
How would the cluster look like to the admins' expectations?
Couldn't a taint be placed on some nodes to resolve capacity before the new product announcement?

[EdDev]

Same, I do not want to argue with an admin on how the cluster should be managed, but this is surely not a recommended way we want to encourage/support.

[tiraboschi]

It seems to me that most, if not all, of the user stories listed in the proposal can already be achieved through existing methods.

Right, I also added this note:

Note

technically all of this can be already achieved manipulating the node affinity rules on the VM object, but as a cluster admin I want to keep a clear boundary between what is a long-lasting setting for a VM, defined by the VM owner, and what is single shot requirement for a one-off migration

[vladikr]

I spoke with @fabiand offline.
Perhaps we can simply copy any provided Affinity and/or Tolerations set but the admin on the VMIM to the target pod -
instead of offering a dedicated API field.

My main concern with this proposal is that it may promote a wrong assumption that manual cluster balancing is preferred instead of relying on the scheduler/descheduler - while this is just a local minimum.

[tiraboschi]

I spoke with @fabiand offline. Perhaps we can simply copy any provided Affinity and/or Tolerations set but the admin on the VMIM to the target pod - instead of offering a dedicated API field.

I think that exposing the whole node affinity/anti-affinity (+ tolerations + ...) grammar on the VirtualMachineInstanceMigration object is by far too much.
At the end, as a cluster admin I want to only to try to migrate that VM to a named node. All the other uses cases are out of scope and should be addressed correctly setting/amending the node affinity on the VM.
I still think that exposing an optional nodeName string on the VirtualMachineInstanceMigration spec is all of what we need to accomplish all the use cases here.

My main concern with this proposal is that it may promote a wrong assumption that manual cluster balancing is preferred instead of relying on the scheduler/descheduler - while this is just a local minimum.

I think it's up to us to emphasize this assumption in the API documentation making absolutely clear that the nodeName field is optional and we recommend to keep it empty to let the scheduler find the best node (if trying to migrate to a specific named node is not strictly needed).

I'm proposing something like:

// NodeName is a request to try to migrate this VMI to a specific node.
// If it is non-empty, the migration controller simply try to configure the target VMI pod to be started onto that node,
// assuming that it fits resource, limits and other node placement constraints; it will override nodeSelector and affinity
// and anti-affinity rules set on the VM.
// If it is empty, recommended, the scheduler becomes responsible for finding the best Node to migrate the VMI to.
// +optional
NodeName string `json:"nodeName,omitempty"`

I'm adding it to this proposal.

[vladikr]

I spoke with @fabiand offline. Perhaps we can simply copy any provided Affinity and/or Tolerations set but the admin on the VMIM to the target pod - instead of offering a dedicated API field.

I think that exposing the whole node affinity/anti-affinity (+ tolerations + ...) grammar on the VirtualMachineInstanceMigration object is by far too much. At the end, as a cluster admin I want to only to try to migrate that VM to a named node.

Setting affinity and toleration is exactly what any other user would need to do to allow scheduling a workload on tainted node, not sure why we need to facilitate this in the migration case.
Also, taking this route would not require us to add any new logic to the migration controller.

Generally speaking, Affinity and nodeSector are the most acceptable ways to influence scheduling decisions.

All the other uses cases are out of scope and should be addressed correctly setting/amending the node affinity on the VM. I still think that exposing an optional nodeName string on the VirtualMachineInstanceMigration spec is all of what we need to accomplish all the use cases here.

My main concern with this proposal is that it may promote a wrong assumption that manual cluster balancing is preferred instead of relying on the scheduler/descheduler - while this is just a local minimum.

I think it's up to us to emphasize this assumption in the API documentation making absolutely clear that the nodeName field is optional and we recommend to keep it empty to let the scheduler find the best node (if trying to migrate to a specific named node is not strictly needed).

From my pov, we could get away without any API changes and without advertising this option at all - making it available for special cases and not a mainstream.

I'm proposing something like:

// NodeName is a request to try to migrate this VMI to a specific node.
// If it is non-empty, the migration controller simply try to configure the target VMI pod to be started onto that node,
// assuming that it fits resource, limits and other node placement constraints; it will override nodeSelector and affinity
// and anti-affinity rules set on the VM.
// If it is empty, recommended, the scheduler becomes responsible for finding the best Node to migrate the VMI to.
// +optional
NodeName string `json:"nodeName,omitempty"`

I'm adding it to this proposal.

[tiraboschi]

Setting affinity and toleration is exactly what any other user would need to do to allow scheduling a workload on tainted node, not sure why we need to facilitate this in the migration case. Also, taking this route would not require us to add any new logic to the migration controller.
...
From my pov, we could get away without any API changes and without advertising this option at all - making it available for special cases and not a mainstream.

I'm sorry but now I'm a bit confused.
As for Kubevirt documentation,
in order to initiate a live migration I'm supposed to create VirtualMachineInstanceMigration (VMIM) object like:

apiVersion: kubevirt.io/v1
kind: VirtualMachineInstanceMigration
metadata:
  name: migration-job
spec:
  vmiName: vmi-fedora

or, more imperatively, executed something like:

$ virtctl migrate vmi-fedora

that under the hood is going to create a VirtualMachineInstanceMigration for me.

This proposal is now about extending it with the optional capability to try to live migrate to a named node.
So thi is proposing to allow the creation of something like:

apiVersion: kubevirt.io/v1
kind: VirtualMachineInstanceMigration
metadata:
  name: migration-job
spec:
  vmiName: vmi-fedora
  nodeName: my-new-target-node

or executing something like:

$ virtctl migrate vmi-fedora --nodeName=my-new-target-node

and this because one of the key point here is that the cluster admin is not supposed to be required to amend the spec of VMs owned by other users in order to try to migrate them to named nodes.

The migration controller will simply notice that nodeName on the VirtualMachineInstanceMigration is not empty and it will inject/replace (still under discussion, we have two alternatives here) something like:

spec:
  nodeName: <nodeName>

or

spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchFields:
              - key: metadata.name
                operator: In
                values:
                  - <nodeName>

on the target virt-launcher pod.

Can you please summarize what do you exactly mean with

Perhaps we can simply copy any provided Affinity and/or Tolerations set but the admin on the VMIM to the target pod - instead of offering a dedicated API field.

?

[fabiand]

@iholder101 sorry for being picky, but it is not "used to it" - There is simply no way to address the outlined use-cases from an admin perspective. And e should reocgnize that the use-cases are simply problems that administrators face when running virtualization platforms. Thus to some extent we could say that KubeVirt is only concerned about parts of the use-case, because some of the causes are actually platform driven, and KubeVirt is just a platform add-on.

Existing functionality allows VM owners to place workloads.
There is no functionality on a VM level, that allows an administrator to place a workload.

[EdDev]

Such an admin should probably run a checkup [1].

[1] https://github.com/kiagnose/kubevirt-dpdk-checkup

[tiraboschi]

I still see this proposed API as an enabler for such eventual automated check.
Without this additional API, the checkup code should handle an imperative flow amending the spec ofthe named VM a few times in a sequence to trigger the behaviour to be verified.

[fabiand]

@EdDev both have a use-case.

But hte check up tool is not helping the admin to test exactly one node in the way s/he needs to test it.

[EdDev]

@fabiand , for me, this is an attempt to explain that the need can be solved with other means.
I agree that the solution of defining a specific target node solves this and other needs. However, some argue that it is not an ideal solution. This repeats itself on several points.

[EdDev]

I do not understand this goal.
It is already the case today, is there a new goal beyond what exists today?

[tiraboschi]

This is stating that in term of exit status, a migration to selected node via the optional API should behave exactly as migration behaves today when the target node is selected by the scheduler: no side effects or try this first and then eventually let the scheduler decide or anything like that.

[fabiand]

Today live migrations can fail or many reasons.
With this change there is no change to this, live migrations can still fail in many ways.

Technically it's really no change at all, because live migration depens on meeting scheduling constraints. constraints are cpu flags, memory, storage etc - and with this feature locality will be simply one additional constratint,. but the failure more is still the same.

[EdDev]

IMO this should be dropped as nothing special is added or changed.

[tiraboschi]

done, dropped

[EdDev]

For long-lasting placement, nodeSelectors or affinity/anti-affinity rules directly set on the VM spec are the only way to go.

As a goal, there is no need to explain how to do something else. I think this is already mentioned well in the motivation.

[fabiand]

There was enough tension about this poin tof why setting affinity on the VM is not a solution. Thus mentioning it here sounds good.

[EdDev]

I won't fight over this, but IMO a detailed explanation does not fit in the goal.

[EdDev]

I recall this from memory, but it is a real world request from multiple end users.

The need of the admin is clear.
But we have better tooling for the admin to use: https://github.com/kiagnose/kiagnose

[EdDev]

I second @iholder101 .

I do not think Kubevirt is interested in suggesting such an action to a cluster admin.
While I have no interest in arguing with an admin on how one should manage the cluster, I prefer not to try and convince our-self that this makes sense and feel it is a valid scenario.

[EdDev]

I do not understand this one.

[tiraboschi]

Let's think about dev-ops, infrastructure as code or platform engineering: as a VM owner (or as the owner of a complex application composed by VMs, regular pods, services...) I want to store and control its configuration by an external mean and I do not like to have the admin of the KubeVirt cluster interfering with my objects to configure the target for a live migration: this should be eventually done on the migration object, not on my own object.
It's a matter of separation of boundaries.

[EdDev]

I find the user-stories always a challenge.
As I see it, the stories (and even the goals) are suppose to describe the need of the user and attempt to avoid providing the solution. Here, the stories provide the solution upfront: Migrate a VM to a selected node.

If we drop that solution, you may see that stories can be resolved by other means, weakening the solution to migrate to a specific node.

Said all that, there is one user story which is not specified here and is convincing me:

As a cluster admin that managed various VMM systems, I have well defined processes and steps that proved useful when managing VMs.
One of the basic actions I require to continue following my processes and steps is to have control over the node target when migrating a VM.
With time, I expect to learn and trust Kubevirt to achieve my needs through other means, but until that time, I would like to keep the capabilities I learned to trust so far.

At its base, I acknowledge that Kubevirt is not a classic VMM, it does things very differently, however it has interest to make it easy for VMs from other platforms to be migrated to it. That means making the admins happy.

[iholder101]

I agree @EdDev. This indeed sounds much more compelling than the other use-cases.
I do think that if that's the rationale I do think we should make it clear documentation-wise.

[tiraboschi]

No, this is not simply about laziness on learning something new.
Many if not all of the user stories presented here are not directly achievable if not installing optional or even to be developed scheduler and descheduler plugins able to observe real time metrics or using imperative multi steps flows.

[EdDev]

Many if not all of the user stories presented here are not directly achievable if not installing optional or even to be developed scheduler and descheduler plugins able to observe real time metrics or using imperative multi steps flows.

As I see it, a user-story should not be interested on how it is implemented (e.g. installing add-ons, depending on features, etc), it is only about expressing the need of a user.

I think it is problematic to decide on the solution and embed it in the user-story and goals.
As I see it, this is the main friction on the proposal. Reviewers have reservations on the solution and not on the need.

[EdDev]

1. Reliability can be mitigated by using a label we control.
2. Having more options is also a con. It opens up options which may be too wide, increasing the chance we will need to help users to use it correctly.
   While I agree we need to keep this as generic as possible, having a simple solution is an advantage.

@iholder101 , you mention that nodeSelector is bad and it is left there for backward compatibility reasons. Is this written in any formal location? If it is, then this section can ref it as a strong no-go reason.

Anyway, I would add these points even if the previous option is chosen/better.

[iholder101]

@EdDev I think you're referring to what I wrote about .spec.nodeName. Unlike node affinities or nodeSelectors, it is used to bypass the scheduler, which is a huge problem and breaks stuff very quickly. It's probably almost a bad idea to use it, except for very extreme cases.

You can look this this conversation which is interesting regarding the topic: kubernetes/enhancements#3521 (comment)

[EdDev]

That’s it, thanks. This should be added here as the main con then.

Edit: strike that, now I see my confusion

[tiraboschi]

This is already tracked in the Exposing spec.nodeName and directly setting it to the target pod bypassing the k8s scheduler section on the proposal.

[EdDev]

• The "one-time" operation convinced me.
• The reasoning for the real need is hard for me, but I did feedback on this proposal what is convincing me.

[EdDev]

I missed to add to my review that I really like this design proposal. The investment in the motivation and alternative solutions is impressive. 💪

[iholder101]

@tiraboschi

but as an expert cluster admin I simply want to live migrate a couple of selected VMs according my observations of the KubeVirt metrics on VMs.

I feel we're always circling back to "the admin just wants to do that".
When I talk about use-cases, this is not what I mean.
I understand that some admins ask for this, but I think the more important thing is to understand why they want it, for which purpose. This is important, since then we can think of providing different ways of achieving the end-goal they seek to achieve with different mechanisms. That should be the basis of this discussion.

Neither the descheduler is really acting according to actual node resources but only to resource requirements,

A descheduler plugin could support that.
As a continuation to the above, if the admin wants to re-balance the cluster according to a certain logic and this is the reason he wants to migrate VMs to a named node in the first place - this mechanism is a good alternative. Even if as you outline the current plugins aren't good enough, it might be in the interest of Kubevirt to write one.

---

@fabiand

There is simply no way to address the outlined use-cases from an admin perspective

I've added comments on how I think we should address these use-cases above.

And e should reocgnize that the use-cases are simply problems that administrators face when running virtualization platforms.

This is correct, however, sometimes an admin wants to achieve a certain goal but chooses to use the wrong mechanisms in order to do so. This is more likely to happen when the admin comes from different virt platforms which makes him less experienced with the Kubernetes world.

---

@fabiand
If I try to stay practical about this proposal, I think that these are needed to be resolved in order to move forward:

• @EdDev's reason to support this is the most compelling in my opinion. The proposal should get rid of the other use-cases which are not compelling IMO.
• We should document that, and perhaps even be more vocal (i.e. using alerts, events, etc) about the fact that using this mechanism is discouraged and encourage alternative mechanisms to achieve the same goals.
• We should resolve who should be allowed to trigger a migration, i.e. if it should be an admin-only operation.
• We should consider introducing an alpha non-API mechanism first (e.g. through annotations?) in order to present this to some users and have an opportunity to gain feedback to ensure the door is opened to make drastic changes in the future or even drop this if proved unuseful.

I believe my views are thoroughly expressed here already, so I'll shut up now and let others express their opinions as well.
@alicefr @jean-edouard @xpivarc @davidvossel @rmohr

[tiraboschi]

• @EdDev's reason to support this is the most compelling in my opinion. The proposal should get rid of the other use-cases which are not compelling IMO.

Other cases are here because they are valid by themselves and not directly addressable without relying on multi-steps imperative flows.
The only use case that is not really helpful in the discussion is let's make the admins happy because they are used to it.

• We should document that, and perhaps even be more vocal (i.e. using alerts, events, etc) about the fact that using this mechanism is discouraged and encourage alternative mechanisms to achieve the same goals.

same.

• We should resolve who should be allowed to trigger a migration, i.e. if it should be an admin-only operation.

I don't see any real implication of that regarding this API.
That topic deserves an independent discussion.

• We should consider introducing an alpha non-API mechanism first (e.g. through annotations?) in order to present this to some users and have an opportunity to gain feedback to ensure the door is opened to make drastic changes in the future or even drop this if proved unuseful.

Honestly abusing annotations to avoid defining APIs is a bad practice.
I think that this deserves to be a first class citizen.
Let's agree on how we can define a good API, but then let's properly implement it.

I believe my views are thoroughly expressed here already, so I'll shut up now and let others express their opinions as well. @alicefr @jean-edouard @xpivarc @davidvossel @rmohr

[EdDev]

• @EdDev's reason to support this is the most compelling in my opinion. The proposal should get rid of the other use-cases which are not compelling IMO.

Other cases are here because they are valid by themselves and not directly addressable without relying on multi-steps imperative flows.

I think there is a disagreement here on the solution, not on the need.
You have provided use cases where choosing a target node solves needs.
Most feedback I saw on this proposal is about solving the need through other means.

Perhaps you can emphasize the troubleshooting need. I guess it will be hard to find alternative solutions to it.
The problem is that some may not see it as worth the effort and API change.

The only use case that is not really helpful in the discussion is let's make the admins happy because they are used to it.

It is helpful if it convinces others that there is no other alternative solution to it.
I agree it will be better to find a need that the only solution (or the best solution out of other alternatives) is this target node option. And that need should be convincing to invest in adding it to the project.

[tiraboschi]

I think there is a disagreement here on the solution, not on the need.
You have provided use cases where choosing a target node solves needs.
Most feedback I saw on this proposal is about solving the need through other means.

Personally I still think that letting the cluster admin inject a NodeSelectorTerm that is valid only on the specific one-off migration attempt is a simple and smart solution: it's a simple and smart solution because the same simple solution applies exactly in the same way to all the use cases detailed in this proposal.
We can for sure assume that each individual use case tracked here can be otherwise solved with a different ad-hoc solution.
But I like the proposed solution because is the most intuitive (at least for cluster admins with a solid background in traditional virtualization systems) one and it applies exactly in the same way to all the use cases mentioned here.

Perhaps you can emphasize the troubleshooting need. I guess it will be hard to find alternative solutions to it.
The problem is that some may not see it as worth the effort and API change.

I want also to better understand the reasons why we should not accept it.
Are we talking about a crazy new API that doesn't exists in the Kubernetes ecosystem?
No, NodeSelectorTerm is part of Kubernetes core v1

Are we saying that propagating that NodeSelectorTerm is to complex/risky for KubeVirt controllers?
But the KubeVirt code is already using exactly this pattern to handle HostModelCpuModel and ForbiddenFeaturePolicy: here we are only talking about using an existing mechanism to let the user influence the target node of a migration attempt but from coding perspective the logic is already there.

[davidvossel]

whoa. I hadn't looked at this thread in awhile. So much discussion. We're hitting analysis paralysis at this point.

At the end of the day, here's the reality as I see it. Admins are asking for this feature... And they're asking for it over and over. For years. I tried to ignore it in the past, and guide people to our preferred way of doing this sort of thing, but the requests keep coming. I'm worn down.

In my opinion I'm at the point where I say let's just do this and move on. The nodeSelectorTerm on the VMIM seems fine. The alternative with nodeSelectors seems fine as well.

Here's what I'm most interested in now... Does anyone see how proceeding with this will cause any future harm or complexity to the project that impacts our ability to maintain KubeVirt?

[fabiand]

@EdDev @vladikr I changed the proposal slightly to highlight: This is about expcetions.
The feature being proposed here is for exceptional cases. Debugging, researching, understanding, directing. As an admin, transpraently to the user.

Today, I am not aware of a procedure or tool to move a VM to a specific node wihtout modifying the VM definition itself.

This tool is for our users, for cluster administrators, it feels like somebody has to beg to get this in.
It feels like the layed out user-stories are not except, while otoh they come from end-users.

Is our answer that: No, KubeVirt does not support this?
We make the lifes of our admins harder, and I do not see this as the mission of KubeVirt.

This feature is

• small
• declarative
• not conflicting
• low risk

I' not asking why this shoul dbe kept out.
Can you please tell me what are the next steps to get this in?

[fabiand]

@EdDev both have a use-case.

But hte check up tool is not helping the admin to test exactly one node in the way s/he needs to test it.

[fabiand]

Today live migrations can fail or many reasons.
With this change there is no change to this, live migrations can still fail in many ways.

Technically it's really no change at all, because live migration depens on meeting scheduling constraints. constraints are cpu flags, memory, storage etc - and with this feature locality will be simply one additional constratint,. but the failure more is still the same.

[fabiand]

There was enough tension about this poin tof why setting affinity on the VM is not a solution. Thus mentioning it here sounds good.

[dankenigsberg]

We can make this even more convincing if we mention: I'm am a long-time cluster-admin of another, well-established VM management system. Missing this functionality, of one-off placement of a VM on a node, in KubeVirt is one of the reasons I haven't expanded my KubeVirt footprint.

[EdDev]

I find it hard to be convinced that the project should provide manual solutions to admins instead of an automated, self managed one.

I do find it convincing to provide reasoning like the one @dankenigsberg raised here, just because the admin wants to have control like he had on other projects. It is convincing for me because I have no alternative to give for that need.

[EdDev]

Can you please tell me what are the next steps to get this in?

We will escalate this enhancement in the next maintainer/approver meeting and update.

I still think that the way forward is to use a goal and user-story that have no better alternatives.
And I think the troubleshooting and the admin experience & expectations from other systems is the best one [1].

[1] #320 (comment)

[tiraboschi]

I still think that the way forward is to use a goal and user-story that have no better alternatives. And I think the troubleshooting and the admin experience & expectations from other systems is the best one [1].

[1] #320 (comment)

I added

• The capability of live migrating a VM to a specific node is a pretty common and accepted feature across traditional virtualization solutions and also cloud native ones. The lack of this functionality on KubeVirt is currently seen as a bold limit by some potential users. Enabling it on KubeVirt, although without recommending it as the default pattern when alternatives are present, could help foster KubeVirt adoption.

to the list of goals and

• As a cluster admin that managed various traditional and cloud native VM management systems, I have well-defined processes and steps that proved useful when managing VMs.
  One of the basic actions I require to continue following my processes and steps is to have control over the node target when migrating a VM.
  With time, I expect to learn and trust Kubernetes and KubeVirt to achieve my needs through other means, but until that time, I would like to keep the capabilities I learned to trust so far.

to the list of user stories.

[vladikr]

@EdDev @vladikr I changed the proposal slightly to highlight: This is about expcetions. The feature being proposed here is for exceptional cases. Debugging, researching, understanding, directing. As an admin, transpraently to the user.

Today, I am not aware of a procedure or tool to move a VM to a specific node wihtout modifying the VM definition itself.

This tool is for our users, for cluster administrators, it feels like somebody has to beg to get this in. It feels like the layed out user-stories are not except, while otoh they come from end-users.

Indeed, but on the other hand, would allow features that may hard key process in, in the requested form?
As KubeVirt relies on live migration for upgrades, wouldn't you agree that being conservative regarding features related to live migration is justified?

I keep wondering if the same feature could not be achieved in a more "cloudy" - however, I'm not blocking on this.

Is our answer that: No, KubeVirt does not support this? We make the lifes of our admins harder, and I do not see this as the mission of KubeVirt.

This is the key issue. Migrations today are not limited to cluster admins.
In reality, any user can be a namespace admin and trigger migrations.

This feature is

• small
• declarative

• not conflicting
• low risk

@fabiand how do you define risk?
Since each user can potentially trigger the migration of all VMs in the namespace, the number of queued migrations may not be low.
Migrations to a specific node may cause the descheduler to act, potentially increasing the migrations queue.
Since the migrations queue is long this may delay system upgrades.
This will also affect the VMs with a higher priority that are relying on eviction to get scheduled to wait (unestimated amount of time?), delaying hotplug.

This new API also opens the door to queueing up migrations and delaying important tasks that rely on live-migration without control - I think it might be a risk.

Perhaps before proceeding with this feature, we should implement a priority queue for important tasks and some logic around handling important and less important migrations.

I' not asking why this shoul dbe kept out. Can you please tell me what are the next steps to get this in?

[tiraboschi]

This new API also opens the door to queueing up migrations and delaying important tasks that rely on live-migration without control - I think it might be a risk.

Honestly I don't think that this feature by itself is going to make this worse.
If not otherwise bound, a user can already enqueue an arbitrary huge amount of migration requests (even if he cannot specify/limit the target node).
Is this going to become worse just because the user will be allowed to limit the migration scope to a subset of the nodes? I don't think so.
We can eventually think about imposing by default a resource quota on VMIM count per namespace.

[dankenigsberg]

Migrations to a specific node may cause the descheduler to act, potentially increasing the migrations queue.
Since the migrations queue is long this may delay system upgrades.

Much like vladik, I don't like it that project owners can migrate a VMI (it's not "cloudy", as is the fact they can set kubernetes.io/hostname), but I don't think that this design proposal makes things worse in this regards. On the other hand, it adds a feature that our users are repeatedly asking form, hence my support.

/lgtm

[iholder101]

@davidvossel

Here's what I'm most interested in now... Does anyone see how proceeding with this will cause any future harm or complexity to the project that impacts our ability to maintain KubeVirt?

Further to the risks @vladikr have mentioned, I think this proposal imposes a "philosophical" question: is it reasonable to add a feature that admins ask for although as developers we believe has no real use-cases and is the wrong way of doing things? Is our responsibility to simply please the users in this case, or try to educate them towards using the right mechanisms, even if they're used to doing things in a different way?

At the end of the day, here's the reality as I see it. Admins are asking for this feature... And they're asking for it over and over. For years. I tried to ignore it in the past, and guide people to our preferred way of doing this sort of thing, but the requests keep coming. I'm worn down.

What you write here is consistent with what @tiraboschi have added to the proposal:

The capability of live migrating a VM to a specific node is a pretty common and accepted
feature across traditional virtualization solutions ...

This text openly admits that the sole use-case here is that admins are used to this feature that's common on other platforms.

So, eventually, we need to decide if this reasoning is enough:

For years. I tried to ignore it in the past, and guide people to our preferred way of doing this sort of thing, but the requests keep coming. I'm worn down.

Is being worn down from user requests justifies adding a feature we believe is wrong?
Or should we instead insist to guide people to our preferred way of doing this sort of thing?

[tiraboschi]

What you write here is consistent with what @tiraboschi have added to the proposal:

The capability of live migrating a VM to a specific node is a pretty common and accepted
feature across traditional virtualization solutions ...

This text openly admits that the sole use-case here is that admins are used to this feature that's common on other platforms.

No, I don't think that we can derive this.
All the goals and the use cases in the proposal are individually valid.
We could eventually state that different use cases expressed here can eventually find alternative solutions with the existing APIs.
The big advantage of this proposal is that a single API and approach can fit them all and this will lower the barrier to entry for new users coming from other solutions.

[EdDev]

We will escalate this enhancement in the next maintainer/approver meeting and update.

Most of what I am going to summarize below has been raised already above, however, I think it is useful to see the whole picture in one place.

The topic has been discussed in the maintainer meeting on the 4th of November:

1. Reasoning needs to be based on something that does not have other better alternatives.
2. Adding the option to set a target node needs to be available only for Cluster Admins but not for Namespace Admins. This is partially an existing problem of who is able to trigger a migration, but the addition will expand the problem.
3. The proposed selector needs to be re-reviewed by @vladikr . The explanation of why it was chosen is well explained at the moment. Most thought that the proposal is fine in this regard.

I think that the 1st point can be easily addressed and probably not a major blocker. I have not re-reviewed the latest changes done in the last few days.

The 2nd point can be solved at the implementation, but it does needs some research, checking if we can validate that the target node selector can only be valid for a cluster-admin.
At the minimum, I think it is a valid concern and if there is a simple solution, I would vote to handle it.

Hopefully the 3rd point will pass @vladikr .

[tiraboschi]

2. Adding the option to set a target node needs to be available only for Cluster Admins but not for Namespace Admins. This is partially an existing problem of who is able to trigger a migration, but the addition will expand the problem.

@EdDev, can you please explain why this proposal will expand the problem?
Currently a VM owner can already amend node affinity on the VM spec of a running VM and it will be eventually propagated down to the VMI object (if the cluster is configured with vmRolloutStrategy: "LiveUpdate"). At that point creating a VMIM object in that namespace will trigger a live migration to the target node that was indirectly selected acting on a different object.
But the result will be exactly the same with the same risks and the same concerns.

[EdDev]

2. Adding the option to set a target node needs to be available only for Cluster Admins but not for Namespace Admins. This is partially an existing problem of who is able to trigger a migration, but the addition will expand the problem.

@EdDev, can you please explain why this proposal will expand the problem? Currently a VM owner can already amend node affinity on the VM spec of a running VM and it will be eventually propagated down to the VMI object (if the cluster is configured with vmRolloutStrategy: "LiveUpdate"). At that point creating a VMIM object in that namespace will trigger a live migration to the target node that was indirectly selected acting on a different object. But the result will be exactly the same with the same risks and the same concerns.

I think @vladikr can explain this better.
While the same outcome can be reached by updating the VMs, with the new option it will be very easy to create an alternative migration controller that will bypass and prioritize a target node over other logic.