-
Notifications
You must be signed in to change notification settings - Fork 14.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Show how to use scheduling profiles on k8s.io/docs/concepts/scheduling-eviction/kube-scheduler/ #21128
Comments
If you're documenting a “how to” guide for configuring the scheduler and its plugins, I recommend adding that to the Tasks section. |
/priority backlog |
Please add me in the reviews |
Just for the record, If you want to configure your scheduler thru KinD tool you can use the following instructions: cat << EOF > scheduler-config.conf
apiVersion: kubescheduler.config.k8s.io/v1beta1
clientConnection:
acceptContentTypes: ""
burst: 100
contentType: application/vnd.kubernetes.protobuf
kubeconfig: /etc/kubernetes/scheduler.conf
qps: 50
disablePreemption: false
enableContentionProfiling: true
enableProfiling: true
extenders: null
healthzBindAddress: 0.0.0.0:10251
kind: KubeSchedulerConfiguration
leaderElection:
leaderElect: true
leaseDuration: 15s
renewDeadline: 10s
resourceLock: endpointsleases
resourceName: kube-scheduler
resourceNamespace: kube-system
retryPeriod: 2s
metricsBindAddress: 0.0.0.0:10251
percentageOfNodesToScore: 0
podInitialBackoffSeconds: 1
podMaxBackoffSeconds: 10
profiles:
- schedulerName: default-scheduler
plugins:
score:
disabled:
- name: DefaultTopologySpread
EOF
cat << EOF | kind create cluster --config=-
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
kubeadmConfigPatches:
- |
kind: ClusterConfiguration
scheduler:
extraArgs:
config: /etc/kubernetes/scheduler-config.conf
extraVolumes:
- name: configuration
hostPath: /etc/kubernetes/scheduler-config.conf
mountPath: /etc/kubernetes/scheduler-config.conf
readOnly: true
pathType: File
- role: worker
- role: worker
EOF
fi
docker cp scheduler-config.conf kind-control-plane:/etc/kubernetes/scheduler-config.conf |
Feel free to submit pull requests |
Issues go stale after 90d of inactivity. If this issue is safe to close now please do so with Send feedback to sig-testing, kubernetes/test-infra and/or fejta. |
/remove-lifecycle stale |
Issues go stale after 90d of inactivity. If this issue is safe to close now please do so with Send feedback to sig-testing, kubernetes/test-infra and/or fejta. |
Stale issues rot after 30d of inactivity. If this issue is safe to close now please do so with Send feedback to sig-contributor-experience at kubernetes/community. |
I've struggled with configuring and working with scheduler plugins for a day or two now because of poor and misleading documentation about scheduler configuration. A link to this github issue from the scheduler documentation would be highly appreciated. Also, I've been trying to configure plugins with minikube but haven't fully understood how to do that, which lead to me trying to work with kubeadm or kind instead. |
/help unfortunately, we don't have bandwidth to improve the docs at the moment, but we welcome PRs |
@alculquicondor: Please ensure the request meets the requirements listed here. If this request no longer meets these requirements, the label can be removed In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
cc @Huang-Wei |
Hi @Frans-Lukas, I'm sorry for your frustration. Being that the scheduler Profiles config is still relatively new (and many are still transitioning to it) we are open to feedback and there is ongoing work to make the documentation better. Could you share more specifically what in the current documentation (mostly https://kubernetes.io/docs/reference/scheduling/config/) was misleading or not helpful, and what parts of this issue were most helpful that you think should be documented for others? At its core, the plugins configuration is not much different than the old Policy configuration (in terms of how you use it), and in general applies across most k8s installs:
There are many ways to do this in different environments, such as in #21128 (comment), and it may be worth us documenting those different paths. Like @alculquicondor mentioned though, any help in improving this documentation is welcome |
Thank you both for your quick replies. Some of the biggest issues were in understanding where the config files are located, how to "mount the configmap" and if the config file should be edited inside the master node or in the hosting machine. I.e. what is a config object and how do I mount it? It felt like some of the documentation was contradictory. I'll make a more detailed write up of how we interpreted the documentation and what we think is missing tomorrow. |
@Frans-Lukas setting up pods to read from configmap data is an essential feature of Kubernetes (not just related to the scheduler, it can be used with your own application pods as well). The options for that are well documented, please see https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/. Or, if you wish to read the file directly from the host, you can mount local directories and files into pods using Volumes https://kubernetes.io/docs/concepts/storage/volumes/ With an understanding of how one can use configmaps and volumes to pass data to your pods, setting the configuration for the scheduler is essentially no different (though it is often run as a static pod, meaning changes will not be automatically picked up by the apiserver). Much of the scheduler documentation does assume that users will already be familiar with these concepts, however it may be helpful to reference them more clearly. |
@damemi Alright, we got it working! Your reply definitely helped, thank you. We originally followed this tutorial: https://github.com/kubernetes-sigs/scheduler-plugins/blob/master/doc/install.md as it was the only one with clear instructions as to how to modify the scheduler plugins, but we got strange errors about
Without any instructions or links on how to enable or disable plugins. It does link to the github repository linked above with instructions that seems to be deprecated. Anyway, for anyone else having similar issues, what we did was that we entered the master node of the cluster, edited
With the contents of
This makes the scheduler pod fail as there must be a queueSort plugin, however, if the scheduler fails because of this, we know that the config works. |
@Frans-Lukas that's great, I'm glad it is working for you :)
Yeah, that's a typo on that page. It should be
kube-scheduler is the scheduler binary, what is running in your pod under
There are a few example configurations on the page linked to from that section, such as multiple profiles which links to another in extension points, the latter specifically showing how to enable and disable by extension point. We could make these examples more prominent. Thank you for the detailed feedback! Knowing exactly where we can improve will make it much easier for future users. |
@damemi Great! I'm glad I'm able to help. The errors we are getting from trying to use the config in https://github.com/kubernetes-sigs/scheduler-plugins/blob/master/doc/install.md after changing to
|
@Frans-Lukas ah, the relevant message there is
This is because the sample config in Since you are using it with the default scheduler, it fails to start because it does not have the |
Note that we cannot say something like that in https://kubernetes.io/docs/reference/scheduling/config/ as users can setup the scheduler in many different ways, using such file or not.
Note that the doc links to https://kubernetes.io/docs/reference/command-line-tools-reference/kube-scheduler/ It seems that you are not trying to develop new scheduling plugins, you just want to configure the k8s scheduler, so most of https://github.com/kubernetes-sigs/scheduler-plugins/blob/master/doc/install.md doesn't apply to you. Again, I encourage you to send a PR to improve https://kubernetes.io/docs/reference/scheduling/config/. Note that we have a long-standing related issue that we are still looking for help with #23889. This would allow to have the code documentation for the config file in the website. For now, to learn the details of this file, you have to look at the code https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/kube-scheduler/config/v1beta1/types.go |
@damemi I compiled the scheduler-plugins image (according to these instructions), and set the image in my
Which I thought would create the scheduler pod with the custom scheduler image but I still get the same error message.
@alculquicondor We are trying to develop a plugin, just starting out with trying to enable the default ones. I'd love to send a PR as soon as I feel like I understand the process. I am just starting to get comfortable working with configuring the scheduler, but I feel like there are still some missing pieces. Again, thanks for the help. |
If you are looking into creating your own plugin, consider opening an issue in https://github.com/kubernetes-sigs/scheduler-plugins/ instead. |
@Frans-Lukas Let's move the discussion to https://github.com/kubernetes-sigs/scheduler-plugins/ as here the topic is more about configuring default scheduler with multiple profiles. |
Rotten issues close after 30d of inactivity. Send feedback to sig-contributor-experience at kubernetes/community. |
@fejta-bot: Closing this issue. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
This is a Feature Request
What would you like to be added
The Predicates/Priorities (also called filtering/scoring) approach to configuring
kube-scheduler
is being deprecated in favor of the Plugins approach. Currently the Plugins approach is sparsely documented. An example of how to configure it would be fantastic.Why is this needed
It is currently hard to find out how to configure the scheduler plugins.
Page to Update:
https://kubernetes.io/docs/concepts/scheduling-eviction/kube-scheduler/
Or a different page in this section.
Related issues:
#21051 covers marking the old approach as deprecated in the reference docs.
Outline solution:
Here is an outline of the steps I took to configure the scheduling plugins. I hope the scheduler developers or someone else can describe a simpler approach. This is for a cluster installed with
kubeadm
.Add the file
/etc/kubernetes/scheduler-config.conf
to the master node(s), with contentReplace the file
/etc/kubernetes/manifests/kube-scheduler.yaml
with contentThe change to the second file will cause the scheduler to restart and the new configuration should be in use.
Notes on
/etc/kubernetes/scheduler-config.conf
:Notes on
/etc/kubernetes/manifests/kube-scheduler.yaml
:kubeadm
, with just the addition of the--config
flag and the extrahostPath
mount.Additional notes:
--v=10
tocommand
causes some of scheduling values to be logged, which was sufficient for me to verify that my changes really had an effect.The text was updated successfully, but these errors were encountered: