Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] NodeSelector not working in SeldonDeployment #2682

Closed
WaterKnight1998 opened this issue Nov 20, 2020 · 3 comments
Closed

[BUG] NodeSelector not working in SeldonDeployment #2682

WaterKnight1998 opened this issue Nov 20, 2020 · 3 comments

Comments

@WaterKnight1998
Copy link

Describe the bug

I am applying a Deployment that uses NodeSelector to select in which node run the model: gpu or cpu ones.

The yaml looks as follows:

apiVersion: machinelearning.seldon.io/v1
kind: SeldonDeployment
metadata:
  name: deploy-1
  namespace: seldon
  labels:
spec:
  name: deploy-1
  protocol: seldon
  transport: rest
  annotations:
  predictors:
    - name: "default"
      graph:
        name: "model"
        implementation: MLFLOW_SERVER
        serviceAccountName: ****
        modelUri: gs://****
        type: "MODEL"
      componentSpecs:
      - spec:
        containers:
          - name: "model"
            image:
            imagePullPolicy:
            env:
              name: "LOG_LEVEL"
              value: "INFO"
            resources:
              requests:
                memory: 1Mi
        nodeSelector:
          cloud.google.com/gke-nodepool: gpu-node-pool
      replicas: 1

Expected behaviour

The deployment should use NodeSelector. However, it is getting removed!

Environment

@WaterKnight1998 WaterKnight1998 added bug triage Needs to be triaged and prioritised accordingly labels Nov 20, 2020
@ukclivecox
Copy link
Contributor

I ran the following against 1.5.0-dev and it worked after labelling one of the nodes to seldon=true

apiVersion: machinelearning.seldon.io/v1
kind: SeldonDeployment
metadata:
  name: seldon-model
spec:
  name: test-deployment
  predictors:
  - componentSpecs:
    - spec:
        containers:
        - image: seldonio/mock_classifier_rest:1.3
          name: classifier
        nodeSelector:
          seldon: "true"
    graph:
      children: []
      endpoint:
        type: REST
      name: classifier
      type: MODEL
    name: example
    replicas: 1

@ukclivecox ukclivecox added awaiting-feedback and removed triage Needs to be triaged and prioritised accordingly labels Nov 20, 2020
@WaterKnight1998
Copy link
Author

I ran the following against 1.5.0-dev and it worked after labelling one of the nodes to seldon=true

apiVersion: machinelearning.seldon.io/v1
kind: SeldonDeployment
metadata:
  name: seldon-model
spec:
  name: test-deployment
  predictors:
  - componentSpecs:
    - spec:
        containers:
        - image: seldonio/mock_classifier_rest:1.3
          name: classifier
        nodeSelector:
          seldon: "true"
    graph:
      children: []
      endpoint:
        type: REST
      name: classifier
      type: MODEL
    name: example
    replicas: 1

Your example is working! What is wrong with my spec?

@ukclivecox
Copy link
Contributor

- componentSpecs: is a list. Bad yaml?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants