feat: update helm chart for k8 RP [DET-3542] #882
Conversation
| {{- end }} | ||
| {{ end }} | ||
|
|
||
| {{- if .Values.telemetry }} |
There was a problem hiding this comment.
non-blocking: is there no way to do an and comparison in one if statement? It's not a big deal but might look cleaner
There was a problem hiding this comment.
doesn't seem like Helm has a and flow control operator unfortunately
| # scheduling multi-GPU tasks. Multi-gpu (distributed training) tasks will be scheduled as | ||
| # slotsPerTask / slotsPerNode separate pods (tasks sizes that are not divisible by slotsPerNode | ||
| # are never scheduled. | ||
| slotsPerNode: |
There was a problem hiding this comment.
question: what happens if the cluster has different instance types? Is there another way we are making sure each node has the same number of GPUs?
There was a problem hiding this comment.
Right now we don't do anything to account for it. One thing users could do is set it to the smallest denominator, then we will just have the overhead of running multiple pods per node. I am planning to document this as part of M5, also will expand the doc-string here with this info.
helm/charts/determined/values.yaml
Outdated
| masterCpuRequest: "4" | ||
| masterMemRequest: "8Gi" | ||
|
|
||
| ## Configure the task container defaults. Tasks include trials, commands, tensorboards and more. |
There was a problem hiding this comment.
non-blocking: more is just notebooks right?
There was a problem hiding this comment.
There is also shells, will just write them all out.
| ## random non-privileged ports, respectively. | ||
| taskContainerDefaults: | ||
| shmSizeBytes: 4294967296 | ||
| # networkMode: bridge |
There was a problem hiding this comment.
question: is bridge the only valid value for networkMode?
There was a problem hiding this comment.
There is also "host" networking mode, but obv that isn't advisable in k8s.
Description
Added several missing configurations, made the k8 RP configurable, and added CPU and MEM request limits for the master.
Test Plan
Tested manually by deploying on a k8 cluster. Automated testing will be added as part of M4.