How to guide on Dask-Nebari config #79
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
✅ Deploy Preview for nebari-docs ready!
To edit notification comments on pull requests, go to your Netlify site settings. |
|
@HarshCasper @iameskild feel free to continue on this PR or close it. |
|
@HarshCasper could you look at integrating these docs into this how to guide? |
✅ Deploy Preview for nebari-dev ready!
To edit notification comments on pull requests, go to your Netlify site settings. |
kcpevey
changed the title
kcpevey
requested changes
Sep 14, 2022
Comment on lines
+8
to
+10
| In this tutorial we will dive into the `Nebari-Dask` configuration details. Nebari config is essentially | ||
| a `yaml` file which is at the heart of all things (most of them) related to configurations. | ||
| Our main focus in this tutorial will be the `profiles` & `dask_worker` section of the config file. |
There was a problem hiding this comment.
Suggested change
| In this tutorial we will dive into the `Nebari-Dask` configuration details. Nebari config is essentially | |
| a `yaml` file which is at the heart of all things (most of them) related to configurations. | |
| Our main focus in this tutorial will be the `profiles` & `dask_worker` section of the config file. | |
| In this tutorial we will dive into the configuration requirements for running Daask on Nebari. The Nebari config (`qhub_config.yml`) is at the heart of all things (most of them) related to configurations. | |
| Our main focus in this tutorial will be the `profiles` & `dask_worker` section of the config file. |
Comment on lines
+14
to
+21
| Before we dive deeper in configuration details, let's understand about how are the core configuration | ||
| components. | ||
|
|
||
| ### Core components: | ||
|
|
||
| - Dask-gateway | ||
| - dask workers | ||
| - Dask scheduler |
There was a problem hiding this comment.
Suggested change
| Before we dive deeper in configuration details, let's understand about how are the core configuration | |
| components. | |
| ### Core components: | |
| - Dask-gateway | |
| - dask workers | |
| - Dask scheduler | |
| There are three core configuration components necessary for setting up Dask on Nebari. | |
| These are: | |
| - [Dask-gateway](https://gateway.dask.org/): provides a secure, multi-tenant server for managing [Dask](https://dask.org/) clusters | |
| - [Dask workers](https://distributed.dask.org/en/stable/worker.html): compute tasks as directed by the scheduler and store/serve results | |
| - [Dask scheduler](https://docs.dask.org/en/stable/scheduler-overview.html): executes the task graph by coordinating task distribution |
| a `yaml` file which is at the heart of all things (most of them) related to configurations. | ||
| Our main focus in this tutorial will be the `profiles` & `dask_worker` section of the config file. | ||
|
|
||
| ### Basic infrastructure details |
There was a problem hiding this comment.
Suggested change
| ### Basic infrastructure details | |
| ## Basic infrastructure details |
| - dask workers | ||
| - Dask scheduler | ||
|
|
||
| ### How to configure dask gateway profiles? |
There was a problem hiding this comment.
Suggested change
| ### How to configure dask gateway profiles? | |
| ## Configuring Dask Gateway profiles |
| scheduler_cores: 6 | ||
| scheduler_memory_limit: 30G | ||
| scheduler_memory: 28G | ||
| ``` |
There was a problem hiding this comment.
Suggested change
| ``` |
Next Steps
Now that you have your Nebari instance all set up to run Dask, check out our user guide on Working with Big Data using Dask!
There was a problem hiding this comment.
Just guessing at that link to the page, you should double check that works
|
|
||
| - `limit` is the absolute max memory a given pod can consume. Suppose a process within the pod consumes more than the `limit` memory. In that case, the Linux OS will kill the process. `limit` is not used for scheduling purposes with Kubernetes. | ||
|
|
||
| - `guarantee`: is the amount of memory the Kubernetes scheduler uses to place a given pod. In general, the `guarantee` will be less than the limit. Often the node itself has less available memory than the node specification. See this [guide from digital ocean](https://docs.digitalocean.com/products/kubernetes/#allocatable-memory), which generally applies to other clouds. |
There was a problem hiding this comment.
Suggested change
| - `guarantee`: is the amount of memory the Kubernetes scheduler uses to place a given pod. In general, the `guarantee` will be less than the limit. Often the node itself has less available memory than the node specification. See this [guide from digital ocean](https://docs.digitalocean.com/products/kubernetes/#allocatable-memory), which generally applies to other clouds. | |
| - `guarantee`: is the amount of memory the Kubernetes scheduler uses to place a given pod. In general, the `guarantee` will be less than the limit. Often the node itself has less available memory than the node specification. You may want to check out this [guide from digital ocean](https://docs.digitalocean.com/products/kubernetes/#allocatable-memory) which also generally applies to other clouds. |
| mem_guarantee: 4G | ||
| ``` | ||
|
|
||
| ### How to configure dask scheduler? |
There was a problem hiding this comment.
Suggested change
| ### How to configure dask scheduler? | |
| ## Configuring the Dask Scheduler |
|
|
||
| ### How to configure dask scheduler? | ||
|
|
||
| In a few instances, the dask worker node-group might be running on quite a large instance, perhaps with 8 CPUs and 32 GB of memory (or more). In this case, you might also want to increase the resource levels associated with the dask scheduler. |
There was a problem hiding this comment.
Suggested change
| In a few instances, the dask worker node-group might be running on quite a large instance, perhaps with 8 CPUs and 32 GB of memory (or more). In this case, you might also want to increase the resource levels associated with the dask scheduler. | |
| For analyses requiring heavy compute, there may be some situations where the Dask worker node-group might be running on quite a large cluster, perhaps with 8 CPUs and 32 GB of memory (or more). In this case, you might also want to increase the resource levels associated with the Dask scheduler. |
| ### How to configure dask scheduler? | ||
|
|
||
| In a few instances, the dask worker node-group might be running on quite a large instance, perhaps with 8 CPUs and 32 GB of memory (or more). In this case, you might also want to increase the resource levels associated with the dask scheduler. | ||
|
|
There was a problem hiding this comment.
Suggested change
| The following is an example of a Dask worker configuration with the Scheduler resources modified. |
Comment on lines
+72
to
+81
| dask_worker: | ||
| "Huge Worker": | ||
| worker_cores_limit: 7 | ||
| worker_cores: 6 | ||
| worker_memory_limit: 30G | ||
| worker_memory: 28G | ||
| scheduler_cores_limit: 7 | ||
| scheduler_cores: 6 | ||
| scheduler_memory_limit: 30G | ||
| scheduler_memory: 28G |
There was a problem hiding this comment.
Is this a jupyterlab profile?
kcpevey
added
needs: changes 🧱
|
The content in this PR is covered in #115. :) |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
resolves #68
This PR is a work in progress. I am currently referring to this guide to pick bits and pieces.