6 scheduling #36

vlerkin · 2024-11-07T17:25:47Z

What happens in the PR:

The logic of event watcher was separated in an observer class; the logic of log watching stayed in a log handler class, but the initialization was changed to subscribe to the event in case jobless feature was configured;
The new class KubernetesScheduler was created to handle logic when jobs must be unsuspended and how (ordered);
scheduler endpoint was modified, logic to set a value for start_suspended parameter was added;
schedule method from k8s launcher has a new start_suspended parameter, it's value is passed when called inside the api; also new methods were added: unsuspend_job patches existing suspended job suspend=False, get_running_jobs_count returns the number of jobs that are currently running, list_suspended_jobs returns the list of jobs where spec.suspend is true, _get_job_name extracts the job name from the metadata, it is then used for unsuspend function;

The big picture:
Event watcher connects to the k8s api and receives the stream of events, it then notifies the subscribers if a new event is received and passes it to the provided callback. The subscriber - KubernetesScheduler - receives event in a handle_pod_event method, this method reacts to the changes in job statuses, and if job completed running or failed it calls another method - check_and_unsuspend_jobs - that checks capacity and unsuspends jobs until the number of allowed parallel jobs is reached, while doing this it relies on another method - get_next_suspended_job_id - to unsuspend the most recent job, to keep the order in which jobs were initially scheduled.
When the job is scheduled, based on the number of currently active jobs and max_proc provided in the config (default is 4), the job runs or goes to the queue of suspended jobs (native k8s queue). Then events that change the number of active jobs trigger the logic of KubernetesScheduler class that unsuspend suspended jobs until the desired state (num of parallel jobs) is achieved.

…logic for observer in a RecourceWatcher class; added method to stop a thread gracefully

…that handles the logic to unsuspend jobs and get the next in order according to the creation timestamp; modify schedule endpoint to start jobs suspended if there is already enogh jobs running; modify corresponding function in k8s launcher; add to k8s launcher methods to unsuspend job, to get current number of running jobs, to list suspended jobs and a private method to get job name to be used for unsuspend function

…source watcher instance to enable_joblogs to subscribe to the event watcher if the log feature is configured; delete logic about event watcher from main; pass container for list objects function instead of container name; remove start methon from log handler class; modify joblogs init to subscribe to event watcher

wvengen

Ah, nice you were able to come up with something so quickly already!
I looked at it from a high level, and noticed that this is currently implemented for Kubernetes only (that makes sense), and also setup in such a way that it needs refactoring for Docker. I would think of the scheduler as something that could work for both Docker and Kubernetes, especially the scheduling decisions. Also, there is now k8s-specific code in the main file (e.g. the import), and the kubernetes scheduler, this makes the code somewhat spaghetti: there are specific implementation-specific classes where responsibility is meant to be delegated. If you need to access the scheduler in the main file, use a generic scheduler, and make the docker-based parts not implemented. I think that would give a much cleaner design.

Also, I would consider making the launcher responsible for scheduling. And then have the scheduler talk to the launcher to actually start jobs.

I'm not yet sure if we should allow running without the scheduler, or if it would always be active.

wvengen · 2024-11-08T08:19:32Z

kubernetes.yaml

@@ -181,7 +183,7 @@ rules:
    verbs: ["get"]
  - apiGroups: ["batch"]
    resources: ["jobs"]
-    verbs: ["get", "list", "create", "delete"]
+    verbs: ["get", "list", "create", "patch", "update", "delete"]


do you need patch and update? what is the difference?

Yes, good point, since we do not update the entire resource, patch is enough for the feature, deleted it

wvengen · 2024-11-08T08:33:12Z

Hope my feedback was at an angle that helps you at this stage. In any case, well done, keep it going!

p.s. the CI error looks like it could be cause by Kubernetes-specific things having entered into the main api code, which wouldn't work when running with Docker.

vlerkin · 2024-11-11T08:12:26Z

Working on Docker implementation to be added to this PR

…rs and run more from the queue of created jobs when capacity is available; add backgroung thread that sleeps for 5 sec and triggers the function that starts additional containers up to capacity; add a method to gracefully stop the background thread that might be used in the future to stop the thread when app stops; encapsulate k8s and docker related schedule functionality in corresponding launchers and keep api.py launcher agnostic; add max_proc to config for docker

…nd patch is sufficient

wvengen

Great to see a working version! Quite readable :)
I think it needs a little cleanup, but you're getting there, I think.

wvengen · 2024-11-12T11:21:49Z

scrapyd_k8s.sample-docker.conf

@@ -16,6 +16,9 @@ repository   = scrapyd_k8s.repository.Local
 # Since this is the Docker example, we choose Docker here.
 launcher     = scrapyd_k8s.launcher.Docker

+# Maximum number of jobs running in parallel
+max_proc     = 1
+


not sure yet if we want to enable this by default
do you know what scrapyd has in its default configuration?

In scrapyd the max_proc parameter is not default. I think there are reasons to consider making it default because it enhances cluster stability but on the other hand, if a user did not think of optimal resource usage this batch limiting can lead to a smaller output at a given time. So it's beneficial if we need to be conscious about recourses of the prod setting but not very handy if we want to go all in and extract as much data as possible and faster.

wvengen · 2024-11-12T11:22:30Z

scrapyd_k8s/api.py

@@ -7,6 +7,7 @@
 from natsort import natsort_keygen, ns

 from .config import Config
+from .k8s_resource_watcher import ResourceWatcher


reference to k8s-specific term k8s_resource_watcher, this file should use the configured backend and launchers instead

It is required for conditional start of joblogs feature in the run method and this method does not rely on a specific launcher, that is why I can't think of a clean way how to remove it from the api.py. But I am open for suggestions if you see a better way.

If the ResourceWatcher is not Kubernetes-specific, perhaps the k8s_ could be dropped from the filename?

Looking at the code, I do see k8s-specific things in there.
I think the launcher needs to be responsible for this somehow. Then the launcher can decide what the k8s and Docker specific parts are.

(sorry, I'm not diving fully now into the whole code, otherwise I could have given a more direct answer making more sense perhaps - can you think of a way to separate the k8s part here?)

I do agree that it shouldn't be here and we don't want to violate SOLID, I will think how to refactor this.

scrapyd_k8s/k8s_resource_watcher.py

wvengen · 2024-11-12T11:24:13Z

scrapyd_k8s/k8s_resource_watcher.py

+            try:
+                subscriber(event)
+            except Exception as e:
+                logger.exception(f"Error notifying subscriber {subscriber.__name__}: {e}")


why catch this? when do you expect this?

This exception is added so we can separate the exception originated in particular subscriber from watcher and from other subscribers. And we also protect watcher from crashing due to problems in subscribers which is external in terms of the design.
If there is an unexpected edge case for subscriber it is nice to catch it and understand where it comes from, also easier to debug.

wvengen · 2024-11-12T11:25:13Z

scrapyd_k8s/k8s_scheduler.py

@@ -0,0 +1,96 @@
+import logging


Can we fit this in the directory structure? I wouldn't expect this in the src root.

For me, this functionality seems related to the kubernetes launcher.

wvengen · 2024-11-12T11:26:32Z

scrapyd_k8s/launcher/k8s.py

+    def enable_joblogs(self, config, resource_watcher):
+        joblogs_init(config, resource_watcher)
+
+    def unsuspend_job(self, job_id: str):


is "unsuspend" the k8s terminology? I would also imagine resume, or start or so.

wvengen · 2024-11-12T11:28:00Z

scrapyd_k8s/launcher/k8s.py

+            return suspended_jobs
+        except Exception as e:
+            logger.exception(f"Error listing suspended jobs: {e}")
+            return []


Is there a situation where you would want to get both the running and suspended job count? Then it could be nice to do one call to list_namespaced_job to obtain both.

wvengen · 2024-11-12T11:28:34Z

scrapyd_k8s/launcher/k8s.py

+        if not jobs.items:
+            logger.error(f"No job found with job_id={job_id}")
+            return None
+        return jobs.items[0].metadata.name


when you're listing jobs, would you also get the name already?

vlerkin added 4 commits November 1, 2024 16:48

extracted resource watching logic into a separate class; implemented …

4c0a82d

…logic for observer in a RecourceWatcher class; added method to stop a thread gracefully

Merge branch 'main' into 6-scheduling

9dae80d

vlerkin requested a review from wvengen November 7, 2024 17:25

change enable_joblogs method signature in docker

963f279

wvengen reviewed Nov 8, 2024

View reviewed changes

vlerkin added 3 commits November 11, 2024 14:50

refactor api.py and launcher/k8s.py to keep api.py launcher agnostic

7dc24bc

remove update from RBAC because we perform patching of the resource a…

01b120b

…nd patch is sufficient

wvengen reviewed Nov 12, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

6 scheduling #36

6 scheduling #36

vlerkin commented Nov 7, 2024

wvengen left a comment •

edited

Loading

wvengen Nov 8, 2024

vlerkin Nov 12, 2024

wvengen commented Nov 8, 2024

vlerkin commented Nov 11, 2024

wvengen left a comment

wvengen Nov 12, 2024

vlerkin Nov 12, 2024 •

edited

Loading

wvengen Nov 12, 2024

vlerkin Nov 12, 2024

wvengen Nov 12, 2024

wvengen Nov 12, 2024

vlerkin Nov 13, 2024

wvengen Nov 12, 2024

vlerkin Nov 13, 2024 •

edited

Loading

wvengen Nov 12, 2024

wvengen Nov 12, 2024

wvengen Nov 12, 2024

wvengen Nov 12, 2024

wvengen Nov 12, 2024

6 scheduling #36

Are you sure you want to change the base?

6 scheduling #36

Conversation

vlerkin commented Nov 7, 2024

wvengen left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wvengen commented Nov 8, 2024

vlerkin commented Nov 11, 2024

wvengen left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vlerkin Nov 12, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vlerkin Nov 13, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wvengen left a comment •

edited

Loading

vlerkin Nov 12, 2024 •

edited

Loading

vlerkin Nov 13, 2024 •

edited

Loading