Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix for job tasks to be sorted by task kay alphabetically #287

Merged
merged 2 commits into from
Oct 30, 2024

Conversation

goda
Copy link
Collaborator

@goda goda commented Oct 29, 2024

Fix for #286 - adding a method in Job class that sorts its tasks by task_key whenever the class is serialized. This will ensure that when no job task's have been changed that terraform doesn't think an update is needed.

@osoucy
Copy link
Contributor

osoucy commented Oct 30, 2024

Can you update the CHANGELOG.md file to include in the updated section:
* Jobto automatically alphabetically sortstasks [[#286](https://github.com/okube-ai/laktory/issues/277)]

@osoucy
Copy link
Contributor

osoucy commented Oct 30, 2024

Can you update the test_job.py file by changing the order of the tasks in the job definition ? This will validate that the new sorting works as expected. Final job definition could be something like:


job = Job(
    name="job-stock-prices",
    clusters=[
        {
            "name": "main",
            "spark_version": "14.0.x-scala2.12",
            "node_type_id": "Standard_DS3_v2",
        },
    ],
    tasks=[
        {
            "depends_ons": [{"task_key": "ingestion"}],
            "pipeline_task": {"pipeline_id": "${resources.pl-spark-dlt.id}"},
            "task_key": "pipeline",
        },
        {
            "task_key": "view",
            "sql_task": {
                "query": {"query_id": "456"},
                "warehouse_id": "123",
            },
        },
        {
            "job_cluster_key": "main",
            "notebook_task": {
                "notebook_path": "job/ingest_stock_prices",
            },
            "task_key": "ingestion",
        },
    ],
    email_notifications={
        "on_duration_warning_threshold_exceededs": ["[email protected]"],
        "on_failures": ["[email protected]"],
        "on_starts": ["[email protected]"],
        "on_successes": ["[email protected]"],
    },
)

@goda
Copy link
Collaborator Author

goda commented Oct 30, 2024

Pushed new commit with the requested changes:

  • CHANGELONG.md updated to reflect the change
  • Changed order of tasks in test_job job object to ensure the sorting of the tasks works
  • Moved the sort_tasks method right after the attribute declarations in Job class

@osoucy osoucy merged commit d833d9d into okube-ai:main Oct 30, 2024
0 of 3 checks passed
@goda goda deleted the fix_job_task_sorting branch October 30, 2024 17:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants