Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use the backend-configured model #12336

Merged
merged 1 commit into from
Nov 13, 2020
Merged

Use the backend-configured model #12336

merged 1 commit into from
Nov 13, 2020

Conversation

mjpieters
Copy link
Contributor

Rather than import the backend Task model directly, use the class that the backend actually uses. This could have been customised, and there is no reason not to use this reference.

Fixes #12138

@boring-cyborg boring-cyborg bot added the area:Scheduler including HA (high availability) scheduler label Nov 13, 2020
turbaszek
turbaszek reviewed Nov 13, 2020
View reviewed changes
Copy link
Member

@turbaszek turbaszek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Make sense to be. Should we add some test around this?

@mjpieters
Copy link
Contributor Author

Make sense to be. Should we add some test around this?

Not sure how much that would achieve, tbh. The change just makes sure to use the same API the Celery db backend uses, and the existing tests already cover this area. Testing that backend.test_cls is different when someone changed the database backend config to use the extended class is not that useful.

@github-actions github-actions bot added the full tests needed We need to run full set of tests for this PR to merge label Nov 13, 2020
@github-actions
Copy link

The PR needs to run all tests because it modifies core of Airflow! Please rebase it to latest master or ask committer to re-run it!

@mjpieters
Use the backend-configured model
6866f7e 
Rather than import the backend Task model directly, use the class that the backend actually uses. This could have been customised, and there is no reason not to use this reference.
@turbaszek turbaszek merged commit d54f087 into apache:master Nov 13, 2020
kaxil added a commit to astronomer/airflow that referenced this pull request Mar 4, 2021
@kaxil
Default to Celery Task model when backend-configured model does not e…
a746c08 
…xist

closes apache#14586

We add this feature in apache#12336
but looks like for some users this attribute does not exist.

I am not sure if they are using a different Celery DB Backend or not
but even Celery > 5 contains this attribute
(https://github.com/celery/celery/blob/v5.0.5/celery/backends/database/__init__.py#L66)

and even Celery 4 but this commits use the Celery Task model when an attribute
error occurs
@kaxil kaxil mentioned this pull request Mar 4, 2021
Merged
kaxil added a commit that referenced this pull request Mar 5, 2021
@kaxil
Default to Celery Task model when backend model does not exist (#14612)
33910d6 
closes #14586

We add this feature in #12336
but looks like for some users this attribute does not exist.

I am not sure if they are using a different Celery DB Backend or not
but even Celery > 5 contains this attribute
(https://github.com/celery/celery/blob/v5.0.5/celery/backends/database/__init__.py#L66)

and even Celery 4 but this commits use the Celery Task model when an attribute
error occurs
ashb pushed a commit that referenced this pull request Mar 19, 2021
@kaxil @ashb
Default to Celery Task model when backend model does not exist (#14612)
0fc6ca3 
closes #14586

We add this feature in #12336
but looks like for some users this attribute does not exist.

I am not sure if they are using a different Celery DB Backend or not
but even Celery > 5 contains this attribute
(https://github.com/celery/celery/blob/v5.0.5/celery/backends/database/__init__.py#L66)

and even Celery 4 but this commits use the Celery Task model when an attribute
error occurs

(cherry picked from commit 33910d6)
ashb pushed a commit that referenced this pull request Apr 15, 2021
@kaxil @ashb
Default to Celery Task model when backend model does not exist (#14612)
5b3ce3f 
closes #14586

We add this feature in #12336
but looks like for some users this attribute does not exist.

I am not sure if they are using a different Celery DB Backend or not
but even Celery > 5 contains this attribute
(https://github.com/celery/celery/blob/v5.0.5/celery/backends/database/__init__.py#L66)

and even Celery 4 but this commits use the Celery Task model when an attribute
error occurs

(cherry picked from commit 33910d6)
leahecole pushed a commit to GoogleCloudPlatform/composer-airflow that referenced this pull request Sep 16, 2021
@kaxil
Default to Celery Task model when backend model does not exist (#14612)
23433aa 
closes apache/airflow#14586

We add this feature in apache/airflow#12336
but looks like for some users this attribute does not exist.

I am not sure if they are using a different Celery DB Backend or not
but even Celery > 5 contains this attribute
(https://github.com/celery/celery/blob/v5.0.5/celery/backends/database/__init__.py#L66)

and even Celery 4 but this commits use the Celery Task model when an attribute
error occurs

(cherry picked from commit 33910d6c699b5528db4be40d31199626dafed912)

GitOrigin-RevId: 5b3ce3ff11e0b6d5d63b10b5b7a3364399c1ed6e
leahecole pushed a commit to GoogleCloudPlatform/composer-airflow that referenced this pull request Sep 17, 2021
@kaxil
Default to Celery Task model when backend model does not exist (#14612)
66fe370 
closes apache/airflow#14586

We add this feature in apache/airflow#12336
but looks like for some users this attribute does not exist.

I am not sure if they are using a different Celery DB Backend or not
but even Celery > 5 contains this attribute
(https://github.com/celery/celery/blob/v5.0.5/celery/backends/database/__init__.py#L66)

and even Celery 4 but this commits use the Celery Task model when an attribute
error occurs

GitOrigin-RevId: 33910d6c699b5528db4be40d31199626dafed912
leahecole pushed a commit to GoogleCloudPlatform/composer-airflow that referenced this pull request Sep 23, 2021
@kaxil
Default to Celery Task model when backend model does not exist (#14612)
b6fac67 
closes apache/airflow#14586

We add this feature in apache/airflow#12336
but looks like for some users this attribute does not exist.

I am not sure if they are using a different Celery DB Backend or not
but even Celery > 5 contains this attribute
(https://github.com/celery/celery/blob/v5.0.5/celery/backends/database/__init__.py#L66)

and even Celery 4 but this commits use the Celery Task model when an attribute
error occurs

GitOrigin-RevId: 33910d6c699b5528db4be40d31199626dafed912
leahecole pushed a commit to GoogleCloudPlatform/composer-airflow that referenced this pull request Nov 27, 2021
@kaxil
Default to Celery Task model when backend model does not exist (#14612)
da95f9e 
closes apache/airflow#14586

We add this feature in apache/airflow#12336
but looks like for some users this attribute does not exist.

I am not sure if they are using a different Celery DB Backend or not
but even Celery > 5 contains this attribute
(https://github.com/celery/celery/blob/v5.0.5/celery/backends/database/__init__.py#L66)

and even Celery 4 but this commits use the Celery Task model when an attribute
error occurs

GitOrigin-RevId: 33910d6c699b5528db4be40d31199626dafed912
leahecole pushed a commit to GoogleCloudPlatform/composer-airflow that referenced this pull request Mar 10, 2022
@kaxil
Default to Celery Task model when backend model does not exist (#14612)
439a3b9 
closes apache/airflow#14586

We add this feature in apache/airflow#12336
but looks like for some users this attribute does not exist.

I am not sure if they are using a different Celery DB Backend or not
but even Celery > 5 contains this attribute
(https://github.com/celery/celery/blob/v5.0.5/celery/backends/database/__init__.py#L66)

and even Celery 4 but this commits use the Celery Task model when an attribute
error occurs

GitOrigin-RevId: 33910d6c699b5528db4be40d31199626dafed912
leahecole pushed a commit to GoogleCloudPlatform/composer-airflow that referenced this pull request Jun 4, 2022
@kaxil
Default to Celery Task model when backend model does not exist (#14612)
67a1b09 
closes apache/airflow#14586

We add this feature in apache/airflow#12336
but looks like for some users this attribute does not exist.

I am not sure if they are using a different Celery DB Backend or not
but even Celery > 5 contains this attribute
(https://github.com/celery/celery/blob/v5.0.5/celery/backends/database/__init__.py#L66)

and even Celery 4 but this commits use the Celery Task model when an attribute
error occurs

GitOrigin-RevId: 33910d6c699b5528db4be40d31199626dafed912
kosteev pushed a commit to GoogleCloudPlatform/composer-airflow that referenced this pull request Jul 9, 2022
@kaxil
Default to Celery Task model when backend model does not exist (#14612)
d284a78 
closes apache/airflow#14586

We add this feature in apache/airflow#12336
but looks like for some users this attribute does not exist.

I am not sure if they are using a different Celery DB Backend or not
but even Celery > 5 contains this attribute
(https://github.com/celery/celery/blob/v5.0.5/celery/backends/database/__init__.py#L66)

and even Celery 4 but this commits use the Celery Task model when an attribute
error occurs

GitOrigin-RevId: 33910d6c699b5528db4be40d31199626dafed912
leahecole pushed a commit to GoogleCloudPlatform/composer-airflow that referenced this pull request Aug 27, 2022
@kaxil
Default to Celery Task model when backend model does not exist (#14612)
af9e7b0 
closes apache/airflow#14586

We add this feature in apache/airflow#12336
but looks like for some users this attribute does not exist.

I am not sure if they are using a different Celery DB Backend or not
but even Celery > 5 contains this attribute
(https://github.com/celery/celery/blob/v5.0.5/celery/backends/database/__init__.py#L66)

and even Celery 4 but this commits use the Celery Task model when an attribute
error occurs

GitOrigin-RevId: 33910d6c699b5528db4be40d31199626dafed912
leahecole pushed a commit to GoogleCloudPlatform/composer-airflow that referenced this pull request Oct 4, 2022
@kaxil
Default to Celery Task model when backend model does not exist (#14612)
667df60 
closes apache/airflow#14586

We add this feature in apache/airflow#12336
but looks like for some users this attribute does not exist.

I am not sure if they are using a different Celery DB Backend or not
but even Celery > 5 contains this attribute
(https://github.com/celery/celery/blob/v5.0.5/celery/backends/database/__init__.py#L66)

and even Celery 4 but this commits use the Celery Task model when an attribute
error occurs

GitOrigin-RevId: 33910d6c699b5528db4be40d31199626dafed912
aglipska pushed a commit to GoogleCloudPlatform/composer-airflow that referenced this pull request Oct 7, 2022
@kaxil
Default to Celery Task model when backend model does not exist (#14612)
1d246b6 
closes apache/airflow#14586

We add this feature in apache/airflow#12336
but looks like for some users this attribute does not exist.

I am not sure if they are using a different Celery DB Backend or not
but even Celery > 5 contains this attribute
(https://github.com/celery/celery/blob/v5.0.5/celery/backends/database/__init__.py#L66)

and even Celery 4 but this commits use the Celery Task model when an attribute
error occurs

GitOrigin-RevId: 33910d6c699b5528db4be40d31199626dafed912
leahecole pushed a commit to GoogleCloudPlatform/composer-airflow that referenced this pull request Dec 7, 2022
@kaxil
Default to Celery Task model when backend model does not exist (#14612)
8c64f41 
closes apache/airflow#14586

We add this feature in apache/airflow#12336
but looks like for some users this attribute does not exist.

I am not sure if they are using a different Celery DB Backend or not
but even Celery > 5 contains this attribute
(https://github.com/celery/celery/blob/v5.0.5/celery/backends/database/__init__.py#L66)

and even Celery 4 but this commits use the Celery Task model when an attribute
error occurs

GitOrigin-RevId: 33910d6c699b5528db4be40d31199626dafed912
leahecole pushed a commit to GoogleCloudPlatform/composer-airflow that referenced this pull request Jan 27, 2023
@kaxil
Default to Celery Task model when backend model does not exist (#14612)
03cbe27 
closes apache/airflow#14586

We add this feature in apache/airflow#12336
but looks like for some users this attribute does not exist.

I am not sure if they are using a different Celery DB Backend or not
but even Celery > 5 contains this attribute
(https://github.com/celery/celery/blob/v5.0.5/celery/backends/database/__init__.py#L66)

and even Celery 4 but this commits use the Celery Task model when an attribute
error occurs

GitOrigin-RevId: 33910d6c699b5528db4be40d31199626dafed912
kosteev pushed a commit to kosteev/composer-airflow-test-copybara that referenced this pull request Sep 12, 2024
@kaxil
Default to Celery Task model when backend model does not exist (#14612)
646b3dc 
closes apache/airflow#14586

We add this feature in apache/airflow#12336
but looks like for some users this attribute does not exist.

I am not sure if they are using a different Celery DB Backend or not
but even Celery > 5 contains this attribute
(https://github.com/celery/celery/blob/v5.0.5/celery/backends/database/__init__.py#L66)

and even Celery 4 but this commits use the Celery Task model when an attribute
error occurs

GitOrigin-RevId: 33910d6c699b5528db4be40d31199626dafed912
kosteev pushed a commit to kosteev/composer-airflow-test-copybara that referenced this pull request Sep 12, 2024
@kaxil
Default to Celery Task model when backend model does not exist (#14612)
0a2a892 
closes apache/airflow#14586

We add this feature in apache/airflow#12336
but looks like for some users this attribute does not exist.

I am not sure if they are using a different Celery DB Backend or not
but even Celery > 5 contains this attribute
(https://github.com/celery/celery/blob/v5.0.5/celery/backends/database/__init__.py#L66)

and even Celery 4 but this commits use the Celery Task model when an attribute
error occurs

GitOrigin-RevId: 33910d6c699b5528db4be40d31199626dafed912
kosteev pushed a commit to GoogleCloudPlatform/composer-airflow that referenced this pull request Sep 17, 2024
@kaxil
Default to Celery Task model when backend model does not exist (#14612)
cebfad3 
closes apache/airflow#14586

We add this feature in apache/airflow#12336
but looks like for some users this attribute does not exist.

I am not sure if they are using a different Celery DB Backend or not
but even Celery > 5 contains this attribute
(https://github.com/celery/celery/blob/v5.0.5/celery/backends/database/__init__.py#L66)

and even Celery 4 but this commits use the Celery Task model when an attribute
error occurs

GitOrigin-RevId: 33910d6c699b5528db4be40d31199626dafed912
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@turbaszek turbaszek turbaszek approved these changes

@mik-laj mik-laj Awaiting requested review from mik-laj

Assignees
No one assigned
Labels
area:Scheduler including HA (high availability) scheduler full tests needed We need to run full set of tests for this PR to merge
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Celery executor should support custom database backend models
2 participants
@mjpieters @turbaszek