Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[kubernetes_generate_dag_yaml] - Fix dag yaml generate func in version 1.10.14 #13816

Merged
merged 1 commit into from
Feb 4, 2021

Conversation

Lord-Y
Copy link

@Lord-Y Lord-Y commented Jan 21, 2021

Signed-off-by: Lord-Y [email protected]

When trying to debug the generated yaml pod to see why I'm not having the right docker image I got this error:

airflow kubernetes generate-dag-yaml DAG_NAME 2021-01-19T20:00:00+00:00

[2021-01-21 09:52:56,314] {{__init__.py:50}} INFO - Using executor KubernetesExecutor
[2021-01-21 09:52:56,317] {{dagbag.py:417}} INFO - Filling up the DagBag from /opt/airflow/dags/airflow-dags.git
Traceback (most recent call last):
  File "/home/airflow/.local/bin/airflow", line 37, in <module>
    args.func(args)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/cli.py", line 233, in wrapper
    func(args)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/cli.py", line 81, in wrapper
    return f(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/bin/cli.py", line 1367, in kubernetes_generate_dag_yaml
    pod = PodGenerator.construct_pod(
TypeError: construct_pod() got an unexpected keyword argument 'kube_executor_config'

After fixing kube_executor_config, I got the same error for worker_config parameter and also kube_image was missing.

@boring-cyborg
Copy link

boring-cyborg bot commented Jan 21, 2021

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst)
Here are some useful points:

  • Pay attention to the quality of your code (flake8, pylint and type annotations). Our pre-commits will help you with that.
  • In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
  • Consider using Breeze environment for testing locally, it’s a heavy docker but it ships with a working Airflow and a lot of integrations.
  • Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
  • Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
  • Be sure to read the Airflow Coding style.
    Apache Airflow is a community-driven project and together we are making it better 🚀.
    In case of doubts contact the developers at:
    Mailing List: [email protected]
    Slack: https://s.apache.org/airflow-slack

@kaxil kaxil force-pushed the kubernetes_generate_dag_yaml branch from 53d4179 to 0f87b84 Compare January 22, 2021 21:53
@kaxil kaxil requested a review from dimberman January 22, 2021 21:54
@kaxil kaxil modified the milestone: Airflow 1.10.15 Jan 22, 2021
@Lord-Y
Copy link
Author

Lord-Y commented Jan 29, 2021

@kaxil Can you get some time to have a look on SQL part :)

Copy link
Member

@kaxil kaxil left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM -- @dimberman any final thoughts ??

@github-actions github-actions bot added the full tests needed We need to run full set of tests for this PR to merge label Jan 29, 2021
@github-actions
Copy link

The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest master at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.

@dimberman dimberman merged commit 28a6a08 into apache:v1-10-stable Feb 4, 2021
@boring-cyborg
Copy link

boring-cyborg bot commented Feb 4, 2021

Awesome work, congrats on your first merged pull request!

linuxfft added a commit to linuxfft/airflow that referenced this pull request Aug 13, 2021
Apache Airflow 1.10.15

- Fix `airflow db upgrade` to upgrade db as intended (apache#13267)
- Moved boto3 limitation to snowflake (apache#13286)
- `KubernetesExecutor` should accept images from `executor_config` (apache#13074)
- Scheduler should acknowledge active runs properly (apache#13803)
- Bugfix: Unable to import Airflow plugins on Python 3.8 (apache#12859)
- Include `airflow/contrib/executors` in the dist package
- Pin Click version for Python 2.7 users
- Ensure all statsd timers use millisecond values. (apache#10633)
- [`kubernetes_generate_dag_yaml`] - Fix dag yaml generate function (apache#13816)
- Fix `airflow tasks clear` cli command wirh `--yes` (apache#14188)
- Fix permission error on non-POSIX filesystem (apache#13121) (apache#14383)
- Fixed deprecation message for "variables" command (apache#14457)
- BugFix: fix the `delete_dag` function of json_client (apache#14441)
- Fix merging of secrets and configmaps for `KubernetesExecutor` (apache#14090)
- Fix webserver exiting when gunicorn master crashes (apache#13470)
- Bump ini from 1.3.5 to 1.3.8 in `airflow/www_rbac`
- Bump datatables.net from 1.10.21 to 1.10.23 in `airflow/www_rbac`
- Webserver: Sanitize string passed to origin param (apache#14738)
- Make `rbac_app`'s `db.session` use the same timezone with `@provide_session` (apache#14025)

- Adds airflow as viable docker command in official image (apache#12878)
- `StreamLogWriter`: Provide (no-op) close method (apache#10885)
- Add 'airflow variables list' command for 1.10.x transition version (apache#14462)

- Update URL for Airflow docs (apache#13561)
- Clarifies version args for installing 1.10 in Docker (apache#12875)
linuxfft added a commit to linuxfft/airflow that referenced this pull request Aug 13, 2021
Apache Airflow 1.10.15

- Fix `airflow db upgrade` to upgrade db as intended (apache#13267)
- Moved boto3 limitation to snowflake (apache#13286)
- `KubernetesExecutor` should accept images from `executor_config` (apache#13074)
- Scheduler should acknowledge active runs properly (apache#13803)
- Bugfix: Unable to import Airflow plugins on Python 3.8 (apache#12859)
- Include `airflow/contrib/executors` in the dist package
- Pin Click version for Python 2.7 users
- Ensure all statsd timers use millisecond values. (apache#10633)
- [`kubernetes_generate_dag_yaml`] - Fix dag yaml generate function (apache#13816)
- Fix `airflow tasks clear` cli command wirh `--yes` (apache#14188)
- Fix permission error on non-POSIX filesystem (apache#13121) (apache#14383)
- Fixed deprecation message for "variables" command (apache#14457)
- BugFix: fix the `delete_dag` function of json_client (apache#14441)
- Fix merging of secrets and configmaps for `KubernetesExecutor` (apache#14090)
- Fix webserver exiting when gunicorn master crashes (apache#13470)
- Bump ini from 1.3.5 to 1.3.8 in `airflow/www_rbac`
- Bump datatables.net from 1.10.21 to 1.10.23 in `airflow/www_rbac`
- Webserver: Sanitize string passed to origin param (apache#14738)
- Make `rbac_app`'s `db.session` use the same timezone with `@provide_session` (apache#14025)

- Adds airflow as viable docker command in official image (apache#12878)
- `StreamLogWriter`: Provide (no-op) close method (apache#10885)
- Add 'airflow variables list' command for 1.10.x transition version (apache#14462)

- Update URL for Airflow docs (apache#13561)
- Clarifies version args for installing 1.10 in Docker (apache#12875)
andrewdanks added a commit to Affirm/airflow that referenced this pull request Mar 18, 2022
Apache Airflow 1.10.15

- Fix `airflow db upgrade` to upgrade db as intended (apache#13267)
- Moved boto3 limitation to snowflake (apache#13286)
- `KubernetesExecutor` should accept images from `executor_config` (apache#13074)
- Scheduler should acknowledge active runs properly (apache#13803)
- Bugfix: Unable to import Airflow plugins on Python 3.8 (apache#12859)
- Include `airflow/contrib/executors` in the dist package
- Pin Click version for Python 2.7 users
- Ensure all statsd timers use millisecond values. (apache#10633)
- [`kubernetes_generate_dag_yaml`] - Fix dag yaml generate function (apache#13816)
- Fix `airflow tasks clear` cli command wirh `--yes` (apache#14188)
- Fix permission error on non-POSIX filesystem (apache#13121) (apache#14383)
- Fixed deprecation message for "variables" command (apache#14457)
- BugFix: fix the `delete_dag` function of json_client (apache#14441)
- Fix merging of secrets and configmaps for `KubernetesExecutor` (apache#14090)
- Fix webserver exiting when gunicorn master crashes (apache#13470)
- Bump ini from 1.3.5 to 1.3.8 in `airflow/www_rbac`
- Bump datatables.net from 1.10.21 to 1.10.23 in `airflow/www_rbac`
- Webserver: Sanitize string passed to origin param (apache#14738)
- Make `rbac_app`'s `db.session` use the same timezone with `@provide_session` (apache#14025)

- Adds airflow as viable docker command in official image (apache#12878)
- `StreamLogWriter`: Provide (no-op) close method (apache#10885)
- Add 'airflow variables list' command for 1.10.x transition version (apache#14462)

- Update URL for Airflow docs (apache#13561)
- Clarifies version args for installing 1.10 in Docker (apache#12875)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:CLI full tests needed We need to run full set of tests for this PR to merge
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants