Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[AIRFLOW-6425] Serialization: Add missing DAG parameters to Json Schema #7002

Merged
merged 2 commits into from
Jan 2, 2020

Conversation

kaxil
Copy link
Member

@kaxil kaxil commented Jan 2, 2020

The following DAG currently fails:

Schema Validation error when DAG.is_paused_upon_creation is set

def make_simple_dag():
    """Make very simple DAG to verify serialization result."""
    dag = DAG(
        dag_id='simple_dag',
        default_args={
            "retries": 1,
            "retry_delay": timedelta(minutes=5),
            "depends_on_past": False,
        },
        start_date=datetime(2019, 8, 1),
        is_paused_upon_creation=True,
    )
    BaseOperator(task_id='simple_task', dag=dag, owner='airflow')
    CustomOperator(task_id='custom_task', dag=dag)
    return {'simple_dag': dag}

Error:

 jsonschema.exceptions.ValidationError: Additional properties are not allowed ('is_paused_upon_creation' was unexpected)

Reason:

is_paused_upon_creation is in DAG.get_serialized_fields() but not in the Schema definition.


In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.
Read the Pull Request Guidelines for more information.

@kaxil kaxil requested review from ashb and potiuk January 2, 2020 14:28
@codecov-io
Copy link

codecov-io commented Jan 2, 2020

Codecov Report

Merging #7002 into master will decrease coverage by 0.27%.
The diff coverage is 100%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #7002      +/-   ##
==========================================
- Coverage   84.85%   84.57%   -0.28%     
==========================================
  Files         679      679              
  Lines       38542    38545       +3     
==========================================
- Hits        32703    32598     -105     
- Misses       5839     5947     +108
Impacted Files Coverage Δ
airflow/models/dag.py 90.95% <ø> (ø) ⬆️
airflow/serialization/json_schema.py 84% <100%> (+2.18%) ⬆️
airflow/kubernetes/volume_mount.py 44.44% <0%> (-55.56%) ⬇️
airflow/kubernetes/volume.py 52.94% <0%> (-47.06%) ⬇️
airflow/kubernetes/pod_launcher.py 45.25% <0%> (-46.72%) ⬇️
airflow/kubernetes/refresh_config.py 50.98% <0%> (-23.53%) ⬇️
...rflow/contrib/operators/kubernetes_pod_operator.py 78.75% <0%> (-20%) ⬇️
airflow/gcp/operators/gcs.py 85.62% <0%> (-0.6%) ⬇️
airflow/jobs/backfill_job.py 91.59% <0%> (-0.29%) ⬇️
airflow/gcp/example_dags/example_gcs.py 100% <0%> (+11.42%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update aa90753...1fa2aac. Read the comment docs.

airflow/serialization/json_schema.py Outdated Show resolved Hide resolved
@kaxil kaxil merged commit 13c21c1 into apache:master Jan 2, 2020
@kaxil kaxil deleted the dag-s10n-bugfix branch January 2, 2020 18:10
kaxil added a commit that referenced this pull request Jan 7, 2020
kaxil added a commit to astronomer/airflow that referenced this pull request Jan 8, 2020
kaxil added a commit that referenced this pull request Jan 10, 2020
potiuk pushed a commit that referenced this pull request Jan 21, 2020
kaxil added a commit that referenced this pull request Jan 23, 2020
galuszkak pushed a commit to FlyrInc/apache-airflow that referenced this pull request Mar 5, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants