Skip to content

Commit

Permalink
Enforce code-block directives in doc (apache#9443)
Browse files Browse the repository at this point in the history
  • Loading branch information
mik-laj authored Jun 20, 2020
1 parent 5442c91 commit 60d19dc
Show file tree
Hide file tree
Showing 16 changed files with 101 additions and 86 deletions.
24 changes: 12 additions & 12 deletions docs/best-practices.rst
Original file line number Diff line number Diff line change
Expand Up @@ -89,13 +89,13 @@ This can result in a lot of open connections.

The best way of using variables is via a Jinja template, which will delay reading the value until the task execution. The template syntax to do this is:

.. code::
.. code-block::
{{ var.value.<variable_name> }}
or if you need to deserialize a json object from the variable :

.. code::
.. code-block::
{{ var.json.<variable_name> }}
Expand All @@ -117,7 +117,7 @@ DAG Loader Test
This test should ensure that your DAG does not contain a piece of code that raises error while loading.
No additional code needs to be written by the user to run this test.

.. code::
.. code-block::
python your-dag-file.py
Expand All @@ -132,7 +132,7 @@ Unit tests ensure that there is no incorrect code in your DAG. You can write uni

**Unit test for loading a DAG:**

.. code::
.. code-block::
from airflow.models import DagBag
import unittest
Expand All @@ -151,7 +151,7 @@ Unit tests ensure that there is no incorrect code in your DAG. You can write uni
**Unit test a DAG structure:**
This is an example test want to verify the structure of a code-generated DAG against a dict object

.. code::
.. code-block::
import unittest
class testClass(unittest.TestCase):
Expand All @@ -172,7 +172,7 @@ This is an example test want to verify the structure of a code-generated DAG aga
**Unit test for custom operator:**

.. code::
.. code-block::
import unittest
from airflow.utils.state import State
Expand Down Expand Up @@ -205,7 +205,7 @@ make sure that the partition is created in S3 and perform some simple checks to

Similarly, if you have a task that starts a microservice in Kubernetes or Mesos, you should check if the service has started or not using :class:`airflow.providers.http.sensors.http.HttpSensor`.

.. code::
.. code-block::
task = PushToS3(...)
check = S3KeySensor(
Expand All @@ -227,7 +227,7 @@ Do not hard code values inside the DAG and then change them manually according t

You can use environment variables to parameterize the DAG.

.. code::
.. code-block::
import os
Expand All @@ -252,7 +252,7 @@ If you want to run production-grade Airflow, make sure you :doc:`configure the b

You can change the backend using the following config

.. code:: ini
.. code-block:: ini
[core]
sql_alchemy_conn = my_conn_string
Expand All @@ -261,7 +261,7 @@ Once you have changed the backend, airflow needs to create all the tables requir
Create an empty DB and give airflow's user the permission to ``CREATE/ALTER`` it.
Once that is done, you can run -

.. code::
.. code-block::
airflow db upgrade
Expand Down Expand Up @@ -305,14 +305,14 @@ Airflow comes bundled with a default ``airflow.cfg`` configuration file.
You should use environment variables for configurations that change across deployments
e.g. metadata DB, password, etc. You can accomplish this using the format :envvar:`AIRFLOW__{SECTION}__{KEY}`

.. code::
.. code-block::
AIRFLOW__CORE__SQL_ALCHEMY_CONN=my_conn_id
AIRFLOW__WEBSERVER__BASE_URL=http://host:port
Some configurations such as the Airflow Backend connection URI can be derived from bash commands as well:

.. code::
.. code-block::
sql_alchemy_conn_cmd = bash_command_to_run
Expand Down
15 changes: 15 additions & 0 deletions docs/build
Original file line number Diff line number Diff line change
Expand Up @@ -333,6 +333,20 @@ def check_exampleinclude_for_example_dags():
)


def check_enforce_code_block():
all_docs_files = glob("**/*rst", recursive=True)

for doc_file in all_docs_files:
assert_file_not_contains(
file_path=doc_file,
pattern=r"^.. code::",
message=(
"We recommend using the code-block directive instead of the code directive. "
"The code-block directive is more feature-full."
)
)


MISSING_GCP_DOC_GUIDES = {
'adls_to_gcs',
'bigquery_to_bigquery',
Expand Down Expand Up @@ -489,6 +503,7 @@ clean_files()
check_guide_links_in_operator_descriptions()
check_class_links_in_operators_and_hooks_ref()
check_guide_links_in_operators_and_hooks_ref()
check_enforce_code_block()
check_exampleinclude_for_example_dags()
check_gcp_guides()

Expand Down
Loading

0 comments on commit 60d19dc

Please sign in to comment.