Check for minimum version of Sqlite (#13496)

Some users testing Airlfow 2.0 with sqlite noticed that for
old versions of sqlite, Airflow does not run tasks and fails with
'sqlite3.OperationalError: near ",": syntax error' when running
tasks. More details about it in #13397.

Bisecting had shown that minimum supported version of sqlite is
3.15.0, therefore this PR adds checking if sqlite version is
higher than that and fails hard if it is not.

Documentation has been updated with minimum requirements, some
inconsisttencies have been removed, also the minimum requirements
for stable 2.0 version were moved to installation.rst because
the requirements were never explicitely stated in the user-facing
documentation.

README.md

@@ -83,15 +83,14 @@ Apache Airflow is tested with:
 | Python       | 3.6, 3.7, 3.8             | 3.6, 3.7, 3.8            | 2.7, 3.5, 3.6, 3.7, 3.8    |
 | PostgreSQL   | 9.6, 10, 11, 12, 13       | 9.6, 10, 11, 12, 13      | 9.6, 10, 11, 12, 13        |
 | MySQL        | 5.7, 8                    | 5.7, 8                   | 5.6, 5.7                   |
-| SQLite       | latest stable             | latest stable            | latest stable              |
+| SQLite       | 3.15.0+                   | 3.15.0+                  | 3.15.0+                    |
 | Kubernetes   | 1.16.9, 1.17.5, 1.18.6    | 1.16.9, 1.17.5, 1.18.6   | 1.16.9, 1.17.5, 1.18.6     |
 
-**Note:** MariaDB and MySQL 5.x are unable to or have limitations with
-running multiple schedulers -- please see the "Scheduler" docs.
+**Note:** MySQL 5.x versions are unable to or have limitations with
+running multiple schedulers -- please see the "Scheduler" docs. MariaDB is not tested/recommended.
 
 **Note:** SQLite is used in Airflow tests. Do not use it in production. We recommend
-using the latest stable version of SQLite for local development. Some older versions
-of SQLite might not work well, however anything at or above 3.27.2 should work fine.
+using the latest stable version of SQLite for local development.
 
 ## Support for Python versions
 

airflow/configuration.py

@@ -30,6 +30,7 @@
 
 # Ignored Mypy on configparser because it thinks the configparser module has no _UNSET attribute
 from configparser import _UNSET, ConfigParser, NoOptionError, NoSectionError  # type: ignore
+from distutils.version import StrictVersion
 from json.decoder import JSONDecodeError
 from typing import Dict, List, Optional, Tuple, Union
 
@@ -230,10 +231,15 @@ def _validate_config_dependencies(self):
             'SequentialExecutor',
         )
         is_sqlite = "sqlite" in self.get('core', 'sql_alchemy_conn')
-        if is_executor_without_sqlite_support and is_sqlite:
-            raise AirflowConfigException(
-                "error: cannot use sqlite with the {}".format(self.get('core', 'executor'))
-            )
+        if is_sqlite and is_executor_without_sqlite_support:
+            raise AirflowConfigException(f"error: cannot use sqlite with the {self.get('core', 'executor')}")
+        if is_sqlite:
+            import sqlite3
+
+            # Some of the features in storing rendered fields require sqlite version >= 3.15.0
+            min_sqlite_version = '3.15.0'
+            if StrictVersion(sqlite3.sqlite_version) < StrictVersion(min_sqlite_version):
+                raise AirflowConfigException(f"error: cannot use sqlite version < {min_sqlite_version}")
 
         if self.has_option('core', 'mp_start_method'):
             mp_start_method = self.get('core', 'mp_start_method')

docs/apache-airflow/installation.rst

@@ -21,6 +21,28 @@ Installation
 
 .. contents:: :local:
 
+
+Prerequisites
+-------------
+
+Airflow is tested with:
+
+* Python: 3.6, 3.7, 3.8
+
+* Databases:
+
+  * PostgreSQL:  9.6, 10, 11, 12, 13
+  * MySQL: 5.7, 8
+  * SQLite: 3.15.0+
+
+* Kubernetes: 1.16.9, 1.17.5, 1.18.6
+
+**Note:** MySQL 5.x versions are unable to or have limitations with
+running multiple schedulers -- please see the "Scheduler" docs. MariaDB is not tested/recommended.
+
+**Note:** SQLite is used in Airflow tests. Do not use it in production. We recommend
+using the latest stable version of SQLite for local development.
+
 Getting Airflow
 '''''''''''''''