Skip to content

Commit

Permalink
Add better description and guidance in case of sqlite version mismatch
Browse files Browse the repository at this point in the history
Closes: #14208
  • Loading branch information
potiuk committed Feb 13, 2021
1 parent 0f384f0 commit 6d74319
Show file tree
Hide file tree
Showing 2 changed files with 50 additions and 1 deletion.
6 changes: 5 additions & 1 deletion airflow/configuration.py
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,7 @@

from airflow.exceptions import AirflowConfigException
from airflow.secrets import DEFAULT_SECRETS_SEARCH_PATH, BaseSecretsBackend
from airflow.utils.docs import get_docs_url
from airflow.utils.module_loading import import_string

log = logging.getLogger(__name__)
Expand Down Expand Up @@ -243,7 +244,10 @@ def _validate_config_dependencies(self):
# Some of the features in storing rendered fields require sqlite version >= 3.15.0
min_sqlite_version = '3.15.0'
if StrictVersion(sqlite3.sqlite_version) < StrictVersion(min_sqlite_version):
raise AirflowConfigException(f"error: cannot use sqlite version < {min_sqlite_version}")
raise AirflowConfigException(
f"error: sqlite C library version too old (< {min_sqlite_version}). "
f"See {get_docs_url('howto/set-up-database.rst#setting-up-a-sqlite-database')}"
)

if self.has_option('core', 'mp_start_method'):
mp_start_method = self.get('core', 'mp_start_method')
Expand Down
45 changes: 45 additions & 0 deletions docs/apache-airflow/howto/set-up-database.rst
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,51 @@ the example below.
The exact format description is described in the SQLAlchemy documentation, see `Database Urls <https://docs.sqlalchemy.org/en/14/core/engines.html>`__. We will also show you some examples below.

Setting up a SQLite Database
----------------------------

SQLite database can be used to run Airflow for development purpose as it does not require any database server
(the database is stored in a local file). There are a few limitations of using the SQLite database (for example
it only works with Sequential Executor) and it should NEVER be used for production.

There is a minimum version of sqlite3 required to run Airflow 2.0+ - minimum version is 3.15.0. Some of the
older systems have an earlier version of sqlite installed by default and for those system you need to manually
upgrade SQLite to use version newer than 3.15.0. Note, that this is not a ``python library`` version, it's the
SQLite system-level application that needs to be upgraded. There are different ways how SQLIte might be
installed, you can find some information about that at the `official website of SQLite
<https://www.sqlite.org/index.html>`_ and in the documentation specific to distribution of your Operating
System.

**Troubleshooting**

Sometimes even if you upgrade SQLite to higher version and your local python reports higher version,
the python interpreter used by Airflow might still use the older version available in the
``LD_LIBRARY_PATH`` set for the python interpreter that is used to start Airflow.

You can make sure which version is used by the interpreter by running this check:

.. code-block:: bash
root@b8a8e73caa2c:/opt/airflow# python
Python 3.6.12 (default, Nov 25 2020, 03:59:00)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sqlite3
>>> sqlite3.sqlite_version
'3.27.2'
>>>
But be aware that setting environment variables for your Airflow deployment might change which SQLite
library is found first, so you might want to make sure that the "high-enough" version of SQLite is the only
version installed in your system.

An example URI for the sqlite database:

.. code-block:: text
sqlite:////home/airflow/airflow.db
Setting up a MySQL Database
---------------------------

Expand Down

0 comments on commit 6d74319

Please sign in to comment.