Python 3.8 - Sqlite3 version error

[gertnerbot]

python 3.8 centos 7 Docker image

Can't update sqlite3. Tried Airflow 2.0.1 and 2.0.0. Same issue on Python 3.6 with Airflow 2.0.0. I was able to force the install on 2.0.0 but when running a task it failed because of the sqlite3 version mismatch.

Am I just stupid? #13496

#```
(app-root) airflow db init
Traceback (most recent call last):
File "/opt/app-root/bin/airflow", line 5, in
from airflow.main import main
File "/opt/app-root/lib64/python3.8/site-packages/airflow/init.py", line 34, in
from airflow import settings
File "/opt/app-root/lib64/python3.8/site-packages/airflow/settings.py", line 37, in
from airflow.configuration import AIRFLOW_HOME, WEBSERVER_CONFIG, conf # NOQA F401
File "/opt/app-root/lib64/python3.8/site-packages/airflow/configuration.py", line 1007, in
conf.validate()
File "/opt/app-root/lib64/python3.8/site-packages/airflow/configuration.py", line 209, in validate
self._validate_config_dependencies()
File "/opt/app-root/lib64/python3.8/site-packages/airflow/configuration.py", line 246, in _validate_config_dependencies
raise AirflowConfigException(f"error: cannot use sqlite version < {min_sqlite_version}")
airflow.exceptions.AirflowConfigException: error: cannot use sqlite version < 3.15.0
(app-root) python -c "import sqlite3; print(sqlite3.sqlite_version)"
3.7.17
(app-root) python --version
Python 3.8.6
(app-root) pip install --upgrade sqlite3
ERROR: Could not find a version that satisfies the requirement sqlite3 (from versions: none)
ERROR: No matching distribution found for sqlite3

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[ashb]

This is the version of the system sqlite library, not the python bindings.

I.e. brew upgrade sqlite or similar.

A PR to make that error message clearer would be happily merged.

[potiuk]

I've added a bit clearer documentation about it in #14209, added troubleshooting section (most likely you have an older version of sqlite library that airflow uses, and it is different than in case of your local "python" execution (LD_LIBRARY_PATH likely is a problem) . You need to remove older version of the libraries you have in your system (or better - upgrade your OS to something a bit "newer" . Centos 7 was relased in 2009 and Centos 8 has been released in 2011, so I believe it is highest time to upgrade.

If you are using such an old OS, you are unfortunately on your own and we will not be able to help, so upgrading OS is a good idea I think.

[pgibert]

Hello close to your problem i think ... .. i got the same error with python 3.8 centos 7 but on a manual installation ( no docker)
resolved with the following steps :

1. wget https://www.sqlite.org/2021/sqlite-autoconf-3340100.tar.gz
2. tar xvf sqlite-autoconf-3340100.tar.gz
3. cd sqlite-autoconf-3340100
4. ./configure
5. make
6. sudo make install
7. ls /usr/local/lib
8. export LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH
9. airfow info

Apache Airflow: 2.0.1
System info
.........
....... etc --> ok

[subashcanapathy]

I used the following on amazonlinux2 to get latest sqlite

    # Pre-req
    yum -y install wget tar gzip gcc make expect

    wget https://www.sqlite.org/src/tarball/sqlite.tar.gz
    tar xzf sqlite.tar.gz
    cd sqlite/
    export CFLAGS="-DSQLITE_ENABLE_FTS3 \
        -DSQLITE_ENABLE_FTS3_PARENTHESIS \
        -DSQLITE_ENABLE_FTS4 \
        -DSQLITE_ENABLE_FTS5 \
        -DSQLITE_ENABLE_JSON1 \
        -DSQLITE_ENABLE_LOAD_EXTENSION \
        -DSQLITE_ENABLE_RTREE \
        -DSQLITE_ENABLE_STAT4 \
        -DSQLITE_ENABLE_UPDATE_DELETE_LIMIT \
        -DSQLITE_SOUNDEX \
        -DSQLITE_TEMP_STORE=3 \
        -DSQLITE_USE_URI \
        -O2 \
        -fPIC"
    export PREFIX="/usr/local"
    LIBS="-lm" ./configure --disable-tcl --enable-shared --enable-tempstore=always --prefix="$PREFIX"
    make
    make install

[potiuk]

Maybe @subashcanapathy an/ @pgibert you could add a PR to guide people in https://github.com/apache/airflow/blob/master/docs/apache-airflow/howto/set-up-database.rst#setting-up-a-sqlite-database ?

Sounds like a useful "howto" part for people who use centos 7 / amazonlinux2 .