Fail when SQLite version is not high enough

[potiuk]

Seems that in Airflow 2.0 we use some features of sqlite3 which do not work with really old versions of sqlite (for sure for 3.11.0 which was released in 2016).

Following #13397 (comment) and few other slack discussions, seems that there are enough systems out there (CentOS, RHEL7) that come with really old sqlite (RHEL7 with a 7 years old one!)

We have so far no formal requirement for sqlite3 but in fact there is one. We just do not know which. I propose to take as a base the one we have in CI in buster debian: 3.27.2:

root@b8a8e73caa2c:/opt/airflow# python 
Python 3.6.12 (default, Nov 25 2020, 03:59:00) 
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sqlite3
>>> sqlite3.sqlite_version
'3.27.2'
>>> 

My propoosal is that we should fail Aiflow with appropriate message if sqlite is the db and the sqlite version is lower than that.

Description

Use case / motivation

Are you willing to submit a PR?

Related Issues

[DuongPTIT]

I upgrade sqlite3 to newest version, but it still throw error:
`Python 3.6.12 (default, Jan 6 2021, 11:20:09)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-36)] on linux
Type "help", "copyright", "credits" or "license" for more information.

import sqlite3
sqlite3.sqlite_version
'3.34.0'

`

[potiuk]

It worked for me when I tsted it. You must have airflow started in the way that uses still old version of sqlite. If you installed it wth the LD_LIBRARY_PATH modification, you need tto make sure your airflow also has the LD_LIBRARY_PATH set to point to the right sqlite library. Ideally - remove all other sqlite libraries to be sure airflow is using the right one.

What also might help if you start using some newer version of RedHat VM that already might have a version of sqlite which is not very old.

Sorry I can't help more. I tested it all and installing older/newer version of sqlite shows/solves the problem. You are on your own to make sure that the right version of sqlite is used. Here is the guide that might hep you.

https://unix.stackexchange.com/questions/434100/updating-the-sqlite-version-used-by-python-3-on-centos-7

[potiuk]

Closed by #13496