[AIRFLOW-5111] Remove apt-get upgrade from Dockerfile

[Fokko]

Make sure you have checked all steps below.

I'm preparing a PR and noticed that we're doing an apt-get upgrade. My memory is a bit blurry on why we do this. AFAIK this is not a good practice since the base-image should be up to date.

I found this discussion: https://github.com/apache/airflow/pull/4936/files#r268649187

But I think we should rely on the base image and the pinned versions.

While preparing the PR, I noticed that it updates debian-archive-keyring nodejs. Two packages that I'm not particularly interested in, and not sure if it worth spending CI cycles on this:

Step 91/104 : RUN apt-get update     && apt-get upgrade -y --no-install-recommends     && apt-get clean     && rm -rf /var/lib/apt/lists/*
 ---> Running in 35d38754d240
�[91m+ apt-get update
�[0mIgn:1 http://deb.debian.org/debian stretch InRelease
Get:2 http://deb.debian.org/debian stretch-updates InRelease [91.0 kB]
Get:3 http://security.debian.org/debian-security stretch/updates InRelease [94.3 kB]
Get:4 http://repo.mysql.com/apt/debian stretch InRelease [21.6 kB]
Get:5 http://deb.debian.org/debian stretch Release [118 kB]
Get:6 http://deb.debian.org/debian stretch Release.gpg [2434 B]
Get:7 https://deb.nodesource.com/node_10.x stretch InRelease [4585 B]
Get:8 http://repo.mysql.com/apt/debian stretch/mysql-5.6 amd64 Packages [4762 B]
Get:9 http://deb.debian.org/debian stretch-updates/main amd64 Packages [27.4 kB]
Get:10 http://security.debian.org/debian-security stretch/updates/main amd64 Packages [501 kB]
Get:11 http://deb.debian.org/debian stretch/main amd64 Packages [7082 kB]
Get:12 https://deb.nodesource.com/node_10.x stretch/main amd64 Packages [767 B]
Fetched 7948 kB in 1s (4242 kB/s)
Reading package lists...
�[91m+ apt-get upgrade -y --no-install-recommends
�[0mReading package lists...
Building dependency tree...
Reading state information...
Calculating upgrade...
The following packages will be upgraded:
  debian-archive-keyring nodejs
2 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 15.9 MB of archives.
After this operation, 60.4 kB of additional disk space will be used.
Get:1 http://deb.debian.org/debian stretch-updates/main amd64 debian-archive-keyring all 2017.5+deb9u1 [73.9 kB]
Get:2 https://deb.nodesource.com/node_10.x stretch/main amd64 nodejs amd64 10.16.1-1nodesource1 [15.8 MB]
Fetched 15.9 MB in 0s (34.4 MB/s)
(Reading database ... 
(Reading database ... 5%
(Reading database ... 10%
(Reading database ... 15%
(Reading database ... 20%
(Reading database ... 25%
(Reading database ... 30%
(Reading database ... 35%
(Reading database ... 40%
(Reading database ... 45%
(Reading database ... 50%
(Reading database ... 55%
(Reading database ... 60%
(Reading database ... 65%
(Reading database ... 70%
(Reading database ... 75%
(Reading database ... 80%
(Reading database ... 85%
(Reading database ... 90%
(Reading database ... 95%
(Reading database ... 100%
(Reading database ... 20651 files and directories currently installed.)

Preparing to unpack .../debian-archive-keyring_2017.5+deb9u1_all.deb ...

Unpacking debian-archive-keyring (2017.5+deb9u1) over (2017.5) ...

Setting up debian-archive-keyring (2017.5+deb9u1) ...

Removing obsolete conffile /etc/apt/trusted.gpg.d/debian-archive-wheezy-automatic.gpg ...

Removing obsolete conffile /etc/apt/trusted.gpg.d/debian-archive-wheezy-stable.gpg ...

(Reading database ... 
(Reading database ... 5%
(Reading database ... 10%
(Reading database ... 15%
(Reading database ... 20%
(Reading database ... 25%
(Reading database ... 30%
(Reading database ... 35%
(Reading database ... 40%
(Reading database ... 45%
(Reading database ... 50%
(Reading database ... 55%
(Reading database ... 60%
(Reading database ... 65%
(Reading database ... 70%
(Reading database ... 75%
(Reading database ... 80%
(Reading database ... 85%
(Reading database ... 90%
(Reading database ... 95%
(Reading database ... 100%
(Reading database ... 20652 files and directories currently installed.)

Preparing to unpack .../nodejs_10.16.1-1nodesource1_amd64.deb ...

Detected old npm client, removing...

Unpacking nodejs (10.16.1-1nodesource1) over (10.16.0-1nodesource1) ...

Setting up nodejs (10.16.1-1nodesource1) ...

�[91m+ apt-get clean
�[0m�[91m+ rm -rf /var/lib/apt/lists/deb.debian.org_debian_dists_stretch-updates_InRelease /var/lib/apt/lists/deb.debian.org_debian_dists_stretch-updates_main_binary-amd64_Packages.lz4 /var/lib/apt/lists/deb.debian.org_debian_dists_stretch_Release /var/lib/apt/lists/deb.debian.org_debian_dists_stretch_Release.gpg /var/lib/apt/lists/deb.debian.org_debian_dists_stretch_main_binary-amd64_Packages.lz4 /var/lib/apt/lists/deb.nodesource.com_node%5f10.x_dists_stretch_InRelease /var/lib/apt/lists/deb.nodesource.com_node%5f10.x_dists_stretch_main_binary-amd64_Packages.lz4 /var/lib/apt/lists/lock /var/lib/apt/lists/partial /var/lib/apt/lists/repo.mysql.com_apt_debian_dists_stretch_InRelease /var/lib/apt/lists/repo.mysql.com_apt_debian_dists_stretch_mysql-5.6_binary-amd64_Packages.lz4 /var/lib/apt/lists/security.debian.org_debian-security_dists_stretch_updates_InRelease /var/lib/apt/lists/security.debian.org_debian-security_dists_stretch_updates_main_binary-amd64_Packages.lz4
�[0mRemoving intermediate container 35d38754d240

Jira

• My PR addresses the following Airflow Jira issues and references them in the PR title. For example, "[AIRFLOW-5111] My Airflow PR"
  • https://issues.apache.org/jira/browse/AIRFLOW-5111
  • In case you are fixing a typo in the documentation you can prepend your commit with [AIRFLOW-5111], code changes always need a Jira issue.
  • In case you are proposing a fundamental code change, you need to create an Airflow Improvement Proposal (AIP).
  • In case you are adding a dependency, check if the license complies with the ASF 3rd Party License Policy.

Description

• Here are some details about my PR, including screenshots of any UI changes:

Tests

• My PR adds the following unit tests OR does not need testing for this extremely good reason:

Commits

• My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":
  1. Subject is separated from body by a blank line
  2. Subject is limited to 50 characters (not including Jira issue reference)
  3. Subject does not end with a period
  4. Subject uses the imperative mood ("add", not "adding")
  5. Body wraps at 72 characters
  6. Body explains "what" and "why", not "how"

Documentation

• In case of new functionality, my PR adds documentation that describes how to use it.
  • All the public functions and the classes in the PR contain docstrings that explain what it does
  • If you implement backwards incompatible changes, please leave a note in the Updating.md so we can assign it to a appropriate release

Code Quality

• Passes flake8

[potiuk]

I am happy to do it - We now have (automated) rebuild of the images whenever new python security release is released. And that should be semi-frequent (every few months so far - see below). Plus we will have cron rebuilt from scratch every day running full install from the scratch + tests so we should detect whenever we have an incompatible package causing test failure.

Do others (@ashb @BasPH) have different opinion on that?

Python 3.6.9, documentation released on 02 July 2019.
Python 3.6.8, documentation released on 24 December 2018.
Python 3.6.7, documentation released on 20 October 2018.
Python 3.6.6, documentation released on 27 June 2018.
Python 3.6.5, documentation released on 28 March 2018.
Python 3.6.4, documentation released on 19 December 2017.
Python 3.6.3, documentation released on 03 October 2017.
Python 3.6.2, documentation released on 17 July 2017.
Python 3.6.1, documentation released on 21 March 2017.

[Fokko]

Awesome. I think the Python image updates more often, for example, if they change something to the base image, or if they rebuild as soon as the upstream container changes (but this requires --pull flag). In the case of Python, this is buildpack-deps:stretch: https://github.com/docker-library/python/blob/master/3.6/stretch/Dockerfile#L7

[BasPH]

@potiuk I prefer pinned versions to avoid random failures. With those mentioned measures we should be okay. For my understanding, where/how does the cron rebuild run?

[potiuk]

@BasPH we have a build that runs regularly https://travis-ci.org/apache/airflow/builds - those are the builds that have "CRON" label - not yet doing full rebuild but they will.