Prepare for Ubuntu 16.04 End of Standard Support

[aahlenst]

Standard support for the Ubuntu 16.04 line ends at the end of April 2021. Afterwards, patches are only available if ESM is purchased from Canonical. Affected machines according to our inventory:

• ci.adoptopenjdk.net (Issue)
• trss.adoptopenjdk.net Already upgraded to 18.04 - will update inventory
• ansible.adoptopenjdk.net (No longer in use, kept for reference when setting up new server)
• ubuntu1604-armv7-1 (Scaleway, replacement in progress)
• ubuntu1604-armv7-2 (Scaleway, replacement in progress)
• build-ubuntu1604-x64-1 (Scaleway) Only machine tagged for fpm (installers) Now offline as test-packet-ubuntu-1804-x64-2 can now do this.
• docker-ubuntu1604-x64-1 (AWS) Marked offline
• ubuntu1604-s390x-1 (Marist) Marked offline
• ubuntu1604-s390x-2 (Marist) Upgraded to 18.04
• ubuntu1604-s390x-3 (Marist) Upgraded to 18.04
• ubuntu1604-s390x-4 (Marist) Upgraded to 18.04
• test-ubuntu1604-armv8-1 (Packet) To be decomissioned when Altras fully in place
• test-ubuntu1604-armv8-2 (Packet) Already upgraded to 18.04 - needs renamed
• docker-ubuntu1604-armv8-1 (Packet) Marked offline - had 3 docker executors
• ubuntu1604-x64-1 (Packet) Marked offline
• ubuntu1604-x64-2 (Packet) Marked offline
• ubuntu1604-x64-3 (Packet) Upgrade completed but won't restart properly
• ubuntu1604-ppc64le-1 (OSUOSL) Marked offline
• ubuntu1604-ppc64le-2 (OSUOSL) Marked offline
• ubuntu1604-ppc64le-3 (OSUOSL) Decomissioned
• ubuntu1604-ppc64le-4 (OSUOSL) Decomissioned
• ubuntu1604-x64-1 (IBM Cloud) Marked offline

[sxa]

Basic upgrade process is as follows, followed by updating the hostname on the system accordingly - more details underneath.

apt-get update
apt-get upgrade
apt-get dist-upgrade
apt-get install ubuntu-release-upgrader-core
do-release-upgrade

In general for upgrades with new configuration files:

• grub: Don't update - may be provider specific things in there (xLinux packet systems will need to have the update accepted)
• sshd: Don't update - may affect our ability to connect to it if our setup is changed
• NTP: Take updates
• sysctl.conf - since we don't edit this in playbooks, accept the updates

I had an issue on one machine where it repeatedly asked about my timezone. In that case I broke out of the do-release-upgrade process and run apt-get dist-upgrade which helped it to go past that point. (If you don't have do-release-upgrade install the ubuntu-release-upgrader-core package to get it)
The marist machines were struggling to connect to the upgrade servers over IPv6 so I added explicit IPv4 entries for the problematic hosts to /etc/hosts which allowed the updates to succeed - at the time of writing these were the values used:

91.189.95.85 ppa.launchpad.net
91.189.88.142 ports.ubuntu.com

[sxa]

Note that after upgrading from 16.04 to 18.04 the git that we build in the GIT_source role does not work:
stderr: /usr/local/libexec/git-core/git-remote-https: /usr/lib/s390x-linux-gnu/libcurl.so.4: version CURL_OPENSSL_3' not found (required by /usr/local/libexec/git-core/git-remote-https)
Fix is to remove the version of git in /usr/local/bin and allow the default installed one with Ubuntu 18.04 (which is 2.17.1) to be used (May require override of the git tool location in jenkins for those agents if it has been explicitly set to /usr/local/bin/git
The files associated with the GIT_Source install are listed in
gitsourcefiles.gz
the attached file (Use some variation of wget -q -O - https://github.com/AdoptOpenJDK/openjdk-infrastructure/files/6386223/gitsourcefiles.gz | xargs rm as a quick way to remove them)
Slack reference thread: https://adoptium.slack.com/archives/C53GHCXL4/p1619448983268400

[sxa]

Also note: At present all xLinux sw.tool.docker nodes are Ubuntu 16.04

[sxa]

Ubuntu 16.04 docker image created for arm32 so they can be "hidden" from the network as we do for CentOS6, which will enable us to decomission the scaleway systems and move to alternate hosts

[sxa]

I believe we only have the 32-bit docker containers used for building arm32 images live running that level of the OS (and are to be retained like the x64 CentOS ones), along with the jenkins server which is covered in #2108, therefore I am closing this.