Distinct repository content views

README.md

@@ -1,8 +1,9 @@
 # ns8-porthos
 
-Host Rocky Linux repository mirrors on Porthos
+Host Distfeed and Rocky Linux repository mirrors on Porthos
 
-This module provides a simple web stack for DNF repositories, based on Nginx + PHP-FPM
+This module provides a simple web stack to serve DNF repositories and NS8
+application Distfeed, based on Nginx + PHP-FPM.
 
 ## Install
 
@@ -56,23 +57,23 @@ Change one or more attributes with a command like this:
 To publish the repository, a HTTP host name route must be configured from
 the cluster-admin Settings page.
 
-## Commands
-
-### `take-snapshot`
+When the `configure-module` action is executed for the first time, service
+and timer units are enabled and started.
 
-To create a snapshot manually run:
+Timers are:
 
-    runagent -m porthos1 take-snapshot
+- `take-snapshot.timer` is a weekly job that update Rocky Linux repository
+  mirror contents and makes a copy of repodata.json and other metadata
+  from NS8 repositories.
 
-As alternative start the equivalent Systemd service:
+- `sync-head.timer` is a frequent job that 4 times per hour, from Monday to
+  Friday and during working hours, makes a copy of repodata.json and other
+  metadata from NS8 repositories.
 
-    runagent -m porthos1 systemctl --user start snapshot
+Services are:
 
-### Inspect the services status
-
-Print the status of relevant Systemd units:
-
-    runagent -m porthos1 systemctl --user status fpm.service nginx.service snapshot.timer
+- `fpm.service`
+- `nginx.service`
 
 ## Environment variables
 
@@ -110,4 +111,129 @@ net.ipv4.tcp_mem = 42456 169824 679296
 net.core.wmem_max = 16482304
 net.core.rmem_max = 16482304
 net.ipv4.tcp_rmem = 4096 16384 16482304
-```
+```
+
+## Content views
+
+Content views are designed for two different purposes. On one hand, a
+subscribed cluster wants to see the latest app updates from the Software
+Center page, which needs the Distfeed content. On the other hand the
+nightly cluster update procedure wants to install managed updates
+automatically for both the base Rocky Linux OS and NS8 applications.
+
+Web clients can send an optional `X-Repo-View` header, with value `latest`
+or `managed`. Any other value is considered like `latest`, but this
+behavior may change in the future.
+
+Header codes:
+
+- `A` Authenticated
+- `U` Anonymous
+- `M` Managed repo view
+- `L` Latest repo view
+- `X` X-Repo-View header not sent
+
+Request types:
+
+- `df` request for Distfeed (like `repodata.json`)
+- `rl` request for rockylinux mirror content
+
+Response types:
+
+- `L` content from the latest snapshot
+- `S` content from a managed (past) snapshot
+- `H` content from Distfeed head (updated multiple times per day from
+  Distfeed upstream)
+
+The following table summarizes the response for every headers/request
+combination.
+
+| headers/request | df | rl |
+|-----------------|----|----|
+| AX              | S  | S  |
+| AM              | S  | S  |
+| AL              | H  | L  |
+| UM              | H  | L  |
+| UL              | H  | L  |
+| UX              | L  | L  |
+
+## Content schedule
+
+NS8 clients run automatic updates from Tuesday to Friday, at some random
+time between 00:00 and 06:00 (client local time). Clients are assigned to
+a content tier, based on a hash function of the credentials used for
+authentication. Tiers have different, increasing size. The smaller one
+receive updates before the larger one:
+
+- Tier 1, 10%. Age 3 days
+- Tier 2, 20%. Age 4 days
+- Tier 3, 70%. Age 5 days
+
+The tier Age defines how old a snapshot must be before being served by the
+`managed` view.
+
+The following table helps to understand the day of the week a snapshot is
+served (values), starting from the day of the week a snapshot was created
+(first column) and the client tier (first row). The table assumes a
+snapshot is not created during the night (from 12 to 6 AM).
+
+| -   | tier 1 | tier 2 | tier 3 |
+|:---:|--------|--------|--------|
+| FRI | Tue*   | Wed*   | Thu*   |
+| SAT | Wed*   | Thu*   | Fri*   |
+| SUN | Wed    | Thu    | Fri    |
+| MON | Fri    | Tue*   | Tue*   |
+| TUE | Tue*   | Tue*   | Tue*   |
+| WED | Tue*   | Tue*   | Tue*   |
+| THU | Tue*   | Tue*   | Wed*   |
+
+The asterisk in a value indicates a day of the next week. The
+`snapshot.timer` runs on Fridays at 21:00 UTC.
+
+The table shows that taking a snapshot on Tuesdays and Wednesdays
+flattenize the day of update for every tier, which may be undesired.
+
+
+## Commands
+
+The Distfeed head is synchoronized by the `sync-head` command. It fetches
+Distfeed from its upstream on GitHub. A new snapshot of Rocky Linux and
+Distfeed is created with `take-snapshot`. The two commands are described in
+the next sections.
+
+The two commands are executed periodically by two Systemd timer units:
+`snapshot.timer` and `sync-head.timer`. They can be run manually at any
+time too, but refer to the Content policy and Content schedule sections to
+understand the implications.
+
+### `take-snapshot`
+
+To create a snapshot manually run:
+
+    runagent -m porthos1 take-snapshot
+
+As alternative start the equivalent Systemd service:
+
+    runagent -m porthos1 systemctl --user start snapshot
+
+Each snapshot contains a copy of NS8 Distfeed data fetched from GitHub and
+an exact copy of Rocky Linux BaseOS and AppStream DNF repositories. To
+save disk space and performance, it is recommended to store the snapshots
+on a filesystem or block device that provides data deduplication.
+
+### `sync-head`
+
+To synchronize the copy of NS8 repodata with its GitHub upstream manually
+run:
+
+    runagent -m porthos1 sync-head
+
+As alternative start the equivalent Systemd service:
+
+    runagent -m porthos1 systemctl --user start sync-head
+
+### Inspect the services status
+
+Print the status of relevant Systemd units:
+
+    runagent -m porthos1 systemctl --user status fpm.service nginx.service snapshot.timer sync-head.timer

imageroot/actions/configure-module/80restart_services

@@ -10,8 +10,8 @@ set -e
 exec 1>&2
 
 if [[ -z ${PORTHOS_FIRST_CONFIG} ]]; then
-    systemctl --user enable --now fpm.service nginx.service snapshot.timer
+    systemctl --user enable --now fpm.service nginx.service snapshot.timer sync-head.timer
     echo 'PORTHOS_FIRST_CONFIG=1' >> environment
 else
-    systemctl --user try-reload-or-restart fpm.service nginx.service snapshot.timer
+    systemctl --user try-reload-or-restart fpm.service nginx.service snapshot.timer sync-head.timer
 fi

imageroot/bin/sync-head

@@ -0,0 +1,29 @@
+#!/bin/bash
+
+#
+# Copyright (C) 2024 Nethesis S.r.l.
+# SPDX-License-Identifier: GPL-3.0-or-later
+#
+
+set -e
+shopt -s nullglob
+
+# shellcheck disable=SC1091
+source /etc/nethserver/core.env
+new_head=tmp.$(date +head.%Y%m%dt%H%I%S%2N)
+rsync_opts=(-aiSH --no-motd --no-super --no-perms --chmod=ugo=rwX --no-g --no-o --delete-after)
+distfeed_url=https://github.com/NethServer/ns8-repomd/archive/refs/heads/repomd.tar.gz
+
+cd /srv/porthos/webroot
+
+trap 'rm -rf remove."${new_head}" "${new_head}"' EXIT
+
+printf "Sync distfeed head from %s\n" "${distfeed_url}"
+mkdir -vp "${new_head}/distfeed" head
+curl --fail -L -O --output-dir "${new_head}" "${distfeed_url}"
+tar -z -x --strip-components=1 -f "${new_head}/repomd.tar.gz" -C "${new_head}/distfeed"
+rm -f "${new_head}/repomd.tar.gz"
+
+# Substitute the head directory with new_head
+mv head remove."${new_head}" || :
+mv -v "${new_head}" head || :

imageroot/script/snapshot.php

@@ -43,36 +43,76 @@ function snapshot_age($snapshot) {
     return $days_age;
 }
 
-function main() {
-    // Minimum age (days) expected by tiers
-    $tier_age = [3, 4, 5];
-
+function serve_from_latest_snapshot() {
     $lsnapshots = array_map('basename', glob('/srv/porthos/webroot/d20*'));
+    if (count($lsnapshots) == 0) {
+        return 'not-found'; // Nginx will fail the file and return 404 for us
+    }
+    sort($lsnapshots);
+    $snapshot = array_pop($lsnapshots);
+    return $snapshot;
+}
 
+function serve_from_snapshots() {
+    $lsnapshots = array_map('basename', glob('/srv/porthos/webroot/d20*'));
     if (count($lsnapshots) == 0) {
-        http_response_code(404);
-        echo "Not found\n";
+        return 'not-found'; // Nginx will not find the file and return 404 for us
+    }
+    sort($lsnapshots);
+    $snapshot = array_pop($lsnapshots);
+    // Minimum age (days) expected by tiers
+    $tier_age = [3, 4, 5];
+    while($snapshot != NULL) {
+        $tier_id = system_tier($_SERVER['PHP_AUTH_USER']);
+        if (snapshot_age($snapshot) >= $tier_age[$tier_id]) {
+            break;
+        }
+        $snapshot = array_pop($lsnapshots);
     }
+    return $snapshot;
+}
 
+function main() {
+    $repo_view = isset($_SERVER['HTTP_X_REPO_VIEW']) ? $_SERVER['HTTP_X_REPO_VIEW'] : "unknown";
     $username = isset($_SERVER['PHP_AUTH_USER']) ? $_SERVER['PHP_AUTH_USER'] : "";
     $password = isset($_SERVER['PHP_AUTH_PW']) ? $_SERVER['PHP_AUTH_PW'] : "";
 
-    sort($lsnapshots);
-
-    $snapshot = array_pop($lsnapshots);
-    if($username && $password) {
-        // Authenticated clients gain access to older tiers
-        while($snapshot != NULL) {
-            $tier_id = system_tier($username);
-            if (snapshot_age($snapshot) >= $tier_age[$tier_id]) {
-                break;
-            }
-            $snapshot = array_pop($lsnapshots);
+    $is_authenticated = ($username != "") && ($password != "");
+    $is_distfeed_request = substr($_SERVER['DOCUMENT_URI'], 0, 10) == "/distfeed/";
+
+    //
+    // This is the enumeration if input => output cases
+    //
+    // AX, AM => SS
+    // UX => LL
+    // AL, UL, UM => HL
+    //
+    // Where A=Authenticated, U=Not authenticated,
+    //       X=no-view, M=managed view, L=latest view
+    //       SS=from snapshots, LL=from latest,
+    //       HL=distfeed from head, other content from latest
+    //
+
+    if($repo_view == "unknown") {
+        // Core <2.10 does not send the X-Repo-View header. We implement
+        // the initial update policy for backward compatibility.
+        if($is_authenticated) {
+            $prefix = serve_from_snapshots(); // AX => SS
+        } else {
+            $prefix = serve_from_latest_snapshot(); // UX => LL
         }
+    } else if($is_authenticated && $repo_view == "managed") {
+        // Automated nightly update job receive managed updates.
+        $prefix = serve_from_snapshots(); // AM => SS
+    } else if($is_distfeed_request) {
+        $prefix = 'head'; // AL, UL, UM => H.
+    } else {
+        // AL, UL, UM => .L
+        $prefix = serve_from_latest_snapshot();
     }
 
     header('Cache-Control: private');
-    header('X-Accel-Redirect: /' . $snapshot . $_SERVER['DOCUMENT_URI']);
+    header('X-Accel-Redirect: /' . $prefix . $_SERVER['DOCUMENT_URI']);
 }
 
 // Run

imageroot/systemd/user/sync-head.service

@@ -0,0 +1,12 @@
+#
+# Copyright (C) 2023 Nethesis S.r.l.
+# SPDX-License-Identifier: GPL-3.0-or-later
+#
+
+[Unit]
+Description=Scheduled Porthos sync-head procedure
+
+[Service]
+Type=oneshot
+ExecStart=runagent sync-head
+SyslogIdentifier=%N

imageroot/systemd/user/sync-head.timer

@@ -0,0 +1,13 @@
+#
+# Copyright (C) 2024 Nethesis S.r.l.
+# SPDX-License-Identifier: GPL-3.0-or-later
+#
+
+[Unit]
+Description=Timer of Porthos sync-head procedure
+
+[Timer]
+OnCalendar=Mon..Fri *-*-* 07..17:00,30:00 UTC
+
+[Install]
+WantedBy=timers.target

imageroot/update-module.d/50restart

@@ -5,9 +5,7 @@
 # SPDX-License-Identifier: GPL-3.0-or-later
 #
 
-set -e
-
 exec 1>&2
 
 # Restart the service
-systemctl --user try-restart fpm.service nginx.service snapshot.timer
+systemctl --user try-restart fpm.service nginx.service snapshot.timer sync-head.timer

ui/index.html

@@ -1 +1,2 @@
-<h1>Porthos</h1>
+<h1>Porthos</h1>
+<p>This application has no UI, see <a target="_blank" href="https://github.com/NethServer/ns8-porthos/blob/main/README.md">README.md</a></p>

ui/metadata.json

@@ -0,0 +1,22 @@
+{
+    "name": "Porthos",
+    "description": {
+        "en": "Managed distribution of NS8 software updates",
+        "it": "Distribuzione gestita degli aggiornamenti software di NS8"
+    },
+    "categories": [
+        "tools"
+    ],
+    "authors": [
+        {
+            "name": "Nethesis",
+            "email": "[email protected]"
+        }
+    ],
+    "docs": {
+        "documentation_url": "https://github.com/NethServer/ns8-porthos/blob/main/README.md",
+        "bug_url": "https://github.com/NethServer/dev",
+        "code_url": "https://github.com/NethServer/ns8-porthos"
+    },
+    "source": "ghcr.io/nethserver/porthos"
+}

ui/shortcuts.json

@@ -0,0 +1 @@
+[]