v3.0.0

Important

Version 3.0.0 includes numerous breaking changes that are not backwards compatible!

Features and Changes

• Replace Temporal with a simpler task system: River.
  • Temporal provided a lot of nice features but ultimately is too difficult for a selfhosted application. Using a simpler task system with custom workflows allows for more customization.
  • Bring back the ability to restart any task from the queue page (except for live video and chat downloads).
    • This brings back the possibility to pull a new version of Ganymede to resolve an issue and then re-run failing archive tasks.
    • 
  • Migrate all but one scheduled tasks to River. The live channel check is not run in River as that will just pollute the queue and some users have a very low interval configured.
  • Optionally deploy River WebUI for a look into the tasks queue
    • 
  • Archive tasks have a heartbeat. If the task times out (crashes, Ganymede restarts, etc) a watchdog will attempt to re-queue the task.
    • Livestream video and chat downloads are not retried, instead they are cancelled and the subsequent workflows (post-process video and convert chat) started.
• Add support to customize all the directories inside the container using environment variables (#337).
  • On start a migration function will check if the paths change. If a change is detected it will attempt to update all paths in the database.
• Category restrictions can now optionally be applied to live streams.
  • On each channel check, the category of the live stream is compared against the list of categories configured in the watched channel settings. If a match is found then the stream is archived. It will not detect if the channel switches to a different category after the live stream archive is created. If this is something that interests you let me know!
  • 
• Add a "Blocked VODs" page in the admin panel.
  • Video IDs can be blocked from being archived.
• Ability to cancel a VOD archive in-progress.
  • Optionally can delete the video along with the video files.
  • Optionally can block the video ID from future archives.
• Add a button to regenerate the static thumbnail for videos.
  • A tasked is queued which uses ffmpeg to extract a screenshot at a random position for the full resolution thumbnail along with the web thumbnail.
  • Browsers along with the nginx config aggressively cache thumbnails so a hard reload may be required to see the updated thumbnail.
  • 

Notable Changes

Important changes that don't break functionality.

• The ganymede-nginx container isn't required anymore. The API container will serve the static files at the VIDEO_DIR env var route. Example: VIDEO_DIR=/data/videos then it would be available at localhost:4800/data/videos. There is no index so you can't browse files. Updating the frontend CDN_URL to localhost:4800/data/videos is the same as serving the files via Nginx.
  • I recommend still using Nginx, it provides caching and sets headers so your browser caches important images like thumbnails. This is mainly for people who don't like running a lot of containers.

Breaking Changes

• Completely different worker system (requires changes to the compose file)
  • Remove temporal containers
  • Add a new river-ui container
• Moved the following settings from the config.json to environment variables (set these as environment variables on the api container)
  • DEBUG
  • OAUTH_ENABLED
• Added environment variables to change the paths of important directory inside the container. Changing these will require changing the mounted volumes.
  • VIDEOS_DIR
  • TEMP_DIR
  • CONFIG_DIR
  • LOGS_DIR
    The defaults of these are:

	VideosDir string `env:"VIDEOS_DIR, default=/data/videos"`
	TempDir   string `env:"TEMP_DIR, default=/data/temp"`
	ConfigDir string `env:"CONFIG_DIR, default=/data/config"`
	LogsDir   string `env:"LOGS_DIR, default=/data/logs"`

• **You will need to update the paths for your containers to match these new defaults, or override the default by specifying the path in the environment variables. Example:

volumes:
-   - /mnt/nas/vods:/vods
+  - /mnt/nas/vods:/data/videos
-   - ./logs:/logs
+  - ./logs:/data/logs
+  - ./temp:/data/temp
+  - ./config:/data/config # be sure to move `config.json` if needed

• I recommend mounting the TEMP_DIR as a volume so temporary files are not lost on container restart. See the below docker-compose diff for an example.

Updating Your Instance to v3.0.0

Important

BACK UP YOUR CURRENT INSTANCE, INCLUDING THE DATABASE!!!

The upgrade is non-reversible. Be sure to create a backup of the database if something goes wrong. Rely on your standard backup procedure or create a database dump by running the following command.

docker exec ganymede-db pg_dump -U ganymede ganymede-prd | gzip > /tmp/dump.sql.gz

Upgrade Steps

Wait until you have no active archives. There is no archive comparability between versions!

It may be simpler to start with a fresh and updated docker-compose.yml file. You will need to update the paths and any env vars from your old compose file.

1. Bring down all containers docker compose down
2. Make modifications to the docker-compose.yml file. Optionally you can start with a fresh docker-compose.yml file. Be sure to update the volume mounts to account for the new *_DIR variables.
   1. Perform the changes to your docker-compose.yml outlined in the breaking changes section above. This includes...
   2. Removing the temporal containers, and any references to it.
   3. Optionally update the paths in the VIDEOS_DIR and TEMP_DIR environment variable. I recommend mounting the TEMP_DIR to a volume on your host. This is to prevent losing data if the container crashes or restarts. Any modification to these variables requires changing the volume mounts as well.
   4. Adding the river-ui container

version: "3.3"
services:
  ganymede-api:
    container_name: ganymede-api
    image: ghcr.io/zibbp/ganymede:latest
    restart: unless-stopped
    depends_on:
-      - ganymede-temporal
+      - ganymede-db
    environment:
      - TZ=America/Chicago # Set to your timezone
+      - DEBUG=false # set to true for debug logs 
+      - VIDEOS_DIR=/data/videos
+      - TEMP_DIR=/data/temp
+      - LOGS_DIR=/data/logs
+      - CONFIG_DIR=/data/config
      - DB_HOST=ganymede-db
      - DB_PORT=5432
      - DB_USER=ganymede
      - DB_PASS=PASSWORD
      - DB_NAME=ganymede-prd
      - DB_SSL=disable
      - JWT_SECRET=SECRET
      - JWT_REFRESH_SECRET=SECRET
      - TWITCH_CLIENT_ID=
      - TWITCH_CLIENT_SECRET=
      - FRONTEND_HOST=http://IP:PORT
-      - COOKIE_DOMAIN=http://domain.com
      # OPTIONAL
+     # - OAUTH_ENABLED=false
      # - OAUTH_PROVIDER_URL=
      # - OAUTH_CLIENT_ID=
      # - OAUTH_CLIENT_SECRET=
      # - OAUTH_REDIRECT_URL=http://IP:PORT/api/v1/auth/oauth/callback # Points to the API service
-     - TEMPORAL_URL=ganymede-temporal:7233
      # WORKER
      - MAX_CHAT_DOWNLOAD_EXECUTIONS=2
      - MAX_CHAT_RENDER_EXECUTIONS=2
      - MAX_VIDEO_DOWNLOAD_EXECUTIONS=3
      - MAX_VIDEO_CONVERT_EXECUTIONS=2
    volumes:
-   - /mnt/nas/vods:/vods
+   - /mnt/nas/vods:/data/videos
-   - ./logs:/logs
+   - ./logs:/data/logs
+   - ./temp:/data/temp
+   - ./config:/data/config # be sure to move `config.json` if needed
    ports:
      - 4800:4000
  ganymede-frontend:
    container_name: ganymede-frontend
    image: ghcr.io/zibbp/ganymede-frontend:latest
    restart: unless-stopped
    environment:
      - API_URL=http://IP:PORT # Points to the API service
      - CDN_URL=http://IP:PORT # Points to the CDN service
      - SHOW_SSO_LOGIN_BUTTON=true # show/hide SSO login button on login page
      - FORCE_SSO_AUTH=false # force SSO auth for all users (bypasses login page and redirects to SSO)
      - REQUIRE_LOGIN=false # require login to view videos
    ports:
      - 4801:3000
-  ganymede-temporal:
-    image: temporalio/auto-setup:1.23
-    container_name: ganymede-temporal
-    depends_on:
-      - ganymede-db
-    environment:
-      - DB=postgres12 # this tells temporal to use postgres (not the db name)
-      - DB_PORT=5432
-      - POSTGRES_USER=ganymede
-      - POSTGRES_PWD=PASSWORD
-      - POSTGRES_SEEDS=ganymede-db # name of the db service
-    ports:
-      - 7233:7233
-  # -- Uncomment below to enable temporal web ui --
-  # ganymede-temporal-ui:
-  #   image: temporalio/ui:latest
-  #   container_name: ganymede-temporal-ui
-  #   depends_on:
-  #     - ganymede-temporal
- #   environment:
-  #     - TEMPORAL_ADDRESS=ganymede-temporal:7233
-  #   ports:
-  #     - 8233:8080
  ganymede-db:
    container_name: ganymede-db
    image: postgres:14
    volumes:
      - ./ganymede-db:/var/lib/postgresql/data
    environment:
      - POSTGRES_PASSWORD=PASSWORD
      - POSTGRES_USER=ganymede
      - POSTGRES_DB=ganymede-prd
    ports:
      - 4803:5432
  ganymede-nginx: # this isn't necessary if you do not want to serve static files via Nginx. The API container will serve the `VIDEO_DIR` environment variable.
    container_name: ganymede-nginx
    image: nginx
    volumes:
      - /path/to/nginx.conf:/etc/nginx/nginx.conf:ro
-      - /pah/to/vod/stoage:/mnt/vods
+     - /mnt/nas/vods:/data/videos
    ports:
      - 4802:8080
+ ganymede-river-ui:
+   image: ghcr.io/riverqueue/riverui:0.3
+   environment:
+     - DATABASE_URL=postgres://ganymede:DB_PASSWORD@ganymede-db:5432/ganymede-prd # update with env settings from the ganymede-db container
+   ports:
+     - 4804:8080

4. Delete unused temporal directory rm -rf temporal (if you still have it).
5. Update your nginx.conf with the contents from https://github.com/Zibbp/ganymede/blob/main/nginx.conf. This updates the paths to use /data/videos. Make sure the nginx containers has your vods mounted to /data/videos.
6. Bring the containers up docker compose up -d
7. Watch the api logs for possible issues docker logs ganymede-api -f

Issues / Questions

Have issues or questions about the upgrade process? Feel free to open a discussion or issue and I can help you out.