Releases: Zibbp/ganymede
v3.0.1
Important
If you have not upgraded to v3.0.0 yet see the release notes for upgrade instructions!
What's Changed
- build(deps): bump golangci/golangci-lint-action from 6.0.1 to 6.1.0 by @dependabot in #479
- build(deps): bump golang.org/x/crypto from 0.25.0 to 0.26.0 by @dependabot in #481
- build(deps): bump github.com/docker/docker from 27.0.3+incompatible to 27.1.1+incompatible by @dependabot in #480
- fix arm64 builds by @Zibbp in #484
- arm64 images will now work with the
:latest
tag
- arm64 images will now work with the
Full Changelog: v3.0.0...v3.0.1
v3.0.0
Important
Version 3.0.0 includes numerous breaking changes that are not backwards compatible!
Features and Changes
- Replace Temporal with a simpler task system: River.
- Temporal provided a lot of nice features but ultimately is too difficult for a selfhosted application. Using a simpler task system with custom workflows allows for more customization.
- Bring back the ability to restart any task from the queue page (except for live video and chat downloads).
- Migrate all but one scheduled tasks to River. The live channel check is not run in River as that will just pollute the queue and some users have a very low interval configured.
- Optionally deploy River WebUI for a look into the tasks queue
- Archive tasks have a heartbeat. If the task times out (crashes, Ganymede restarts, etc) a watchdog will attempt to re-queue the task.
- Livestream video and chat downloads are not retried, instead they are cancelled and the subsequent workflows (post-process video and convert chat) started.
- Add support to customize all the directories inside the container using environment variables (#337).
- On start a migration function will check if the paths change. If a change is detected it will attempt to update all paths in the database.
- Category restrictions can now optionally be applied to live streams.
- On each channel check, the category of the live stream is compared against the list of categories configured in the watched channel settings. If a match is found then the stream is archived. It will not detect if the channel switches to a different category after the live stream archive is created. If this is something that interests you let me know!
- Add a "Blocked VODs" page in the admin panel.
- Video IDs can be blocked from being archived.
- Ability to cancel a VOD archive in-progress.
- Optionally can delete the video along with the video files.
- Optionally can block the video ID from future archives.
- Add a button to regenerate the static thumbnail for videos.
Notable Changes
Important changes that don't break functionality.
- The
ganymede-nginx
container isn't required anymore. The API container will serve the static files at theVIDEO_DIR
env var route. Example:VIDEO_DIR=/data/videos
then it would be available atlocalhost:4800/data/videos
. There is no index so you can't browse files. Updating the frontendCDN_URL
tolocalhost:4800/data/videos
is the same as serving the files via Nginx.- I recommend still using Nginx, it provides caching and sets headers so your browser caches important images like thumbnails. This is mainly for people who don't like running a lot of containers.
Breaking Changes
- Completely different worker system (requires changes to the compose file)
- Remove temporal containers
- Add a new river-ui container
- Moved the following settings from the
config.json
to environment variables (set these as environment variables on the api container)DEBUG
OAUTH_ENABLED
- Added environment variables to change the paths of important directory inside the container. Changing these will require changing the mounted volumes.
VIDEOS_DIR
TEMP_DIR
CONFIG_DIR
LOGS_DIR
The defaults of these are:
VideosDir string `env:"VIDEOS_DIR, default=/data/videos"`
TempDir string `env:"TEMP_DIR, default=/data/temp"`
ConfigDir string `env:"CONFIG_DIR, default=/data/config"`
LogsDir string `env:"LOGS_DIR, default=/data/logs"`
- **You will need to update the paths for your containers to match these new defaults, or override the default by specifying the path in the environment variables. Example:
volumes:
- - /mnt/nas/vods:/vods
+ - /mnt/nas/vods:/data/videos
- - ./logs:/logs
+ - ./logs:/data/logs
+ - ./temp:/data/temp
+ - ./config:/data/config # be sure to move `config.json` if needed
- I recommend mounting the
TEMP_DIR
as a volume so temporary files are not lost on container restart. See the below docker-compose diff for an example.
Updating Your Instance to v3.0.0
Important
BACK UP YOUR CURRENT INSTANCE, INCLUDING THE DATABASE!!!
The upgrade is non-reversible. Be sure to create a backup of the database if something goes wrong. Rely on your standard backup procedure or create a database dump by running the following command.
docker exec ganymede-db pg_dump -U ganymede ganymede-prd | gzip > /tmp/dump.sql.gz
Upgrade Steps
Wait until you have no active archives. There is no archive comparability between versions!
It may be simpler to start with a fresh and updated docker-compose.yml file. You will need to update the paths and any env vars from your old compose file.
- Bring down all containers
docker compose down
- Make modifications to the
docker-compose.yml
file. Optionally you can start with a fresh docker-compose.yml file. Be sure to update the volume mounts to account for the new*_DIR
variables.- Perform the changes to your
docker-compose.yml
outlined in the breaking changes section above. This includes... - Removing the temporal containers, and any references to it.
- Optionally update the paths in the
VIDEOS_DIR
andTEMP_DIR
environment variable. I recommend mounting theTEMP_DIR
to a volume on your host. This is to prevent losing data if the container crashes or restarts. Any modification to these variables requires changing the volume mounts as well. - Adding the river-ui container
- Perform the changes to your
version: "3.3"
services:
ganymede-api:
container_name: ganymede-api
image: ghcr.io/zibbp/ganymede:latest
restart: unless-stopped
depends_on:
- - ganymede-temporal
+ - ganymede-db
environment:
- TZ=America/Chicago # Set to your timezone
+ - DEBUG=false # set to true for debug logs
+ - VIDEOS_DIR=/data/videos
+ - TEMP_DIR=/data/temp
+ - LOGS_DIR=/data/logs
+ - CONFIG_DIR=/data/config
- DB_HOST=ganymede-db
- DB_PORT=5432
- DB_USER=ganymede
- DB_PASS=PASSWORD
- DB_NAME=ganymede-prd
- DB_SSL=disable
- JWT_SECRET=SECRET
- JWT_REFRESH_SECRET=SECRET
- TWITCH_CLIENT_ID=
- TWITCH_CLIENT_SECRET=
- FRONTEND_HOST=http://IP:PORT
- - COOKIE_DOMAIN=http://domain.com
# OPTIONAL
+ # - OAUTH_ENABLED=false
# - OAUTH_PROVIDER_URL=
# - OAUTH_CLIENT_ID=
# - OAUTH_CLIENT_SECRET=
# - OAUTH_REDIRECT_URL=http://IP:PORT/api/v1/auth/oauth/callback # Points to the API service
- - TEMPORAL_URL=ganymede-temporal:7233
# WORKER
- MAX_CHAT_DOWNLOAD_EXECUTIONS=2
- MAX_CHAT_RENDER_EXECUTIONS=2
- MAX_VIDEO_DOWNLOAD_EXECUTIONS=3
- MAX_VIDEO_CONVERT_EXECUTIONS=2
volumes:
- - /mnt/nas/vods:/vods
+ - /mnt/nas/vods:/data/videos
- - ./logs:/logs
+ - ./logs:/data/logs
+ - ./temp:/data/temp
+ - ./config:/data/config # be sure to move `config.json` if needed
ports:
- 4800:4000
ganymede-frontend:
container_name: ganymede-frontend
image: ghcr.io/zibbp/ganymede-frontend:latest
restart: unless-stopped
environment:
- API_URL=http://IP:PORT # Points to the API service
- CDN_URL=http://IP:PORT # Points to the CDN service
- SHOW_SSO_LOGIN_BUTTON=true # show/hide SSO login button on login page
- FORCE_SSO_AUTH=false # force SSO auth for all users (bypasses login page and redirects to SSO)
- REQUIRE_LOGIN=false # require login to view videos
ports:
- 4801:3000
- ganymede-temporal:
- image: temporalio/auto-setup:1.23
- container_name: ganymede-temporal
- depends_on:
- - ganymede-db
- environment:
- - DB=postgres12 # this tells temporal to use postgres (not the db name)
- - DB_PORT=5432
- - POSTGRES_USER=ganymede
- - POSTGRES_PWD=PASSWORD
- - POSTGRES_SEEDS=ganymede-db # name of the db service
- ports:
- - 7233:7233
- # -- Uncomment below to enable temporal web ui --
- # ganymede-temporal-ui:
- # image: temporalio/ui:latest
- # container_name: ganymede-temporal-ui
- # depends_on:
- # - ganymede-temporal
- # environment:
- # - TEMPORAL_ADDRESS=ganymede-temporal:7233
- # ports:
- # - 8233:8080
ganymede-db:
container_name: ganymede-db
image: postgres:14
volumes:
- ./ganymede-db:/var/lib/postgresql/data
environment:
- POSTGRES_PASSWORD=PASSWORD
- POSTGRES_USER=ganymede
- POSTGRES_DB=ganymede-prd
ports:
- 4803:5432
ganymede-nginx: # this isn't necessary if you do not want to serve static files via Nginx. The API container will serve the `VIDEO_DIR` environment variable.
container_name: ganymede-nginx
image: nginx
volumes:
- /path/to/ng...
v2.3.2
Tip
Now is the best time to switch back to the :latest
image tags if you are on :main
for testing.
v2.3.2
Important
Required Change!
The following changes must be made to your docker-compose.yml
file if you have not already done so. Do not make this change if you have archives running. Wait until nothing is being archived then perform the changes. See #441 for more information.
ganymede-temporal:
- image: temporalio/auto-setup:1
+ image: temporalio/auto-setup:1.23
container_name: ganymede-temporal
depends_on:
- ganymede-db
environment:
- - DB=postgresql # this tells temporal to use postgres (not the db name)
+ - DB=postgres12 # this tells temporal to use postgres (not the db name)
- DB_PORT=5432
- POSTGRES_USER=ganymede
- POSTGRES_PWD=PASSWORD
Features
Audio Only
Support for audio-only archives.
Bug Fixes
A bunch of bug fixes are included in this release, most important are #437 and #413.
What's Changed
- feat: audio only archive support by @Zibbp in #431
- build(deps): bump go.temporal.io/api from 1.32.0 to 1.34.0 by @dependabot in #432
- build(deps): bump golangci/golangci-lint-action from 4.0.0 to 6.0.1 by @dependabot in #428
- build(deps): bump github.com/go-playground/validator/v10 from 10.19.0 to 10.20.0 by @dependabot in #421
- feat(archive): chat conversion improvements by @Zibbp in #433
- fix(archive): delete postprocessed video file after converting to hls by @Zibbp in #438
- feat(activities): handle and kill multiple chat process ids by @Zibbp in #439
- fix(docker): pin temporal image by @Zibbp in #440
- fix: re-add tasks by @Zibbp in #442
Full Changelog: v2.3.1...v2.3.2
v2.3.1
Tip
Now is the best time to switch back to the :latest
image tags if you are on :main
for testing.
What's Changed
-
Update TwitchDownloader version to fix new VOD ids being greater than what int32 supports lay295/TwitchDownloader#1057
-
fix : #416 oidc by @boringwolf in #417
New Contributors
- @boringwolf made their first contribution in #417
Full Changelog: v2.3.0...v2.3.1
v2.3.0
Tip
Now is the best time to switch back to the :latest
image tags if you are on :main
for testing.
New Features
Muted Segments
Going forward, all video (vod) downloads will also save the muted segments. These are saved to the database in the table muted_segments
. They are also saved to the info.json
file. These are not displayed in the frontend yet, I'm still trying to find a good way to display the segments. For now you can use the API or query the database.
/api/v1/vod/<id>?with_channel=true&with_chapters=true&with_muted_segments=true
Title Regex
Watched channels now have a new option called "title regex". Custom regex can be applied to either live streams or video downloads. This allows you to customize what is downloaded by applying regex to the title. The obvious use case is to ignore reruns as the title usually contains it, but it's all customizable to fit your needs.
To get started, edit a "watched channel" and click the plus button under Advanced > Title Regex
A new entry will be visible allowing you to enter a regex and apply some optional settings.
negative
: Invert the match, meaning you do not want the regexapply to video downloads
: Apply the regex to video downloads. By default the regex is only applied to live streams. If this is checked then it will not be used for live streams. You need to make two regexes if you want to apply to both live stream and videos.
A title regex to not download any rerun livestreams would look like the following
For more information please see the wiki page https://github.com/Zibbp/ganymede/wiki/Watched-Channel-Title-Regex. If you have any useful regexes, please share in a discussion!
What's Changed
- feat: save muted segments by @Zibbp in #406
- feat: title regex filter by @Zibbp in #409
- build(deps): bump golang.org/x/net from 0.21.0 to 0.23.0 by @dependabot in #410
- build: package updates by @Zibbp in #411
- build(deps): bump docker/setup-buildx-action from 3.2.0 to 3.3.0 by @dependabot in #403
Full Changelog: v2.2.0...v2.3.0
v2.2.0
What's Changed
- The Twitch API is slow to update when a stream goes offline, this is especially noticeable with a low "live check interval". #385 adds some logic to detect if a stream gets archived when it's truly offline, but the API hasn't updated yet. This includes stopping the queue/workflows and deleting the video entry.
- Add a "max video age" to watched channels when archiving previous VODs. VODs are only archived if they are less than X number of days old. (#395).
- Re-implement the dark/light mode theme toggle (#397).
Commits
- build(deps): bump docker/login-action from 3.0.0 to 3.1.0 by @dependabot in #393
- build(deps): bump docker/build-push-action from 5.1.0 to 5.3.0 by @dependabot in #392
- build(deps): bump docker/setup-buildx-action from 3.0.0 to 3.2.0 by @dependabot in #391
- build(deps): bump github.com/go-jose/go-jose/v3 from 3.0.1 to 3.0.3 by @dependabot in #384
- build(deps): bump github.com/go-playground/validator/v10 from 10.16.0 to 10.19.0 by @dependabot in #381
- build(deps): bump github.com/google/uuid from 1.5.0 to 1.6.0 by @dependabot in #375
- build(deps): bump github.com/rs/zerolog from 1.31.0 to 1.32.0 by @dependabot in #374
- Delete archive if stream is ended but Twitch API is slow to update by @Zibbp in #385
Full Changelog: v2.1.0...v2.2.0
v2.1.0
Important
Temporal container changes
It's been reported that the sqlite version of the Temporal container that was previously shipped with Ganymede caused issues, see #339 and #353 for more information. To fix this, I'm now recommending everyone to switch to the more "production-like" temporal setup, which uses a database. To perform this change, follow the below.
No workflow data will be saved from the previous container, so ensure that nothing is actively being archived
- Bring down Ganymede
docker compose down
- Perform the following change to your
docker-compose.yml
file. The new Temporal container requires access to your database, make sure to update thePOSTGRES_USER
andPOSTGRES_PWD
with the username and password of yourganymede-db
container.
- ganymede-temporal:
- container_name: ganymede-temporal
- image: ghcr.io/zibbp/ganymede-temporal:latest
- restart: unless-stopped
- volumes:
- - ./temporal:/data
- ports:
- - 7233:7233
- - 8233:8233 # web ui - optional
+ ganymede-temporal:
+ image: temporalio/auto-setup:1
+ container_name: ganymede-temporal
+ depends_on:
+ - ganymede-db
+ environment:
+ - DB=postgresql # this tells temporal to use postgres (not the db name)
+ - DB_PORT=5432
+ - POSTGRES_USER=ganymede
+ - POSTGRES_PWD=PASSWORD
+ - POSTGRES_SEEDS=ganymede-db # name of the db service
+ ports:
+ - 7233:7233
- Optionally add the web-ui service if you wish to use that.
+ ganymede-temporal-ui:
+ image: temporalio/ui:latest
+ container_name: ganymede-temporal-ui
+ depends_on:
+ - ganymede-temporal
+ environment:
+ - TEMPORAL_ADDRESS=ganymede-temporal:7233
+ ports:
+ - 8233:8080
- Delete the
temporal
directory - Bring everything back up
docker compose up -d
- Ensure the temporal container is working fine
docker logs ganymede-temporal
and check that the API container is able to connectdocker logs ganymede-api
Refer to the docker-compose.yml file for a full example.
What's Changed
If you've been running the :main
image for either the API or frontend, now is a good time to switch back to the :latest
tag.
Frontend
- Bump package versions
- Place theater mode button next to full screen button
- Show thumbnail when video hasn't begun playing yet
- Add
sharp
and ensure/app
directory is writable - Fix bug the prevented the VOD title component from coming back after exiting theater mode
API
- Use debian as the x86 docker base image
- Re-run live thumbnail download 10 minutes after the archive starts
Lots of bug fixes and package version bumps:
- feat(workflow): update live stream archives with correct vod ids by @Zibbp in #351
- Fixes 354 by @Zibbp in #357
- Fixes 2024 02 04 by @Zibbp in #361
- fix(exec): check if livechatworkflowid is populated by @Zibbp in #364
- fix: add comments by @Zibbp in #365
- feat(workflow): re-download live thumbnail again after 10 minutes by @Zibbp in #372
- build(deps): bump golangci/golangci-lint-action from 3.7.0 to 4.0.0 by @dependabot in #368
- build(deps): bump golang.org/x/crypto from 0.17.0 to 0.19.0 by @dependabot in #367
- build(deps): bump docker/metadata-action from 5.0.0 to 5.5.1 by @dependabot in #363
- build(deps): bump github.com/mattn/go-sqlite3 from 1.14.17 to 1.14.22 by @dependabot in #362
- build(deps): bump github.com/prometheus/client_golang from 1.17.0 to 1.18.0 by @dependabot in #338
- build(deps): bump github.com/labstack/echo/v4 from 4.11.3 to 4.11.4 by @dependabot in #332
- build(deps): bump golang.org/x/oauth2 from 0.15.0 to 0.17.0 by @dependabot in #366
- build(deps): bump actions/setup-go from 4 to 5 by @dependabot in #322
- build(deps): bump github/codeql-action from 2 to 3 by @dependabot in #326
- build(docker): use debian as base image by @Zibbp in #373
Full Changelog: v2.0.2...v2.1.0
v2.0.2
What's Changed
- feat(workflows): paginate executions by @Zibbp in #341
- fix(channel): allow specifying an external id when creating a channel… by @Zibbp in #344
- feat(activity/livevideo): attempt to update livestream archive with the external vod id by @Zibbp in #347
- Fix live stream chat update if the channel does not have any previous videos
Frontend
What's Changed
- feat(workflows): paginate workflows by @Zibbp in Zibbp/ganymede-frontend#19
- feat(admin/channel): allow setting external id when creating channel by @Zibbp in Zibbp/ganymede-frontend#20
- fix(chat): fallback emote tooltip text to fragment text when empty + use mantine tooltips by @Entrivax in Zibbp/ganymede-frontend#23
- fix(channel): add more elements per row on bigger screens by @Entrivax in Zibbp/ganymede-frontend#21
- fix(vod): try improve responsiveness of the vod page by @Entrivax in Zibbp/ganymede-frontend#22
New Contributors
- @Entrivax made their first contribution in Zibbp/ganymede-frontend#23
Full Changelog: v2.0.1...v2.0.2
v2.0.1
If you have not read the 2.0.0 release notes, please start there! https://github.com/Zibbp/ganymede/releases/tag/v2.0.0
v2.0.1 Changes
- add
procps
package to arm build - Add retry logic to viper config parsing
- Do not run
refreshConfig
when worker starts - Intelligently detect if the database needs to be seeded
- Remove unused config options
What's Changed
Full Changelog: v2.0.0...v2.0.1
v2.0.0
2.0.0
Merry Christmas everyone, I hope you enjoy this update! 🎅🎄
Breaking Change!
Important
Version 2.0.0 has a change to the docker compose that must be applied. The API container will not start until this change is applied
A few environment variables need to be added to the API compose service. As well as adding a new Temporal service.
ganymede-api:
container_name: ganymede-api
image: ghcr.io/zibbp/ganymede:latest
restart: unless-stopped
environment:
- TZ=America/Chicago # Set to your timezone
- DB_HOST=ganymede-db
- DB_PORT=5432
- DB_USER=ganymede
- DB_PASS=PASSWORD
- DB_NAME=ganymede-prd
- DB_SSL=disable
- JWT_SECRET=SECRET
- JWT_REFRESH_SECRET=SECRET
- TWITCH_CLIENT_ID=
- TWITCH_CLIENT_SECRET=
- FRONTEND_HOST=http://IP:PORT
# OPTIONAL
# - OAUTH_PROVIDER_URL=
# - OAUTH_CLIENT_ID=
# - OAUTH_CLIENT_SECRET=
# - OAUTH_REDIRECT_URL=http://IP:PORT/api/v1/auth/oauth/callback # Points to the API service
+ - TEMPORAL_URL=ganymede-temporal:7233
+ # WORKER
+ - MAX_CHAT_DOWNLOAD_EXECUTIONS=5
+ - MAX_CHAT_RENDER_EXECUTIONS=3
+ - MAX_VIDEO_DOWNLOAD_EXECUTIONS=5
+ - MAX_VIDEO_CONVERT_EXECUTIONS=3
volumes:
- /path/to/vod/storage:/vods
- ./logs:/logs
- ./data:/data
# Uncomment below to persist temp files
#- ./tmp:/tmp
ports:
- 4800:4000
+ ganymede-temporal:
+ container_name: ganymede-temporal
+ image: ghcr.io/zibbp/ganymede-temporal:latest
+ restart: unless-stopped
+ volumes:
+ - ./temporal:/data
+ ports:
+ - 7233:7233
+ - 8233:8233 # web ui - optional
See the full docker-compose.yml for a full example (see commit history for new additions).
Features
New Workflow System
I'm now utilizing Temporal to create and run workflows. This replaces the queue system with something that will hopefully be more robust.
There is now a new "Workflows" page that shows active and closed workflows.
The "Queue" page is still available which provides an easy-to-see status of an archive. Workflows are still in an early stage so if you'd like to see more information about the workflows I suggest exposing the "web ui" port of the Temporal container (8233) and visiting that. Manually restarting a task is now accomplished on the workflow page for that specific execution.
Most workflows now have auto retries, currently configured up to 3 retries before the workflow errors out. You can now be more strict on the maximum number of workflow types as seen in the new environment variables for the API container.
The worker is currently bundled in the API server container and they both run at once. In the future I will probably add support to break this up and allow for distributed worker nodes.
Getting the existing code moved over to Temporal workflows has been a massive lift. It will pay off over time though as it will be significantly easier to setup new workflows in the future. This change will likely introduce some bugs that I wasn't able to catch. If you encounter any issues, please create an issue with as many logs and information as possible.
Chapters / Categories
Requested in #317, Game categories are now saved to the info.json
file as well as the database if the video has categories. These chapters / categories are visible in the video player.
A new workflow exists to update existing videos with available categories (if the video still exists and has categories).
A future update will allow creating, editing, and deleting chapters in the frontend.
Frontend Updates
I've given the frontend a new color scheme and updated some components, including the landing page.
Known Issues
- Live stream chat will not render if the channel does not have any videos.
- TwitchDownloader now requires a video ID to render chat. I'm exploring possible workarounds for this.
- Workflows on the "workflows" page disappear after a day.
- Temporal auto removes old workflows. I'm hoping to have a fix for this in the next release.
- Mobile player
- The vertical layout for watching a video is a bit messed up right now. Rotate your device so it's in landscape and press the theater mode icon. This will play the video next to the chat player.
What's Changed
- fix(archive): remove delete vod by @Zibbp in #309
- build(deps): bump github.com/rs/zerolog from 1.30.0 to 1.31.0 by @dependabot in #303
- build(deps): bump golang.org/x/net from 0.15.0 to 0.17.0 by @dependabot in #310
- build(deps): bump github.com/go-playground/validator/v10 from 10.15.4 to 10.16.0 by @dependabot in #312
- build(deps): bump docker/build-push-action from 5.0.0 to 5.1.0 by @dependabot in #315
- build(deps): bump github.com/go-jose/go-jose/v3 from 3.0.0 to 3.0.1 by @dependabot in #316
- feat: temporal queue by @Zibbp in #328
- docs: temporal docker by @Zibbp in #330
Full Changelog: v1.4.3...v2.0.0