v1.2.3 - Maintenance

What's Changed

• Miscellaneous by @Zibbp in #99
  • Added golangci-lint.
• build(deps): bump docker/build-push-action from 3.3.0 to 4.0.0 by @dependabot in #100
• build(deps): bump github.com/rs/zerolog from 1.28.0 to 1.29.0 by @dependabot in #98
• build(deps): bump github.com/go-playground/validator/v10 from 10.11.1 to 10.11.2 by @dependabot in #97
• build(deps): bump docker/setup-buildx-action from 2.2.1 to 2.4.0 by @dependabot in #96
• More tests by @Zibbp in #102
  • Added swagger API docs.
• fix(live): use cursor var

Fixes

• fix(live): use cursor var
  • Archiving a channel's videos that has more than 100 videos now properly works.

API Docs

Swagger API docs have been implemented. I will try to keep these up to date. To view the docs visit the API container http://IP:PORT/swagger/index.html.

Maintenance

This release is mainly focused on cleaning up and improving the CI process. Linting has been added locally and as a CI job. Basic unit tests have also been added. These tests do not cover any Twitch integrations yet.

Full Changelog: v1.2.2...v1.2.3

v1.2.2 - ARM Build Fix

What's Changed

• ops(action): overhaul build action by @Zibbp in #95
  • Images now use a single manifest with different architecture types. Visit the PR for more information.
• build(deps): bump entgo.io/ent from 0.11.6 to 0.11.7 by @dependabot in #94

Full Changelog: v1.2.1...v1.2.2

v1.2.1 - ARM Builds

What's Changed

• Tests by @Zibbp in #88
• build(deps): bump entgo.io/ent from 0.11.5 to 0.11.6 by @dependabot in #89
• build(deps): bump github.com/spf13/viper from 1.14.0 to 1.15.0 by @dependabot in #90
• Create Dockerfile.aarch64 by @Smallinger in #87
• fix: use "render_chat" variable when archiving videos from a watched channel 2c32b7b.

ARM Builds

With every new release an ARM64 image is built with a tag of :latest-arm64. This includes both the API and frontend images. Thanks to @Smallinger for PR.

New Contributors

• @Smallinger made their first contribution in #87

Full Changelog: v1.2.0...v1.2.1

v1.2.0 - Watched Channel Sub Only Video

What's Changed

• Watched channels subscriber only videos by @Zibbp in #83
  • Fixes #82.
• Add another fallback font cffa1ca.

Features

Pull Request #83 adds a field when adding a watched channel enabling the option of downloading "subscriber-only" videos. Previously this was not checked for when archiving a channel's videos. This resulted in those videos failing in the queue. If the option is disabled (default) any video that has a "sub-only" restriction provided by Twitch's GraphQL API will be skipped.

Full Changelog: v1.1.9...v1.2.0

v1.1.9 - Improvements

Changes

• VOD pagination route now accepts video types 206bcbb.
• Add optional caption_path database column e319030.
• Add back some of the fallback fonts d678d14.

Features

Video Type Filtering

When viewing a channel's videos a new multi-select dropdown is available which allows filtering by video types (archive, live, highlight, upload, or clip).

Video Player

This release moves the video player to VideoJS. There was some issues with the recently implemented Ovenplayer. VideoJS is also a more mature and has a larger community.

Captions Path

An optional caption_path field has been added to video database entries. Twitch has no captions for live streams but I was tinkering with Whisper to create vtt caption files for different video and audio files. I supplied Whisper with an archive video and it did a great job at transcribing the audio to a vtt caption file. The file was placed in the video directory and I populated the caption_path in Admin > Settings. If the caption_path is populated the video player will load the captions.

I will not be adding functionality to generate captions after an archive completed. This something you will need to script, possibly using the success webhooks to trigger a job.

v1.1.8 - Fixes & Video Download Token

Changes

• Switch back to the main chat-downloader repository 16afe24.
• Create Twitch token config entry b33a17e.
• Use Twitch token for VOD/video downloads, this allows sub-only downloads c800041.
• Fix: Check if pos2+1 is out of bounds for emotes 2ff1efb. Fixes #79

v1.1.7 Storage Templates & Chat Render Option

What's Changed

• Storage Templates by @Zibbp in #76
• "Render Chat" Options by @Zibbp in #78

Frontend

• Storage template settings Zibbp/ganymede-frontend@381c42b.
• Storage template task Zibbp/ganymede-frontend@182cf7b.
• Render chat options Zibbp/ganymede-frontend@2383ede.
• Query parameters are now removed if a link is pasted in the archive page Zibbp/ganymede-frontend@aa91caf.

Storage Templates

Storage templates allows users to customize how folders and files are named. Previously folders were created using the naming scheme of {id}-{uuid}. Now this can be customized. To get started customizing the storage templates navigate to Admin > Settings and click the "Storage Template Settings" button. Here you are able to customize templates for folders and files. Any template changes apply only for new files. To apply your current storage templates to existing files, execute the "storage migration" task at Admin > Tasks. More information about this can be found on the storage template wiki page. One requirement of the folder template is {{uuid}} must be included somewhere in the template. This is to ensure unique folders for each archive so file do not get accidently overwritten.

Render Chat Option

When manually archiving a video or using the "watched channels" feature, an option has been added to disable chat rendering. This feature allows users to skip the chat rendering task while still downloading the chat json file. Chat rendering is enabled by default and you will need to disable it for every archive if you wish to not have the chat rendered.

Full Changelog: v1.1.6...v1.1.7

v1.1.6 - Performance Improvements and Optimizations

Changes

API

• Chat playback now uses embedded badges if available in the chat file 053ae5c.
  • If both emotes and badges are embedded in a chat file, no requests to Twitch or any other external services is called when watching a video.
  • Use channel's specific badges instead of global versions in 53edd7b.
• Improve performance of real time chat playback routes ebb4b4b.
  • Before this release, playback of large chat files would be slow and consume a lot of memory.
  • Improvements:
    • Revert back to Go's encoding/json package for better memory management.
    • Chat comments are sorted and stored in the cache using an interface on the initial request.
      • Before the comments were stored as a byte which needed to be unmarshalled each time which was terrible for performance.
    • A binary search is now used to find the index of a comment that is closest to the request offset time.
      • Before the comments were looped through to find the requested time. This was not efficient for large chat files.
    • The chat comment cache has a 10 minute expiration time that is refreshed each time a route that uses it is called.
    • The badge and emote routes have been improved to handle memory better.

Frontend

• Add "loading chat" text when the real time chat player is loading Zibbp/ganymede-frontend@aff4dd0.
• Add placeholder images for video thumbnails and channel profile images Zibbp/ganymede-frontend@666f657.
  • A blank image is used as a placeholder while the thumbnail is loading.
  • This results in a more pleasant experience when loading images as content is not jumping all over the page.
• Styling fixes on the admin "watched channels" page Zibbp/ganymede-frontend@9d30103.

Information about Performance Improvements

How the real time chat player works

When navigating to a video to playback, the first two requests the real time chat player makes is for the emotes and badges. Each one of these requests opens the chat file and checks if there are embedded emotes and badges. If embedded images are found, those are sent back to the frontend chat player. If no images are found, the emotes or badges are fetched from various providers. This process, especially extracting the embedded images results in a temporary increase of ram usage as it has to load a large chat file (250mb) into memory and extract/process the embedded images which is a majority of the chat file size. Once the image are sent back to the client Go's garbage collector will free up the memory when it sees fit.

After the emotes and badges are loaded into the player, the first comments are fetched. Comments are fetched in interval of 10 seconds using this API route /api/v1/vod/id/chat?start=0&end=10. In the backend, this route will check if the comments are stored in the cache. If they are not then the chat file is opened, comments extracted, formatted, and placed into the cache for subsequent requests. Once placed or read from the cache, a binary search is used to find a comment's index which is closest to the requested start location. The index is then used to iterate up until the end location is satisfied.

Performance numbers

This test occurred locally using a 251mb chat file which has 3,379 embedded images.

Endpoint used: /chat/seek?start=12352&count=30.

Old Performance:

Initial Request: 19.5 seconds
Cached Requests: ~8 seconds

New Performance:

Initial Request: 17.4 seconds
Cached Requests: 50-60 milliseconds.

With a ~3x reduction of memory usage.

Summary

The initial chat request when starting a real time chat playback will still take some time as it has to open, unmarshall, and sort the 10's of thousands of comments. Once that is complete the comments are stored in a 10 minute cache. Any subsequent request that uses that chat file will pull the comments from the chat resulting in <100 millisecond response times of the requested chat comments.

Notice any issues or bugs with this release? Please open an issue.

Full Changelog: v1.1.5...v1.1.6

v1.1.5 - New Player & HLS Support

What's Changed

• New Video Player Zibbp/ganymede-frontend@075fe19.
  • Now using OvenPlayer.
  • Better support for HLS playlists.
  • The chat player still uses Plyr for now.
• Add an option to convert the archived video from mp4 to a HLS playlist 0e8803e.
  • Disabled by default. This setting can be enabled at Admin > Settings.
  • The conversion takes place during the "Video Convert" queue job.
  • The mp4 is deleted and not moved to the final destination as to avoid having two identical large video files in two formats.
  • This option is useful if you publicly host Ganymede. It will significantly decrease the initial load time of the video.
    • A basic test of restricting the browser to 50mb/s loading a 20gb video.
      • MP4 - 10 seconds until playable.
      • HLS - 1.8 seconds until playable.
  • Not of much significance if you only use Ganymede on your LAN as the load times between an HLS playlist and large MP4 is under a second.
  • M3U8 file and segments are stored in a sub folder with a naming scheme of externalId-video_hls.
• Bump frontend packages Zibbp/ganymede-frontend@631dd19.

v1.1.4 - API Static Files

Changes

• feat(http): allow the api to serve static files in 3f0b230.
  • This change allows the API container to serve static files similar to the Nginx container. The Nginx container will still be preferred, this is for users who wish to not use the Nginx container or have issues with it. To use in place of the Nginx container, set the frontend environment variable CDN_URL to be equal the API host with /static at the end like so: http://192.168.1.150:4800/static.