[Bug]: Issue with Container Failing to Start After Upgrade

[AemonCao]

What happened?

After updating to versions 2.12.0, 2.12.1, 2.12.2, and 2.12.3, I encountered an issue where the container fails to start. Below is the content of my crash_logs.txt file. Please let me know if you need any additional information.

What did you expect to happen?

The container can start normally.

Steps to reproduce the issue

After encountering the issue described in #3251, I upgraded to version 2.12.2, and the container failed to start properly. I then downgraded to version 2.11.0, which is currently working fine.

Audiobookshelf version

v2.12.x

How are you running audiobookshelf?

Docker

What OS is your Audiobookshelf server hosted from?

Linux

If the issue is being seen in the UI, what browsers are you seeing the problem on?

None

Logs

{"timestamp":"2024-08-10 22:01:02.973","source":"Server.js:170","message":"[Server] Unhandled rejection: AxiosError: Request failed with status code 403, promise: Promise {\n  <rejected> [AxiosError: Request failed with status code 403] {\n    code: 'ERR_BAD_REQUEST',\n    config: {\n      transitional: [Object],\n      adapter: [Function: httpAdapter],\n      transformRequest: [Array],\n      transformResponse: [Array],\n      timeout: 0,\n      xsrfCookieName: 'XSRF-TOKEN',\n      xsrfHeaderName: 'X-XSRF-TOKEN',\n      maxContentLength: -1,\n      maxBodyLength: -1,\n      env: [Object],\n      validateStatus: [Function: validateStatus],\n      headers: [Object],\n      method: 'get',\n      url: 'https://api.github.com/repos/ffbinaries/ffbinaries-prebuilt/releases/tags/v5.1',\n      data: undefined\n    },\n    request: ClientRequest {\n      _events: [Object: null prototype],\n      _eventsCount: 7,\n      _maxListeners: undefined,\n      outputData: [],\n      outputSize: 0,\n      writable: true,\n      destroyed: true,\n      _last: true,\n      chunkedEncoding: false,\n      shouldKeepAlive: true,\n      maxRequestsOnConnectionReached: false,\n      _defaultKeepAlive: true,\n      useChunkedEncodingByDefault: false,\n      sendDate: false,\n      _removedConnection: false,\n      _removedContLen: false,\n      _removedTE: false,\n      strictContentLength: false,\n      _contentLength: 0,\n      _hasBody: true,\n      _trailer: '',\n      finished: true,\n      _headerSent: true,\n      _closed: true,\n      socket: [TLSSocket],\n      _header: 'GET /repos/ffbinaries/ffbinaries-prebuilt/releases/tags/v5.1 HTTP/1.1\\r\\n' +\n        'Accept: application/vnd.github.v3+json\\r\\n' +\n        'User-Agent: axios\\r\\n' +\n        'Host: api.github.com\\r\\n' +\n        'Connection: keep-alive\\r\\n' +\n        '\\r\\n',\n      _keepAliveTimeout: 0,\n      _onPendingData: [Function: nop],\n      agent: [Agent],\n      socketPath: undefined,\n      method: 'GET',\n      maxHeaderSize: undefined,\n      insecureHTTPParser: undefined,\n      joinDuplicateHeaders: undefined,\n      path: '/repos/ffbinaries/ffbinaries-prebuilt/releases/tags/v5.1',\n      _ended: true,\n      res: [IncomingMessage],\n      aborted: false,\n      timeoutCb: null,\n      upgradeOrConnect: false,\n      parser: null,\n      maxHeadersCount: null,\n      reusedSocket: false,\n      host: 'api.github.com',\n      protocol: 'https:',\n      _redirectable: [Writable],\n      [Symbol(shapeMode)]: false,\n      [Symbol(kCapture)]: false,\n      [Symbol(kBytesWritten)]: 0,\n      [Symbol(kNeedDrain)]: false,\n      [Symbol(corked)]: 0,\n      [Symbol(kOutHeaders)]: [Object: null prototype],\n      [Symbol(errored)]: null,\n      [Symbol(kHighWaterMark)]: 16384,\n      [Symbol(kRejectNonStandardBodyWrites)]: false,\n      [Symbol(kUniqueHeaders)]: null\n    },\n    response: {\n      status: 403,\n      statusText: 'rate limit exceeded',\n      headers: [Object],\n      config: [Object],\n      request: [ClientRequest],\n      data: [Object]\n    }\n  }\n}","levelName":"FATAL","level":5}

Additional Notes

No response

[devnoname120]

You are rate-limited. Unfortunately this is something that happens for unauthenticated API calls to GitHub…

[devnoname120]

It's a regression caused by: 329e9c9#diff-4699c7a67ec78e60e1158c6809e244b0c927674d0d7fb340e582eae313069aacL9

It changes the https://ffbinaries.com/api/v1 API to GitHub's unauthenticated API, which is rate-limited.

[seszele64]

Had the same issue,
downgrading to 2.11 solved the problem
logs.txt

[MaximUltimatum]

My own curiosity question, why is Audiobookshelf downloading the ffmpeg binary instead of including it in the container image?

[CraigMThomas]

I've hit this as well but I'm running on VM. The reason it gets rate limited seems to be because of a loop...

Aug 12 12:33:09 [2024-08-12 12:33:09.413] ERROR: [Binary] Failed to check version of ffmpeg (BinaryManager.js:248)
Aug 12 12:33:09 [2024-08-12 12:33:09.417] INFO: [BinaryManager] ffmpeg not found or version too old
Aug 12 12:33:09 [2024-08-12 12:33:09.420] ERROR: [Binary] Failed to check version of ffprobe (BinaryManager.js:248)
Aug 12 12:33:09 [2024-08-12 12:33:09.423] INFO: [BinaryManager] ffprobe not found or version too old
Aug 12 12:33:09 [2024-08-12 12:33:09.424] INFO: [BinaryManager] Installing binaries: ffmpeg, ffprobe

If I monitor /usr/share/audiobookshelf/ I can it downloading temp.zip and extracting ffmpeg and ffprobe, then it appears to fail the version check deletes the binaries and downloads again. Over and over until you get rate limited and hit the 403. Then you finally get a failed status on the AudioBookshef service.

I have also tried downloading the binaries manually and specifying the path in the config.

[wangzhili888]

downgrading to 2.11 solved the problem

[CraigMThomas]

Yeah, thanks wangzhili888. I've already done that too. Just been trying different things to try and narrow it down in case any of the info was helpful.

[CraigMThomas]

This is the code throwing the error:

async isGood(binaryPath) {
  if (!binaryPath || !(await fs.pathExists(binaryPath))) return false
  if (!this.validVersions.length) return true
  if (this.type === 'library') return true
  try {
    const { stdout } = await this.exec('"' + binaryPath + '"' + ' -version')
    const version = stdout.match(/version\s([\d\.]+)/)?.[1]
    if (!version) return false
    return this.validVersions.some((validVersion) => version.startsWith(validVersion))
  } catch (err) {
    Logger.error(`[Binary] Failed to check version of ${binaryPath}`)
    return false
  }
}

Which is being called by new Binary('ffmpeg', 'executable', 'FFMPEG_PATH', ['5.1'], ffbinaries)

For the life of me I cannot see why the version check is failing.
The response from ffmpeg -version is "ffmpeg version 5.1.1-static https://johnvansickle.com/ffmpeg/ Copyright (c) 2000-2022 the FFmpeg developers" which should match perfectly.

Only thing I can think of is failing execute access on within the scope of this function. Perhaps passing through the exception would be helpful rather than only returning the custom error message.

--- Edit ---
It was fapolicy (facepalm)
so solution for me was this:
sudo systemctl stop audiobookshelf
sudo chown audiobookshelf:audiobookshelf /usr/share/audiobookshelf/ -R
sudo vi /etc/fapolicyd/rules.d/80-myapps.rules
Add the following lines and save:
allow perm=execute exe=/usr/bin/bash trust=1 : path=/usr/share/audiobookshelf/ffmpeg ftype=application/x-executable trust=0
allow perm=execute exe=/usr/bin/bash trust=1 : path=/usr/share/audiobookshelf/ffprobe ftype=application/x-executable trust=0
sudo systemctl restart fapolicyd
sudo systemctl restart audiobookshelf

Done working on latest 2.12.3

[advplyr]

This is updated for the next release already so that docker skips the binary manager step. The binaries are included in the image.

You can use edge docker image to use it before the next release

[MaximUltimatum]

Thank you @advplyr ! (And other maintainers)

[luevano]

My own curiosity question, why is Audiobookshelf downloading the ffmpeg binary instead of including it in the container image?

Same thoughts... I do have ffmpeg installed and somehow it sill misses it (but actually tries to unlink it from /usr/bin/ffmpeg ????)
EDIT: works fine on 2.11.0... didn't care to check other vers

[github-actions]

Fixed in v2.13.0.