Recommended Settings for a GIT-LFS Server Proxy Host

[alex-ibb]

I used the following content as the Custom Nginx Configuration for the proxy host gitea.mysite.com on Nginx-Proxy-Manager. This host runs a Docker-based Gitea server with Git LFS enabled:

proxy_read_timeout 3600s;
proxy_connect_timeout 3600s;
proxy_send_timeout 3600s;
client_header_timeout 120m;
client_max_body_size 1024M;
client_body_buffer_size 512M;
fastcgi_intercept_errors on;

However, when I try to push a new repository with large files (ranging from 30MB to 92GB, averaging around 10GB, totaling 160GB) to the host, it eventually fails with errors like this:

LFS: Client error &{%!!(string=https) %!!(string=) %!!(*url.Userinfo=<nil>) %!!(string=gitea.mysite.com:10443) %!!(string=/your.name/repo.name.git/info/lfs/objects/0c0d735294888209ddc297a15bea40eb0ba9a34b446848fe3ec76601d1234567/2341234567) %!!(string=) %!!(bool=false) %!!(bool=false) %!!(string=) %!!(string=) %!!(string=)}s(MISSING) from HTTP 413
error: failed to push some refs to 'https://gitea.mysite.com:10443/your.name/repo.name.git'

Observations:

1. Initial Upload and Resource Usage:
   After running the git push command from the client, the Nginx-Proxy-Manager server (running in Docker) shows increased CPU and disk usage, suggesting it receives the data.
   The Gitea server also exhibits similar behavior initially. The Gitea GUI confirms that some segments of the repository are pushed. This indicates that initial content is transferred to the Gitea server via the Nginx-Proxy-Manager.
2. Eventual Failure:
   Despite the initial progress, the push operation eventually fails with the HTTP 413 error mentioned above. No new content appears on the Gitea server after the error occurs.
3. Nginx-Proxy-Manager Resource Spike:
   After removing all previously pushed LFS files from the Gitea server and retrying the git push,
   the Nginx-Proxy-Manager server again experiences noticeable CPU and disk usage spikes,
   while the Gitea server remains relatively unaffected.
   This suggests that the Nginx-Proxy-Manager might be receiving the large files but failing to forward them to the Gitea server.
4. Decreasing client_max_body_size to, for example, 512M leads to a considerable decrease in pushing speed.
5. Increasing client_max_body_size to, for example, 10240M, leads to higher pushing speed. However, it appears that the disk space on the NPM server becomes exhausted, preventing new content from being added to Gitea.
   • tail data/logs/proxy-host-7_error.log on the NPM server prints:

2024/09/25 01:18:01 [crit] 1349#1349: *16381 pwrite() "/tmp/nginx/body/7/22/0000000227" failed (28: No space left on device), client: 192.168.100.1, server: gitea.mysite.com, request: "PUT /your.name/repo.name.git/info/lfs/objects/eb308a3fd45d9360f4f78dcdffc8233e6675500e8df32710a026d3fa16ebc8a0/9364376858 HTTP/1.1", host: "gitea.mysite.com:10443"
2024/09/25 01:18:09 [crit] 1349#1349: *16385 pwrite() "/tmp/nginx/body/8/22/0000000228" failed (28: No space left on device), client: 192.168.100.1, server: gitea.mysite.com, request: "PUT /your.name/TechPortal-Qualcomm.git/info/lfs/objects/ee157261d7b7432ecaf8a3e78df79f634d8b6aa89e6ceba414e0beca11301234/3333717351 HTTP/1.1", host: "gitea.mysite.com:10443"
2024/09/25 01:19:25 [warn] 1353#1353: *16491 a client request body is buffered to a temporary file /tmp/nginx/body/3/23/0000000233, client: 192.168.100.1, server: gitea.mysite.com, request: "PUT /your.name/repo.name.git/info/lfs/objects/fec309bb45b4c285f57f8b2f659c63a006987edf1b0c0c19e33140b6e4201b10/10524559770 HTTP/1.1", host: "gitea.mysite.com:10443"

• docker logs mysite-gitea -n 10 on the Gitea server prints logs likes below:

2024/09/25 01:18:56 ...lfs/content_store.go:58:Put() [E] Whilst putting LFS OID[314e54970b454815af9f8ee8f527e9c087939d75785db0bd01fc6b38a1234567]: Failed to copy to tmpPath: 31/4e/314e54970b454815af9f8ee8f527e9c087939d75785db0bd01fc6b38a1234567 Error: unexpected EOF
2024/09/25 01:18:56 ...rvices/lfs/server.go:325:func1() [E] Error putting LFS MetaObject [314e54970b454815af9f8ee8f527e9c087939d75785db0bd01fc6b38a1234567] into content store. Error: unexpected EOF
2024/09/25 01:18:56 ...rvices/lfs/server.go:338:UploadHandler() [E] Error whilst uploadOrVerify LFS OID[314e54970b454815af9f8ee8f527e9c087939d75785db0bd01fc6b38a1234567]: unexpected EOF
2024/09/25 01:19:15 ...ices/context/repo.go:944:func88() [E] GetBranchCommit: io: read/write on closed pipe

Any advices?

[alex-ibb]

After applying the following Nginx configuration changes, the issues described previously appear to be resolved:

client_body_buffer_size  128k;
client_max_body_size     50G;
ignore_invalid_headers   off;
proxy_buffer_size        16k;
proxy_buffers           32 16k;
proxy_busy_buffers_size  64k;
proxy_request_buffering  off; 

Key Observation:

• Disabling proxy_request_buffering seems to have played a significant role in addressing the problems. By turning off request buffering, Nginx directly streams large LFS objects to the upstream Gitea server, preventing excessive disk I/O on the proxy.