Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sign in not possible behind tinyproxy with sub-path #31202

Closed
LauKr opened this issue May 31, 2024 · 20 comments · Fixed by #31306
Closed

Sign in not possible behind tinyproxy with sub-path #31202

LauKr opened this issue May 31, 2024 · 20 comments · Fixed by #31306
Assignees
Labels
issue/needs-feedback For bugs, we need more details. For features, the feature must be described in more detail issue/not-a-bug The reported issue is the intended behavior or the problem is not inside Gitea

Comments

@LauKr
Copy link

LauKr commented May 31, 2024

Description

Running Gitea behind a Reverse Proxy using a sub-path doesn't allow login.
This issue was first occurring with the update 1.21.10 -> 1.21.11 and
continues with 1.22.0.

After entering credentials and pressing the "Sign In" button, the
unauthenticated "root" site is shown again, like after a logout.

From the release notes
we noted the following change which seems suspicious:

Fix to delete the cookie when AppSubURL is non-empty (#30375) (#30468)

Therefore, we checked our configurations and ensured, they used trailing slashes:

$ grep -r "git-staging" /etc/gitea-staging /etc/tinyproxy/ 
/etc/gitea-staging/app.ini:ROOT_URL = https://192.168.1.12/git-staging/
/etc/tinyproxy/tinyproxy.conf:ReversePath "/git-staging/" "http://127.0.0.1:3333/"

The logs from our reverse proxy show a behavior as requested by the
documentation:

May 29 16:16:31 debian-git tinyproxy[390434]: Connect (file descriptor 5): 127.0.0.1
May 29 16:16:31 debian-git tinyproxy[390434]: Request (file descriptor 5): POST /git-staging/user/login HTTP/1.1
May 29 16:16:31 debian-git tinyproxy[390434]: Rewriting URL: /git-staging/user/login -> http://127.0.0.1:3333/user/login
May 29 16:16:31 debian-git tinyproxy[390434]: No upstream proxy for 127.0.0.1
May 29 16:16:31 debian-git tinyproxy[390434]: opensock: opening connection to 127.0.0.1:3333
May 29 16:16:31 debian-git tinyproxy[390434]: opensock: getaddrinfo returned for 127.0.0.1:3333
May 29 16:16:31 debian-git tinyproxy[390434]: Established connection to host "127.0.0.1" using file descriptor 6.
May 29 16:16:31 debian-git tinyproxy[390434]: Closed connection between local client (fd:5) and remote client (fd:6)
May 29 16:16:31 debian-git tinyproxy[390434]: Connect (file descriptor 5): 127.0.0.1
May 29 16:16:31 debian-git tinyproxy[390434]: Request (file descriptor 5): GET /git-staging/ HTTP/1.1
May 29 16:16:31 debian-git tinyproxy[390434]: Rewriting URL: /git-staging/ -> http://127.0.0.1:3333/
May 29 16:16:31 debian-git tinyproxy[390434]: No upstream proxy for 127.0.0.1
May 29 16:16:31 debian-git tinyproxy[390434]: opensock: opening connection to 127.0.0.1:3333
May 29 16:16:31 debian-git tinyproxy[390434]: opensock: getaddrinfo returned for 127.0.0.1:3333
May 29 16:16:31 debian-git tinyproxy[390434]: Established connection to host "127.0.0.1" using file descriptor 6.
May 29 16:16:31 debian-git tinyproxy[390434]: Closed connection between local client (fd:5) and remote client (fd:6)

We see the equivalent to

Make the reverse-proxy pass https://common.example.com/gitea/foo to http://gitea:3000/foo

Rewriting URL: /git-staging/user/login -> http://127.0.0.1:3333/user/login

The corresponding Gitea logs are:

May 29 16:53:07 debian-git gitea[390439]: 2024/05/29 16:53:07 ...eb/routing/logger.go:102:func1() [I] router: completed POST /user/login for 127.0.0.1:56322, 303 See Other in 68.6ms @ auth/auth.go:196(auth.SignInPost)
May 29 16:53:07 debian-git gitea[390439]: 2024/05/29 16:53:07 .../context_response.go:70:HTML() [D] Template: home
May 29 16:53:07 debian-git gitea[390439]: 2024/05/29 16:53:07 ...eb/routing/logger.go:102:func1() [I] router: completed GET / for 127.0.0.1:56324, 200 OK in 3.6ms @ web/home.go:32(web.Home)

Access logs in Gitea:

127.0.0.1 - - [29/May/2024:16:52:57 +0200] "GET /user/login HTTP/1.1" 200 10629 "" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:126.0) Gecko/20100101 Firefox/126.0"
127.0.0.1 - - [29/May/2024:16:53:00 +0200] "POST /user/login HTTP/1.1" 303 0 "" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:126.0) Gecko/20100101 Firefox/126.0"
127.0.0.1 - - [29/May/2024:16:53:00 +0200] "GET / HTTP/1.1" 200 14214 "" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:126.0) Gecko/20100101 Firefox/126.0"
127.0.0.1 - - [29/May/2024:16:53:02 +0200] "GET /user/login?redirect_to=%2fgit-staging%2f HTTP/1.1" 200 10629 "" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:126.0) Gecko/20100101 Firefox/126.0"
127.0.0.1 - - [29/May/2024:16:53:07 +0200] "POST /user/login HTTP/1.1" 303 0 "" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:126.0) Gecko/20100101 Firefox/126.0"
127.0.0.1 - - [29/May/2024:16:53:07 +0200] "GET / HTTP/1.1" 200 14214 "" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:126.0) Gecko/20100101 Firefox/126.0"

Resolving

After reverting commit b18c04e (and commenting
out two sections undefined afterwards, see patch), we are able to use the normal
login again, when using our custom built binary. Thus, this also shows that our
error is connected to this change.

Gitea Version

1.21.11 and 1.22.0

Can you reproduce the bug on the Gitea demo site?

No

Log Gist

No response

Screenshots

No response

Git Version

2.39.2

Operating System

Debian Bookworm

How are you running Gitea?

The binary is pulled from the official server. Execution is happening through systemd on Debian 6.1.85-1 (2024-04-11) 86_64.
The Gitea server is listening locally via HTTP and is served by a tinyproxy reverse proxy. TLS is terminated by stunnel in front of tinyproxy.

Database

SQLite

@LauKr LauKr added the type/bug label May 31, 2024
@wxiaoguang
Copy link
Contributor

After reverting commit b18c04e, we are able to use the normal login again,
" fix: Fix to delete cookie when AppSubURL is non-empty #30375 "

@jtran would you like to take a look?

@wxiaoguang
Copy link
Contributor

wxiaoguang commented May 31, 2024

@LauKr could you help to elaborate some details?

  1. Does the "login" work in a private window (without any existing cookie)?
  2. If the login doesn't work, what are the cookies sent from server (Set-Cookie) and sent from client (Cookie)? Do they match?

@LauKr
Copy link
Author

LauKr commented May 31, 2024

Sadly GitHub doesn't let me upload a file with .path extension. I changed it to .txt instead.
diff.txt

This diff shows the revert plus two comments that allowed us to login again.

@LauKr
Copy link
Author

LauKr commented May 31, 2024

  1. No, deleting cookies, using a private window etc. did not work. We already tried this first, due to [E] UserSignIn: not in 2FA session #29757 and AAD Login via OAuth fails with Dockerhub image "gitea/gitea"  #29881.
  2. The POST request for login has the following cookies set:
    Set-Cookie: yummy_magical_cookie=/git-staging/; path=/
    Cookie: i_like_gitea=XXX; _csrf=XXX; yummy_magical_cookie=/git-staging/
    

@bohde
Copy link
Contributor

bohde commented May 31, 2024

I worked with @jtran on this change for bug we were seeing in a deployment that is also behind a reverse proxy (nginx in our case) with a subpath of /app_path/. The set-cookie headers as a result of POST /app_path/user/login look like the following in my Firefox network tab:

set-cookie i_like_gitea=; Path=/app_path/; Max-Age=0; HttpOnly; Secure; SameSite=Lax
set-cookie i_like_gitea=<redacted>; Path=/app_path; HttpOnly; Secure; SameSite=Lax
set-cookie lang=; Path=/app_path/; Max-Age=0; HttpOnly; Secure; SameSite=Lax
set-cookie_csrf=; Path=/app_path; Max-Age=0; HttpOnly; Secure; SameSite=Lax
set-cookie redirect_to=; Path=/app_path; Max-Age=0; HttpOnly; Secure; SameSite=Lax
set-cookie redirect_to=; Path=/app_path/; Max-Age=0; HttpOnly; Secure; SameSite=Lax

@LauKr based on my reading of your config, it appears like your cookies should set Path=/git-staging, but it appears that's not happening from the Set-Cookie header your posted.

@wxiaoguang
Copy link
Contributor

wxiaoguang commented Jun 1, 2024

@LauKr it doesn't seem right in your reverse proxy config.

Did you see any Set-Cookie for i_like_gitea? Which path is it on? It is directly related to Gitea login.

And, did you set any ReversePath or related config for your reverse proxy? I think they should be removed. UPDATE: maybe it should set ReverseMagic No, see below.


Since you are using tinyproxy, could you prepare a reproducible setup with detailed steps ? If you could provide a docker compose to reproduce, then I could try to debug it on my machine.

@wxiaoguang
Copy link
Contributor

wxiaoguang commented Jun 1, 2024

And one more thing, It seems that tinyproxy does some trick (black magic?) on the "cookie path". https://github.com/search?q=repo%3Atinyproxy%2Ftinyproxy%20REVERSE_COOKIE&type=code

I do not feel it is quite right. If there is a chance, I would recommend to try to use some modern and standard reverse proxies.

ps: I guess setting ReverseMagic No should resolve the problem, according to the code in tinyproxy:

image

@wxiaoguang wxiaoguang added issue/needs-feedback For bugs, we need more details. For features, the feature must be described in more detail and removed type/bug labels Jun 1, 2024
@wxiaoguang
Copy link
Contributor

Any update?

@LauKr
Copy link
Author

LauKr commented Jun 4, 2024

Any update?

Sorry, I need some time for the reproducible build.

Changing ReverseMagic to No sadly didn't resolve the issue. I'll try debugging where the cookie went as soon as I find the time.

@LauKr
Copy link
Author

LauKr commented Jun 9, 2024

Hi,
sorry for the delay. I wrote a Dockerfile to reproduce the issue. Necessary files are attached:
setup_for_debugging.tar.gz

Building and running was tested using Podman, but should work with Docker as well:

podman build -t gitea_proxy_testing --file Dockerfile
podman run --rm -it --publish 8888:8888 gitea_proxy_testing:latest

The Dockerfile creates a container with Gitea and tinyproxy. The tinyproxy is accessible via port 8888.
Accessing Gitea via 127.0.0.1:8888/git-staging/ was successful, but login was not possible, as described in the issue.

@wxiaoguang
Copy link
Contributor

I guess the problem is caused by that the new cookie is correctly removed by the "deleting legacy code". So I think we could try to flip the order: first delete the legacy cookie, then set new cookie, then the new cookie won't be deleted.

-> Delete legacy cookie before setting new cookie #31306

@wxiaoguang
Copy link
Contributor

Wait for backport to 1.22

@LauKr
Copy link
Author

LauKr commented Jun 11, 2024

When trying use a custom build on commit 5342a61, the problem was not fixed in the provided container setup.
I'll have a look at the proxy again, just in case.

@lunny
Copy link
Member

lunny commented Jun 11, 2024

Backport merged, wait for @LauKr 's confirmation and feedback.

@wxiaoguang
Copy link
Contributor

Backport merged, wait for @LauKr 's confirmation and feedback.

It's said that the "fix" doesn't work.

@wxiaoguang
Copy link
Contributor

wxiaoguang commented Jun 11, 2024

I think it is a bug in tinyproxy. By using tcpdump, I can see:

Gitea tells tinyproxy:

HTTP/1.1 303 See Other
Cache-Control: max-age=0, private, must-revalidate, no-transform
Content-Type: text/plain; charset=utf-8
Location: /git-staging/
Set-Cookie: _csrf=en3AFom-bfjvNKBwckDCoPm17uQ6MTcxODA5OTk4NzE4NzcxMzAwMA; Path=/git-staging; Max-Age=86400; HttpOnly; SameSite=Lax
Set-Cookie: i_like_gitea=; Path=/git-staging/; Max-Age=0; HttpOnly; SameSite=Lax
Set-Cookie: i_like_gitea=fb422c79cf11bd1f; Path=/git-staging; HttpOnly; SameSite=Lax
Set-Cookie: lang=; Path=/git-staging/; Max-Age=0; HttpOnly; SameSite=Lax
Set-Cookie: lang=en-US; Path=/git-staging; HttpOnly; SameSite=Lax
Set-Cookie: _csrf=; Path=/git-staging; Max-Age=0; HttpOnly; SameSite=Lax
Vary: Accept-Encoding
X-Frame-Options: SAMEORIGIN
X-Gitea-Debug: RUN_MODE=dev
Date: Tue, 11 Jun 2024 09:59:47 GMT
Content-Length: 0
Connection: close

Tinyproxy tells browsers

HTTP/1.1 303 See Other
Via: 1.1 tinyproxy (tinyproxy/1.11.2)
Cache-Control: max-age=0, private, must-revalidate, no-transform
Content-Type: text/plain; charset=utf-8
Location: /git-staging/
Set-Cookie: _csrf=en3AFom-bfjvNKBwckDCoPm17uQ6MTcxODA5OTk4NzE4NzcxMzAwMA; Path=/git-staging; Max-Age=86400; HttpOnly; SameSite=Lax

!!!!!!!! HERE, all other cookies are lost !!!!!!!!!!

Vary: Accept-Encoding
X-Frame-Options: SAMEORIGIN
X-Gitea-Debug: RUN_MODE=dev
Date: Tue, 11 Jun 2024 09:59:47 GMT
Content-Length: 0

Tinyproxy loses most cookies for browsers.

@wxiaoguang
Copy link
Contributor

I would suggest to use a professional reverse proxy like nginx, which is also small enough and I guess it won't cost more resources than tinyproxy.

@wxiaoguang wxiaoguang added the issue/not-a-bug The reported issue is the intended behavior or the problem is not inside Gitea label Jun 11, 2024
@wxiaoguang wxiaoguang changed the title Sign in not possible behind reverse proxy with sub-path Sign in not possible behind tinyproxy with sub-path Jun 11, 2024
@LauKr
Copy link
Author

LauKr commented Jun 11, 2024

Thanks for the effort.
We will try to debug tinyproxy and create a separate issue there. Hopefully observing tinyproxy will show a clear reason why it cannot handle the changes mode by the mentioned commit.
Testing with NGINX looks promising, however we will have to consider the feature set etc. first, before switching.

@LauKr
Copy link
Author

LauKr commented Jun 11, 2024

For reference, if anyone happens to have the same issue:
tinyproxy/tinyproxy#403 describes a relevant issue:

All but the first of Set-Cookie headers in an HTTP response will be silently dropped by tinyproxy

The corresponding PR is still open: tinyproxy/tinyproxy#435

However, we were able to confirm the fix by building a custom version of tinyproxy. As we use Debian, the Debian repository with corresponding release is used (https://salsa.debian.org/debian/tinyproxy/-/tags/debian%2F1.11.1-2) and the commits are applied on top.

Using this custom version of tinyproxy a login into Gitea is possible.

@wxiaoguang from my side this issue can be closed, as this is evidently no Gitea issue.

@wxiaoguang
Copy link
Contributor

Thank you very much for the details!

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
issue/needs-feedback For bugs, we need more details. For features, the feature must be described in more detail issue/not-a-bug The reported issue is the intended behavior or the problem is not inside Gitea
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants