Use a more general (and faster) method to sanitize URLs with credentials #19239

wxiaoguang · 2022-03-28T05:58:49Z

As discussed in

Hide sensitive content on admin panel progress monitor #19218

We can use a more general (and faster) method to sanitize URLs with credentials

Benefits:

Simple and intuitive
- before: util.NewStringURLSanitizer(opts.Remote, true).Replace(opts.Remote)
- now: util.SanitizeCredentialURLs(opts.Remote)
For errors:
- before: get a remoteAddr; util.NewURLSanitizedError(err, remoteAddr, true)
- now: util.SanitizeErrorCredentialURLs(err)
Faster, fast return when no credential, one time allocation for a bytes buffer for most cases.
The new implemention can remove all credentials in all URLs, while the old one could only remove the only one targeted URL.

lunny · 2022-03-28T06:54:11Z

modules/util/sanitize_test.go

 			"https://" + userPlaceholder + "@github.com/go-gitea/test_repo.git",
 		},
-		// case 3


We still need this normal cases.

Which case? I think I have covered most.

I mean why we delete these original test cases?

All code are rewritten, all new cases cover old ones. If you feel which is missing, please just point out and add it.

And many cases are indeed the old cases, for example, these github.com cases.

modules/util/sanitize.go

Co-authored-by: zeripath <[email protected]>

Gusted

Seems okay... The current code seems to be indeed faster:

BenchmarkNew-12          3575206               368.0 ns/op           320 B/op          2 allocs/op
BenchmarkOld-12           373596              2956 ns/op            3288 B/op          9 allocs/op

A relative minor issue that comes with this, is that the current code will "handle" URL's like ://@ into -> ://sanitized-credential@ whereby the old function would return a error that it is a incorrect URL, but like I said relative minor issue.

wxiaoguang · 2022-03-28T13:38:34Z

Seems okay... The current code seems to be indeed faster:
BenchmarkNew-12          3575206               368.0 ns/op           320 B/op          2 allocs/op
BenchmarkOld-12           373596              2956 ns/op            3288 B/op          9 allocs/op
A relative minor issue that comes with this, is that the current code will "handle" URL's like ://@ into -> ://sanitized-credential@ whereby the old function would return a error that it is a incorrect URL, but like I said relative minor issue.

Thank you for the benchmark!

In my mind the major purpose is to make code more simple and intuitive, the performance is just a bonus.

And the edge case ://@ is fixed now, the minimal pattern is "s://u@h" now

Gusted · 2022-03-29T21:19:08Z

Why the skip-changelog?

wxiaoguang · 2022-03-30T00:01:57Z

Why the skip-changelog?

this PR is just a refactoring, it doesn't change the expected output, and doesn't improve performance obviously (because the sanitizer isn't called too much), so I think it's not necessary to mention it in the changelog, it doesn't provide useful information to end users.

feel free to adjust the labels.

zeripath · 2022-03-30T09:23:22Z

modules/util/sanitize.go

-		u.User = url.User(userPlaceholder)
-	} else {
-		u.User = nil
+// SanitizeCredentialURLs remove all credentials in URLs for the input string: "https://user:[email protected]" => "https://[email protected]"


This code only removes the credential from the first URL it finds in the string

If we want to strip out credentials from all urls in the string we're gonna need to do something else. (Possibly iterate across all "://" in the string however see below...)

One other issue is that there are URLs which do not have the ":", for example //username:[email protected] which says use the current protocol. I think /\username:[email protected] also works.

IMO I don't think we need to necessarily sanitize these - we should probably just make it clear that we're not going to do this.

OK, I will add it to comment. IMO "//xxxx" couldn't be treated as a valid URL in strings. It's only valid in a context with scheme already (say, inside a browser)

And /\ is not standard, it's just a browser's dirty hack. Golang's url.Parse doesn't support /\ either.

Suggested change

// SanitizeCredentialURLs remove all credentials in URLs for the input string: "https://user:[email protected]" => "https://[email protected]"

// SanitizeCredentialURLs remove all credentials in URLs (starting with "scheme://") for the input string: "https://user:[email protected]" => "https://[email protected]"

"This code only removes the credential from the first URL it finds in the string"

No, it removes all. See the test case

{ "URLs in log https://u:b@h and https://u:b@h:80/, with https://h.com and [email protected]", "URLs in log https://" + userPlaceholder + "@h and https://" + userPlaceholder + "@h:80/, with https://h.com and [email protected]", },

Co-authored-by: zeripath <[email protected]>

…L in strings because it doesn't have a valid scheme context.

…into refactor-url-sanitizer

* giteaofficial/main: Update reserved usernames list (go-gitea#18438) Configure OpenSSH log level via Environment in Docker (go-gitea#19274) Use a more general (and faster) method to sanitize URLs with credentials (go-gitea#19239) [skip ci] Updated translations via Crowdin fix link to package registry docs (go-gitea#19268) Add Redis Sentinel Authentication Support (go-gitea#19213)

wxiaoguang added type/refactoring Existing code has been cleaned up. There should be no new functionality. skip-changelog This PR is irrelevant for the (next) changelog, for example bug fixes for unreleased features. labels Mar 28, 2022

wxiaoguang added this to the 1.17.0 milestone Mar 28, 2022

wxiaoguang force-pushed the refactor-url-sanitizer branch from 7558472 to e6dd7c2 Compare March 28, 2022 06:07

wxiaoguang mentioned this pull request Mar 28, 2022

Hide sensitive content on admin panel progress monitor #19218

Merged

Use a more general (and faster) method to sanitize URLs with credentials

3d3f223

wxiaoguang force-pushed the refactor-url-sanitizer branch from e6dd7c2 to 3d3f223 Compare March 28, 2022 06:15

lunny reviewed Mar 28, 2022

View reviewed changes

GiteaBot added the lgtm/need 2 This PR needs two approvals by maintainers to be considered for merging. label Mar 28, 2022

zeripath reviewed Mar 28, 2022

View reviewed changes

modules/util/sanitize.go Outdated Show resolved Hide resolved

Update modules/util/sanitize.go

26cf692

Co-authored-by: zeripath <[email protected]>

Gusted approved these changes Mar 28, 2022

View reviewed changes

GiteaBot added lgtm/need 1 This PR needs approval from one additional maintainer to be merged. and removed lgtm/need 2 This PR needs two approvals by maintainers to be considered for merging. labels Mar 28, 2022

wxiaoguang added 2 commits March 28, 2022 21:21

Merge branch 'main' into refactor-url-sanitizer

7c7116c

fix edge case "://@"

ded6969

fix typo

dafa3f0

wxiaoguang requested review from lunny and zeripath March 28, 2022 16:51

wxiaoguang added 5 commits March 29, 2022 01:11

Merge branch 'main' into refactor-url-sanitizer

9580607

Merge branch 'main' into refactor-url-sanitizer

80719d7

Merge branch 'main' into refactor-url-sanitizer

ca05837

Merge branch 'main' into refactor-url-sanitizer

fe5e461

Merge branch 'main' into refactor-url-sanitizer

9f5fa58

Merge branch 'main' into refactor-url-sanitizer

26f05b5

zeripath reviewed Mar 30, 2022

View reviewed changes

Update modules/util/sanitize.go

fc167f3

Co-authored-by: zeripath <[email protected]>

wxiaoguang added 3 commits March 30, 2022 17:26

Merge branch 'main' into refactor-url-sanitizer

70bc63d

add a test for string "//u:p@h", which is not treated as a "valid" UR…

59046a5

…L in strings because it doesn't have a valid scheme context.

Merge branch 'refactor-url-sanitizer' of github.com:wxiaoguang/gitea …

0252a7b

…into refactor-url-sanitizer

zeripath approved these changes Mar 30, 2022

View reviewed changes

GiteaBot added lgtm/done This PR has enough approvals to get merged. There are no important open reservations anymore. and removed lgtm/need 1 This PR needs approval from one additional maintainer to be merged. labels Mar 30, 2022

Merge branch 'main' into refactor-url-sanitizer

74ea0d5

wxiaoguang merged commit c831681 into go-gitea:main Mar 31, 2022

wxiaoguang deleted the refactor-url-sanitizer branch March 31, 2022 02:25

go-gitea locked and limited conversation to collaborators Apr 28, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use a more general (and faster) method to sanitize URLs with credentials #19239

Use a more general (and faster) method to sanitize URLs with credentials #19239

wxiaoguang commented Mar 28, 2022

lunny Mar 28, 2022

wxiaoguang Mar 28, 2022

lunny Mar 28, 2022

wxiaoguang Mar 28, 2022

wxiaoguang Mar 28, 2022

Gusted left a comment

wxiaoguang commented Mar 28, 2022

Gusted commented Mar 29, 2022

wxiaoguang commented Mar 30, 2022

zeripath Mar 30, 2022

wxiaoguang Mar 30, 2022 •

edited

Loading

wxiaoguang Mar 30, 2022

wxiaoguang Mar 30, 2022

	// SanitizeCredentialURLs remove all credentials in URLs for the input string: "https://user:[email protected]" => "https://[email protected]"
	// SanitizeCredentialURLs remove all credentials in URLs (starting with "scheme://") for the input string: "https://user:[email protected]" => "https://[email protected]"

Use a more general (and faster) method to sanitize URLs with credentials #19239

Use a more general (and faster) method to sanitize URLs with credentials #19239

Conversation

wxiaoguang commented Mar 28, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Gusted left a comment

Choose a reason for hiding this comment

wxiaoguang commented Mar 28, 2022

Gusted commented Mar 29, 2022

wxiaoguang commented Mar 30, 2022

Choose a reason for hiding this comment

wxiaoguang Mar 30, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wxiaoguang Mar 30, 2022 •

edited

Loading