Fix conflict between param and exact path #2706

g1eny0ung · 2021-04-25T09:35:35Z

Signed-off-by: Yue Yang [email protected]

Close #2682.
Close #2696.

This PR solves the above issues.

~~For a detailed description, see #2682 (comment).~~

Signed-off-by: Yue Yang <[email protected]>

codecov · 2021-04-25T09:43:00Z

Codecov Report

Merging #2706 (808f903) into master (4fe5f3e) will increase coverage by 0.01%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master    #2706      +/-   ##
==========================================
+ Coverage   98.67%   98.68%   +0.01%     
==========================================
  Files          41       41              
  Lines        2036     2054      +18     
==========================================
+ Hits         2009     2027      +18     
  Misses         15       15              
  Partials       12       12

Impacted Files	Coverage Δ
tree.go	`100.00% <100.00%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4fe5f3e...808f903. Read the comment docs.

Signed-off-by: Yue Yang <[email protected]>

g1eny0ung · 2021-04-28T02:42:40Z

@thinkerou Would you please take a look when you are free? Very thx. 😃

appleboy · 2021-04-28T03:26:21Z

@g1eny0ung Can you also send the PR to https://github.com/julienschmidt/httprouter?

g1eny0ung · 2021-04-28T04:05:50Z

@g1eny0ung Can you also send the PR to https://github.com/julienschmidt/httprouter?

Because the PR julienschmidt/httprouter#329 still not be merged, so it's needed to wait for it or let @rw-access help to update the related code.

tree.go

tree_test.go

appleboy · 2021-04-28T05:00:04Z

@g1eny0ung Please provide the benchmark result like #2663 (comment) and #2663 (comment)

g1eny0ung · 2021-04-28T05:37:24Z

@rw-access @appleboy Thanks for the notes mentioned. I will try to add more tests and the benchmark later to make sure the change code is stable.

Signed-off-by: Yue Yang <[email protected]>

g1eny0ung · 2021-04-29T03:25:32Z

@appleboy After @rw-access noted (backtracking) and some thoughts 🤔, I update the code to save a temporary skipped value when the children have a param node at the last.

The skipped contains the current node path and a param node (same as the current node except no indices). When a param path has the same prefix with the exact path, the prefix tree will work as before, keep searching downwards, but it will fail.

Then the follow-up logic will check the skipped value is not nil to perform a backtracking search. The param node in skipped will be set to the current node. Because it has no indices, it will jump the non-wildcard children check, go directly to the place of param judgment.

Compared to before, I have added more tests, if there are omissions, please help me to add. 😆

The benchmarks:

master - https://travis-ci.com/github/justttype/go-http-routing-benchmark/builds/224357913
this PR - https://travis-ci.com/github/justttype/go-http-routing-benchmark/builds/224455608

The benchstat report:

name              old time/op    new time/op    delta
Gin_Param           66.6ns ± 0%    67.6ns ± 0%   ~     (p=1.000 n=1+1)
Gin_Param5           117ns ± 0%     112ns ± 0%   ~     (p=1.000 n=1+1)
Gin_Param20          303ns ± 0%     298ns ± 0%   ~     (p=1.000 n=1+1)
Gin_ParamWrite       123ns ± 0%     119ns ± 0%   ~     (p=1.000 n=1+1)
Gin_GithubStatic    77.4ns ± 0%    90.0ns ± 0%   ~     (p=1.000 n=1+1)
Gin_GithubParam      133ns ± 0%     156ns ± 0%   ~     (p=1.000 n=1+1)
Gin_GithubAll       26.8µs ± 0%    31.1µs ± 0%   ~     (p=1.000 n=1+1)
Gin_GPlusStatic     61.7ns ± 0%    66.8ns ± 0%   ~     (p=1.000 n=1+1)
Gin_GPlusParam      82.5ns ± 0%    92.2ns ± 0%   ~     (p=1.000 n=1+1)
Gin_GPlus2Params     103ns ± 0%     116ns ± 0%   ~     (p=1.000 n=1+1)
Gin_GPlusAll        1.13µs ± 0%    1.29µs ± 0%   ~     (p=1.000 n=1+1)
Gin_ParseStatic     62.3ns ± 0%    65.5ns ± 0%   ~     (p=1.000 n=1+1)
Gin_ParseParam      71.4ns ± 0%    74.3ns ± 0%   ~     (p=1.000 n=1+1)
Gin_Parse2Params    84.4ns ± 0%    84.7ns ± 0%   ~     (p=1.000 n=1+1)
Gin_ParseAll        2.01µs ± 0%    2.16µs ± 0%   ~     (p=1.000 n=1+1)
Gin_StaticAll       18.5µs ± 0%    23.7µs ± 0%   ~     (p=1.000 n=1+1)

name              old alloc/op   new alloc/op   delta
Gin_Param            0.00B          0.00B        ~     (all equal)
Gin_Param5           0.00B          0.00B        ~     (all equal)
Gin_Param20          0.00B          0.00B        ~     (all equal)
Gin_ParamWrite       0.00B          0.00B        ~     (all equal)
Gin_GithubStatic     0.00B          0.00B        ~     (all equal)
Gin_GithubParam      0.00B          0.00B        ~     (all equal)
Gin_GithubAll        0.00B          0.00B        ~     (all equal)
Gin_GPlusStatic      0.00B          0.00B        ~     (all equal)
Gin_GPlusParam       0.00B          0.00B        ~     (all equal)
Gin_GPlus2Params     0.00B          0.00B        ~     (all equal)
Gin_GPlusAll         0.00B          0.00B        ~     (all equal)
Gin_ParseStatic      0.00B          0.00B        ~     (all equal)
Gin_ParseParam       0.00B          0.00B        ~     (all equal)
Gin_Parse2Params     0.00B          0.00B        ~     (all equal)
Gin_ParseAll         0.00B          0.00B        ~     (all equal)
Gin_StaticAll        0.00B          0.00B        ~     (all equal)

name              old allocs/op  new allocs/op  delta
Gin_Param             0.00           0.00        ~     (all equal)
Gin_Param5            0.00           0.00        ~     (all equal)
Gin_Param20           0.00           0.00        ~     (all equal)
Gin_ParamWrite        0.00           0.00        ~     (all equal)
Gin_GithubStatic      0.00           0.00        ~     (all equal)
Gin_GithubParam       0.00           0.00        ~     (all equal)
Gin_GithubAll         0.00           0.00        ~     (all equal)
Gin_GPlusStatic       0.00           0.00        ~     (all equal)
Gin_GPlusParam        0.00           0.00        ~     (all equal)
Gin_GPlus2Params      0.00           0.00        ~     (all equal)
Gin_GPlusAll          0.00           0.00        ~     (all equal)
Gin_ParseStatic       0.00           0.00        ~     (all equal)
Gin_ParseParam        0.00           0.00        ~     (all equal)
Gin_Parse2Params      0.00           0.00        ~     (all equal)
Gin_ParseAll          0.00           0.00        ~     (all equal)
Gin_StaticAll         0.00           0.00        ~     (all equal)

Thank you all for your reviews.

rw-access · 2021-04-29T03:49:50Z

tree_test.go

 		"/cmd/:tool/",
+		"/cmd/:tool/:sub",


Thanks for all the updates, I really appreciate it.

I think we could use another test to check edge case to check:

if you have routes like this set up:

/cmd/whoami/foo

/cmd/:tool/

/cmd/:tool/:subtool

What happens if someone hits the route:

/cmd/whoami/bar?

It should cause a 404 and not hit /cmd/:tool/:subtool.

Basically, whenever the tree is exhausted or a the end of a path segment is found, the backtrack should be invalidated. Once you hit /cmd/whoami/, it shouldn't be possible to fall back to /cmd/tool/.

~~This is the problem.~~

~~For example, /cmd/who/ will not hit /cmd/:tool/ because it has the same prefix as /cmd/whoami/foo, but it should match.~~

Know what you mean. I will think about this situation.

But what still let me confused is: with the above routes, whether /cmd/whoami/bar hit or not, seems they are all meaningful?

According to my understanding and my actual use, if I don’t have an exact route definition like /cmd/whoami/foo, then /cmd/whoami/bar needs to be matched to the wildcard route.

rw-access · 2021-04-29T03:51:58Z

tree.go

@@ -411,6 +418,21 @@ walk: // Outer loop for walking the tree
 				idxc := path[0]
 				for i, c := range []byte(n.indices) {
 					if c == idxc {
+						if strings.HasPrefix(n.children[len(n.children)-1].path, ":") {


I think you want to check if the last child is of type Param, right?

and then when you need to backtrack, then you can go to the Param child directly?

So I think is needed to save the Param child to a place? Otherwise, I will not be able to find it because the walk loop will replace the previous node with the current.

g1eny0ung · 2021-05-10T02:35:47Z

@thinkerou @appleboy What do you think of these changes? Should it be merged?

appleboy · 2021-05-10T04:53:12Z

@rw-access needs your feedback.

vaimr · 2021-05-12T11:13:21Z

It's very important fix for everybody!

daheige · 2021-05-14T13:38:30Z

nice,great job!

…

------------------ Original ------------------ From: Denis Saponenko ***@***.***> Date: Wed,May 12,2021 7:13 PM To: gin-gonic/gin ***@***.***> Cc: heige ***@***.***>, Comment ***@***.***> Subject: Re: [gin-gonic/gin] Fix conflict between param and exact path (#2706) It's very important fix for everybody! — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

rw-access · 2021-05-18T04:06:11Z

tree.go

+						if strings.HasPrefix(n.children[len(n.children)-1].path, ":") {
+							skipped = &skip{
+								path: prefix + path,
+								paramNode: &node{


why not just make this data structure take n?

Because this operation removes indices to prevent prefix checking.

tree.go

rw-access

LGTM. Thanks for the fixes

appleboy · 2021-05-18T13:35:10Z

I will bump the new version v1.7.2 if the PR has been merge. Waiting for @thinkerou approval.

thinkerou

lgtm

appleboy · 2021-05-21T23:45:10Z

Hi All,

We already released https://github.com/gin-gonic/gin/releases/tag/v1.7.2 version, just include the changes.

* Fix conflict between param and exact path Signed-off-by: Yue Yang <[email protected]> * Add test Signed-off-by: Yue Yang <[email protected]> * Fix prefix conflict in exact paths Signed-off-by: Yue Yang <[email protected]> * Use backtracking Signed-off-by: Yue Yang <[email protected]> * Fix panic Signed-off-by: Yue Yang <[email protected]>

* Fix conflict between param and exact path Signed-off-by: Yue Yang <[email protected]> * Add test Signed-off-by: Yue Yang <[email protected]> * Fix prefix conflict in exact paths Signed-off-by: Yue Yang <[email protected]> * Use backtracking Signed-off-by: Yue Yang <[email protected]> * Fix panic Signed-off-by: Yue Yang <[email protected]> (cherry picked from commit 2921582)

* Fix conflict between param and exact path Signed-off-by: Yue Yang <[email protected]> * Add test Signed-off-by: Yue Yang <[email protected]> * Fix prefix conflict in exact paths Signed-off-by: Yue Yang <[email protected]> * Use backtracking Signed-off-by: Yue Yang <[email protected]> * Fix panic Signed-off-by: Yue Yang <[email protected]>

Fix conflict between param and exact path

801d7b3

Signed-off-by: Yue Yang <[email protected]>

Add test

39fa5a3

Signed-off-by: Yue Yang <[email protected]>

appleboy mentioned this pull request Apr 28, 2021

[Bug] Mixed params and exact paths can cause a 404 #2682

Closed

appleboy requested review from thinkerou and appleboy April 28, 2021 03:29

appleboy added the bug label Apr 28, 2021

appleboy added this to the v1.8 milestone Apr 28, 2021

rw-access reviewed Apr 28, 2021

View reviewed changes

tree.go Outdated Show resolved Hide resolved

tree_test.go Show resolved Hide resolved

g1eny0ung added 3 commits April 28, 2021 14:56

Fix prefix conflict in exact paths

7c9e3b0

Signed-off-by: Yue Yang <[email protected]>

Use backtracking

3bb5308

Signed-off-by: Yue Yang <[email protected]>

Fix panic

e75fe4a

Signed-off-by: Yue Yang <[email protected]>

Merge branch 'master' into fix/issue#2682

47b1ea4

rw-access reviewed Apr 29, 2021

View reviewed changes

daheige approved these changes May 1, 2021

View reviewed changes

Merge branch 'master' into fix/issue#2682

808f903

lallinger-tech mentioned this pull request May 14, 2021

Add support for wildcard paths with other children julienschmidt/httprouter#329

Closed

rw-access reviewed May 18, 2021

View reviewed changes

tree.go Show resolved Hide resolved

rw-access approved these changes May 18, 2021

View reviewed changes

appleboy approved these changes May 18, 2021

View reviewed changes

appleboy modified the milestones: v1.8, v1.7.2 May 18, 2021

thinkerou approved these changes May 19, 2021

View reviewed changes

appleboy merged commit 2921582 into gin-gonic:master May 19, 2021

appleboy modified the milestones: v1.7.2, v1.8 May 19, 2021

g1eny0ung deleted the fix/issue#2682 branch May 19, 2021 02:13

nnntlh mentioned this pull request Jun 18, 2021

wildcard path param #2754

Closed

rw-access mentioned this pull request Jun 25, 2021

404 when using dynamic routing #2762

Closed

appleboy mentioned this pull request Jun 28, 2021

fix #2762 #2767

Merged

g1eny0ung mentioned this pull request Jul 1, 2021

chore(api): use direct routes chaos-mesh/chaos-mesh#2066

Merged

5 tasks

ti-srebot mentioned this pull request Jul 2, 2021

chore(api): use direct routes (#2066) chaos-mesh/chaos-mesh#2072

Merged

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix conflict between param and exact path #2706

Fix conflict between param and exact path #2706

g1eny0ung commented Apr 25, 2021 •

edited

Loading

codecov bot commented Apr 25, 2021 •

edited

Loading

g1eny0ung commented Apr 28, 2021 •

edited

Loading

appleboy commented Apr 28, 2021

g1eny0ung commented Apr 28, 2021 •

edited

Loading

appleboy commented Apr 28, 2021

g1eny0ung commented Apr 28, 2021

g1eny0ung commented Apr 29, 2021

rw-access Apr 29, 2021

g1eny0ung Apr 29, 2021 •

edited

Loading

g1eny0ung Apr 29, 2021

g1eny0ung Apr 29, 2021 •

edited

Loading

rw-access Apr 29, 2021

rw-access Apr 29, 2021

g1eny0ung Apr 29, 2021

g1eny0ung commented May 10, 2021

appleboy commented May 10, 2021

vaimr commented May 12, 2021

daheige commented May 14, 2021 via email

rw-access May 18, 2021

g1eny0ung May 18, 2021

rw-access left a comment

appleboy commented May 18, 2021

thinkerou left a comment

appleboy commented May 21, 2021

Fix conflict between param and exact path #2706

Fix conflict between param and exact path #2706

Conversation

g1eny0ung commented Apr 25, 2021 • edited Loading

codecov bot commented Apr 25, 2021 • edited Loading

Codecov Report

g1eny0ung commented Apr 28, 2021 • edited Loading

appleboy commented Apr 28, 2021

g1eny0ung commented Apr 28, 2021 • edited Loading

appleboy commented Apr 28, 2021

g1eny0ung commented Apr 28, 2021

g1eny0ung commented Apr 29, 2021

rw-access Apr 29, 2021

Choose a reason for hiding this comment

g1eny0ung Apr 29, 2021 • edited Loading

Choose a reason for hiding this comment

g1eny0ung Apr 29, 2021

Choose a reason for hiding this comment

g1eny0ung Apr 29, 2021 • edited Loading

Choose a reason for hiding this comment

rw-access Apr 29, 2021

Choose a reason for hiding this comment

rw-access Apr 29, 2021

Choose a reason for hiding this comment

g1eny0ung Apr 29, 2021

Choose a reason for hiding this comment

g1eny0ung commented May 10, 2021

appleboy commented May 10, 2021

vaimr commented May 12, 2021

daheige commented May 14, 2021 via email

rw-access May 18, 2021

Choose a reason for hiding this comment

g1eny0ung May 18, 2021

Choose a reason for hiding this comment

rw-access left a comment

Choose a reason for hiding this comment

appleboy commented May 18, 2021

thinkerou left a comment

Choose a reason for hiding this comment

appleboy commented May 21, 2021

g1eny0ung commented Apr 25, 2021 •

edited

Loading

codecov bot commented Apr 25, 2021 •

edited

Loading

g1eny0ung commented Apr 28, 2021 •

edited

Loading

g1eny0ung commented Apr 28, 2021 •

edited

Loading

g1eny0ung Apr 29, 2021 •

edited

Loading

g1eny0ung Apr 29, 2021 •

edited

Loading