fiberi18n error on concurrent requests #594

tran-the-lam · 2023-05-16T08:56:08Z

Hi team!

I have an issue. When there are many concurrent requests to our system, fiberi18n throws error like this

[2023-05-16T04:55:32.634958995Z] 57398fb9-da68-4ea2-b008-51da27c5a0ab:: 200  -  GET      /threads 7.703849ms -
fatal error: concurrent map writes
fatal error: concurrent map writes

goroutine 172 [running]:
github.com/gofiber/contrib/fiberi18n.New.func1(0xc0002282c0)
	/go/pkg/mod/github.com/gofiber/contrib/[email protected]/i18n.go:42 +0x209
github.com/gofiber/fiber/v2.(*App).next(0xc0001b8480, 0xc0002282c0)
	/go/pkg/mod/github.com/gofiber/fiber/[email protected]/router.go:134 +0x1b6
github.com/gofiber/fiber/v2.(*App).handler(0xc0001b8480, 0x4cc317?)
	/go/pkg/mod/github.com/gofiber/fiber/[email protected]/router.go:160 +0x87
github.com/valyala/fasthttp.(*Server).serveConn(0xc00054c200, {0xf7e6f8?, 0xc0000aa1b0})
	/go/pkg/mod/github.com/valyala/[email protected]/server.go:2371 +0x11d3
github.com/valyala/fasthttp.(*workerPool).workerFunc(0xc000539720, 0xc0000c0b60)
	/go/pkg/mod/github.com/valyala/[email protected]/workerpool.go:224 +0xa9
github.com/valyala/fasthttp.(*workerPool).getCh.func1()
	/go/pkg/mod/github.com/valyala/[email protected]/workerpool.go:196 +0x38
created by github.com/valyala/fasthttp.(*workerPool).getCh
	/go/pkg/mod/github.com/valyala/[email protected]/workerpool.go:195 +0x1b0

goroutine 1 [IO wait, 1 minutes]:
internal/poll.runtime_pollWait(0x7fb9247119b8, 0x72)
	/usr/local/go/src/runtime/netpoll.go:306 +0x89
internal/poll.(*pollDesc).wait(0xc0004ad100?, 0x4?, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0xc0004ad100)
	/usr/local/go/src/internal/poll/fd_unix.go:614 +0x2bd
net.(*netFD).accept(0xc0004ad100)
	/usr/local/go/src/net/fd_unix.go:172 +0x35
net.(*TCPListener).accept(0xc0004c5950)
	/usr/local/go/src/net/tcpsock_posix.go:148 +0x25
net.(*TCPListener).Accept(0xc0004c5950)
	/usr/local/go/src/net/tcpsock.go:297 +0x3d
github.com/valyala/fasthttp.acceptConn(0xc00054c200, {0xf7a710, 0xc0004c5950}, 0xc0006938b8)
	/go/pkg/mod/github.com/valyala/[email protected]/server.go:1930 +0x62
github.com/valyala/fasthttp.(*Server).Serve(0xc00054c200, {0xf7a710?, 0xc0004c5950})
	/go/pkg/mod/github.com/valyala/[email protected]/server.go:1823 +0x4f4
github.com/gofiber/fiber/v2.(*App).Listen(0xc0001b8480, {0xe3e28a?, 0xd52f40?})
	/go/pkg/mod/github.com/gofiber/fiber/[email protected]/listen.go:82 +0x110
main.main()
	/build/cmd/api/api.go:119 +0x357

My code implemented:

app.Use(
		fiberi18n.New(&fiberi18n.Config{
			RootPath:        "./localize",
			AcceptLanguages: []language.Tag{language.Vietnamese, language.English},
			DefaultLanguage: language.English,
		}),
	)

 msg := fiberi18n.MustGetMessage("successfully")

Please help me to solve this problem.

The text was updated successfully, but these errors were encountered:

ReneWerner87 · 2023-05-16T09:02:20Z

@Skyenought can you help here

Skyenought · 2023-05-16T10:11:39Z

let me see.

Skyenought · 2023-05-16T11:31:26Z

I will be making a pull request to fix the known issues

tran-the-lam · 2023-05-17T09:33:22Z

@Skyenought I got the code from master branch but when I run the request concurrently, the system still gives i18n error

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x13e3e79]

goroutine 173 [running]:
github.com/valyala/fasthttp.(*RequestCtx).URI(...)
        /Users/user/go/pkg/mod/github.com/valyala/[email protected]/server.go:952
github.com/valyala/fasthttp.(*RequestCtx).QueryArgs(0x0)
        /Users/user/go/pkg/mod/github.com/valyala/[email protected]/server.go:991 +0x19
github.com/gofiber/fiber/v2.(*Ctx).Query(0xc000402840, {0x1a3ee1f, 0x4}, {0x0, 0x0, 0x0?})
        /Users/user/go/pkg/mod/github.com/gofiber/fiber/[email protected]/ctx.go:1054 +0x45
my-fiberi18n/pkg/i18n.defaultLangHandler(0xc000860086?, {0x1a80bae, 0x2})

Skyenought · 2023-05-17T09:35:32Z

@tran-the-lam show your code ?

ReneWerner87 · 2023-05-17T09:56:40Z

had you read article -> https://docs.gofiber.io/#zero-allocation

tran-the-lam · 2023-05-17T10:10:52Z

@tran-the-lam show your code ?

You can see my demo(https://github.com/tran-the-lam/debug-fiberi18n)
Step 1: go run main.go
Step 2: . scripts/concurrent_request.sh
Error:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x135e679]

goroutine 10 [running]:
github.com/valyala/fasthttp.(*RequestCtx).URI(...)
        /Users/user/go/pkg/mod/github.com/valyala/[email protected]/server.go:952
github.com/valyala/fasthttp.(*RequestCtx).QueryArgs(0x0)
        /Users/user/go/pkg/mod/github.com/valyala/[email protected]/server.go:991 +0x19
github.com/gofiber/fiber/v2.(*Ctx).Query(0xc00020c000, {0x150c47b, 0x4}, {0x0, 0x0, 0x0?})
        /Users/user/go/pkg/mod/github.com/gofiber/fiber/[email protected]/ctx.go:1054 +0x45
test/fiberi18n.defaultLangHandler(0xc000860086?, {0x152a6d4, 0x2})
        /Users/user/Documents/workspace/ PiScale/Source Code/CHAT/debug_fiberi18n/fiberi18n/config.go:80 +0x3f
test/fiberi18n.GetMessage({0x1488ce0?, 0x15bf2b0?})
        /Users/user/Documents/workspace/ PiScale/Source Code/CHAT/debug_fiberi18n/fiberi18n/i18n.go:105 +0x86
test/fiberi18n.MustGetMessage(...)
        /Users/user/Documents/workspace/ PiScale/Source Code/CHAT/debug_fiberi18n/fiberi18n/i18n.go:85
main.main.func1(0xc0000e0840)
        /Users/user/Documents/workspace/ PiScale/Source Code/CHAT/debug_fiberi18n/main.go:24 +0x6a
github.com/gofiber/fiber/v2.(*App).next(0xc000080d80, 0xc0000e0840)
        /Users/user/go/pkg/mod/github.com/gofiber/fiber/[email protected]/router.go:144 +0x1bf
github.com/gofiber/fiber/v2.(*Ctx).Next(0xc0001de000?)
        /Users/user/go/pkg/mod/github.com/gofiber/fiber/[email protected]/ctx.go:913 +0x53
test/fiberi18n.New.func1(0x14a5680?)
        /Users/user/Documents/workspace/ PiScale/Source Code/CHAT/debug_fiberi18n/fiberi18n/i18n.go:25 +0x5e
github.com/gofiber/fiber/v2.(*App).next(0xc000080d80, 0xc0000e0840)
        /Users/user/go/pkg/mod/github.com/gofiber/fiber/[email protected]/router.go:144 +0x1bf
github.com/gofiber/fiber/v2.(*App).handler(0xc000080d80, 0x10860f7?)
        /Users/user/go/pkg/mod/github.com/gofiber/fiber/[email protected]/router.go:171 +0x87
github.com/valyala/fasthttp.(*Server).serveConn(0xc0001bc000, {0x15c5640?, 0xc00028c170})
        /Users/user/go/pkg/mod/github.com/valyala/[email protected]/server.go:2365 +0x11d3
github.com/valyala/fasthttp.(*workerPool).workerFunc(0xc0000ccf00, 0xc0000720c0)
        /Users/user/go/pkg/mod/github.com/valyala/[email protected]/workerpool.go:224 +0xa9
github.com/valyala/fasthttp.(*workerPool).getCh.func1()
        /Users/user/go/pkg/mod/github.com/valyala/[email protected]/workerpool.go:196 +0x38
created by github.com/valyala/fasthttp.(*workerPool).getCh
        /Users/user/go/pkg/mod/github.com/valyala/[email protected]/workerpool.go:195 +0x1b0
exit status 2

ReneWerner87 · 2023-05-17T10:13:41Z

here you should do a copy
https://github.com/tran-the-lam/debug-fiberi18n/blob/main/fiberi18n/config.go#L85
because of our zero allocation concept -> read the article

tran-the-lam · 2023-05-17T10:25:19Z

here you should do a copy https://github.com/tran-the-lam/debug-fiberi18n/blob/main/fiberi18n/config.go#L85 because of our zero allocation concept -> read the article

I tried but response still same error
My code:

lang = c.Get("Accept-Language")
buffer := make([]byte, len(lang))
copy(buffer, lang)
langRs := string(buffer)
if langRs != "" {
	return langRs
}

tran-the-lam · 2023-05-17T10:27:17Z

Maybe gofiber/contrib/fiberi18n has the same problem

contrib/fiberi18n/config.go

Line 85 in 2bad1b1

lang = c.Get("Accept-Language")

ReneWerner87 · 2023-05-17T11:03:54Z

https://github.com/tran-the-lam/debug-fiberi18n/blob/c9740c79fa469dc833cb466f5db3528e5b578d2a/fiberi18n/i18n.go#L106

problem is that you store the value in a map which survives longer than the request response loop, so you should make a copy here with our copy function

ReneWerner87 · 2023-05-17T11:04:48Z

@Skyenought same problem is here
https://github.com/gofiber/contrib/blob/2bad1b1d10b4f1bbc0196947feb679d0b8f81c9d/fiberi18n/i18n.go#LL106C45-L106C49

we need better unittests

Skyenought · 2023-05-17T11:33:11Z

yes, i will

Skyenought · 2023-05-17T11:54:34Z

@tran-the-lam show your code ?

You can see my demo(https://github.com/tran-the-lam/debug-fiberi18n) Step 1: go run main.go Step 2: . scripts/concurrent_request.sh Error:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x135e679]

goroutine 10 [running]:
github.com/valyala/fasthttp.(*RequestCtx).URI(...)
        /Users/user/go/pkg/mod/github.com/valyala/[email protected]/server.go:952
github.com/valyala/fasthttp.(*RequestCtx).QueryArgs(0x0)
        /Users/user/go/pkg/mod/github.com/valyala/[email protected]/server.go:991 +0x19
github.com/gofiber/fiber/v2.(*Ctx).Query(0xc00020c000, {0x150c47b, 0x4}, {0x0, 0x0, 0x0?})
        /Users/user/go/pkg/mod/github.com/gofiber/fiber/[email protected]/ctx.go:1054 +0x45
test/fiberi18n.defaultLangHandler(0xc000860086?, {0x152a6d4, 0x2})
        /Users/user/Documents/workspace/ PiScale/Source Code/CHAT/debug_fiberi18n/fiberi18n/config.go:80 +0x3f
test/fiberi18n.GetMessage({0x1488ce0?, 0x15bf2b0?})
        /Users/user/Documents/workspace/ PiScale/Source Code/CHAT/debug_fiberi18n/fiberi18n/i18n.go:105 +0x86
test/fiberi18n.MustGetMessage(...)
        /Users/user/Documents/workspace/ PiScale/Source Code/CHAT/debug_fiberi18n/fiberi18n/i18n.go:85
main.main.func1(0xc0000e0840)
        /Users/user/Documents/workspace/ PiScale/Source Code/CHAT/debug_fiberi18n/main.go:24 +0x6a
github.com/gofiber/fiber/v2.(*App).next(0xc000080d80, 0xc0000e0840)
        /Users/user/go/pkg/mod/github.com/gofiber/fiber/[email protected]/router.go:144 +0x1bf
github.com/gofiber/fiber/v2.(*Ctx).Next(0xc0001de000?)
        /Users/user/go/pkg/mod/github.com/gofiber/fiber/[email protected]/ctx.go:913 +0x53
test/fiberi18n.New.func1(0x14a5680?)
        /Users/user/Documents/workspace/ PiScale/Source Code/CHAT/debug_fiberi18n/fiberi18n/i18n.go:25 +0x5e
github.com/gofiber/fiber/v2.(*App).next(0xc000080d80, 0xc0000e0840)
        /Users/user/go/pkg/mod/github.com/gofiber/fiber/[email protected]/router.go:144 +0x1bf
github.com/gofiber/fiber/v2.(*App).handler(0xc000080d80, 0x10860f7?)
        /Users/user/go/pkg/mod/github.com/gofiber/fiber/[email protected]/router.go:171 +0x87
github.com/valyala/fasthttp.(*Server).serveConn(0xc0001bc000, {0x15c5640?, 0xc00028c170})
        /Users/user/go/pkg/mod/github.com/valyala/[email protected]/server.go:2365 +0x11d3
github.com/valyala/fasthttp.(*workerPool).workerFunc(0xc0000ccf00, 0xc0000720c0)
        /Users/user/go/pkg/mod/github.com/valyala/[email protected]/workerpool.go:224 +0xa9
github.com/valyala/fasthttp.(*workerPool).getCh.func1()
        /Users/user/go/pkg/mod/github.com/valyala/[email protected]/workerpool.go:196 +0x38
created by github.com/valyala/fasthttp.(*workerPool).getCh
        /Users/user/go/pkg/mod/github.com/valyala/[email protected]/workerpool.go:195 +0x1b0
exit status 2

In fact, although I didn't use methods like utils.CopyString(), I didn't get a panic when I executed this script 🤔

Skyenought · 2023-05-17T12:17:54Z

In fact, I used ab for the stress test, and there was no panic

> ab -n 1000 -c 50  http://127.0.0.1:3000/
This is ApacheBench, Version 2.3 <$Revision: 1843412 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 127.0.0.1 (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests


Server Software:
Server Hostname:        127.0.0.1
Server Port:            3000

Document Path:          /
Document Length:        5 bytes

Concurrency Level:      50
Time taken for tests:   0.021 seconds
Complete requests:      1000
Failed requests:        0
Total transferred:      140000 bytes
HTML transferred:       5000 bytes
Requests per second:    46777.06 [#/sec] (mean)
Time per request:       1.069 [ms] (mean)
Time per request:       0.021 [ms] (mean, across all concurrent requests)
Transfer rate:          6395.30 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.2      0       1
Processing:     0    1   0.8      1       7
Waiting:        0    1   0.7      1       7
Total:          0    1   0.8      1       7

Percentage of the requests served within a certain time (ms)
  50%      1
  66%      1
  75%      1
  80%      1
  90%      2
  95%      3
  98%      3
  99%      3
 100%      7 (longest request)

tran-the-lam · 2023-05-19T02:49:25Z

@Skyenought Is this line already in your source test?

Skyenought · 2023-05-20T06:41:58Z

@tran-the-lam ok, i will fix them

Skyenought · 2023-05-20T09:36:38Z

i think you can add recover to your code, after you do this you can run service success.

I don't know what it's about 😖

tran-the-lam · 2023-05-21T11:12:50Z

Recover link not found

Skyenought · 2023-05-22T06:25:36Z

https://docs.gofiber.io/api/middleware/recover

ReneWerner87 · 2023-08-16T12:45:00Z

@tran-the-lam show your code ?

You can see my demo(https://github.com/tran-the-lam/debug-fiberi18n) Step 1: go run main.go Step 2: . scripts/concurrent_request.sh Error:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x135e679]

goroutine 10 [running]:
github.com/valyala/fasthttp.(*RequestCtx).URI(...)
        /Users/user/go/pkg/mod/github.com/valyala/[email protected]/server.go:952
github.com/valyala/fasthttp.(*RequestCtx).QueryArgs(0x0)
        /Users/user/go/pkg/mod/github.com/valyala/[email protected]/server.go:991 +0x19
github.com/gofiber/fiber/v2.(*Ctx).Query(0xc00020c000, {0x150c47b, 0x4}, {0x0, 0x0, 0x0?})
        /Users/user/go/pkg/mod/github.com/gofiber/fiber/[email protected]/ctx.go:1054 +0x45
test/fiberi18n.defaultLangHandler(0xc000860086?, {0x152a6d4, 0x2})
        /Users/user/Documents/workspace/ PiScale/Source Code/CHAT/debug_fiberi18n/fiberi18n/config.go:80 +0x3f
test/fiberi18n.GetMessage({0x1488ce0?, 0x15bf2b0?})
        /Users/user/Documents/workspace/ PiScale/Source Code/CHAT/debug_fiberi18n/fiberi18n/i18n.go:105 +0x86
test/fiberi18n.MustGetMessage(...)
        /Users/user/Documents/workspace/ PiScale/Source Code/CHAT/debug_fiberi18n/fiberi18n/i18n.go:85
main.main.func1(0xc0000e0840)
        /Users/user/Documents/workspace/ PiScale/Source Code/CHAT/debug_fiberi18n/main.go:24 +0x6a
github.com/gofiber/fiber/v2.(*App).next(0xc000080d80, 0xc0000e0840)
        /Users/user/go/pkg/mod/github.com/gofiber/fiber/[email protected]/router.go:144 +0x1bf
github.com/gofiber/fiber/v2.(*Ctx).Next(0xc0001de000?)
        /Users/user/go/pkg/mod/github.com/gofiber/fiber/[email protected]/ctx.go:913 +0x53
test/fiberi18n.New.func1(0x14a5680?)
        /Users/user/Documents/workspace/ PiScale/Source Code/CHAT/debug_fiberi18n/fiberi18n/i18n.go:25 +0x5e
github.com/gofiber/fiber/v2.(*App).next(0xc000080d80, 0xc0000e0840)
        /Users/user/go/pkg/mod/github.com/gofiber/fiber/[email protected]/router.go:144 +0x1bf
github.com/gofiber/fiber/v2.(*App).handler(0xc000080d80, 0x10860f7?)
        /Users/user/go/pkg/mod/github.com/gofiber/fiber/[email protected]/router.go:171 +0x87
github.com/valyala/fasthttp.(*Server).serveConn(0xc0001bc000, {0x15c5640?, 0xc00028c170})
        /Users/user/go/pkg/mod/github.com/valyala/[email protected]/server.go:2365 +0x11d3
github.com/valyala/fasthttp.(*workerPool).workerFunc(0xc0000ccf00, 0xc0000720c0)
        /Users/user/go/pkg/mod/github.com/valyala/[email protected]/workerpool.go:224 +0xa9
github.com/valyala/fasthttp.(*workerPool).getCh.func1()
        /Users/user/go/pkg/mod/github.com/valyala/[email protected]/workerpool.go:196 +0x38
created by github.com/valyala/fasthttp.(*workerPool).getCh
        /Users/user/go/pkg/mod/github.com/valyala/[email protected]/workerpool.go:195 +0x1b0
exit status 2

In fact, although I didn't use methods like utils.CopyString(), I didn't get a panic when I executed this script 🤔

@Skyenought In your test you only request with the same local, you can make it random and send several different requests with several different locals at the same time

think you need to copy the value and use a safe map for concurrency

problem should be the place already mentioned by me

Skyenought · 2023-08-17T16:23:54Z

@ReneWerner87 I'll try to fix it again.

KompocikDot · 2023-08-17T17:16:17Z

Hi, I think I have found the bug and it's placed here.

contrib/fiberi18n/i18n.go

Line 24 in 2bad1b1

appCfg.ctx = c

When using utils.CopyString on c.Query and c.Get and assigning header and query there to appCfg instead of assigning whole ctx to appCfg I could not face this error again. But I'm new to go and I might be wrong 😅

ReneWerner87 added the ☢️ Bug Something isn't working label May 16, 2023

Skyenought mentioned this issue May 16, 2023

fix fiberi18n error on concurrent requests #595

Merged

Skyenought mentioned this issue Aug 18, 2023

fix(fiberi18n): use sync.Map and utils.CopyString to fix concurrent problem #718

Merged

Skyenought closed this as completed in #718 Aug 22, 2023

ReneWerner87 mentioned this issue Aug 25, 2023

🐛 [Bug]: i18n middleware panics #721

Closed

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fiberi18n error on concurrent requests #594

fiberi18n error on concurrent requests #594

tran-the-lam commented May 16, 2023

ReneWerner87 commented May 16, 2023

Skyenought commented May 16, 2023

Skyenought commented May 16, 2023

tran-the-lam commented May 17, 2023

Skyenought commented May 17, 2023

ReneWerner87 commented May 17, 2023

tran-the-lam commented May 17, 2023

ReneWerner87 commented May 17, 2023

tran-the-lam commented May 17, 2023 •

edited

Loading

tran-the-lam commented May 17, 2023

ReneWerner87 commented May 17, 2023

ReneWerner87 commented May 17, 2023

Skyenought commented May 17, 2023

Skyenought commented May 17, 2023

Skyenought commented May 17, 2023

tran-the-lam commented May 19, 2023 •

edited

Loading

Skyenought commented May 20, 2023

Skyenought commented May 20, 2023 •

edited

Loading

tran-the-lam commented May 21, 2023

Skyenought commented May 22, 2023

ReneWerner87 commented Aug 16, 2023

Skyenought commented Aug 17, 2023

KompocikDot commented Aug 17, 2023

fiberi18n error on concurrent requests #594

fiberi18n error on concurrent requests #594

Comments

tran-the-lam commented May 16, 2023

ReneWerner87 commented May 16, 2023

Skyenought commented May 16, 2023

Skyenought commented May 16, 2023

tran-the-lam commented May 17, 2023

Skyenought commented May 17, 2023

ReneWerner87 commented May 17, 2023

tran-the-lam commented May 17, 2023

ReneWerner87 commented May 17, 2023

tran-the-lam commented May 17, 2023 • edited Loading

tran-the-lam commented May 17, 2023

ReneWerner87 commented May 17, 2023

ReneWerner87 commented May 17, 2023

Skyenought commented May 17, 2023

Skyenought commented May 17, 2023

Skyenought commented May 17, 2023

tran-the-lam commented May 19, 2023 • edited Loading

Skyenought commented May 20, 2023

Skyenought commented May 20, 2023 • edited Loading

tran-the-lam commented May 21, 2023

Skyenought commented May 22, 2023

ReneWerner87 commented Aug 16, 2023

Skyenought commented Aug 17, 2023

KompocikDot commented Aug 17, 2023

tran-the-lam commented May 17, 2023 •

edited

Loading

tran-the-lam commented May 19, 2023 •

edited

Loading

Skyenought commented May 20, 2023 •

edited

Loading