runtime: support unwinding logical cgo stack transitions using frame pointers

[qmuntal]

Note: follow-up issue for #40044 (comment)

When using generic third-party debuggers (e.g. dbg or windbg) to debug a amd64 Go binary, it is currently not possible to know which piece of Go code started a cgo call once runtime.asmcgocall has switched from the Go routine stack to the system stack. This is because asmcgocall is marked as NOFRAME, so the frame pointer is not set on the function prologue, confusing the debugger unwinding process.

Notice that this is only true for amd64 binaries, as all other architectures supporting frame pointers do not mark asmcgocall as NOFRAME. Yet, at least windows/arm64 neither supports cgo stack unwinding because asmcgocall calls asmstdcall, which is marked as NOFRAME.

IMO we should at least make sure that asmcgocall and related-auxiliary functions can be unwound from the system stack up to the Go stack by placing the right frame pointer in their prologue. I'm leaving out of this discussion other functions that perform a system stack on purpose, as they are not so fundamental as asmcgocall and correctly setting the frame pointer of low-level assembly functions is not straight-forward.

I'm particularly interested on this because on Windows, WinDbg uses the frame pointer to unwind the stack (see #57302) to unwound the stack of a debugged binary or crash minidump (#57441, #49471, #20498).

It would also help fixing at least #40044 and #57698.

@golang/runtime @golang/compiler

[mknyszek]

@qmuntal I've assigned this to you for now in triage; do you plan to work on it? (Feel free to unassign and leave a "help wanted" tag.)

[qmuntal]

@qmuntal I've assigned this to you for now in triage; do you plan to work on it?

Yep. In fact I've already submitted a bunch of CLs (not merged yet) that will make this issue easier to implement (although their are valuable on their own). Will update them referencing this issue.

[gopherbot]

Change https://go.dev/cl/466835 mentions this issue: runtime: remove unnecessary NOFRAME flags on windows

[gopherbot]

Change https://go.dev/cl/466396 mentions this issue: runtime: use explicit NOFRAME on netbsd/amd64

[gopherbot]

Change https://go.dev/cl/466457 mentions this issue: runtime: use explicit NOFRAME on plan9/amd64

[gopherbot]

Change https://go.dev/cl/466455 mentions this issue: runtime: use explicit NOFRAME on openbsd/amd64

[gopherbot]

Change https://go.dev/cl/466316 mentions this issue: runtime: use explicit NOFRAME on linux/amd64

[gopherbot]

Change https://go.dev/cl/466458 mentions this issue: runtime: remove implicit NOFRAME heuristic support

[gopherbot]

Change https://go.dev/cl/466395 mentions this issue: runtime: use explicit NOFRAME on freebsd/amd64

[gopherbot]

Change https://go.dev/cl/466355 mentions this issue: runtime: use explicit NOFRAME on dragonfly/amd64

[gopherbot]

Change https://go.dev/cl/466456 mentions this issue: runtime: use explicit NOFRAME on solaris/amd64

[gopherbot]

Change https://go.dev/cl/472195 mentions this issue: runtime: remove NOFRAME from asmcgocall and systemstack

[felixge]

👋 It's great to see your CL above. I'm currently working on adding frame pointer unwind support to the execution tracer (much faster than gentraceback), see #16638 . systemstack() not pushing frame pointers is an issue I also encountered, causing the caller frame of systemstack to be missing.

[qmuntal]

It was been quite a journey getting to that CL 🥲, glad that it also helps other effords other than my goal of enabling SEH unwinding on Windows.

Now that I almost managed to get rid of the implicit NOFRAME rule, it would be easier to have frame pointers in the runtime functions.

[nightlyone]

Please consider a release note about that semantic change that may surprise assembler code outside of the tree. @qmuntal

[qmuntal]

Please consider a release note about that semantic change that may surprise assembler code outside of the tree. @qmuntal

Sure

@aclements I'm having problems on CL 472195 trying to get TestTracebackSystemstack happy. It seems that gentraceback doesn't like the fact that systemstack now has a frame pointer. Manage to fix it, but without knowing why and probably by chance. As you are working on refactoring that gentraceback, could you provide some guidance on how to properly fix that?

[cherrymui]

FWIW, I'm still not convinced that saving frame pointers is a good idea for stack transition functions like systemstack, at least not implicitly. If we want to use the frame pointer to express stack transition, I think we may want to be very explicit.

[qmuntal]

FWIW, I'm still not convinced that saving frame pointers is a good idea for stack transition functions like systemstack, at least not implicitly. If we want to use the frame pointer to express stack transition, I think we may want to be very explicit.

Your opinion worth alot 😸. What I don't understand is what could go wrong when saving the frame pointer on functions that change the stack. And, if there are caveats, what could be worse than not having any stack unwinding at all?

[cherrymui]

I'm not sure if it could go wrong, just that as the stack transition is special enough I think it is clearer that we do it more explicitly, with comments. It may also be simpler if we save the FP only when the stack transition actually happens, and not on the tail call case.

For the traceback failure (I didn't read the code or the failure message, just guessing), could it be that the frame layout of systemstack_switch need to match? Also the comment at gosave_systemstack_switch may be helpful.

[aarzilli]

I remembered that one of the problems we had with delve's stack unwinder was the morestack/newstack path. Maybe you have already accounted for it, I thought it mention it just in case.

[felixge]

the stack transition is special enough I think it is clearer that we do it more explicitly, with comments

Not sure I follow. Are you talking about some sort of compiler directive like this?

//go:emitfp
systemstack(func() { ... })

Sorry if this makes no sense, just trying to make sure I understand your comments 😅 .

If we want to use the frame pointer to express stack transition, I think we may want to be very explicit.

FWIW, while working on CL 463835 I found that frame pointer unwinding through systemstack seems to be working already because systemstack doesn't clobber the BP register, so it gets saved by the callee of systemstack. This causes an unwinder to miss the caller of systemstack, but the stack transition itself seems to be working.

[qmuntal]

For the traceback failure (I didn't read the code or the failure message, just guessing), could it be that the frame layout of systemstack_switch need to match? Also the comment at gosave_systemstack_switch may be helpful.

Thanks for the advice, the frame layout of systemstack_switch did have to match. Tests are green now without having to modify gentraceback.

I'm not sure if it could go wrong, just that as the stack transition is special enough I think it is clearer that we do it more explicitly, with comments.

Have the same questions as @felixge, plus one data point more: arm64 versions of asmcgocall and systemstack already place the frame pointer on the stack.

It may also be simpler if we save the FP only when the stack transition actually happens, and not on the tail call case.

I tried this at some point, but it would create another (and probably bigger) issue: the FP would be stored outside the function prologue, which is non-standard. Doing so will make it difficult to programmatically generate DWARF unwinding info using the FP instead of stack deltas, and AFAIK will not work on Windows with SEH unwinding info, which requires the FP to be on the prologue, and the prologue is limited to 255 bytes.

[cherrymui]

//go:emitfp
systemstack(func() { ... })

No. I was just thinking explicitly saving FP in the assembly code. If you all prefer the implicit saving, that is fine. But I think somewhere in the code we should have a comment mentioning that the frame pointer is chained through the stack transition.

arm64 versions of asmcgocall

Not sure if that is actually correct (https://golang.org/cl/241080). It does do the implicit saving, though.

[felixge]

No. I was just thinking explicitly saving FP in the assembly code. If you all prefer the implicit saving, that is fine.

Ah, got it! I don't have a preference on that. If it's better to do this "by hand" in the assembly rather than letting the compiler generate it, that works for me. As long as the frame pointer gets pushed I'm happy 😄.

[qmuntal]

Closing this issue as done. There has been many CLs during go1.21 dev cycle to improve the frame pointer unwinding through the cgo transition, and there are even some tests verifying that there are no regressions in this area, i.e. @felixge's CL 474916.

Thanks everyone for the discussion and the reviews!

[gopherbot]

Change https://go.dev/cl/501516 mentions this issue: doc/go1.21: NOFRAME heuristic changes