proposal: Go2: goroutines execute defer statements when main exits

[songmelted]

Currently, Goroutines do not execute defer statements when main exits.

https://play.golang.org/p/pBWU5gcnA4C

package main

import (
	"fmt"
)

var intChan chan int

func main() {
	intChan = make(chan int)
	go worker()
	intChan<-4
	close(intChan)
}

func worker() {
	defer fmt.Println("clean up before exiting worker") //never gets executed
	for msg := range intChan {
		fmt.Println(msg)
	}
}

It seems reasonable for Goroutines to exit and execute defer statements as part of main exiting.

This would be a straightforward way guarantee any clean up is performed before main exits.

Are there compelling reasons not to do this?

[smasher164]

That defer statement does get executed. It’s just that the process exits before the worker goroutine finishes, which is to be expected. Add a sleep after closing the channel and you’ll see what I mean.

As such, this issue seems to be a question about how to use Go, rather than a feature request or defect report about the Go language and/or toolchain. For questions about Go, please visit:

• The golang-nuts mailing list.

• Stack Overflow, with questions tagged go.

• The Go Forum, a web-based forum hosted by GoBridge.

• The Gophers Slack hosted by GopherAcademy; use the invite app for access.
  The #general channel is a good starting point.

• Go Community on Hashnode with questions and posts tagged with "go"

• IRC channel #go-nuts on Freenode

Thanks

[songmelted]

@seankhliao I appreciate the perspective.

That defer statement does get executed. It’s just that the process exits before the worker goroutine finishes, which is to be expected. Add a sleep after closing the channel and you’ll see what I mean.

My issue isn't with defer statements being called after a blocking channel is closed (causing a Goroutine to exit). As you suggest, I can put a sleep after that point in time. It is with defer in Goroutines not being called when main exits. There is nowhere to put a sleep call at or after the main exiting that would allow a Goroutine to execute defer statements.

I think it is sensible for the following to execute the Goroutine defer statement. Note that a sleep call has been added in the very last place it could possibly be and does not allow the Goroutine's defer statement to execute.

https://play.golang.org/p/OXU92BBl4Yo

 package main

import (
	"fmt"
	"time"
)

var intChan chan int

func main() {
	defer time.Sleep(8 * time.Second) 
	intChan = make(chan int)
	go worker()
	intChan<-4
	//close(intChan) //removed to illustrate a point
}

func worker() {
	defer fmt.Println("clean up before exiting worker")//never gets executed
	for msg := range intChan {
		fmt.Println(msg)
	}
}

https://blog.golang.org/defer-panic-and-recover

A defer statement pushes a function call onto a list. The list of saved calls is executed after the surrounding function returns. Defer is commonly used to simplify functions that perform various clean-up actions.

From my inexperienced viewpoint, it seems incongruent to have a keyword "defer" which is "commonly used to simplify clean-up activities" not work with Goroutines when main exits.

Go should exit all Goroutines and let defer statements run prior to exiting main. This is how a defer statement within main acts, why would any other defer statement work differently? (Not in a technical sense as they all obviously follow the "same" set of rules, but in a descriptive and philosophical sense "defer executes after the end.............except when it doesn't?"

Is there a compelling reason why it isn't this way?

[smasher164]

Is there a compelling reason why it isn't this way?

A goroutine may be in an infinite loop when the process exits. How would it reach a deferred function call?
The purpose of defer is to be scoped to the function-level, not process-level. Go doesn't have an equivalent of atexit.

[ianlancetaylor]

I agree that this seems impossible. It would mean that on a call to os.Exit we would have to interrupt every existing goroutine and let it exit first. But that goroutine might be running C code or be stuck in a system call, so that seems infeasible. If we just run the defer functions without interrupting the goroutine then it may not be in a valid state for running the deferred functions.

[songmelted]

There is a feeling associated with defer that this function will definitely execute.

Deferring a call to a function such as Close has two advantages. First, it guarantees that you will never forget to close the file, a mistake that's easy to make if you later edit the function to add a new return path. Second, it means that the close sits near the open, which is much clearer than placing it at the end of the function.

Consider the following code:

package main

import (
	"fmt"
)

func main() {
	defer fmt.Println("main defer")
	
	go goRoutine()
	
	function()
	
	//function body
	
	panic("main panic")
}

func goRoutine() {
	defer fmt.Println("Goroutine defer")
	
	//function body
	
	panic("Goroutine panic")
}

func function() {
	defer fmt.Println("function defer")
	
	//function body
	
	panic("function panic")
}

Naively, I would expect a defer in main(), and Goroutine() to execute regardless of where the panic is called (once Goroutine() has been called).

But I find that neither always run. This seems contrary to the feeling Go gives me.

If the panic is called from inside function(), the deferred statements of both function() and main() and get called.

But if the panic is called within goRoutine(), the deferred statement in goRoutine() is called, but not for main().

And, of course, if the panic is called by main() while goRoutine() is running, the deferred statement in main() is called, but not for goRoutine().

@smasher164

A goroutine may be in an infinite loop when the process exits. How would it reach a deferred function call?
The purpose of defer is to be scoped to the function-level, not process-level. Go doesn't have an equivalent of atexit.

I do not have the expertise to suggest how this would work. But I am questioning if it should work. There are always clever people who accomplish things I would have been certain were impossible.

To me, "can it" and "should it" are orthogonal questions. Of course, there are situations where something should be, but also cannot be.

Thinking out loud:

If it becomes certain that it is too difficult to guarantee a Goroutine will execute defer statements when the program is terminated elsewhere, is it worthwhile just to try and report the result? What if, when the program exits, it sends a please terminate now signal to the Goroutines. If a Goroutine does not return, exit with a warning Disorderly shutdown: not all Goroutines could exit prior to the program terminating?

[ianlancetaylor]

It will always be the case that in exceptional circumstances defer functions will not run. If someone pulls the plug on the computer, they will not run. If the program is killed by a SIGKILL signal, they will not run. So what we are talking about is not some way to ensure that they will always run, but the exact circumstances in which they will run.

Personally I do not think it would be a good way to notify goroutines on program exit and wait for them to respond. Many server programs have tens of thousands of goroutines. We want programs to exit in microseconds, not seconds.

Taking this together, if it is essential for some operation to occur when a program exits, that should be handled outside of the program.

[songmelted]

@ianlancetaylor When I read about defer for Go, it seems like a clear use case is simple, orderly, clean-up almost regardless of how the program exits. Honestly, not having dug deeply into it, I expected a Goroutine to always execute a defer statement (except, of course, the most extreme cases). I do not consider main exiting while a Goroutine is running an "extreme" circumstance.

Suppose I have a Goroutine that consumes data generating by workers and writes them to file. The canonical example of defer seems to be closing a file. I cannot open the file in main and defer its close there - if a Goroutine causes a termination, it won't run (does that seem right to you?) I cannot open it in the Goroutine and defer the close there - if main (or anything else) causes an exit, it won't run.

Of course I can bake in a bunch of flow controls and error / panic handling to make sure the file is closed. But, isn't defer meant to be the easy way to do this?

Isn't it weird that defer doesn't play well with raison d'etre of Go?

I am sure, since "this is the way it was designed / has always been" this behavior seems very reasonable to seasoned veterans. But what do you think about it if you pretend to have fresh eyes?

Obviously, there are going to be some practical considerations, such as the ones you bring up. But can we overcome those issues in a way that allows defer to be a great way to close open files (etc) except in the most exceptional of circumstances?

Wouldn't it be nice if I could open a file in a Goroutine, defer its close and be done? Instead of adding a bunch of boilerplate to be able to signal to the routine that main is going to exit and it should close the file? Or instead of opening the file in main even though the file is used nowhere else?

[ianlancetaylor]

All files are closed anyhow when a program exits, so that doesn't seem like a helpful example.

I'm not trying to argue that defer should work a certain way because it has always worked that way. I'm arguing that defer was never intended to solve the problem of "this code must always run," because, as discussed above, that problem can't be solved within a single process. We've always been very aware of that fact when developing Go.

Another problem we've always been aware of is that many C++ programs take a long time to exit, because they must run all global destructors. This is such a well-known problem that C++ introduced a way to avoid that problem: a quick_exit function (https://en.cppreference.com/w/cpp/utility/program/quick_exit). We explicitly want to avoid that problem for Go programs by not providing any equivalent to C++ global destructors (and not providing any equivalent to C/C++ atexit).

I believe that any changes made here should be made considering those potential problems.

[songmelted]

@ianlancetaylor I really appreciate you taking the time to share your expertise here. It means a lot to me.

All files are closed anyhow when a program exits, so that doesn't seem like a helpful example.

Well, you got me there. I should have chosen a better example.

I'm arguing that defer was never intended to solve the problem of "this code must always run,"

Can you share with me what the motivation for defer is? While looking into to this, I read some articles about defer, the origins of Go, and the "philosophy" of Go to try to figure out what I was missing. I was still left with my question afterward.

Another problem we've always been aware of is that many C++ programs take a long time to exit

We explicitly want to avoid that problem for Go programs by not providing any equivalent to C++ global destructors (and not providing any equivalent to C/C++ atexit).

Can you point me to a conversation that discusses this in more detail (preferably written with a reasonably general audience in mind)?

[bcmills]

Programs today may rely on the fact that the code in a goroutine executes before its defer statements are run. Consider a very simple example:

func f() (err error) {
	errc := make(chan error, 1)
	defer func() { err = <-errc }
	errc <- nil
}

If that defer is executed before errc <- nil, it will deadlock. In contrast, under normal execution it cannot deadlock.

(Compare #41891 (comment).)

[ianlancetaylor]

Can you share with me what the motivation for defer is? While looking into to this, I read some articles about defer, the origins of Go, and the "philosophy" of Go to try to figure out what I was missing. I was still left with my question afterward.

Ken's initial motivation for the construct that became defer was a dynamic form of the try/finally construct found in some other languages. His initial proposal was catch { statements } and signal string-expression. That is, catch provided a dynamic finally clause, and signal was similar to throw. This then evolved into defer and panic, and later we added recover.

So the goal was to provide a way to execute a series of statements upon function return, typically to preserve a set of invariants across calls to the function.

(For what it's worth, in other languages a finally clause of a try/finally is not executed if the try clause exits the program.)

Can you point me to a conversation that discusses this in more detail (preferably written with a reasonably general audience in mind)?

I don't know of one, sorry.

[ianlancetaylor]

Based on this discussion above, and the lack of support based on emoji voting, this is a likely decline. Leaving open for four weeks for final comments.

[ianlancetaylor]

No further comments.