path/filepath: Glob should support ** for zero or more directories

[ascarter]

Go version 1.4.2
Mac OS X 10.10

Example:

package main

import "fmt"
import "path/filepath"
import "os"

func main() {
    files, err := filepath.Glob("/usr/local/go/src/**/*.go")
    if err != nil {
            fmt.Print(err)
            os.Exit(1)
    }
    fmt.Printf("files: %d\n", len(files))
    for _, f := range files {
        fmt.Println(f)
    }
}

Expected:

% ls /usr/local/go/src/**/*.go | wc -l
    1633

Actual:

files: 732

It seems that ** is equivalent to *. The extended ** pattern is common in shells and is supported in Rust and Java for example.

[ascarter]

For reference, here is Rust's implementation. I think these rules would be a good description of how Go's filepath.Glob should work:

https://github.com/rust-lang/glob/blob/master/src/lib.rs#L369

[ianlancetaylor]

For reference, this is supported by bash when invoked as "bash -O globstar" or after running "shopt -s globstar".

[nodirt]

OK to implement? @adg

[nodirt]

@bradfitz for opinion

[ianlancetaylor]

I think this is OK to implement.

[adg]

Yep, I agree.

[gopherbot]

CL https://golang.org/cl/16280 mentions this issue.

[dotcomputercraft]

all, I wanted to check in to see if this issue (#11862) was fixed. Please let me know status of this ticket. Is there a work around to this problem? Talk to you soon. Thanks in advance.

[nodirt]

Status: currently I'm not working on this. Feel free to unassign.

[rsc]

I certainly see the utility here but adding ** has various knock-on effects that may not be obvious. For example, * matches symlinks to other directories. ** cannot, at least not without extra work to avoid file system cycles. Also it's not obvious this should be considered a backwards compatible change, both for Glob and for Match. And you'd probably want to do path.Match as well. There's a lot to consider here, even though it looks trivial.

[slimsag]

@rsc closed the most recent CL for this with the comment:

I really don't think we should do any of this. This is much less understandable than the current code. If you need a very fast, very complex filepath.Glob, it is of course possible to write one outside the standard library.

Should this issue remain open?

[rsc]

This issue does remain open. What's confusing is that GitHub has linked just above your comment to issue Shopify/themekit#102, which is closed.

[slimsag]

@rsc that is because someone linked to it in a Markdown link / comment: Shopify/themekit#102 (comment)

What can be done here? A better implementation? It's not clear to me.

[extemporalgenome]

I think it's useful, but I wouldn't consider it backwards compatible. ** already works and does something (which is the same something as a single star).

[ascarter]

@extemporalgenome At best, I'd argue that ** is undefined and it just happens to do the same thing as *. I don't think this should represent an issue in regards to backwards compatibility.

[ascarter]

@slimsag the shopify issue I think is different. They chose to walk the tree and that solved their case - I don't think that applies to the general problem. I think @rsc was saying that the solution here is non trivial (and it's not just filepath.Glob but also path.Match).

I'd like it to stay on the list. Coming from other languages, I think this is a pretty common feature for glob syntax.

Note that the downside of attempting to write your own implementation is that I think you would wind up re-implementing glob entirely. I'm trying to figure out how to solve this for http://github.com/jstemmer/gotags since it won't be solved in the standard library anytime soon.

[extemporalgenome]

@ascarter * is very well defined as matching any sequence (including an empty sequence) of non-separator characters. Thus ** is equally well defined as matching zero or more non-separator characters, followed by a match of zero characters on the tail of the same filename.

The proposed special-casing of ** will certainly not match less than it does now, but may match more, and the question is whether existing programs could become broken through what constitutes a change in behavior.

[mattn]

I'm thinking this feature should be provided from third-party products not core.
for example, you can use https://github.com/mattn/go-zglob

[soypat]

Could this be needed to match only main.go, hey.go files in the following case?

- unwanted.go
- dir1
| - main.go
| - dir2
   | - hey.go

• Glob("dir1/*.go") only matches main.go
• Glob("dir1/*/*.go") only matches hey.go
• Glob("*.go") matches them all, including the unwanted unwanted.go

[mih-kopylov]

It's a shame such an famous Glob feature not implemented in Go for 7 years.

@adg @rsc is there a chance to prioritize it please?

[adg]

@mih-kopylov There are some concerns that Russ outlined in an earlier comment and also there are compatibility promises that we must heed. So if someone wants to address this issue they need should probably write it up as a proposal that addresses all these potential issues.

In the meantime, if there is a (seemingly, from my brief research) mature doublestar package that provides a glob that handles **. It doesn't have any dependencies and has extensive unit tests. Give it a shot.

[guillermo]

I needed something more straightforward than the solution from @adg without adding an external lib. Maybe this is enough for most people:

https://github.com/guillermo/doubleglob/blob/main/double_glob.go

var replaces = regexp.MustCompile(`(\.)|(\*\*\/)|(\*)|([^\/\*]+)|(\/)`)

func toRegexp(pattern string) string {
	pat := replaces.ReplaceAllStringFunc(pattern, func(s string) string {
		switch s {
		case "/":
			return "\\/"
		case ".":
			return "\\."
		case "**/":
			return ".*"
		case "*":
			return "[^/]*"
		default:
			return s
		}
	})
	return "^" + pat + "$"
}

// Glob returns a list of files matching the pattern.
// The pattern can include **/ to match any number of directories.
func Glob(inputFS fs.FS, pattern string) ([]string, error) {
	files := []string{}

	regexpPat := regexp.MustCompile(toRegexp(pattern))

	err := fs.WalkDir(inputFS, ".", func(path string, d fs.DirEntry, err error) error {
		if d.IsDir() || err != nil {
			return nil
		}
		if regexpPat.MatchString(path) {
			files = append(files, path)
		}
		return nil
	})

	return files, err
}

[claytonchew]

Was wondering why ** doesn't work until I came across this issue.

To see that such famous ** glob pattern wasn't considered since 2015. I hope it gets prioritised.

The use case for this is plain obvious and should not introduce backward compatibility issue.

Instead of having to specify glob patterns like this to match all directories and subdirectories within a directory like this:

dir/*
dir/*/*
dir/*/*/*
dir/*/*/*/*

☝️ another limitation: only allows matches up to 4 nested levels

It could have done away with this instead:

dir/**/*

[benstigsen]

This issue has been open since 2015, but it is not clear whether adding support for ** is planned or considered.