Packr embed silently fails when project dir is a symlink

[jgillich]

Gitlab CI does not support custom checkout directories (like in $GOPATH), which meant my application directory was a symlink. Which seemed to work just fine, but the binary lacked the embedded files. For example:

$ packr -i opencl
$ ls opencl
opencl-packr.go
# all good
$ go build
$ ./myapp
# ok
$ rm -rf opencl
# let's try this again wiithout the files on disk
$ ./myapp
panic: file does not exist

Real build log here: https://gitlab.com/blockforge/blockforge/-/jobs/50043767 (relevant portion at the bottom)

As a workaround, I've symlinked the parent directory instead, and it's working now. Not sure if there's anything that can be done about this.

[AaronOmanLendesk]

This just bit me, too. We aren't primarily a Go shop, so $GOPATH doesn't conform with the way we typically lay out our project directories. As a result, we symlink our Go projects into our regular workspaces.
If I hadn't found this issue I would have been scratching my head for some time!

[jgillich]

I wasted like two hours today wondering why my Gitlab builds were broken, glad I saved you from that haha.

I just looked into this a bit more. With the project being a symlink, this is what's generated:

packr.PackJSONBytes("opencl", "amd.cl", ...

With a regular folder:

packr.PackJSONBytes("gitlab.com/blockforge/blockforge/opencl", "amd.cl" ...

So the package path is just wrong, which explains why lookup at runtine fails even though the embed files are being generated.

[markbates]

Go doesn't like symlinks for GOPATHs, as you've learned. :)

@AaronOmanLendesk a lot of people new to Go do that, and that ends up hurting them more than helping them. I would recommend that you buy into the concept of the GOPATH as the Go team expects it to be used. You'll have a lot less headaches and bugs to deal with it.

@jgillich can you provide a simple example to reproduce the issue? maybe we can find a way to either make it work, or at the very least, return an error about it.

[jgillich]

Sure, create the following main.go at $GOPATH/foo:

package main

import (
	"fmt"

	"github.com/gobuffalo/packr"
)

func main() {
	box := packr.NewBox(".")
	str, err := box.MustString("main.go")
	if err != nil {
		panic(err)
	}
	fmt.Println(str)
}

Then symlink it to another package and try running it without the original file existing (just using the go file itself as an example, the file could be anything):

ln -s $GOPATH/foo $GOPATH/bar
cd $GOPATH/bar
packr build
./bar # this should work
mv main.go main.go.bak
./bar # this panics

In this example, the box directory itself is a symlink, while in my original case it was a real directory and the parent (main project dir) was a symlink.

[markbates]

That is a strange little bug, isn't it? I couldn't reproduce on my Mac, but in a docker container I had problem.

The file is never actually embedded it looks like. If you run just packr in the foo directory it creates a foo-packr.go file, but it doesn't do that in the bar directory.

Apparently the https://golang.org/pkg/path/filepath/#Walk function doesn't follow symlinks, which is what's happening.

Going to have to see if there's an alternative out there.

[markbates]

I just posted a PR that I think fixes the issue #40 I would love if one of you wouldn't mind trying it out to make sure.

[jgillich]

Now you'll get the path of the resolved directory in the box file, which still fails at runtime if the file isn't on disk. For example, when I move the original package out of $GPATH, I get:

packr.PackJSONBytes("/home/jgillich/packr-demo

I'm not sure how the path is generated, but it should be based on the relative location from $GOPATH and not on the path on disk, right? Since that's what it uses at runtime to look it up.

The file is never actually embedded it looks like.

That is most likely a separate but similar issue, because it is being generated when you do packr -i . - just the paths in the file are wrong.

[markbates]

Got it. The problem seems to be packr is expecting you to be in $GOPATH/src/... but you're in $GOPATH/.... Let's see what we can do about that, between the symlinks and the not being in $GOPATH/src/..., this is definitely stepping out of the bounds of normal Go usage, but let me see what I can do.

[markbates]

The fix for this is elsewhere, in a different package.

$ go get -u github.com/markbates/inflect

Then recompile packr from this branch. I believe that should fix it, but I want to make sure before I merge.

Thanks.

[jgillich]

No sorry that's my bad, of course I'm in $GOPATH/src. Well, at least my symlink is.

As I said above, GitLab CI checks out to a path like:

/builds/org-name/repo-name

This directory is completely outside $GOPATH. You cannot change that, and moving the files breaks a lot of GitLab CI's other features like artifact upload and caching, so the easiest solution is to soft link it. This is what I originally did:

ln -s /builds/org-name/repo-name $GOPATH/src/gitlab.com/org-name/repo-name

This works completely fine with the rest of the Go ecosystem, but not with packr:

cd $GOPATH/src/gitlab.com/org-name/repo-name
packr -i some-box-dir
go build

The resulting binary now does not work on any other system because packr cannot find the files.

You should still be able to reproduce my example above with the files being in $GOPATH/src, that was only a typo by me.

[markbates]

Ok, Gotcha!

I didn't realize you were trying to use packr -i this whole time. I thought you cd into the build directory and ran packr install or packr build.

I didn't even look at the flags, because I was focused, originally on the symlink issue.

Yes, -i doesn't seem to work. That should probably be a different issue/request for that.

That would also explain why I struggled with trying to reproduce I was running packr build instead of packr -i.

I've merged in the symlink PR, since it did fix that issue. I'm going to close this issue and make a new focusing on adding -i support.

[jgillich]

Still can't get it to work though, even without -i. Here's a working example this time: https://github.com/jgillich/packr-demo

Just clone and do docker build .. While the box file is being generated, the file path is still wrong.

[markbates]

Thanks, that docker file is a huge help! I'll let you know when I have something.

[markbates]

@jgillich I've rolled back the v2 changes, as there is clearly issues with it. after rolling back your Dockerfile example works perfectly. Thanks for that, it was really helpful.