cmd/go: plugin versions do not match when built within vs. outside a module

[zimnx]

What did you do?

I have simple application that loads Go plugins and allows them to communicate with app via exported interface.

Simplified version can be found here.
Shared interface is stored under ext directory, real implementation is located underplugin directory.
Application passes pointer to real implementation to dynamically loaded plugin which expects interface.

Example plugin code can be found here.

Unfortunately combination of Go Modules and Go Plugins doesn't work unless go.mod of plugins module has replace entry with local relative path for shared interface path.
When I use remote location in plugins module, application crashes because of wrong version of packages.

Build app:

~/tmp/go-plugin-bug/central on  master ⌚ 16:12:19
$ go clean -modcache 

~/tmp/go-plugin-bug/central on  master ⌚ 16:12:21
$ git checkout v1.0.0
Note: checking out 'v1.0.0'.

~/tmp/go-plugin-bug/central on  f1d7e9f ⌚ 16:12:22
$ go install -a

Then build plugin

~/tmp/go-plugin-bug/central on  f1d7e9f ⌚ 16:12:27
$ cd ../plugins 

~/tmp/go-plugin-bug/plugins on  master ⌚ 16:12:28
$ cat go.mod 
module github.com/zimnx/plugins

go 1.12

require github.com/zimnx/central v1.0.0


~/tmp/go-plugin-bug/plugins on  master ⌚ 16:12:30
$ go build -buildmode=plugin -o plugin.so
go: finding github.com/zimnx/central v1.0.0
go: downloading github.com/zimnx/central v1.0.0
go: extracting github.com/zimnx/central v1.0.0

~/tmp/go-plugin-bug/plugins on  master ⌚ 16:12:39
$ central plugin.so 
2019/04/08 16:12:42 cant open plugin: plugin.Open("plugin"): plugin was built with a different version of package github.com/zimnx/central/ext

When I change plugins go mod to use local path instead of remote one everything works.

~/tmp/go-plugin-bug/plugins on  master ⌚ 16:12:42
$ cat go.mod                             
module github.com/zimnx/plugins

go 1.12

require github.com/zimnx/central v1.0.0

replace github.com/zimnx/central => ../central

~/tmp/go-plugin-bug/plugins on  master! ⌚ 16:14:30
$ go build -buildmode=plugin -o plugin.so

~/tmp/go-plugin-bug/plugins on  master! ⌚ 16:14:33
$ central plugin.so 
hello = world

What did you expect to see?

Go Modules and plugins working fine when remote path is used.

What did you see instead?

Error about different package versions.

Does this issue reproduce with the latest release (go1.12.3)?

Yes.

System details

go version go1.12.3 linux/amd64
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/maciej/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/maciej/work"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/maciej/tmp/go-plugin-bug/plugins/go.mod"
GOROOT/bin/go version: go version go1.12.3 linux/amd64
GOROOT/bin/go tool compile -V: compile version go1.12.3
uname -sr: Linux 4.14.13-041413-generic
Distributor ID:	Ubuntu
Description:	Ubuntu 17.10
Release:	17.10
Codename:	artful
/lib/x86_64-linux-gnu/libc.so.6: GNU C Library (Ubuntu GLIBC 2.26-0ubuntu2.1) stable release version 2.26, by Roland McGrath et al.
gdb --version: GNU gdb (Ubuntu 8.0.1-0ubuntu1) 8.0.1

[zimnx]

Bump. Any plans to fix this? Seems that algorithm which verifies package version integrity needs alignment.

[bcmills]

This is the same underlying problem as #29814.

The workaround should be to build both the plugin and the main binary from outside of their respective repositories, or (as you note) to use a replace directive when building locally.

[bcmills]

CC @jayconrod

[bcmills]

See also #31278, of which this may be a duplicate.

[juhwany]

@bcmills

Is this issue related to #16860 ?

Building plugin and main binary with -trimpath flag doesn't solve this issue even using a go toolchain built from head.

[xxxserxxx]

There are several similar tickets; apologies if this is not the right place to comment.

If the path is different but the version of the libraries are identical, we get version mismatch errors. A way of replicating this is to build the calling program in a container, and build the plugin outside of the container. The paths may look like this:

# Calling
/go/pkg/mod/github.com/shirou/[email protected]+incompatible/internal/common/common.go
# Called
/home/ser/go/pkg/mod/github.com/shirou/[email protected]+incompatible/internal/common/common.go

Building locally, and where:

• the plugin is built using a go.mod with a specific date+revision version for the calling main program, and
• the calling application is built from a checkout at that version, verified that the version in the plugin go.mod matches the checked-out version with:

echo $(TZ=UTC git show -s --format=%cd --date=format-local:%Y%m%d%H%M%S HEAD | cut -b -19 |  tr -cd '[:digit:]')-$(git rev-parse HEAD | cut -b -12)

• and both programs being compiled with the same go executable, and
• using --trimpath on both, and
• both go.mod list the same go version dependency
  I get the same version incompatibility error with a different package:

# Error:
2:31:33 main.go:485: plugin.Open("dummy"): plugin was built with a different version of package github.com/xxxserxxx/gotop/v3/devices
$ #######################################################
$ strings ./gotop | grep 'v3/devices$'
go.link.pkghashbytes.github.com/xxxserxxx/gotop/v3/devices
go.link.pkghash.github.com/xxxserxxx/gotop/v3/devices
%github.com/xxxserxxx/gotop/v3/devices
go.link.pkghashbytes.github.com/xxxserxxx/gotop/v3/devices
go.link.pkghash.github.com/xxxserxxx/gotop/v3/devices
$ #######################################################
$ strings dummy.so | grep 'v3/devices$'
go.link.pkghashbytes.github.com/xxxserxxx/gotop/v3/devices
go.link.pkghash.github.com/xxxserxxx/gotop/v3/devices
github.com/xxxserxxx/gotop-dummygithub.com/xxxserxxx/gotop-dummygithub.com/xxxserxxx/gotop/v3/devices
%github.com/xxxserxxx/gotop/v3/devices
go.link.pkghashbytes.github.com/xxxserxxx/gotop/v3/devices
go.link.pkghash.github.com/xxxserxxx/gotop/v3/devices

I've played with copying the go.sum into the plugin directory so that it's identical to the other project, and have made sure that the entries in both go.mods are the same (except for the plugin's dependency on the main program).

If I add the following to the plugin's go.mod:

replace github.com/xxxserxxx/gotop/v3 => ../gotop

then it works.

Edit

Rewrote some of the text to be more readable, I hope.

Edit 2

I made an error: I need to remove the -trimpath argument in addition to adding the replace directive in the go.mod, or else I still get a version incompatibility error. -trimpath appears to make things worse.

[xplodwild]

Same thing happens for me for a go.mod-based project regarding the -trimpath option. When building both the main app and the plugin with -trimpath enabled, I get a plugin was built with a different version of package github.com/me/myapp/mypackage. Removing -trimpath makes it load fine, however I'm afraid people with other setups might have trouble getting their plugins loaded due to paths inconsistencies.

[xxxserxxx]

Another data point which I can't even begin to interpret:

$ git clone https://github.com/xxxserxxx/gotop
$ git clone https://github.com/xxxserxxx/gotop-dummy
$ cd gotop && git checkout v3.5.0
$ go build -o gotop ./cmd/gotop
$ cd ../gotop-dummy && grep gotop go.mod
require github.com/xxxserxxx/gotop/v3 v3.5.0
$ go mod download
$ go mod vendor
$ go build --buildmode=plugin -o ../gotop/dummy.so .
$ cd ../gotop
$ ./gotop -X dummy    # this hangs for some reason; ^C to break
$ cat ~/.local/state/gotop/errors.log
13:03:16 main.go:485: plugin.Open("dummy"): plugin was built with a different version of package github.com/shirou/gopsutil/internal/common
$ go mod graph | grep gopsutil
github.com/xxxserxxx/gotop/v3 github.com/shirou/[email protected]+incompatible
➜  gotop git:(be42ba5) ✗ git log -1
commit be42ba538cefc892d1dc3d18c783483f4875baff (HEAD, tag: v3.5.0)
$ cd ../gotop-dummy
$ go mod graph | grep gopsutil
github.com/xxxserxxx/gotop/[email protected] github.com/shirou/[email protected]+incompatible

These are the identical dependencies. Why is the system claiming that they are not?

[hoijui]

This is the same underlying problem as #29814.

The workaround should be to build both the plugin and the main binary from outside of their respective repositories, or (as you note) to use a replace directive when building locally.

do you mean to say, that the ext folder should be its own module, separate from the main app?
would that work?

[mattn]

The current plugin mechanism does not allow you to avoid mixed package hashes. For example, this error occurs when a main application that references package A loads plugins that references A. I often use gophernotes to try to write notebook with Go. gophernotes reference mattn/go-runewidth. So gophernotes can not import mattn/go-runewidth on the notebooks. I have to disable Go code to avoid this.

diff --git a/src/runtime/plugin.go b/src/runtime/plugin.go
index cd7fc5f848..ace1b13ed9 100644
--- a/src/runtime/plugin.go
+++ b/src/runtime/plugin.go
@@ -48,12 +48,14 @@ func plugin_lastmoduleinit() (path string, syms map[string]interface{}, errstr s
                        throw("plugin: new module data overlaps with previous moduledata")
                }
        }
-       for _, pkghash := range md.pkghashes {
-               if pkghash.linktimehash != *pkghash.runtimehash {
-                       md.bad = true
-                       return "", nil, "plugin was built with a different version of package " + pkghash.modulename
+       /*
+               for _, pkghash := range md.pkghashes {
+                       if pkghash.linktimehash != *pkghash.runtimehash {
+                               md.bad = true
+                               return "", nil, "plugin was built with a different version of package " + pkghash.modulename
+                       }
                }
-       }
+       */

        // Initialize the freshly loaded module.
        modulesinit()

[devsergiy]

@rsc Hi Russ,
Is this issue could be fixed in some foreseen future?

[bcmills]

A fix for this issue would require changes to the compiler (to prevent cross-package inlining when building plugins), and probably the linker (to interpret and record a signature for the package ABI). That's possible, but it would be a lot of work to support a fairly esoteric build mode.

@jeremyfaller could perhaps speak to whether the compiler and linker folks are planning to implement that support, but to my knowledge it's not on the roadmap at this time.

[jeremyfaller]

@bcmills's knowledge is correct. We've kicked around improving this for some time, but those things are just thoughts. It's a complicated problem, and it's almost certainly not on the roadmap for 2022.

[marcusholmessumup]

just ran into this problem and cannot get it working even with a replace directive in the plugin's go mod file.

[septatrix]

A fix for this issue would require changes to the compiler (to prevent cross-package inlining when building plugins), and probably the linker (to interpret and record a signature for the package ABI). That's possible, but it would be a lot of work to support a fairly esoteric build mode.

This is mostly a chicken and egg problem where many projects are not utilizing plugins due to their complicated build setup and other limitation (version incompatibility, requiring a main packet, linux only) while the plugin system does not get improved because very few projects are using it. However the demand certainly exists.

[antst]

If look around, there are plenty of incomplete and partially-working attempt to deal with plugins, just because current system is way too restricting. At the end, a lot of efforts lost for nothing and 1) adoption of current plugins system is subpar, and 2) people waste time re-introducing wheel.
I think, for most of it, if we introduce some kind of "unsafe" switch which disables next lines in src/runtime/plugin.go

	for _, pkghash := range md.pkghashes {
		if pkghash.linktimehash != *pkghash.runtimehash {
			md.bad = true
			return "", nil, "plugin was built with a different version of package " + pkghash.modulename
		}
	}

giving user options to make unsafe (and unrecommended) choice and deal with potential fallout on it's own, adoptions of plugin system will jump tremendously, which on it's own will lead to improvement of it. Solving chicken and egg issue from above.

Of course, it is not even close to be real solution. But rather temporary patch. But it give chance to actually use it.

In a current form it is not plugin system, it is merely lazy loading. The only realistic option to ensure than plugin will load, is to build is a part of source tree of the main code as part of the same compilation process.

or just rename to "lazy_load" to stop confusing people :)

[mark4z]

I try to import this and finally give up, maybe try the wasm.

[voodooEntity]

So its 2023 and i did run in the same issue as the rest here. I have a repostiory for my application and a respostiory for the plugin code. The idea was to write an application that can be fed by custom developed plugins for execution. Tho even considering the workarround with the alias it would be quite uncomfortable for people to use Plugins and this way of extending the apllication than.

I think this definatly should be considered to be solved since the net is allover filled with people hitting this issue and this thread now is ongoing for about 4 years with people asking for a solution.

[sridgeway7]

A fix for this issue would require changes to the compiler (to prevent cross-package inlining when building plugins), and probably the linker (to interpret and record a signature for the package ABI). That's possible, but it would be a lot of work to support a fairly esoteric build mode.

@jeremyfaller could perhaps speak to whether the compiler and linker folks are planning to implement that support, but to my knowledge it's not on the roadmap at this time.

Hi there, just wanted to see if it would be possible to get an update on this since, AFAICT, there has been a good amount of development in this area with the versions released after this comment (1.16+). Particularly since, in looking through the 1.16 release notes, I found this document outlining the design goals for linker improvements, particularly with this section which relates to this issue.

In using Go for ~7 years now, I've definitely reached for the plugin package a few times, but had to give up due to the difficulty in getting a plugin to load, so I can definitely echo that the demand here may seem artificially low due people either giving up on trying to use it, or finding an alternative like hashicorp/go-plugin (the popularity of which may be a better indicator for demand for this functionality) for non performance-critical applications.

Are there any public documents on what it would take to better support the plugin package (either as a quick fix with sharp edges, or a formal fix)? Is this something the community could help with at all, or is there still a fundamental limitation in the compiler/linker that the go maintainers would need to address first?

Thanks, as always, for your time and consideration 🙏

EDIT: To clarify that final question: is this a situation where the go maintainers have an idea for how this should be implemented, and its just a matter or prioritization/implementation? Or is it still up for debate how this functionality should be implemented?

[ianlancetaylor]

My personal opinion is that plugins need to be rethought completely. The current approach absolutely requires using the exact same version of Go to build the plugin and the application. That restriction makes plugins significantly less useful. Basically plugins were implemented as "this is something we can do with existing technology" rather than "this is something that people really need."

So I think we need to go back to the beginning and decide what plugins are for, and then think about how to implement them. And we have to understand why plugins are better than simply executing a separate program and passing data back and forth on standard input and output.

To be clear, this issue is not the place for any of those discussions.

[e-nikolov]

I think in the current design, the memory layouts of all plugins are "merged" into the main program. Meaning that a global variable in the main program (e.g. http.DefaultClient, or shared.GlobalVariable) can be modified by the plugins as well.

This causes some of the usability issues with the plugin package like the requirement for the main program and all plugins to have the exact same versions of all shared dependencies. Otherwise the memory layouts could get misaligned if there are changes like variables or new function definitions.

When it comes to applications that want to support third party plugins, the shared memory layout is probably not really necessary and the plugin package is not a good fit.

An alternative might be to compile the Go plugins with -buildmode=c-shared and then dynamically load them in the main program via CGO and dlopen.

Some examlpes:

• https://github.com/e-nikolov/go-plugins - loading Go plugins from Go code via C
• https://github.com/vladimirvivien/go-cshared-examples - loading Go plugins in other languages

[kevin-postman]

It is interesting to see this issue come up so many times over the past few years (or longer) and yet.. little to no movement by the go team to fix this properly once and for all. The use of plugins if properly implemented allows any application that supports it the ability to be extended dynamically outside the context of the original creators of the application. Applications like IDEs, DAWs, and more make great use of extensibility in this way and provides a path of enhancement the original developers may not thought of or had the manpower/etc to do themselves. Similar to an open source project where many contribute.

I've seen in Rust a similar issue, where by code built as a dynamic loadable module using Rust's ABI is "unstable" and only able to be loaded with specific versions that the app and module are compiled in. Sound familiar? Rust (and Zig) seem to have the ability to also load stable C ABI module, and compile to them however, though I won't pretend to be an expert in how all that works and if it's as good and easy to use.

As far as I can tell, this is the one built in area of Go that was never complete as there is no Windows implementation (not sure if ARM is supported?). So it's like a half baked piece of work that the Go team gave up on and I am surprised it was never deprecated given its lack of support on Windows and last I read (some time ago) not fully working on Mac. Maybe that changed?

The Go team should figure out some sort of stable ABI option like Rust/zig have, but lets make it easier to use.. following how easy Go typically is to use. Make it so we do something like go build -plugin ... and we do NOT have to deal with any version issues. There should be no reason an app build in go 1.22 (or later since I will assume this "updated" idea I have here wouldn't be possible to make it in 1.21 if even 1.22) can't load a plugin built in go 1.23, 1.37, etc. As long as the tool/process builds it the same way, we developers shouldn't have to do a bunch of fudgery crap to make it work across versions.

Until then, I don't feel like there is much use in the built in plugin concept with the version mismatch issues. I am not sure if it works across minor versions.. but how awful is it that you build an app in go 1.19 and a plugin compiled in go 1.19.1 or even 1.20 does not load/work in the app? It makes no sense really.. unless the idea by the go team who started on this area figured it would only be used for in house use cases and thus no real need to make it work across versions. Granted, I wont pretend to know everything about why it doesn't work, only that if it was truly impossible to do (which it isnt since rust/zig/c are able to do it), why put it in half baked and now see people trying to use it but running in to issues.

[ExtraE113]

I've also run into this issue many times in 2024.