Undocumented / impossible: override go modules #86349

doronbehar · 2020-04-29T19:54:28Z

Describe the bug

I'm trying to write an overlay that will override attributes of a package that is built with buildGoModule and I'm having trouble overriding the modules derivation of it.

It seems the author's intention was that I'd use overrideModAttrs as an argument to buildGoModule? But I guess this doesn't work with overlays? Whatever I do, I get either an evaluation error or a build error because the old go-modules is used. Here are a few examples I tried:

overridedPackage = super.original.overrideAttrs(oldAttrs: rec {
  src = ....;
  modSha256 = "0000000000000000000000000000000000000000000000000000";

But the build still took the old go modules derivation.

I also tried instead:

  overrideModAttrs = (_oldAttrs : {
    modSha256 = "0r062nka72ah2nb2gf8dfrrj4sxadkykcqjzkp4c9vwk93mhw41k";
  });

Which failed with:

error: cannot coerce a function to a string, at ...

And I even went as far as:

  configurePhase = (let
    go-modules = self.pistol.go-modules.overrideAttrs(oldAttrs: {
      modSha256 = "0000000000000000000000000000000000000000000000000000";
    });
  in
    ''
      echo checking I override configure phase
      runHook preConfigure
  
      export GOCACHE=$TMPDIR/go-cache
      export GOPATH="$TMPDIR/go"
      export GOSUMDB=off
      export GOPROXY=file://${go-modules}
  
      cd "$modRoot"
  
      runHook postConfigure
    ''
  );
});

But that also failed because the build used the old go-modules. I can confirm though, that it did override the configurePhase which is the only place ${go-modules} is evaluated.

Expected behavior

It shouldn't be that difficult to override a go modules derivation.

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context

I'm trying to override a package I maintain which it's update is stale (#85452), due to an unresolved issue with the version build flags :(.

Notify maintainers

@kalbasit ? As the author of buildGoModule according to git blame...

The text was updated successfully, but these errors were encountered:

kalbasit · 2020-05-01T03:47:11Z

I have the same issue, unfortunately, and I have not been able to look into it.

kalbasit · 2020-05-06T07:34:43Z

@doronbehar this worked for me

  (self: super: {
    bazel-gazelle = (super.callPackage "${super.path}/pkgs/development/tools/bazel-gazelle" {
      buildGoModule = args: super.buildGoModule (args // {
        modSha256 = "13yx70p0x7hpdp2zn7zrl0i88fk6dhcryp0lmn0zwdn92ldm3n5y";
        patches = (args.patches or []) ++ super.lib.lists.singleton (super.fetchpatch {
          url = "https://github.com/bazelbuild/bazel-gazelle/pull/749.patch";
          sha256 = "03sxbgwvb6hxwpmk9mckwjzgxank9difc0bgvzl6x3rir76ivwmj";
        });
      });
    });
  })

doronbehar · 2020-05-16T19:43:19Z

Damn you are overriding the goddamn function. It's nice to know this is not "Impossible". Do you think @kalbasit this interface could be improved somehow?

doronbehar · 2020-09-13T13:04:18Z

@kalbasit I've stumbled upon another point to improve the golang docs at #97856 . I'm not sure whether this should be documented or not along with the other changes - overrideModAttrs doesn't seem to be able to provide what it promises.

nixos-discourse · 2020-10-06T12:15:50Z

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/inconsistent-vendoring-in-buildgomodule-when-overriding-source/9225/4

joulaud · 2021-01-14T11:10:10Z

The problem with pythonBuildPackage described at https://discourse.nixos.org/t/un-disabling-a-python-package-in-an-overlay/10593 seems similar to me. Perhaps overrideAttrs documentation should be improved to better explain those traps.

fiksn · 2021-01-16T23:06:11Z

I've also stumbled on this when trying to override some go package which involved changing vendorSha256 too.
So if I understood this correctly the problem with overrideAttrs is that it works just with a simple mkDerivation, but buildGoModule internally does more and calls mkDerivation also for the vendored dependencies?

Then I suppose in the same way as there is overridePythonAttrs there should be some overrideGoAttrs.

doronbehar · 2021-01-20T20:32:30Z

Then I suppose in the same way as there is overridePythonAttrs there should be some overrideGoAttrs.

I think we should strive to have less of such functions with different names but the same meaning.

Mic92 · 2021-06-01T11:23:40Z

This one is a bit shorter:

(pkgs.istioctl.overrideAttrs (old: let
  version = "1.7.1";
  src = pkgs.fetchFromGitHub {
    owner = "istio";
    repo = "istio";
    rev = version;
    sha256 = "sha256-CvYqYxb19lSc61SrvGcf/kG9e9zosHz4JbByA+bN600=";
  };
in rec {
  name = "istioct-${version}";
  inherit src;
  go-modules = (pkgs.buildGoModule {
    inherit name src;
    vendorSha256 = "sha256-4Z4Fgv9zmAwp3tEvHj8yLOWkFY/zFz5VfehSWCuIUcI=";
  }).go-modules;
}))

nh2 · 2021-06-14T01:17:26Z

This one is a bit shorter:

@Mic92 this doesn't work for me, trying with:

{
  kopia_unstable = nixos-2105.kopia.overrideAttrs (old:
    let
      version = "ede09c15854ce74fde40603a6857ed3991ba666c";
      src = nixos-2105.fetchFromGitHub {
        owner = "kopia";
        repo = "kopia";
        rev = version;
        sha256 = "0wdkwpsgmycbiq8dvng3d0ps5qp87b7k1xvw8rv2wk1qzrrp6d93";
      };
    in
      rec {
        name = "kopia-${version}";
        inherit src;
        go-modules = (nixos-2105.buildGoModule {
          inherit name src;
          vendorSha256 = "1idg6l9cnii6694kdmv9sc42c0bzk7pk67sdlglkaz52kg5hc6bd";
      }).go-modules;
  });
}

Error output:

these derivations will be built:
  /nix/store/d5ada6p0wl6pnz6xm4r5ws50yvha1c36-kopia-ede09c15854ce74fde40603a6857ed3991ba666c-go-modules.drv
  /nix/store/nvixyjbfji4ljlyqkhx32323ip5zq6a7-kopia-ede09c15854ce74fde40603a6857ed3991ba666c.drv
building '/nix/store/d5ada6p0wl6pnz6xm4r5ws50yvha1c36-kopia-ede09c15854ce74fde40603a6857ed3991ba666c-go-modules.drv'...
copying 1 paths...
copying path '/nix/store/mbhc8w8qxckv4j1rwz7mlaljahv6m46w-kopia-ede09c15854ce74fde40603a6857ed3991ba666c-go-modules' from 'ssh://...
building '/nix/store/nvixyjbfji4ljlyqkhx32323ip5zq6a7-kopia-ede09c15854ce74fde40603a6857ed3991ba666c.drv'...
copying 1 paths...
copying path '/nix/store/0y74vrvy799phqqwl781q9xadzqaj5ly-kopia-0.8.4-go-modules' to 'ssh://...
unpacking sources
unpacking source archive /nix/store/w9zd6h50l258cdzkmja19pfvzih51ywj-source
source root is source
patching sources
configuring
building
Building subPackage ./.
go: inconsistent vendoring in /build/source:
        cloud.google.com/go/[email protected]: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
        github.com/Azure/[email protected]: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
        github.com/aws/[email protected]: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
        github.com/dustinkirkland/[email protected]: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
        github.com/fatih/[email protected]: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
        github.com/golang/[email protected]: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
        github.com/klauspost/[email protected]: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
        github.com/pierrec/[email protected]+incompatible: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
        github.com/pkg/[email protected]: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
        github.com/sanity-io/[email protected]: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
        [email protected]: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
        golang.org/x/[email protected]: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
        golang.org/x/[email protected]: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
        golang.org/x/[email protected]: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
        golang.org/x/[email protected]: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
        google.golang.org/[email protected]: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
        google.golang.org/[email protected]: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
        cloud.google.com/go/[email protected]: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        github.com/Azure/go-autorest/[email protected]: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        github.com/aws/[email protected]: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        github.com/fatih/[email protected]: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        github.com/golang/[email protected]: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        github.com/google/[email protected]: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        github.com/klauspost/[email protected]: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        github.com/pkg/[email protected]: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        [email protected]: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        golang.org/x/[email protected]: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        golang.org/x/[email protected]: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        golang.org/x/[email protected]: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        golang.org/x/[email protected]: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        google.golang.org/[email protected]: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        google.golang.org/[email protected]: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod

        To ignore the vendor directory, use -mod=readonly or -mod=mod.
        To sync the vendor directory, run:
                go mod vendor

nh2 · 2021-06-14T01:39:36Z

The approach from #86349 (comment) worked for me (replacing modSha256 by its successor vendorSha256):

{
  kopia_unstable =
    let
      version = "ede09c15854ce74fde40603a6857ed3991ba666c";
      src = nixos-2105.fetchFromGitHub {
        owner = "kopia";
        repo = "kopia";
        rev = version;
        sha256 = "0wdkwpsgmycbiq8dvng3d0ps5qp87b7k1xvw8rv2wk1qzrrp6d93";
      };
    in
      (nixos-2105.callPackage "${nixos-2105.path}/pkgs/tools/backup/kopia" {
        buildGoModule = args: nixos-2105.buildGoModule (args // {
          vendorSha256 = "1idg6l9cnii6694kdmv9sc42c0bzk7pk67sdlglkaz52kg5hc6bd";
          inherit src version;
        });
      });
}

danielphan2003 · 2021-09-03T15:49:58Z

The approach from #86349 (comment) worked for me (replacing modSha256 by its successor vendorSha256):

{
  kopia_unstable =
    let
      version = "ede09c15854ce74fde40603a6857ed3991ba666c";
      src = nixos-2105.fetchFromGitHub {
        owner = "kopia";
        repo = "kopia";
        rev = version;
        sha256 = "0wdkwpsgmycbiq8dvng3d0ps5qp87b7k1xvw8rv2wk1qzrrp6d93";
      };
    in
      (nixos-2105.callPackage "${nixos-2105.path}/pkgs/tools/backup/kopia" {
        buildGoModule = args: nixos-2105.buildGoModule (args // {
          vendorSha256 = "1idg6l9cnii6694kdmv9sc42c0bzk7pk67sdlglkaz52kg5hc6bd";
          inherit src version;
        });
      });
}

What about using nixos-2105.kopia.override instead of nixos-2105.callPackage "${nixos-2105.path}/pkgs/tools/backup/kopia"? This works when Nix is in restricted mode (in my case I was using flake).

Also Go 1.17 is already out, and latest Tailscale requires it so it makes sense to use buildGoModule = args: (nixos-2105.buildGoModules.override { go = go_1_17; }) (args // { ... }); (waiting #127519 to merge, so we can ditch overriding Go in the first place)

It now requires esbuild, which wants to download stuff as part of install script. Fortunately, it can be bypassed by an environment variable. evanw/esbuild#881 Also, annoyingly, Nixpkgs’s Go infrastructure does not support overriding attributes of buildGoPackage: NixOS/nixpkgs#86349

evanjs · 2021-10-17T22:58:48Z

Original Comment

The approach from #86349 (comment) worked for me (replacing modSha256 by its successor vendorSha256):
{
  kopia_unstable =
    let
      version = "ede09c15854ce74fde40603a6857ed3991ba666c";
      src = nixos-2105.fetchFromGitHub {
        owner = "kopia";
        repo = "kopia";
        rev = version;
        sha256 = "0wdkwpsgmycbiq8dvng3d0ps5qp87b7k1xvw8rv2wk1qzrrp6d93";
      };
    in
      (nixos-2105.callPackage "${nixos-2105.path}/pkgs/tools/backup/kopia" {
        buildGoModule = args: nixos-2105.buildGoModule (args // {
          vendorSha256 = "1idg6l9cnii6694kdmv9sc42c0bzk7pk67sdlglkaz52kg5hc6bd";
          inherit src version;
        });
      });
}
What about using nixos-2105.kopia.override instead of nixos-2105.callPackage "${nixos-2105.path}/pkgs/tools/backup/kopia"? This works when Nix is in restricted mode (in my case I was using flake).

Also Go 1.17 is already out, and latest Tailscale requires it so it makes sense to use buildGoModule = args: (nixos-2105.buildGoModules.override { go = go_1_17; }) (args // { ... }); (waiting #127519 to merge, so we can ditch overriding Go in the first place)

Stumbled across this comment after realizing I couldn't build tailscale, but before I realized that's what this comment was about (tailscale specifically)😂

Seems to work fine with override, and no wonder it was failing to build until I overrode the go version used.

Tailscale 1.16.0 Override

tailscaleLatest =
    let
      version = "1.16.0";
      src = pkgs.fetchFromGitHub {
        owner = "tailscale";
        repo = "tailscale";
        rev = "v${version}";
        sha256 = "sha256-wrC6bvC1f6SFY/SwVUW1+xuF1sk2jA9apMpo4jE2bFw=";
      };
    in
    (pkgs.tailscale.override rec {
      buildGoModule = args: pkgs.buildGoModule.override { go = pkgs.go_1_17; } (args // {
        inherit src version;
        vendorSha256 = "sha256-HJLT0CGUVzJZ56vS/v3FZ7svxyzZ+wlXvrC2MExTLM4=";
      });
    });

Thanks for the pointer on overriding go in buildGoModule, I had not dug into Tailscale changes that far and did not realize a new Go version was required 🤪

Inline geth nix derivation in repo and use submodule for code We can't simply override the `src` attribute of the go-ethereum derivation because this doesn't work with vendored go dependencies. For more info: NixOS/nixpkgs#86349 . Here we copy the nix derivation from nixpkgs to ./nix/go-ethereum folder of this repo. Nix will print a mismatch to the console if the vendored geth dependencies change. In that case the new vendorSha256 in the inlined geth nix derivation in ./nix/go-ethereum/ needs to be updated. This is a stop-gap solution so we can all work with the same version of geth until we find a better way to enable that.

ShamrockLee · 2022-03-22T19:45:09Z

In #158486, I work around this issue by passing related attributes including veniorSha256 as inputs, so that they can be owerrided with the override attribute of the resulted package. An example can be found here:

https://github.com/ShamrockLee/nixpkgs/blob/singularity-apptainer/pkgs/applications/virtualization/singularity/generic.nix#L21-L24=

Nevertheless, I still hope that this issue can be fixed. If not, we should at least document the workaround.

Overlay-style overridable mkDerivation (#119942) might be one of the ultimate solutions to issues of this kind. See #171586.

Lillecarl · 2022-12-13T16:12:34Z

@Mic92 Do you have a complete example for those of us who are still learning the language? :)

    zabbix-overlay =
      (self: super: super.zabbix.agent2.overrideAttrs (
            let
              version = "5.4.11";
              src = super.fetchurl {
                url = "https://cdn.zabbix.com/zabbix/sources/oldstable/5.4/zabbix-5.4.11.tar.gz";
                sha256 = "sha256-mxEsUFlnDB8MHWX6zWZaETNzNW6ut9ro28hXGG490kQ=";
              };
            in
            rec {
              name = "zabbix.agent2";
              inherit src;
              go-modules = (self.buildGoModule {
                inherit name src;
                vendorSha256 = "sha256-4Z4Fgv9zmAwp3tEvHj8yLOWkFY/zFz5VfehSWCuIUcI=";
              }).go-modules;
            }
        )
      );

I'm getting an infinite loop here.

Mic92 · 2022-12-13T20:09:15Z

@Mic92 Do you have a complete example for those of us who are still learning the language? :)

    zabbix-overlay =
      (self: super: super.zabbix.agent2.overrideAttrs (
            let
              version = "5.4.11";
              src = super.fetchurl {
                url = "https://cdn.zabbix.com/zabbix/sources/oldstable/5.4/zabbix-5.4.11.tar.gz";
                sha256 = "sha256-mxEsUFlnDB8MHWX6zWZaETNzNW6ut9ro28hXGG490kQ=";
              };
            in
            rec {
              name = "zabbix.agent2";
              inherit src;
              go-modules = (self.buildGoModule {
                inherit name src;
                vendorSha256 = "sha256-4Z4Fgv9zmAwp3tEvHj8yLOWkFY/zFz5VfehSWCuIUcI=";
              }).go-modules;
            }
        )
      );

I'm getting an infinite loop here.

Try packageOverrides instead to avoid infinite recursions due the overlay.

nixos-discourse · 2023-02-02T15:59:50Z

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/inconsistent-vendoring-in-buildgomodule-when-overriding-source/9225/7

nixos-discourse · 2023-05-04T19:43:28Z

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/use-a-specific-version-of-terraform-in-a-nix-shell/27880/1

overrideAttrs does not combine well with buildGoModule (see NixOS/nixpkgs#86349 for details), so I went with a simple approach of defining a new derivation instead.

* this is a terrible hack but unfortunately it's necessary because buildGoModule computes the vendorHash after installing all Go modules * although overriding the vendorHash works just fine, there's still something wrong because the overlay is not applied properly, as per kluctl --version kluctl version v2.22.1 Resolves: Related: NixOS/nixpkgs#86349 Signed-off-by: Daniel-Andrei Minca <[email protected]>

ShamrockLee · 2024-07-29T01:20:49Z

While #234651 is under discussion, I reimplemented #225051 to address this issue with minimal expression changes. #225051 is now ready for review.

doronbehar · 2024-08-17T16:55:02Z

Closing as this is fixed since #225051 is merged.

* this is a terrible hack but unfortunately it's necessary because buildGoModule computes the vendorHash after installing all Go modules * although overriding the vendorHash works just fine, there's still something wrong because the overlay is not applied properly, as per kluctl --version kluctl version v2.22.1 Resolves: Related: NixOS/nixpkgs#86349 Signed-off-by: Daniel-Andrei Minca <[email protected]>

doronbehar added the 0.kind: bug Something is broken label Apr 29, 2020

emilazy mentioned this issue Apr 29, 2020

Recommend buildGoPackage instead of buildGoModule in the nixpkgs manual #84826

Open

veprbl added the 6.topic: golang label Apr 30, 2020

zimbatm mentioned this issue Apr 30, 2020

Modify buildGoModules to use -mod=vendor instead of the go package cache. #86376

Merged

10 tasks

ljani mentioned this issue Sep 22, 2021

caddy: add all addons #14671

Open

vojta001 mentioned this issue Mar 22, 2022

buildGoModule: Unable to override vendorSha256 when specified as null #159569

Open

ShamrockLee mentioned this issue Apr 28, 2022

singularity: fix defaultPath and reflect upstream changes #158486

Merged

13 tasks

tjni mentioned this issue May 4, 2022

stdenv.mkDerivation: overlay style overridable recursive attributes #119942

Merged

10 tasks

dotlambda mentioned this issue Aug 1, 2022

deltachat-desktop: fix build #184589

Merged

13 tasks

KFearsoff mentioned this issue Dec 11, 2022

Coredns add plugins #205649

Closed

13 tasks

ShamrockLee mentioned this issue Apr 6, 2023

buildGoModule: Fix overriding with overlay-style stdenv #225051

Merged

12 tasks

jakubgs mentioned this issue Jan 10, 2024

chore: use go 1.20 and latest go-waku status-im/status-go#4522

Merged

dminca mentioned this issue Jul 7, 2024

feat(nix): implement overlays dminca/nix-config#12

Open

doronbehar closed this as completed Aug 17, 2024

qezz mentioned this issue Aug 18, 2024

Go replace take 2 qezz/nixpkgs#1

Draft

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Undocumented / impossible: override go modules #86349

Undocumented / impossible: override go modules #86349

doronbehar commented Apr 29, 2020

kalbasit commented May 1, 2020

kalbasit commented May 6, 2020

doronbehar commented May 16, 2020

doronbehar commented Sep 13, 2020

nixos-discourse commented Oct 6, 2020

joulaud commented Jan 14, 2021

fiksn commented Jan 16, 2021

doronbehar commented Jan 20, 2021

Mic92 commented Jun 1, 2021

nh2 commented Jun 14, 2021

nh2 commented Jun 14, 2021 •

edited

Loading

danielphan2003 commented Sep 3, 2021

evanjs commented Oct 17, 2021

ShamrockLee commented Mar 22, 2022 •

edited

Loading

Lillecarl commented Dec 13, 2022

Mic92 commented Dec 13, 2022

nixos-discourse commented Feb 2, 2023

nixos-discourse commented May 4, 2023

ShamrockLee commented Jul 29, 2024

doronbehar commented Aug 17, 2024

Undocumented / impossible: override go modules #86349

Undocumented / impossible: override go modules #86349

Comments

doronbehar commented Apr 29, 2020

kalbasit commented May 1, 2020

kalbasit commented May 6, 2020

doronbehar commented May 16, 2020

doronbehar commented Sep 13, 2020

nixos-discourse commented Oct 6, 2020

joulaud commented Jan 14, 2021

fiksn commented Jan 16, 2021

doronbehar commented Jan 20, 2021

Mic92 commented Jun 1, 2021

nh2 commented Jun 14, 2021

nh2 commented Jun 14, 2021 • edited Loading

danielphan2003 commented Sep 3, 2021

evanjs commented Oct 17, 2021

ShamrockLee commented Mar 22, 2022 • edited Loading

Lillecarl commented Dec 13, 2022

Mic92 commented Dec 13, 2022

nixos-discourse commented Feb 2, 2023

nixos-discourse commented May 4, 2023

ShamrockLee commented Jul 29, 2024

doronbehar commented Aug 17, 2024

nh2 commented Jun 14, 2021 •

edited

Loading

ShamrockLee commented Mar 22, 2022 •

edited

Loading