Takes a really long time to install

[cah6]

I'm fairly new to nix, so apologies if this is just how it is.

I just finished installing with nix-env -f https://github.com/obsidiansystems/obelisk/archive/master.tar.gz -iA command and it probably took around ~8-12 hours. Is this normal? I did make sure to edit my nix.conf as required, and if it makes a difference I'm on OSX 10.13.4 tracking the nixpkgs-unstable channel.

From looking at the output as it was going on, it looks like for a lot of stuff it missed the caches and was compiling C / Haskell files.

[alexfmpe]

You probably need to restart the daemon after editing /etc/nix/nix.conf: I've updated the instructions in #53
This will probably be automated soon, since https://github.com/reflex-frp/reflex-platform already handles setting up the caches

[cah6]

Ah yep I believe that's it, I didn't know before that it wouldn't automatically pick up changes to the conf file.

Btw even smaller thing (don't want to make a new "issue" for it) that I just ran into is that for me I had to turn the sandbox parameter off when installing hub as it was giving:
error: derivation '/nix/store/rkpli4272qmz5nsfw0a0wq5nbxjjhj9r-user-environment.drv' specifies a sandbox profile, but this is only allowed when 'sandbox' is 'relaxed'
Might be nice to mention this since the instructions before specifically say to turn sandboxing on.

[cah6]

Actually, this is impacting my initial setup more than I thought. For me ob init and ob run also required sandbox = false (same error as above). But when I run ob run with this, it takes a very long time since, again, it's manually compiling a lot of haskell dependencies. I don't know if this is normal, i.e. if it takes a long time first time but then gets significantly better. Thoughts?

[alexfmpe]

I've also run into that sandbox issue on mac - we might want to recommend sandbox = relaxed instead of sandbox = true.

AFAIK lack of sandboxing has so far only been a problem when trying ob deploy push in Ubuntu: #6 (comment).

I'm not sure but I think sandboxing doesn't change any hashes, in which case it wouldn't affect whether caches work. I'll do some experiments. FWIW I've managed to get caches for everything on mac without sandboxing.

I don't know if this is normal, i.e. if it takes a long time first time but then gets significantly better.

It shouldn't take that long even on the first time (expected behaviour is minutes of downloads from cache), but the second time should take only a few seconds.

[matthewbauer]

If we want to avoid asking users to restart, we could give them the Nix arguments directly like is done in cachix:

Installation

1. Install Cachix client using Nix 2.0 or greater:
   $ nix-env -if https://github.com/cachix/cachix/tarball/master --substituters https://cachix.cachix.org --trusted-public-keys cachix.cachix.org-1:eWNHQldwUO7G2VkjpnjDbWwy4KQ/HNxht7H4SSoMckM=
2. Login via https://www.cachix.org/api/v1/login to start using the service

[alexfmpe]

That seems like it'd solve the problem for nix commands run inside of ob - trivial to update, cross-platform, etc.
But won't --substituters always be needed for user-initiated nix-build and nix-shell ?

[cah6]

Fwiw for me it actually takes a long time to run ob run even with sandbox = relaxed. The beginning of output is:

$ ob run
building '/nix/store/vyag6xfwx30cb3g0s82qlfy5cbvvygvh-cabal2nix-backend.drv'...
installing
these derivations will be built:
  /nix/store/x646y17cngmxhnrnb953aha44prdnc8l-linear-1.20.7.drv
  /nix/store/rj4yqwfvwhghc5vbxdyk0kdpmz8kyidy-active-0.2.0.13.drv
  /nix/store/zc9kq005cgsma3060jvp16zcr07r6y2n-diagrams-core-1.4.0.1.drv
  /nix/store/gzrk5fqm7vy98jq6dwhzyvygp0j3ln7j-diagrams-lib-1.4.1.2.drv
  /nix/store/rz5nmy0hsp38jx9iznf57cyp15bnpkp7-diagrams-svg-1.4.1.1.drv
  /nix/store/73fdybgg2m91gj6jf9gv61rqc9w5p49v-obelisk-snap-0.1.drv
  /nix/store/7sx04p9pxyz43qdmimdf3pjl2z278zxx-obelisk-backend-0.1.drv
  /nix/store/9k5cacp22gq3lfqww751p92hk8lih7mf-ghc-8.0.2-with-packages.drv
building '/nix/store/x646y17cngmxhnrnb953aha44prdnc8l-linear-1.20.7.drv'...

which seems wrong as it needs to manually build all those packages.

[ryantrinkle]

Does --trusted-public-keys work even if the user isn't a trusted user by nix? I like the idea of having a more bulletproof installation command (even if it's longer, since people will mostly cut-and-paste), but if it's not actually bulletproof, that's not quite as useful.

sandbox = relaxed sounds fine; I didn't realize there was anything other than true or false - i was just trying to prevent #6

[alexfmpe]

I just did a nix-collect-garbage on my mac, after which, Installing ob via nix-env took 5-10mins, but ob run is compiling a bunch of haskell packages from source.
Maybe somethings aren't actually in the cache?

[ryantrinkle]

@alexfmpe I think mac stuff is not getting fully cached right now. Can you try building release.nix and looking at its closure to see whether the missing things are accounted for in there? If not, we need to add them.

[JBetz]

Is there anything that needs to be done on Linux to deal with this problem? I did a fresh install of nix and obelisk on Ubuntu today, and ob run has been running for hours, compiling a bunch of C and Haskell stuff that I'd expect it to get from the cache.

Also, when I ran nix-build -A exe, I could see that it's actually using the cache, so the problem might be caused by something that ob run is doing differently.

[ryantrinkle]

@JBetz Ah, that's odd! Can you run nix-build --version? I have a suspicion that you might need to change your /etc/nix/nix.conf to say substituters = ... instead of binary-caches = .... I believe this name changed in Nix 2.0

[JBetz]

@ryantrinkle I'm on nix v2.0.4, but my /etc/nix/nix.conf already uses substituters.

Possibly relevant, I see a bunch of warnings in the beginning of the ob run output about locale:

$ ob run       
rning: Setting locale failed.
perl: warning: Please check that your locale settings:
        LANGUAGE = "en_US",
        LC_ALL = (unset),
        LANG = "en_US.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").

[JBetz]

I think the locale errors were a red herring. After running ./try-reflex in reflex-platform, the caches started working. So whatever script reflex-platform runs to set them up should also be used in obelisk.

For comparison, here's what /etc/nix/nix.conf looked like before and after running ./try-reflex:

$ cat /etc/nix/nix.conf.2018-07-19T14\:58\:53Z.baksandbox = true
substituters = https://cache.nixos.org https://nixcache.reflex-frp.org
trusted-public-keys = cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY= ryantrinkle.com-1:JJiAKaRv9mWgpVAz8dwewnZe0AzzEAzPkagE9SP5NWI=

$ cat /etc/nix/nix.conf
sandbox = true
substituters = https://cache.nixos.org https://nixcache.reflex-frp.org
trusted-public-keys = cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY= ryantrinkle.com-1:JJiAKaRv9mWgpVAz8dwewnZe0AzzEAzPkagE9SP5NWI=
binary-caches = https://cache.nixos.org https://nixcache.reflex-frp.org
binary-cache-public-keys = cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY= ryantrinkle.com-1:JJiAKaRv9mWgpVAz8dwewnZe0AzzEAzPkagE9SP5NWI=
binary-caches-parallel-connections = 40

[chengzh2008]

Forgive me if a wrong place for this question. After the completion of obelisk command, issueing 'ob init' gave me this error:

✖ Setting up obelisk
ob: HTTPError (HttpExceptionRequest Request {
host = "api.github.com"
port = 443
secure = True
requestHeaders = [("Authorization","token 2ee01722177e68ac7cbe4719a98b232648ff2992"),("User-Agent","github.hs/0.7.4"),("Accept","application/vnd.github.preview")]
path = "/repos/obsidiansystems/obelisk/commits"
queryString = "?sha=master"
method = "GET"
proxy = Nothing
rawBody = False
redirectCount = 10
responseTimeout = ResponseTimeoutDefault
requestVersion = HTTP/1.1
}
(InternalException (HandshakeFailed (Error_Protocol ("certificate has unknown CA",True,UnknownCa)))))

Any guide and helps are really appreciated.
Thanks!

[e0]

Perhaps the README could be updated with a note about this. I don't mind the long duration for installing or ob run, but it would help to know a rough time frame so I can decide how to allocate this time for something else.

[ElvishJerricco]

@e0 Thing is, it's not supposed to take very long. If binary caches are set up right, installation should be a few minutes.

[e0]

@ElvishJerricco I see, that makes sense. I want to add that when running the initial ob run, I get the following warning message: warning: in configuration file '/etc/nix/nix.conf': unknown setting 'trusted-public-keys'.

I'm running macOS 10.13.6 and Nix 2.0.4. Please let me know if there is anything I can do to help investigate this problem. Obelisk runs beautifully after ob run's initial installation process completes and it's a pity that certain developers are running into this problem.

[conklech]

I'm seeing the same unknown setting 'trusted-public-keys' issue with ob run, which does not happen with the installed nix tools. Is ob run not using the system nix tools?

[3noch]

@conklech That's right. What version of nix is on your system?

[conklech]

$ nix --version
nix (Nix) 2.0.4

[ElvishJerricco]

I thought ob had a hard requirement on Nix >= 2.0? So if it's pinning Nix itself, wouldn't it be pinning 2.0? Also... it's not generally advisable to use non-system-wide Nix tools...

[3noch]

@ElvishJerricco @conklech I filed an issue to track this. We hoped it would avoid the hard requirement but it might cause more trouble than it's worth.

[conklech]

For reference, I installed obsidian per the README (nix-env -f https://github.com/obsidiansystems/obelisk/archive/master.tar.gz -iA command) but did not have the binary substituter correctly configured at that time, so it built a lot of stuff from source.

I also tried running nix-build -A exe -o result-exe from a clean project directory created with ob init. (I have not yet had ob run complete successfully, so there are a few packages to build.) It tries to build different derivations:

$ ob run
./.obelisk/impl: command not cached, building ...
warning: in configuration file '/etc/nix/nix.conf': unknown setting 'trusted-public-keys'
these derivations will be built:
  /nix/store/9slb2qmywph77lsal4lj2vp6spwzv2hx-intervals-0.8.1.drv
  /nix/store/ccp9bymib8p1si7lhzgsrmaf1l6ykw0r-linear-1.20.7.drv
  /nix/store/baxyrydl428qm09rpx5vb95msipn86kp-diagrams-core-1.4.0.1.drv
  /nix/store/vqxfg1vvg8pmd749hxp59g38xwl0ixbf-active-0.2.0.13.drv
  /nix/store/crfn7jsajjgsfjhybl7ih8n6vwhvvrvy-diagrams-lib-1.4.1.2.drv
  /nix/store/injj6qv8qfrgi576h7bvs7yhn0dc4pyi-diagrams-svg-1.4.1.1.drv
  /nix/store/bidigrab8bhlc6jndhsf284id33nq424-obelisk-snap-0.1.drv
  /nix/store/lv21zmr0avxjvgq9389mw91phbjrmn4c-obelisk-backend-0.1.drv
  /nix/store/xpa5mzjxb69qbbajbcd95wr1gx3k543v-ghc-8.0.2-with-packages.drv

$ nix-build -A exe -o result-exe
these derivations will be built:
  /nix/store/lwwxnw81gqff1hw2hcdwas3kwqxzbarg-static-0.drv
  /nix/store/pm8nx1bfn5hr5rmn2gbccjkkj4dmv0f1-common-0.1.drv
  /nix/store/9lv4j8lxhzhzwy52hjlvhpqdl6ff8n63-frontend-0.1.drv
  /nix/store/knlapfz6j9afv0h7iyxmwh6hg71kqass-compressedJs.drv
  /nix/store/izikg0ckn1g7hs692z4akq7ws3ly6nix-intervals-0.8.1.drv
  /nix/store/x646y17cngmxhnrnb953aha44prdnc8l-linear-1.20.7.drv
  /nix/store/rj4yqwfvwhghc5vbxdyk0kdpmz8kyidy-active-0.2.0.13.drv
  /nix/store/zc9kq005cgsma3060jvp16zcr07r6y2n-diagrams-core-1.4.0.1.drv
  /nix/store/gzrk5fqm7vy98jq6dwhzyvygp0j3ln7j-diagrams-lib-1.4.1.2.drv
  /nix/store/rz5nmy0hsp38jx9iznf57cyp15bnpkp7-diagrams-svg-1.4.1.1.drv
  /nix/store/73fdybgg2m91gj6jf9gv61rqc9w5p49v-obelisk-snap-0.1.drv
  /nix/store/7sx04p9pxyz43qdmimdf3pjl2z278zxx-obelisk-backend-0.1.drv
  /nix/store/l4dsjxjac84ljqz9lr9nc92k630q5s2l-common-0.1.drv
  /nix/store/p7bm8djgl5ajpwsmk0gj7y58kjvvd22v-frontend-0.1.drv
  /nix/store/ml8cx7sir3ckspgq3kg0b55s5yq3wll7-backend-0.1.drv
  /nix/store/122vips5zmc36xxy5623yhjnpvdvc055-serverExe.drv

Note that the hashes for, e.g., linear differ. I do not know if that is normal. Also note that the binary substituter is not being hit even with nix-build. I don't know if that's expected.

I don't think there's anything unusual about my setup, but I don't have enough familiarity with nix to say. The ./try-reflex script in reflex-platform works fine. I'm MacOS 10.13.6.

[e0]

@conklech I think we have quite similar setups and experiences. I did have the binary substituter setup but there were still some packages that were being built from source.

ob run took about an hour to complete for me (I have a quad-core 3,5 GHz Intel Core i5). I also tried nix-build -A exe -o result-exe after ob run completed and it built fine, and quickly, for me. I don't remember the exact derivations though but I imagine nothing new was being built since the nix-build -A exe -o result-exe completely so quickly.

[luigy]

Note that the hashes for, e.g., linear differ. I do not know if that is normal.

This is prob ghc vs ghcjs pkg dependency. Ob run only makes use of the ghc environment whereas nix-build -A exe builds everything required to make a deployment which includes the ghcjs build of your frontend

[conklech]

@luigy That makes sense.

To get a clearer report since I'd had substitutes misconfigured(?) before, I ran nix-env --uninstall obelisk-command to remove obelisk, and nix-collect-garbage, which apparently deleted all Haskell artifacts since I don't have anything else meaningful in my environment. Then I ran the try-reflex script, which only had to compile a few packages, cabal2nix and partition-all-cabal-hashes being the main holdups.

I then re-installed obelisk-command, which only took a minute. I then ran ob run in a preexisting obsidian project folder. The result included the warning: in configuration file '/etc/nix/nix.conf': unknown setting 'trusted-public-keys' message. But after building a few packages from source, it then proceeded to get a few dozen packages from the reflex-frp.org and nixos.org binary caches—leaving, however, dozens of packages to build from source. Here is a partial log of the output. The job hasn't finished (yet?).

It might be that the unknown-settings message is a red herring, since at least in this case the warning was followed by pulling some packages from the caches. Are these just ordinary cache misses, i.e. unfortunate but not a bug unless there's a commitment that the cache is up-to-date? Is there anything further I can do to help test/clarify what's going on?

[e0]

@conklech I don’t think the unknown-settings issue is the only thing causing problems with cache missses since some building from source is also happening during the installation of Obelisk.

[davetapley]

+1 for confused first time user, I let it sit there building for an hour before checking log and spotting

warning: in configuration file '/etc/nix/nix.conf': unknown setting 'trusted-public-keys'

Looks like in the same place as @conklech: MacOS 10.13.6, Nix 2.0.4.

[conklech]

This issue is still pending, and does not appear to be caused by configuration issues. It appears that the binary cache at nixcache.reflex-frp.org does not include the majority of derivations necessary for obelisk-command itself or for building obelisk projects.

Looking at default.nix, there are a number of TODOs about upstreaming various overrides and modifications to reflex-platform. Is only reflex-platform is being used to populate the reflex-frp.org binary cache? In that case, either upstreaming to reflex-platform or changing the caching infrastructure would seem to be necessary.

If there's any way for me to help, I'd be glad to.

[ElvishJerricco]

It appears that the binary cache at nixcache.reflex-frp.org does not include the majority of derivations necessary for obelisk-command itself or for building obelisk projects

What makes you say this? It's common that the daemon does not get restarted, giving this illusion.

[conklech]

@ElvishJerricco : Because trying to install (upgrade) obsidian requires building a big slice of the Haskell ecosystem, despite pulling some derivations from the reflex-frp.org cache. See this gist from a run just now.

This is all on macOS, by the way; the title of this issue isn't very informative. I just spun up a NixOS virtual machine to test, and when installing obelisk itself it pulls considerably more from the binary cache; the first (only?) uncached build I ran into was cabal2nix.

BTW, I don't understand your reference to the daemon not being restarted, so maybe I'm not responding to your actual question.

[ElvishJerricco]

@ryantrinkle I'm experiencing the same thing on darwin as @conklech. Those derivations simply aren't cached on nixcache.reflex-frp.org. Is Hydra not doing darwin builds?

[ryantrinkle]

@ElvishJerricco Hmm, you know what, reflex-platform is built on darwin, but maybe that's too far from what obelisk needs. We should definitely cache obelisk specifically for darwin, as well.

[conklech]

I tried making a fork with Travis CI darwin builds enabled. The Travis OSX builder does not have the cache miss problem that @ElvishJerricco and I are seeing. See this log; ignore the cachix stuff, which isn't actually playing a meaningful role.. As expected, it pulls everything but cabal2nix from https://nixcache.reflex-frp.org. Running the same build command on my darwin machine, with what should be identical nix settings, gives different results; it's building different derivations.

I'm at a loss as to what's different, since default.nix doesn't look like it has any impure bits. Are there maybe some system dependencies that change the derivations? A source of nondeterminism in nix that I don't know about? Any pointers where I might look next?

Off-topic: Any interest in a pull request to add .travis.yml to this repository (without the cachix stuff), and maybe trying to fix the reflex CI build that's been burning through a bunch of CPU cycles before timing out after every commit? The latter would involve some serious work, more or less backporting stuff from reflex-platform.

[conklech]

Please disregard the previous comment. The Travis build is not pulling everything from the cache, and is in fact showing the same behavior as I'm seeing locally. I should have waited for the build to finish before jumping to conclusions.

[conklech]

I've populated a cachix cache at conklech-reflex with, I think, everything necessary to install and run obsidian as of 24688a9 on either linux or darwin. See the successful Travis builds here. The travis builder also updates the conklech-reflex binary cache, but there are some significant limitations to its ability to deal with cache misses, discussed below. No promises that I'll keep it up-to-date, but hopefully the heavy dependencies in there will continue to be usable for a while.

I have a few notes from this exercise that may be useful to others:

• I think the correct builds to test and cache are:

nix-build ./default.nix -A command
nix-build ./skeleton/default.nix -A ghc.backend
nix-build ./skeleton/default.nix -A ghcjs.frontend
nix-build ./default.nix -A haskellPackageSets.ghc.obelisk-migration

It appears that the other relevant derivations are covered by these four, but I haven't exhaustively checked.

• On my Macbook Pro (2015), building with a cold cache fails nondeterministically due to long-running tests in system dependencies with too-aggressive timeouts. go-1.4 and skylighting were both repeat violators here. With go-1.4, I had to disable parallel building to get it to build successfully at all. (If you're curious, go-1.4 is built in order to bootstrap go-1.9, which is necessary to build hub. )

• Travis needs an almost completely warm cache in order to complete successfully. Just installing all the cached derivations takes over twenty minutes. I think the Linux builder would run successfully with the current state of https://nixcache.reflex-frp.org, but the OSX builder definitely can't.

• The cache needs to contain both run-time and build-time dependencies, or building on Travis will fail if there are any changes that require recompilation. With cachix, it's necessary to run something like nix-store -qR --include-outputs $(nix-instantiate -A command) | xargs -o cachix push REPONAME, which itself takes a long time. I did this manually for conklech-reflex; I don't have it in my current .travis.yml, because if the build deps aren't already cached the Travis build will eventually fail without making any further progress. Both go-1.4 and ghcjs take more than 50 minutes to compile, and the intermediate state is lost. I don't know how often those dependencies would get invalidated.

For end users, the build-time deps of obelisk itself are probably not necessary, but without the build-time deps of obelisk-run (e.g. ghc.backend and ghcjs.frontend) it'll be impossible to use obelisk until all the deps are rebuilt.

• Just as a general comment, the dependency graph is huge, and because we're pinned to such an old version of nixpkgs none of it is going to be shared with other nix packages. For example, on darwin we need to build our own versions of all sorts of system packages. Unless somebody tells me it's a crazy idea, I'm tempted to poke around to see if we can move to a newer nixpkgs and narrow the delta a bit. Obviously in this context "we" also means reflex-platform.

[ryantrinkle]

@conklech Awesome! Thanks for diving into this.

I just merged #220, which should provide some additional caching for Darwin, but probably not enough. Everything in release.nix will be built by hydra.reflex-frp.org, and everything it builds goes into the cache, so adding things to release.nix is the easiest way to improve the cache. It'll also ensure everything gets built and tested, of course, as well.

• Regarding go-1.4 and other packages with nondeterministic tests, I think we should disable those tests, probably in upstream nixpkgs.
• We're working towards getting on a more recent nixpkgs, and progress is being tracked here: Reunification with NixOS/nixpkgs reflex-frp/reflex-platform#312 It's looking pretty good!

[conklech]

Awesome. OSX install time is down to about twenty minutes with the reflex-frp.org cache: see this successful build. Not everything is cached, either on Linux or OSX. On both platforms, the main thing missing is cabal2nix, which takes maybe five minutes to compile. See build logs for linux and darwin. There are some other time-consuming builds, primarily all-cabal-hashes. Searching through the build logs for "building '/nix*" shows the cache misses. On darwin, there are a handful of Haskell libraries inexplicably not cached; other than being odd, this doesn't seem like a problem.

Regarding adding things to release.nix, I think the only thing would be to capture the build-time dependencies of skeleton. But since the big dependencies (go, ghcjs) get caught by something else, presumably the shells, the only thing missing seems to be cabal2nix and its ecosystem. Since our clean install time is roughly twenty minutes anyway, shaving off an extra few minutes is probably not a high priority.

What might be nice would be for the initial installation step to install not just command but also everything required for ob run to work. Otherwise the user has to sit through two long-running builds, which is not a great experience. But this is now splitting hairs.

From my end, I think this issue is ready to be closed. I'd be glad to submit a pull request for my add-travis branch, which would give us a regression test if the binary cache starts failing again.

Thanks very much!

[ryantrinkle]

Great to hear it! We'll keep improving from here, and please keep reporting any issues you find!

[ali-abrar]

Recent testing seems to indicate that caching for macOS users has improved dramatically. Closing this, but please report any new issues.

[raptazure]

Hi, I'm trying to run calculator-tutorial on macOS 10.15 & Manjaro Linux 20.2 and find it is building a lot of things. But ob init and then ob run is fairly fast. Is there a way to fix it? Thanks.
Here is my nix.conf file:

substituters = https://nixcache.reflex-frp.org https://cache.nixos.org
trusted-public-keys = cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY= ryantrinkle.com-1:JJiAKaRv9mWgpVAz8dwewnZe0AzzEAzPkagE9SP5NWI=

sandbox = relaxed