Is there a way to let 'elm make' behave like a compiler without package manager smartness?

[EdSchouten]

Quick Summary:
I am trying to write build rules for Elm for the Bazel build system. Bazel is a pretty awesome build system: it scales to large projects, provides reproducibility, permits distributed builds, allows you to build many programming languages in a single project through a single build rule language, provides hermeticity, etc. To guarantee hermeticity, build actions may be executed with file system/network sandboxing in place.

My plan is therefore to invoke build actions for Elm sources by providing copies of everything that is necessary:

• The Elm compiler binary,
• Source code for elm/core and any other dependency,
• Source code for the application itself.

Before running elm make, my plan is to generate an elm.json that looks like this:

    {
        "type": "application",
        "dependencies": {"direct": {}, "indirect": {}},
        "elm-version": "0.19.0",
        "source-directories": [
            "/path/of/elm-core",
            "/path/of/elm-json",
            "/path/of/elm-...",
            "/path/of/some/library/declared/as/part/of/the/bazel/project",
        ],
        "test-dependencies": {"direct": {}, "indirect": {}},
    }

So far I'm not having a lot of luck following this approach:

• Elm always wants to phone home to https://package.elm-lang.org/all-packages, even if there are no dependencies. This causes builds to fail with network sandboxing enabled.
• Elm insists that elm/core (and elm/json?) are always provided through "dependencies". Because the "dependencies" system only seems to provide support for listing version numbers (and not, say, allowing you to specify a custom directory of a checkout), there's effectively no way to build code in an isolated environment.

SSCCE

• Elm: 0.19.0
• Operating System: Linux

$ bazel build --config=local-debian8 :demo_app
INFO: Invocation ID: ba6a1d74-00c8-4fc2-aa58-7b5955c0a926
INFO: Streaming Build Event Protocol to http://localhost:7984/build_events/bb-event-service/ba6a1d74-00c8-4fc2-aa58-7b5955c0a926
INFO: Analysed target //:demo_app (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
ERROR: /home/ed/projects/bazel_elm_example/BUILD.bazel:16:1: Elm demo_app failed (Exit 1)
Action details (uncached result): http://localhost:7984/uncached_action_result/debian8/38bc22cd7d13531753bd246955ecf8f186fed216ec2a614ee17b90180e5cd7b1/146/
-- HTTP PROBLEM ----------------------------------------------------------------

The following HTTP request failed:

    <https://package.elm-lang.org/all-packages>

Here is the error message I was able to extract:

    HttpExceptionRequest Request { host = "package.elm-lang.org" port = 443
    secure = True requestHeaders =
    [("User-Agent","elm/0.19.0"),("Accept-Encoding","gzip")] path =
    "/all-packages" queryString = "" method = "GET" proxy = Nothing rawBody =
    False redirectCount = 10 responseTimeout = ResponseTimeoutDefault
    requestVersion = HTTP/1.1 } (ConnectionFailure Network.Socket.getAddrInfo
    (called with preferred socket type/protocol: AddrInfo {addrFlags =
    [AI_ADDRCONFIG], addrFamily = AF_UNSPEC, addrSocketType = Stream,
    addrProtocol = 6, addrAddress = <assumed to be undefined>, addrCanonName =
    <assumed to be undefined>}, host name: Just "package.elm-lang.org", service
    name: Just "443"): does not exist (Try again))

Target //:demo_app failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 5.185s, Critical Path: 5.10s
INFO: 0 processes.
INFO: Build Event Protocol upload finished successfully
INFO: Build Event Protocol results available at http://localhost:7984/build_events/bb-event-service/ba6a1d74-00c8-4fc2-aa58-7b5955c0a926
FAILED: Build did NOT complete successfully

Additional Details

Background: I happen to be the author of Buildbarn, one of the more popular remote build cluster implementations for Bazel. I am currently investigating rewriting Buildbarn's web frontend in Elm, so that you can use it to view build progress/completion in real-time. As Buildbarn is built using (surprise!) Bazel, a prerequisite is to have decent rules for Elm.

[EdSchouten]

For the time being, I'm experimenting with allowing build actions to access the internet. Also: instead of managing dependencies on elm/json and elm/core through Bazel and adding them to source-directories, I am falling back to specifying those through elm.json.

Now I run into another issue. If I manually download packages such as elm/url and elm/browser and try building them as part of a project by adding them to source-directories, it can't find any modules underneath Elm.Kernel. This seems to be due to some special treatment that libraries under elm/* get. Only those packages are permitted to implement modules in Javascript.

I fully understand that the Elm project wants to protect its ecosystem, but the combination of the facts that it's not possible to override dependencies to explicit checkouts, the hardcoded requirement of using elm/core and elm/json, and that only elm/* can declare Elm.Kernel makes it pretty hard to embed builds of Elm projects into larger projects.

It looks like this issue is already known within the Elm community. I stumbled upon this issue report that seems strongly related to what I'm running into: gdotdesign/elm-github-install#62 (comment). Considering that this issue is already known for some time, I don't think I have a lot of choice here but to (hopefully temporarily) stop my efforts integrating Elm into Bazel.

[zwilias]

@EdSchouten Not sure to what degree this alleviates your problems, but using the ELM_HOME env variable, you can control where Elm expects the global cache to be.

I'm not sure what the phone-home behaviour is about, exactly. As far as I know, this happens when the project does not have up to date info in elm-stuff, at which point Elm checks if the global registry (ELM_HOME or ~/.elm) is up to date.

[srobertson]

For what it's worth ... here's some snippets of Bazel I use. It relies on npm_install and npm_binary ... it seems to avoid hitting the network unless something changes in elm.json or package.json but I haven't checked to see if sandboxing is being strictly enforced.

nodejs_binary(
    name = "elm",
    entry_point = "elm/bin/elm",
    node_modules = "@npm_deps//:node_modules",
)


genrule(
    # Elm  0.19  has no method to prefetch depdencies
    # w/o compiling... and we don't want to refetch
    # deps on every src change so we create a dummy
    # module 
    name="elm-stuff",
    srcs=["elm.json"],
    outs=["elm-stuff.tgz"],
    tools=[":elm"],
    cmd="""
   
    ELM=$$PWD/$(location :elm)
    OUTPUT=$$PWD/$@

    cd service/angler
    mkdir src

    echo "module Ignore exposing (..)\n\nx=5\n" > src/ignore.elm
    HOME=. $$ELM make src/ignore.elm 
    tar  -czf $$OUTPUT .elm/ elm-stuff/
    """

)

[EdSchouten]

Just to give a status update on this. I managed to get these build rules to work in the end and have published them here. They should be pretty awesome to use. There were some ugly things I had to do, though:

• I'm handcrafting my own ELM_HOME directory that contains manually downloaded copies of the core libraries, placed in the exact locations where Elm expects them.
• Still, elm make would try to download the package index. I've solved this by handcrafting my own copy of versions.dat, stored in ELM_HOME. The handcrafted copy only contains a list of the packages that are listed as dependencies. This tricks elm make into no longer phoning home.
• For running unit tests, I had to do some ugly things for which I'm likely going to hell. The elm-test utility is a monolithic monstrosity. It builds the sources, then extracts a list of top-level test declarations through the .elmi file, generates a main function that lists all tests, builds the main function together with all dependencies, and runs it through Node.js. There is absolutely no way this tool can be instructed to do those tasks separately, which is a nightmare from a build system's point of view. I therefore had to resort to reimplementing all these features myself as part of the Bazel rulesets.
• elm make doesn't like it that I have to handcraft my elm.json based on dependencies/actions in Bazel. When it notices that, it seems to become completely hostile in the sense that it doesn't want to return proper diagnostics in the case things go wrong. For example, in case versions don't match constraints or dependencies fail to build, it intentionally hides build error messages. To prevent elm make from doing this, I've implemented logic to build libraries not owned by elm or elm-explorations not as separate libraries, but as part of the application itself.

This is actually the first time I'm using Elm. So far I think the language is pretty awesome. It's very likely that I'm going to be using it going forward. That said, I'm baffled by the totalitarian mindset that went into designing the build tool. Especially for a language that is almost never used on its own (read: people likely want to use an Elm web application in combination with a backend service written in another language), I fail to understand why its build process is so hard to integrate.

[evancz]

To answer the initial question, there is not a way to do this. Folks on the Elm Slack can probably give more information on why the build system works as it does. Different people have different opinions about the design tradeoffs faced by Elm in particular when it comes to packages. Many people have yelled very aggressively about how it really should be, but I do not think it cannot be made to work for everyone. Folks on slack can probably point you to some of the posts if you'd like to learn more.

Folks are trying to do a good job, but there are different definitions of good job. We may not ultimately agree about things, but I ask from a place of deep weariness that it can be discussed without calling any names.

Anyway, I think slack is the best place for background info on this.

[turboMaCk]

@EdSchouten I think you should primary check this project https://github.com/hercules-ci/elm2nix
Many of things do map quite naturally between nix and bazel. I understand your frustration from trying to get this work. On the other hand, demanding changes in elm is probably not a way forward. You simply need to write tooling to work around the issues you have. Changing elm is more work than to write any new type of tool as a lot of other things already depends on the way it works.