[Feature Request] Offer pre-built binaries for Lotus

[atopal]

Is your feature request related to a problem? Please describe.
Currently people who are interesting in giving Filecoin a try have to build it from scratch, which is quite time consuming and unusual.

Describe the solution you'd like
Offer pre-built binaries for common OSes to download

Describe alternatives you've considered
We've written instructions for how to build it from scratch

[f8-ptrk]

@atopal the problem i see with this is that the binaries are highly processor architecture dependent. there would need to be a wide array of binaries to build or at least a target subset of architectures.

and all those binaries multiplied by 3 for the 3 different networks

[atopal]

I think it would be fine to offer a subset of all possible binaries, limited to just the most popular ones we see on our site, and also to limit to just the network we want to use in our "getting started" guide. We'd still have docs for how you can build it yourself, if you don't have a matching binary.

[travisperson]

We have binaries for linux and macos on x86, the macos builds look to have been broken for a bit though when the hwloc dependency was introduced. This was resolved in PR #5934 though, so we should see build going through now.

[atopal]

@travisperson Do you have a link? I couldn't find the binaries.

[terichadbourne]

@f8-ptrk For some clarity, our goal is specifically to the needs of storage clients who we're currently teaching to use Lotus Lite on mainnet, with greatly reduced hardware requirements. Totally understand that miners with intense hardware setups will want a lot of customization, and should continue to have it.

@travisperson If we indeed are already offering binaries, where would a user go to find them?

cc @dkkapur re golden path backlog - we expect that using binaries would drop 20-30 minutes from our current hour-long path to making starting one's first deal

[BigLep]

Per verbal standup on 2021-04-27, @travisperson is going to respond with:

1. Where to find the binaries?
2. Steps that would need to be taking to get binaries created for Lotus 1.8
3. Steps to take for adding additional hardware types (e.g., arm_64)

With 2 and 3, a maintenance tribute can pick this up.

[travisperson]

As mentioned in my previous comment this issue was fixed in #5934. However, due to whatever reason, this never made it into the v1.8.0 release tree and therefore no binaries were able to be produced due to an error while building the MacOS binaries (lack of hwloc). This means there are no binaries available for v1.8.0 at this moment.

1. Binaries (when available) can be found in the Github Releases. Currently we only build binaries for tagged versioned releases, but I think we want to build these for the release candidates as well. This will require a change to the tag matching regex in the CircleCI configuration.
   There are various places that need to be updated in the CircleCI configuration, I believe the following should work (https://regex101.com/r/10ERHJ/1)

 tags:
   only:
-    - /^v\d+\.\d+\.\d+$/
+    - /^v\d+\.\d+\.\d+(-rc\d+)?$/

2. Adding binaries to previously releases can be done in a few different ways. The simplest solutions and probably the safest would be to cherry-pick the fix from fix ci build-macos #5934 onto the v1.8.0 tag, make a few modifications to the CircleCI configuration (to ensure that binaries will be built), retag as something like v1.8.0-binaries and allow CircleCI to build the binaries and then add them to the v1.8.0 release with a note to explain the disparity in the version commit and link back to the CircleCI build + tag for people to verify the tree.

3. Supporting additional hardware requires first that the Filecoin FFI (https://github.com/filecoin-project/filecoin-ffi) supports the hardware. From a lotus integration perspective we are limited to instruction sets supported by CircleCI executors at the moment. There is further possibility of supporting cross compiling with additional engineering effort, though the amount is unknown to me at this time.

[f8-ptrk]

please make sure to proper document how those binaries were build. build frags matter a lot. with GPU2 making it into 1.9.0-rc1 right now we need to make sure one knows what they are actually running

[travisperson]

These binaries are only built for mainnet. They are built with no special flags, and consume the filecoin-ffi as distributed by the https://github.com/filecoin-project/filecoin-ffi/releases for the matching release of the filecoin-ffe submodule at ./externs/filecoin-ffi.

We should be able to support additional networks though without much issue, and we should at least support the calibration network at the moment.

[arajasek]

We should probably introduce binaries for RCs in a v1.9.0-rc3, just so that we can test them before binaries return in v1.9.0.

[BigLep]

We will resolve this issue when:

1. Binary has been created for 1.8.0 (currently being done by @coryschwartz )
2. Binaries will be created for all RC's going forward (with first evidence in 1.9.0-rc3 per lotus v1.9.0 #5977)

Creating binaries for Windows, even just as a lotus lite node as storage client, is a much larger endeavor. General Windows support is tracked in #2133

@atopal : we had verbal conversations about getting Lotus into package managers like brew. Assuming that is still wanted, please create a separate project for that or a https://github.com/protocol/web3-dev-team project proposal.

[atopal]

Thanks @BigLep I'll follow up on package managers. Re binaries, rc-3 didn't have any binaries as far as I can tell, are we still on track for the 1.9 release?

[BigLep]

Thanks for calling this out @atopal . I think the regex mentioned in #6030 (comment) hasn't happened given no recent changes to https://github.com/filecoin-project/lotus/commits/master/.circleci/config.yml . We're discussing internally how to get this done.

I would like to confirm RC binaries work before we do the release. If we don't get to this before the 1.9.0 release, the regexes do match for full (non RC) release. (But again, preference is to test that binaries get created and that they work during RC phase and not after)

[BigLep]

@jennijuju : I added steps to the release template about checking/verifying binaries here: #6236

[placer14]

From my interpretation here, looks like we just needed to enable the test/build/publish steps for tags with the RC shape (as mentioned in #6030 (comment)). This can't be tested without a tag, so we should pay close attention next RC tagging to ensure CI runs as desired.

[jennijuju]

resolved by #6256