Enhancement: Add experimental support for Yul compilation

[haltman-at]

This PR addresses #1906, as requested by @axic, adding Yul support, or at least initial experimental Yul support, into Truffle.

To do this I did not add a new package, but rather altered compile-solidity so that it would compile Yul as well as Solidity. This is a bit of a hack, but it works.

Specifically, compile-solidity will now separate out Solidity (and JSON) files from Yul files. Solidity (and JSON) files will be compiled as normal. Yul files will be run one-by-one (since Yul only supports single-source compilation) through the compiler. (This change is made both to sources() and to sourcesWithDependencies().) So, each Yul file will result in a separate compilation in the output.

To make this work, the run function now accepts an optional language parameter (defaulting to "Solidity"); you can set it to "Yul" to compile Yul. Also, since Yul does not support imports, I simply skipped import processing for Yul files, rather than attempt to ensure that import processing would work with them. (I also loaded sources via fs rather than the resolver... let me know if that's not OK!) Also, using Yul is disallowed with pragma-based compilation, as Yul has no version pragmas.

One complication is that Yul does not currently provide sources output. So I had to alter processContracts so that if, if sources is absent, it simply won't populate the fields that rely on sources. But also, I had to make sure we still emitted sources output somehow. Fortunately, since Yul is single-source, I just added a check: If sources is absent and there was exactly one source, use that to populate our sources array.

There was another issue: Currently, any attempt to compile Yul produces a compiler warning, warning you that Yul is still experimental. Since Yul must be invoked one file at a time, this meant that compiling a project with multiple Yul files would produce multiple copies of this compiler warning. So I made it so that we suppress this repetitive compiler warning, and instead emit our own warning if any Yul files are compiled. It's not clear how we'd handle this in the future, once Yul is less experimental, but @gnidan has discussed creating a more general system for compiler warning consolidation (that could e.g. also be applied to license warnings), so if we do that then I guess that would cover it.

I also had to add workarounds for this issue with Yul sourcemaps.

While this makes it possible to compile, migrate, and debug Yul, using Yul definitely comes with a number of caveats; I'm not going to claim this is full Yul support or anything. Still, it's a start, at least. Here are some caveats I can think of:

1. Solc emits no ABI when compiling Yul, because it has no way to. Since we require an ABI, we set it equal to []. But this basically means all contract interactions must be performed manually.
2. Moreover, because there's no ABI, if your Yul contract's constructor is supposed to take arguments, you won't be able to deploy it with Truffle. The lack of a constructor ABI entry will cause Truffle to assume that the constructor takes no arguments. I don't believe we offer any way to deploy a contract while manually giving it arguments in hexadecimal.
3. When using Yul, there is no guarantee that you will get a deployed bytecode for your contract; or, if you do, that it will be correct. This would make the deployed contract impossible to recognize, so transactions against the contract could not meaningfully be debugged.
4. If you use Yul's immutables mechanism, the compiler does not emit immutableReferences, potentially again leading to the deployed bytecode being unrecognizable, with no ability to debug transactions against the contract. (Then again, maybe this doesn't matter that much, because why would you use immutables when you can't deploy contracts that require arguments...?)
5. Solc currently does not emit ASTs when compiling Yul, so the debugger won't be able to track variables when debugging it. (And certain breakpoints won't work due to debugger bug Make sub-line breakpoints work when debugging Vyper or other languages #3827.)
6. Similarly, debugger bug Debugger won't decode return values from non-Solidity transactions #3831 also applies, although this is only relevant when debugging a deployment.
7. Our Yul syntax highlighting is currently only intended to work for Solidity-assembly Yul, not standalone Yul, so standalone Yul keywords and builtins like object or linkersymbol won't be highlighted.

...honestly, I'm not even sure that's all of them. I'm calling this "experimental" for a reason. But, here it is, for those who want it...

[haltman-at]

Hm. The test I added seems to be running out of memory on Node 10...? No idea why, it runs fine locally using Node 10.

[haltman-at]

So, um, I don't know what to do about this. It's only failing on Node 10 CI (although it wasn't failing locally), but now I can't even bootstrap locally on Node 10, so I'm kind of stuck here. I mean notionally I could try just think up a less memory-intensive version by examining the code for where lots of memory is used (maybe I could run some sort of memory profiling under Node 12 under the assumption the high-memory-use spots are the same), but that sounds fairly dicey. I could just remove the test, but, uh, I don't really want to do that for obvious reasons. Or we could just wait until we drop Node 10 support in August...? I don't like leaving PRs outstsanding that long, that's a good way to get nasty merge conflicts. We can't merge it as-is; it'll break all our builds. @gnidan, any ideas? :-/

[coveralls]

Coverage decreased (-0.3%) to 68.127% when pulling ab5114c on compile-yul into 2079746 on develop.

[haltman-at]

OK, I got this passing by downgrading the solc used in the test to 0.5.17, removing the source map part of the test (no Yul sourcemaps before 0.6.4), and also stripping it down and turning up the timeout -- those last two parts may have been unnecessary given that the real solution was the downgrade, oh well. But I can beef the test back up if people want?

[gnidan]

This generally seems reasonable, but we can't leave sourcesWithDependencies so trawling

[gnidan]

Cool, thanks for splitting out the Yul stuff!