Support Yul as a language

[axic]

With Solidity 0.5.7 it is possible to compile Yul contracts via "Standard JSON".

It follows the very same interface as compiling Solidity contracts and as a result it should be simple enough to support it in Truffle.

Please let me know about any questions here, I'm happy to answer any.

[CruzMolina]

Hey @axic! Been taking a look at getting this setup. I noticed a couple of things that appear to be off but I'm not sure if I'm doing something wrong.

If I have a sample.json that includes:

`{
    "language": "Yul",
    "sources": { "input.yul": { "content": "{ sstore(0, 1) }" } },
    "settings": {
        "outputSelection": { "*": { "*": ["*"], "": [ "*" ] } },
        "optimizer": { "enabled": true, "details": { "yul": true } }
    }
}`

and I run solc --standard-json sample.json using 0.5.7+commit.6da8b019.Darwin.appleclang, solc hangs indefinitely.

Also, if I have a sample.yul that includes:

{ sstore(0:u256, 1:u256) }

and I run solc --yul sample.yul, solc outputs:

Warning: Yul and its optimizer are still experimental. Please use the output with care.
input.yul:1:3: Error: Function not found.
{ sstore(0:u256, 1:u256) }
  ^----^

(the above output also occurs if I try solc --evm-version <version> <yulFile> or solc --machine <target> <yulFile>)

[haltman-at]

OK @axic (I'll also ping @chriseth here :) ) so I took a stab at implementing this the other day. Unfortunately it turns out that there are some issues with supporting Yul and I don't think we're going to do this right now, but perhaps in the future once some of this is resolved. Note I tried this with solc 0.8.1.

The biggest problem is that the Yul compiler only accepts a single file at a time right now. It sounds like you're intending on supporting multiple file compilation at some future point, and I think at the least it would make sense for us to wait until that exists. I'm wary of supporting the single-file version if the multi-file version is going to happen later, because... well, I'll skip the details, but supporting both wouldn't play nicely with how we have compile-solidity laid out, and we'd have to reorganize some things that I'd hope to avoid.

But, there are smaller problems as well, that aren't quite so critical but which I'd also want to see something done about. Some of these omissions would seem to be necessary due to the inherent flexibility of Yul (e.g. there's no guarantee that the compiler can correctly determine the deployed bytecode -- although I wonder whether it should emit it in less situations than it currently does as currently it's pretty easy to fool into getting it wrong), but some seem like things that could be present but aren't. I'm going to focus on the latter as those are the ones that seem potentially fixable.

1. There's no sources output, only contracts. So there's no source IDs (although since it's single-source at the moment I suppose that technically isn't necessary at the moment!), but also no ASTs. There are Yul ASTs when Yul is used inside Solidity, or in Solidity generated sources, but not here, apparently. This maybe isn't the biggest deal, but the AST is necessary for the debugger to track variables. And of course, if multiple-source were supported, the source IDs would be necessary for interpreting the source maps.
2. There's no immutable references, even when deployed bytecode is emitted and the immutables mechanism is used. This is a problem for when we perform bytecode matching; there are various places where we have to determine what contract is represented by a piece of unknown bytecode, and to do this we have to know which parts of the bytecode can potentially vary, like link references and immutable references. (And also the delegatecall guard, but let's ignore that for now.) Well, compiling Yul gives us the link references, but not the immutable references. Of course, there is the problem of what these would be keyed by, since at the moment there's no AST, and if there were there'd still be the problem that Yul ASTs don't have node IDs, but it ought to be possible to come up with a substitute. Maybe the immutable name itself can just be used here, IDK.
3. This is truly a minor one, but there's no metadata for whatever reason? This one's not any sort of blocker I don't think but it's a puzzling omission.

It also might be nice if there were a better way to input link references -- having to type out the full file path and library name each time is quite a pain. OTOH, I think Truffle's linker just goes by name and ignores file path, so maybe that wouldn't be such a problem for someone doing this with Truffle after all; they could just use the library name. On the third hand, that seems like something of a problem with Truffle's linker! But I also think it's not changing anytime soon...

[axic]

It sounds like you're intending on supporting multiple file compilation at some future point, and I think at the least it would make sense for us to wait until that exists.

I am not sure if we have such intent, we did keep the JSON format similar to the one required by Solidity to ease integration. Supporting multiple files would be useful if we introduce some kind of import statement in Yul, or to allow compilation of a large number of Yul sources, which produce independent output. While this is an important feature for Solidity given the processing, analysis, and code generation is heavy, with Yul it is rather simple and straightfoward.

Regarding sources, AST, references, etc: can you create an issue on the Solidity repository?

Regarding metadata: there is currently no way in the internal infrastructure to pass metadata to yul, but it is planned in order to have the IR completed. Not sure however if we plan to emit that for a single run of Yul compilation, not originating from Solidity.

[haltman-at]

Yeah, the multi-source thing here is obviously just to keep things more unified on our side. We could also handle things one source at a time if we wanted to get this working right now; it's just that, due to how our code is organized, switching over later would be a pain. :-/

But you're right, I misremembered the error message -- for some reason I thought the error message implied that this was coming later, and it doesn't. So, if you actually don't have any such intent anytime soon, yeah, I could totally do it one source at a time. :) I'd probably want some of the other issues mentioned above resolved first, particularly the lack of sources output. It's not crucial, and I think it would be doable without that, but having the sources output would definitely make things easier (as in, less need to fill with special cases some already-intimidating code).

Anyway yeah I'll go file issues for those. Thanks!

[axic]

We discussed briefly internally (with @chriseth) and we do not have a short-term plan for supporting multiple Yul files in the same run. I could see it happen in the long-term, but we have so many other things taking priority right now. If you are still interested in multi-source could you please open an issue for it on Solidity so it does not get forgotten?

[haltman-at]

Well, I'm not sure how interested we are in multi-source... after all, my upcoming PR to deal with this is made to work with single-source! I can file an issue anyway if you think I should?

[haltman-at]

Hm, actually, there is perhaps a minor issue with single-file after all... well, I'll talk about this with @gnidan and others and we'll figure out what we're doing here.

[cds-amal]

Hi @axic the new release should address this issue.