enable hooks for fetching code behind solidity import statements #80

serapath · 2018-12-19T03:52:25Z

`@todo`

enable hooks for fetching and caching code behind solidity import statements

about imports

The text was updated successfully, but these errors were encountered:

alincode · 2018-12-23T04:01:27Z

ethereum solution:
truffle solution: compile once, then get the error message to find import path.
- https://github.com/trufflesuite/truffle/blob/develop/packages/truffle-compile/parser.js
- https://github.com/trufflesuite/truffle/tree/develop/packages/truffle-resolver
etheratom solution:
- code
  - https://github.com/0mkara/etheratom/blob/7479f2f9a003616bf5220b70ca8503d88f560834/lib/helpers/compiler-imports.js
  - https://github.com/0mkara/etheratom/blob/c2523b28d2587da9f9f7ef02dca98e383a00f6e0/lib/web3/web3.js#L242
- relation issue
  - combineSources for large library imports does not work 0mkara/etheratom#201
resolver-engine solution
- https://github.com/resolver-engine/resolver-engine
- relation issue
  - FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory pubkey/solidity-cli#2
features
- Support async read callbacks. solc-js#140

alincode · 2018-12-23T04:47:24Z

keywords

combineSource
resolver
import parser

alincode · 2018-12-27T09:28:31Z

https://docs.google.com/document/d/1PZBSCBWBwd6AqWCgXqLnw8FNQ4HRurP5usrXuKuU0a0/edit#heading=h.qp8fp07k2avk

https://blog.zeppelin.solutions/solidity-compiler-audit-8cfc0316a420

alincode · 2018-12-31T14:55:05Z

@serapath

readCallback function can't be a async function, because emscripten doesn't support it very well.

but I think another workaround is provided "getReadCallback" API.

example 1:

let readCallback = await getReadCallback(sourceCode);
let output = await compiler(sourceCode, readCallback);

example 2:

let localSources = [{
       path: 'lib.sol',
       content: 'library L { function f() internal returns (uint) { return 7; } }'
     }];
 
let readCallback = await getReadCallback(sourceCode, localSources);
let output = await compiler(sourceCode, readCallback);

https://github.com/ethereum-play/solc-js/pull/1/commits/1db7eb24755acf1446487e7693073e2ca1be7301#diff-3b8192acb0d491039ab68dea0b430bfcR62

https://github.com/ethereum-play/solc-js/pull/1/commits/1db7eb24755acf1446487e7693073e2ca1be7301#diff-3b8192acb0d491039ab68dea0b430bfcR39

serapath · 2019-01-01T22:37:52Z

what about

const output = await compile(sourceode, async (importpath) => await fetch(importpath))

async function compile (source, cb) {
  const X = await getAllImports(source)
  const files = {}
  for (p of X) files[path] = await cb(p)
  const output = solcjs(source, X)
  return output
}

serapath · 2019-01-01T22:39:45Z

Are you very familiar with C++ ? ...because one other option would be to ask the solidity team whether a pull request to make imports async would be welcome :-)

alincode · 2019-01-02T16:55:51Z

@serapath
no, I don't familiar with C++.
but about asked solidity team "support async read callback".
in the past, someone asked the same question, the answer is NO.

@chriseth answered this question in a long time ago.
ethereum/solc-js#140

alincode · 2019-01-07T07:25:35Z

example1: `await compiler(sourceCode, localSources)`

let compiler = await solcjs(version);

const sourceCode = `
  pragma solidity >0.4.99 <0.6.0;

  import "lib.sol";

  library OldLibrary {
    function someFunction(uint8 a) public returns(bool);
  }

  contract NewContract {
    function f(uint8 a) public returns (bool) {
        return OldLibrary.someFunction(a);
    }
  }`;

let localSources = [{
  path: 'lib.sol',
  content: 'library L { function f() internal returns (uint) { return 7; } }'
}];

let output = await compiler(sourceCode, localSources);

example2: `await compiler(sourceCode);`

let compiler = await solcjs(version);

const sourceCode = `
  import 'https://github.com/OpenZeppelin/openzeppelin-solidity/contracts/math/SafeMath.sol';

  library OldLibrary {
      function someFunction(uint8 a) public returns(bool);
  }

  contract NewContract {
      function f(uint8 a) public returns (bool) {
          return OldLibrary.someFunction(a);
      }
  }`;

let output = await compiler(sourceCode);

serapath · 2019-01-09T03:30:59Z

I would still propose that we hide the fact that the compiler is synchronous and use an API that enables to switch to async in the future if they ever enable it, so I would love to see something like:

const output = await compile(sourcecode, async (importpath) => await fetch(importpath))

async function compile (source, cb) {
  const X = await getAllImports(source)
  const files = []
  for (p of X) {
    var content = await cb(p)
    files.push({ path: p , content})
  }
  // let localSources = [{
    // path: 'lib.sol',
    // content: 'library L { function f() internal returns (uint) { return 7; } }'
  // }];
  const output = solcjs(source, files)
  return output
}

alincode · 2019-01-09T03:56:51Z

=== solc-js ====

const compile = require('solc-js')

async function compile (source, resolver) {
  const localSources = await resolver.getLocalSources();
  const readcallback = await resolver.combineSource(source, localSources)
  const output = solcjs(source, readcallback);
  return output;
}

=== user test case ====

const resolver = new resolver(localSources);
let output = await solc.compile(source, resolver);

serapath · 2019-01-09T04:00:13Z

const compile = require('solc-js')
/*
function compile (sourcecode, getContent) {
  var allimports = parse(sourcecode)
  var localSources = []
  // e.g. runs 10x when 10 imports in sourcecode
  for (path of allimports) {
    localSources.push({ path, content: await getContent(path) })
  }
  return solcjs(sourcecode, localSources)  
}
*/

const myDB = { } // maybe it looks like:

const sourcecode = `
  import 'https://github.com/OpenZeppelin/openzeppelin-solidity/contracts/math/a.sol';
  import 'https://github.com/OpenZeppelin/openzeppelin-solidity/contracts/math/b.sol';
  import 'https://github.com/OpenZeppelin/openzeppelin-solidity/contracts/math/c.sol';
  import 'https://github.com/OpenZeppelin/openzeppelin-solidity/contracts/math/d.sol';
  import 'https://github.com/OpenZeppelin/openzeppelin-solidity/contracts/math/e.sol';
  import 'lib1.sol'
  import 'lib2.sol'
  import 'lib3.sol'
  import 'lib4.sol'
  import 'lib5.sol'

  library OldLibrary {
      function someFunction(uint8 a) public returns(bool);
  }

  contract NewContract {
      function f(uint8 a) public returns (bool) {
          return OldLibrary.someFunction(a);
      }
  }`

test('sourcecode with 10 imports calls callback 10 times', function (t) {
  t.plan(10)
  const output = await compile(sourcecode, async (path) => {
    t.assert(true)
    console.log(path)
    var output
    if (myDB.contains(path)) output = await myDB.get(path)
    else output = new Promise(resolve => {
      resolver.import(path, null, ({ content }) => resolve(content)
    })
    return output
  })
})

/*
  'https://github.com/OpenZeppelin/openzeppelin-solidity/contracts/math/a.sol';
  'https://github.com/OpenZeppelin/openzeppelin-solidity/contracts/math/b.sol';
  'https://github.com/OpenZeppelin/openzeppelin-solidity/contracts/math/c.sol';
  'https://github.com/OpenZeppelin/openzeppelin-solidity/contracts/math/d.sol';
  'https://github.com/OpenZeppelin/openzeppelin-solidity/contracts/math/e.sol';
  'lib1.sol'
  'lib2.sol'
  'lib3.sol'
  'lib4.sol'
  'lib5.sol'
*/

const output = await compile(sourcecode, async (path) => 
    myDB.contains(path) ? await myDB.get(path) : await getImportContent(path)
)

// so "getImportContent(...)" is a replacement for
new Promise(resolve => resolver.import(path, null, ({ content }) => resolve(content))

alincode · 2019-01-11T09:21:22Z

https://github.com/ethereum-play/solc-js/pull/2

@serapath now, it like this. (import for loop not finish yet, so just check getContent function interface.)

describe('compiler', () => {

  const version = 'v0.5.1-stable-2018.12.03';

  it('with local import', async () => {
    let compiler = await solcjs(version);
    compiler.should.be.a('function');

    const sourceCode = `
    pragma solidity >0.4.99 <0.6.0;

    import "lib.sol";

    library OldLibrary {
      function someFunction(uint8 a) public returns(bool);
    }

    contract NewContract {
      function f(uint8 a) public returns (bool) {
          return OldLibrary.someFunction(a);
      }
    }`;

    let myDB = new Map();
    myDB.set('lib.sol', 'library L { function f() internal returns (uint) { return 7; } }');

    const getImportContent = async function (path) {
      return myDB.has(path) ? myDB.get(path) : await solcResolver.getImportContent(path);
    };

    let output = await compiler(sourceCode, getImportContent);
    let item = output[0];
    item.should.have.all.keys('name', 'abi', 'sources', 'compiler', 'assembly', 'binary', 'metadata');
    item.metadata.analysis.should.have.all.keys('warnings', 'others');
  });

  it('with github import', async () => {
    let compiler = await solcjs('v0.4.25-stable-2018.09.13');
    compiler.should.be.a('function');
    const sourceCode = `
    import 'https://github.com/OpenZeppelin/openzeppelin-solidity/contracts/math/SafeMath.sol';

    library OldLibrary {
        function someFunction(uint8 a) public returns(bool);
    }

    contract NewContract {
        function f(uint8 a) public returns (bool) {
            return OldLibrary.someFunction(a);
        }
    }`;

    let output = await compiler(sourceCode);
    let item = output[0];
    item.should.have.all.keys('name', 'abi', 'sources', 'compiler', 'assembly', 'binary', 'metadata');
    item.metadata.analysis.should.have.all.keys('warnings', 'others');
  });

});

solc-js still will a little depend solc-resolver, because I will check.

https://github.com/ethereum-play/solc-js/blob/feature/getContent/src/lib/getCompile.js#L18

if (solcImport.isExistImport(sourcecode)) {
   if (getImportContent == undefined) {
      // solution1: auto fixed getImportContent
      getImportContent = solcResolver.getImportContent;
      // solution2: just throw error, don't use solcResolver inside the compiler.
   } else if (typeof getImportContent !== 'function') {
      throw Error('getContent should be a funcion.');
    }
    readCallback = await solcImport.getReadCallback(sourcecode, getImportContent);
}

or you don't want I auto fixed getImportContent(just throw error), then I can remove solc-js depend solc-resolver. but that also meaning, if sourcecode exist import, then compile function two parameter must be require.

now

* if not exist import
  * await compiler(sourceCode);
* if exist import
  * await compiler(sourceCode);  // auto fixed getImportContent;
  * await compiler(sourceCode, getImportContent);

or you change lik this

* if not exist import
  * await compiler(sourceCode);
* if exist import, getImportContent is require.
  * await compiler(sourceCode, getImportContent);

serapath · 2019-01-11T16:37:05Z

I think we should make another module which imports/requires solc-js and many resolvers, and exports a compile function, which takes sourcecode and uses the correct resolvers, so users dont have to bother

serapath · 2019-01-11T16:45:00Z

const solcjs = require('solc-js')
const resolve_http = async x => await fetch(x).then(r => r.text())

const resolve_swarm = require('reaolve-sawrm')

module.exports = sourcecode => {
  const imports = getAllImports(sourcecode)

  for (path of imports) {
    if ( isSwarm(path)) resolve_swarm(path)
    // ...

serapath · 2019-01-23T07:42:42Z

maybe here is some resolver code

serapath mentioned this issue Dec 19, 2018

fix and improve solidity compiler #46

Closed

27 tasks

alincode self-assigned this Dec 23, 2018

ethereum deleted a comment from alincode Jan 9, 2019

serapath added the #compiler specification category label Feb 20, 2019

serapath added this to the v0.0.0 - first public play release milestone Feb 20, 2019

serapath mentioned this issue Aug 16, 2019

transition ethereum/play/issues and ethereum-play issues to playproject-io playproject-io/roadmapping#2

Closed

serapath closed this as completed Aug 16, 2019

serapath mentioned this issue Aug 26, 2019

2019.08.20-15:00Z | CANCELED playproject-io/roadmapping#25

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

enable hooks for fetching code behind solidity import statements #80

enable hooks for fetching code behind solidity import statements #80

serapath commented Dec 19, 2018 •

edited

Loading

alincode commented Dec 23, 2018 •

edited

Loading

alincode commented Dec 23, 2018 •

edited

Loading

alincode commented Dec 27, 2018 •

edited

Loading

alincode commented Dec 31, 2018 •

edited

Loading

serapath commented Jan 1, 2019

serapath commented Jan 1, 2019

alincode commented Jan 2, 2019 •

edited

Loading

alincode commented Jan 7, 2019

serapath commented Jan 9, 2019 •

edited

Loading

alincode commented Jan 9, 2019 •

edited

Loading

serapath commented Jan 9, 2019 •

edited

Loading

alincode commented Jan 11, 2019 •

edited

Loading

serapath commented Jan 11, 2019

serapath commented Jan 11, 2019

serapath commented Jan 23, 2019 •

edited

Loading

enable hooks for fetching code behind solidity import statements #80

enable hooks for fetching code behind solidity import statements #80

Comments

serapath commented Dec 19, 2018 • edited Loading

@todo

alincode commented Dec 23, 2018 • edited Loading

alincode commented Dec 23, 2018 • edited Loading

alincode commented Dec 27, 2018 • edited Loading

alincode commented Dec 31, 2018 • edited Loading

serapath commented Jan 1, 2019

serapath commented Jan 1, 2019

alincode commented Jan 2, 2019 • edited Loading

alincode commented Jan 7, 2019

example1: await compiler(sourceCode, localSources)

example2: await compiler(sourceCode);

serapath commented Jan 9, 2019 • edited Loading

alincode commented Jan 9, 2019 • edited Loading

serapath commented Jan 9, 2019 • edited Loading

alincode commented Jan 11, 2019 • edited Loading

serapath commented Jan 11, 2019

serapath commented Jan 11, 2019

serapath commented Jan 23, 2019 • edited Loading

serapath commented Dec 19, 2018 •

edited

Loading

`@todo`

alincode commented Dec 23, 2018 •

edited

Loading

alincode commented Dec 23, 2018 •

edited

Loading

alincode commented Dec 27, 2018 •

edited

Loading

alincode commented Dec 31, 2018 •

edited

Loading

alincode commented Jan 2, 2019 •

edited

Loading

example1: `await compiler(sourceCode, localSources)`

example2: `await compiler(sourceCode);`

serapath commented Jan 9, 2019 •

edited

Loading

alincode commented Jan 9, 2019 •

edited

Loading

serapath commented Jan 9, 2019 •

edited

Loading

alincode commented Jan 11, 2019 •

edited

Loading

serapath commented Jan 23, 2019 •

edited

Loading