Reading of URL is not allowed with custom filesystem

[jazdw]

With Graal 20.2.0 Creating a context with a custom FS like this

Context.newBuilder("js")
        .allowHostAccess(HostAccess.ALL)
        .allowAllAccess(true)
        .fileSystem(customFs)
        .option(JSContextOptions.LOAD_FROM_URL_NAME, "true")

Then trying to to load a file from a URL results in the following:

EvalError: Reading of URL https://abc:8443/xyz.js is not allowed.
	at <js> :program(filestore\default\test-script\load_http.js:1:0-84)
	at org.graalvm.polyglot.Context.eval(Context.java:345)
	at com.oracle.truffle.js.scriptengine.GraalJSScriptEngine.eval(GraalJSScriptEngine.java:405)
	at com.oracle.truffle.js.scriptengine.GraalJSScriptEngine.eval(GraalJSScriptEngine.java:347)
	at java.scripting/javax.script.AbstractScriptEngine.eval(AbstractScriptEngine.java:249)

Is there any way to allow loading from HTTP with a custom FS?

[eleinadani]

@jazdw, using URLs from a custom FS should work out of the box. For example, this is a unit test demonstrating how to import an ES module, while this is a test that we just added that uses the load builtin. You can have a look and see how polyglot contexts are created and initialized; if you still have problems, perhaps could you share your custom FS implementation and your JS sources so that we can try to reproduce the problem?

[jazdw]

@eleinadani
Custom FS does nothing but delegate to an instance of FileSystem.newDefaultFileSystem()
Note that I am not talking about module imports here, I am talking about the load() function. e.g. load(https://abc:8443/xyz.js) Sorry I should have been more clear.

Related to imports and URLs though is #257 which I still don't think are fixed. I made a comment there but you may not have seen it as it is closed.

[jazdw]

Here's a quick test to demonstrate:
https://github.com/infiniteautomation/ma-modules-public/blob/8fc371b4b52e4d9116430aa4bdc7bfc8372571d6/graaljs/src-test/com/infiniteautomation/mango/graaljs/CustomFileSystemTest.java

[eleinadani]

Thanks for sharing your code @jazdw. The default FS (i.e., delegate in your tests) does not support HTTP URLs; this is intentional, because by default a Polyglot Context uses the local FS to load sources, but cannot download source files from the network.

If you want to load your sources from the network, you can implement a custom FS that maps HTTP URLs to some Path on your local filesystem (e.g., some temporary file created after downloading your sources from the network).

Your load() test, fails because delegate.parsePath(uri) used in public Path parsePath(URI uri) (here) uses the default FS, which cannot parse HTTP URLs and throws UnsupportedOperationException. You can change your custom FS implementation to something like:

@Override
public Path parsePath(URI uri) {
    return Paths.get("/path/to/tmp/downloaded/file.js");
}

and the call to load('https://abc:8443/xyz.js') in your test should work. Conceptually, this is what we do in our test FS, where we map an HTTP URL to some testing Path.

[jazdw]

Thanks for sharing your code @jazdw. The default FS (i.e., delegate in your tests) does not support HTTP URLs; this is intentional, because by default a Polyglot Context uses the local FS to load sources, but cannot download source files from the network.

If you want to load your sources from the network, you can implement a custom FS that maps HTTP URLs to some Path on your local filesystem (e.g., some temporary file created after downloading your sources from the network).

Yep I understand this, this is what #255 is about.

You can change your custom FS implementation to something like:

@Override
public Path parsePath(URI uri) {
    return Paths.get("/path/to/tmp/downloaded/file.js");
}

and the call to load('https://abc:8443/xyz.js') in your test should work.

And indeed it does. I have updated my tests to more explicitly show what does and what does not work. I've summarized the results and copied the output below.

• load(http) with custom filesystem and mapping to Path - WORKS
• load(http) with custom filesystem without mapping - ERROR (Reading of URL is not allowed)
• load(http) with default filesystem - WORKS, loads from actual URL
• import from http with custom filesystem and mapping to Path - ERROR (InvalidPathException: Illegal char <:> at index 5)

[INFO] Running com.infiniteautomation.mango.graaljs.CustomFileSystemTest
[ERROR] Tests run: 6, Failures: 1, Errors: 2, Skipped: 0, Time elapsed: 2.044 s <<< FAILURE! - in com.infiniteautomation.mango.graaljs.CustomFileSystemTest
[ERROR] testLoadHttpCustomFs(com.infiniteautomation.mango.graaljs.CustomFileSystemTest)  Time elapsed: 0.035 s  <<< FAILURE!
org.junit.ComparisonFailure: expected:<EvalError: [abc.example.org]> but was:<EvalError: [Reading of URL https://abc.example.org/xyz.js is not allowed.]>
	at com.infiniteautomation.mango.graaljs.CustomFileSystemTest.testLoadHttpCustomFs(CustomFileSystemTest.java:64)

[ERROR] testImportHttpCustomFsWithMapping(com.infiniteautomation.mango.graaljs.CustomFileSystemTest)  Time elapsed: 0.043 s  <<< ERROR!
org.graalvm.polyglot.PolyglotException: java.nio.file.InvalidPathException: Illegal char <:> at index 5: https://abc.example.org/xyz.mjs
	at com.infiniteautomation.mango.graaljs.CustomFileSystemTest.testImportHttpCustomFsWithMapping(CustomFileSystemTest.java:136)

[ERROR] testLoadHttpRealUrlCustomFs(com.infiniteautomation.mango.graaljs.CustomFileSystemTest)  Time elapsed: 0.015 s  <<< ERROR!
org.graalvm.polyglot.PolyglotException: EvalError: Reading of URL https://unpkg.com/is-number is not allowed.
	at com.infiniteautomation.mango.graaljs.CustomFileSystemTest.testLoadHttpRealUrlCustomFs(CustomFileSystemTest.java:112)

[INFO] 
[INFO] Results:
[INFO] 
[ERROR] Failures: 
[ERROR]   CustomFileSystemTest.testLoadHttpCustomFs:64 expected:<EvalError: [abc.example.org]> but was:<EvalError: [Reading of URL https://abc.example.org/xyz.js is not allowed.]>
[ERROR] Errors: 
[ERROR]   CustomFileSystemTest.testImportHttpCustomFsWithMapping:136 » Polyglot java.nio...
[ERROR]   CustomFileSystemTest.testLoadHttpRealUrlCustomFs:112 » Polyglot EvalError: Rea...
[INFO] 
[ERROR] Tests run: 6, Failures: 1, Errors: 2, Skipped: 0

[eleinadani]

And indeed it does.

Great, happy to hear that load() now works for you.

load(http) with custom filesystem and mapping to Path - WORKS
load(http) with custom filesystem without mapping - ERROR (Reading of URL is not allowed)
load(http) with default filesystem - WORKS, loads from actual URL

As discussed above, the 2nd failure (Reading of URL is not allowed) is expected, because the default Truffle FS does not support HTTP URLs.

import from http with custom filesystem and mapping to Path - ERROR (InvalidPathException: Illegal char <:> at index 5)

I had a look at this other failure, and indeed this was a bug on our side: I have fixed it in this commit -- Let us know if the fix works for you, and thanks for reporting the issue!

[jazdw]

As discussed above, the 2nd failure (Reading of URL is not allowed) is expected, because the default Truffle FS does not support HTTP URLs.

This is not correct, please see the test case testLoadHttpRealUrlDefaultFs() it does work with the default FS!

I had a look at this other failure, and indeed this was a bug on our side: I have fixed it in this commit -- Let us know if the fix works for you, and thanks for reporting the issue!

Thank you! Is there a nightly snapshot build of Graal/Truffle/GraalJS to test with?

[jazdw]

This is how loading from HTTP works with the default filesystem - https://github.com/oracle/graal/blob/fed1f661c5770c01a365d9eaf69b296bdc960759/truffle/src/com.oracle.truffle.api/src/com/oracle/truffle/api/source/Source.java#L1028