Broken handling of AWS S3 urls

[hindog]

Guidelines

Please note that GitHub issues are only meant for bug reports/feature requests. If you have questions on how to use APOC, please ask on the Neo4j Discussion Forum instead of creating an issue here.

Expected Behavior (Mandatory)

When using AWS S3 url with credentials embedded in the url (eg: s3://accessKey:secretKey[:sessionToken]@endpoint:port/bucket/key), procedures such as apoc.load.json should be able to read the data if secret key contains url-escaped characters.

Actual Behavior (Mandatory)

With a secret key contains a / (which they sometimes do), I tried two approaches:

1. Passing the secret UNESCAPED in the url string (eg [not a real secret]: ABCDefGHikbWUyzkbsIKapTBr/fk4VkGsGWDOS/2XYZ). This fails with:

Caused by: java.lang.NumberFormatException: Error at index 1 in: "ABCDefGHikbWUyzkbsIKapTBr"
	at java.lang.NumberFormatException.forCharSequence(NumberFormatException.java:81) ~[?:?]
	at java.lang.Integer.parseInt(Integer.java:735) ~[?:?]
	at java.net.URLStreamHandler.parseURL(URLStreamHandler.java:223) ~[?:?]
	at java.net.URL.<init>(URL.java:674) ~[?:?]
	at java.net.URL.<init>(URL.java:541) ~[?:?]
	at java.net.URL.<init>(URL.java:488) ~[?:?]
	at apoc.util.FileUtils$SupportedProtocols.from(FileUtils.java:119) ~[apoc-4.4.0.2.jar:4.4.0.2]
	at apoc.util.FileUtils.isFile(FileUtils.java:243) ~[apoc-4.4.0.2.jar:4.4.0.2]
	at apoc.ApocConfig.checkReadAllowed(ApocConfig.java:263) ~[apoc-4.4.0.2.jar:4.4.0.2]
	at apoc.util.FileUtils.inputStreamFor(FileUtils.java:163) ~[apoc-4.4.0.2.jar:4.4.0.2]
	at apoc.util.JsonUtil.loadJson(JsonUtil.java:95) ~[apoc-4.4.0.2.jar:4.4.0.2]
	at apoc.load.LoadJson.loadJsonStream(LoadJson.java:71) ~[apoc-4.4.0.2.jar:4.4.0.2]
	at apoc.load.LoadJson.jsonParams(LoadJson.java:60) ~[apoc-4.4.0.2.jar:4.4.0.2]
	at apoc.load.LoadJson.json(LoadJson.java:49) ~[apoc-4.4.0.2.jar:4.4.0.2]
	at org.neo4j.kernel.impl.proc.GeneratedProcedure_json21076469570321.apply(Unknown Source) ~[?:?]
	at org.neo4j.procedure.impl.ProcedureRegistry.callProcedure(ProcedureRegistry.java:235) ~[neo4j-procedure-4.4.3.jar:4.4.3]
	at org.neo4j.procedure.impl.GlobalProceduresRegistry.callProcedure(GlobalProceduresRegistry.java:352) ~[neo4j-procedure-4.4.3.jar:4.4.3]
	at org.neo4j.kernel.impl.newapi.AllStoreHolder.callProcedure(AllStoreHolder.java:1092) ~[neo4j-kernel-4.4.3.jar:4.4.3]
	at org.neo4j.kernel.impl.newapi.AllStoreHolder.procedureCallRead(AllStoreHolder.java:1004) ~[neo4j-kernel-4.4.3.jar:4.4.3]
	at org.neo4j.cypher.internal.runtime.interpreted.CallSupport$.$anonfun$callReadOnlyProcedure$1(CallSupport.scala:47) ~[neo4j-cypher-interpreted-runtime-4.4.3.jar:4.4.3]
	at org.neo4j.cypher.internal.runtime.interpreted.CallSupport$.callProcedure(CallSupport.scala:70) ~[neo4j-cypher-interpreted-runtime-4.4.3.jar:4.4.3]
	at org.neo4j.cypher.internal.runtime.interpreted.CallSupport$.callReadOnlyProcedure(CallSupport.scala:47) ~[neo4j-cypher-interpreted-runtime-4.4.3.jar:4.4.3]
	at org.neo4j.cypher.internal.runtime.interpreted.TransactionBoundReadQueryContext.callReadOnlyProcedure(TransactionBoundQueryContext.scala:1134) ~[neo4j-cypher-interpreted-runtime-4.4.3.jar:4.4.3]
	... 87 more

2. Passing the secret UNESCAPED in the url string (eg [this not a real secret]): ABCDefGHikbWUyzkbsIKapTBr%2Ffk4VkGsGWDOS%2F2XYZ). This fails with:

Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: The AWS Access Key Id you provided does not exist in our records.

So S3ParamsExtractor doesn't url-decode the secret key and apoc.util.FileUtils$SupportedProtocols.from expects the url to be encoded, so they are in conflict with each other.

How to Reproduce the Problem

1. Construct an S3 URL with a / in the secret key and call a procedure such as apoc.load.csv.
2. Construct an S3 URL with a %2F in the secret key and call a procedure such as apoc.load.csv.

Simple Dataset (where it's possible)

//Insert here a set of Cypher statements that helps us to reproduce the problem

Steps (Mandatory)

1. Obtain an AWS secret key with a / in the value
2. Try to load any S3 url (doesn't have to exist) while passing the credentials as part of the URL (either url-encoded or not)

Screenshots (where it's possibile)

Specifications (Mandatory)

• The values extracted from S3 URL should be url-decoded

Currently used versions

Versions

• OS: macOS 11.2.3
• Neo4j: 4.4.4
• Neo4j-Apoc: 4.4.0.2