File System settings

[stefanofornari]

AS A user
I WANT to pass file system configuration at file system creation (i.e. FileSystems.newFileSystem())
SO THAT I can set the run time behaviour of the file system

NOTES:

1. we can use aws.region, aws.accessKey and aws.accessKeySecret for region and credentials
2. we could introduce aws.protocol = http/https to specify which protocol to use for endpoint different than QWS S3
3. we could introduce aws.endpoint to specify the endpoint to use if not otherwise given
4. properties passed to newFileSystem() overwrites environment variables and system properties
5. properties passed to newFileSystem() are overwritten by the values provided in URIs

@markjschreiber can you please provide feedback on the notes above?

[markjschreiber]

Some guidance

1. The AWS SDK will pickup the region and credentials from the standard environment parameters, config files and EC2 instance role, additional configuration options for these isn't needed.
2. The protocol can just be derived from the endpoint URL/URI it is not needed separately
3. We should not use aws because it may one day collide with another AWS configuration variable. We already have other environment variables like S3_SPI_READ_MAX_FRAGMENT_SIZE so maybe use s3spi.endpoint or S3_SPI_ENDPOINT
4. Order of precedence is documented in the README.md and is currently:
   1 Java properties
   2 Environment variables (System properties)
   3 Default values

[stefanofornari]

Thanks, this helps. There is only one bullet I am not sure I have got it right. bullet 2. URIs to access a bucket even in the case of an endpoint would be something like:

s3://endpoint.somewhere.com/bucket

The scheme/protocol is going to to be s3 anyway. Are you suggesting we should use s3://https://endpoint.somewhere.com/bucket ? This would probably not be parsed as a URI.

[markjschreiber]

Can you give an example of an endpoint that you are wanting to process where you need to determine the protocol?

[dannycjones]

Thanks, this helps. There is only one bullet I am not sure I have got it right. bullet 2. URIs to access a bucket even in the case of an endpoint would be something like:

s3://endpoint.somewhere.com/bucket

The scheme/protocol is going to to be s3 anyway. Are you suggesting we should use s3://https://endpoint.somewhere.com/bucket ? This would probably not be parsed as a URI.

Does it make more sense to keep endpoint as a separate parameter?

For example, configuring an environment variable like this:

S3_SPI_ENDPOINT=https://endpoint.somewhere.com/

Then, just go ahead and use the URI s3://bucket/.

If you needed to use HTTP rather than HTTPS, then that would be configurable in a consistent way with other S3 tooling. I believe this is what @markjschreiber is suggesting.

---

This is similar to AWS CLI where you can set the endpoint separately using a CLI option:

aws s3 --endpoint-url https://endpoint.somewhere.com ls s3://bucket/

Or Apache Hadoop's S3A connector using a system property:

bin/hadoop fs -D fs.s3a.endpoint=https://endpoint.somewhere.com -ls s3a://bucket/

[markjschreiber]

Agree, I think endpoint is separate to the s3 bucket URI

[stefanofornari]

Ok, I'll go with a separate parameter.

[stefanofornari]

The name of the property is going to be s3.spi.endpoint