Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add NIO write operations #5

Merged
merged 13 commits into from
Feb 23, 2023
Merged
Show file tree
Hide file tree
Changes from 11 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 21 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -115,16 +115,32 @@ reduce the number of requests to the S3 service.

Ensure sufficient memory is available to your JVM if you increase the fragment size or fragment number.

## Writing Files

The mode of the channel is controlled with the `StandardOpenOptions`. To open a channel for write access you need to
supply the option `StandardOpenOption.READ`. All write operations on the channel will be gathered in a temporary file,
which will be uploaded to S3 upon closing the channel.

Be aware, that the current implementation only supports channels to be used either for read or write due to potential
consistency issues we may face in some cases.
markjschreiber marked this conversation as resolved.
Show resolved Hide resolved

### Configuration

The handling of large files could take potentially very long, therefore there are currently no timeouts configured per
markjschreiber marked this conversation as resolved.
Show resolved Hide resolved
default. However, you may configure timeouts via the `S3SeekableByteChannel`.

#### Timeouts
To configure timeouts for writing files or opening files for write access, you may use the `Long timeout` and
`TimeUnit timeUnit` parameters of the `S3SeekableByteChannel` constructor.
```java
new S3SeekableByteChannel(s3Path, s3Client, channelOpenOptions, timeout, timeUnit);
```

## Design Decisions

As an object store, S3 is not completely analogous to a traditional file system. Therefore, several opinionated decisions
were made to map filesystem concepts to S3 concepts.

### Read Only

The current implementation only supports read operations. It would be possible to add write operations, however special consideration
would be needed due to the lack of support for random writes in S3 and the read-after-write consistency of S3 objects.

### A Bucket is a `FileSystem`

An S3 bucket is represented as a `java.nio.spi.FileSystem` using an `S3FileSystem`. Although buckets are globally
Expand Down
10 changes: 6 additions & 4 deletions build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -44,9 +44,11 @@ dependencies {
testImplementation 'junit:junit:4.13.2'
testImplementation 'org.mockito:mockito-core:4.3.1'

implementation platform('software.amazon.awssdk:bom:2.17.285')
implementation platform('software.amazon.awssdk:bom:2.19.26')

implementation 'software.amazon.awssdk:s3'
implementation 'software.amazon.awssdk:s3-transfer-manager:2.19.26'
implementation 'software.amazon.awssdk.crt:aws-crt:0.21.7'
implementation 'org.slf4j:slf4j-api:1.7.36'
implementation 'ch.qos.logback:logback-classic:1.2.10'
implementation 'ch.qos.logback:logback-core:1.2.10'
Expand All @@ -65,7 +67,7 @@ publishing {
'access to S3 objects for Java applications using Java NIO.2 for file access.'
url = 'https://github.com/awslabs/aws-java-nio-spi-for-s3'
inceptionYear = '2022'
scm{
scm {
url = 'https://github.com/awslabs/aws-java-nio-spi-for-s3/tree/main'
connection = 'scm:git:ssh://[email protected]/awslabs/aws-java-nio-spi-for-s3.git'
developerConnection = 'scm:git:ssh://[email protected]/awslabs/aws-java-nio-spi-for-s3.git'
Expand All @@ -90,7 +92,7 @@ publishing {
}
}
repositories {
maven{
maven {
def releasesRepoUrl = 'https://aws.oss.sonatype.org/service/local/staging/deploy/maven2/'
def snapshotsRepoUrl = 'https://aws.oss.sonatype.org/content/repositories/snapshots'
url = version.endsWith('SNAPSHOT') ? snapshotsRepoUrl : releasesRepoUrl
Expand All @@ -104,7 +106,7 @@ signing {
}

javadoc {
if(JavaVersion.current().isJava9Compatible()) {
if (JavaVersion.current().isJava9Compatible()) {
options.addBooleanOption('html5', true)
}
}
Expand Down
5 changes: 1 addition & 4 deletions src/main/java/software/amazon/nio/spi/s3/S3ClientStore.java
Original file line number Diff line number Diff line change
Expand Up @@ -260,11 +260,8 @@ private S3AsyncClient asyncClientForRegion(String regionString){
Region region = regionString.equals("") ? Region.US_EAST_1 : Region.of(regionString);
logger.debug("bucket region is: '{}'", region.id());

return S3AsyncClient.builder()
return S3AsyncClient.crtBuilder()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice update

.region(region)
.overrideConfiguration(conf -> conf.retryPolicy(builder -> builder
.retryCondition(retryCondition)
.backoffStrategy(backoffStrategy)))
.build();
}

Expand Down
Loading