Skip to content

Commit

Permalink
Merge pull request Azure#334 from rickle-msft/New-Storage-SDK-V10-Pre…
Browse files Browse the repository at this point in the history
…view-dev

New storage sdk v10 preview dev
  • Loading branch information
rickle-msft authored Jul 3, 2018
2 parents db3f3f7 + f72ada5 commit 6a5ffc4
Show file tree
Hide file tree
Showing 74 changed files with 3,653 additions and 1,246 deletions.
5 changes: 4 additions & 1 deletion BreakingChanges.txt
Original file line number Diff line number Diff line change
@@ -1,4 +1,7 @@
XXXX.XX.XX Version XX.X.X-Preview
2018.07.03 Version 10.0.1-Preview
* Created the StorageException type, which deserializes the XML payload in an error response if present and gives access to the ErrorCode header as a property.
* Changed the AppendBlobAccessConditions field types to be Long instead of Int.
* Changed RequestRetryOptions maxTries and tryTimeout fields to be Integer instead of int. 0 is no longer allowed.
* Changed the return type of BlobURL.download to be a DownloadResponse instead of BlobsDownloadResponse for integration with RetryReader.
* Changed CommonRestResponse.lastModifiedTime to be lastModified.
* Changed the dateProperty field in all auto-generated files to be date.
9 changes: 8 additions & 1 deletion ChangeLog.txt
Original file line number Diff line number Diff line change
@@ -1,10 +1,17 @@
XXXX.XX.XX Version XX.X.X-Preview
2018.07.03 Version 10.0.1-Preview
* Added the RetryReader class to allow for more reliable streaming on large downloads. This is now the return type of blobURL.download
* Fixed a bug that caused generation of signatures to fail at high levels of parallelism.
* Created the StorageException type to give easy access to the ErrorCode, StatusCode, and Message as available for unsuccessful responses.
* Added the StorageErrorCode type for checking against error codes returned by the service.
* Changed the AppendBlobAccessConditions field types to be Long instead of Int.
* Upgraded Netty dependency to allow uploading memory mapped files with https.
* Upgraded the autorest runtime dependency to fix a dependency bug in their package.
* Changed RequestRetryOptions maxTries and tryTimeout fields to be Integer instead of int. 0 is no longer allowed.
* Changed CommonRestResponse.lastModifiedTime to be lastModified.
* Added statusCode property to CommonRestResponse.
* Change dateProperty to be date on all generated types.
* Fixed a bug that prevented proper reset of body stream upon retry.
* Updated the defaults for RequestRetryOptions.

2018.04.27 Version 10.0.0-preview
* Initial Release. Please see the README and wiki for information on the new design.
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ To get the binaries of this library as distributed by Microsoft, ready for use w
<dependency>
<groupId>com.microsoft.azure</groupId>
<artifactId>azure-storage-blob</artifactId>
<version>10.0.0-Preview</version>
<version>10.0.1-Preview</version>
</dependency>
```

Expand Down
103 changes: 51 additions & 52 deletions pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@

<groupId>com.microsoft.azure</groupId>
<artifactId>azure-storage-blob</artifactId>
<version>10.0.0-Preview</version>
<version>10.0.1-Preview</version>

<name>Azure Storage Blob</name>
<description>The Azure Storage Java Blob library.</description>
Expand Down Expand Up @@ -65,7 +65,7 @@
<dependency>
<groupId>com.microsoft.rest.v2</groupId>
<artifactId>client-runtime</artifactId>
<version>2.0.0-beta2</version>
<version>2.0.0-beta3</version>
</dependency>
<dependency>
<groupId>junit</groupId>
Expand All @@ -89,55 +89,54 @@

<build>
<testSourceDirectory>src/test/java</testSourceDirectory>
<pluginManagement>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.1</version>
<configuration>
<compilerId>groovy-eclipse-compiler</compilerId>
<compilerArgument>-Xlint:unchecked</compilerArgument>
<source>1.8</source>
<target>1.8</target>
<showDeprecation>true</showDeprecation>
</configuration>
<dependencies>
<dependency>
<groupId>org.codehaus.groovy</groupId>
<artifactId>groovy-eclipse-compiler</artifactId>
<version>2.9.3-01</version>
</dependency>
<dependency>
<groupId>org.codehaus.groovy</groupId>
<artifactId>groovy-eclipse-batch</artifactId>
<version>2.4.15-01</version>
</dependency>
</dependencies>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.20.1</version>
<configuration>
<includes>
<include>**/Test*.*</include>
<include>**/*Test.*</include>
<include>**/*Tests.*</include>
<include>**/*Samples*.*</include>
</includes>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-release-plugin</artifactId>
<version>2.5.2</version>
</plugin>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>build-helper-maven-plugin</artifactId>
</plugin>
</plugins>
</pluginManagement>

<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.1</version>
<configuration>
<compilerId>groovy-eclipse-compiler</compilerId>
<compilerArgument>-Xlint:unchecked</compilerArgument>
<source>1.8</source>
<target>1.8</target>
<showDeprecation>true</showDeprecation>
</configuration>
<dependencies>
<dependency>
<groupId>org.codehaus.groovy</groupId>
<artifactId>groovy-eclipse-compiler</artifactId>
<version>2.9.3-01</version>
</dependency>
<dependency>
<groupId>org.codehaus.groovy</groupId>
<artifactId>groovy-eclipse-batch</artifactId>
<version>2.4.15-01</version>
</dependency>
</dependencies>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.20.1</version>
<configuration>
<includes>
<include>**/Test*.*</include>
<include>**/*Test.*</include>
<include>**/*Tests.*</include>
<include>**/*Samples*.*</include>
</includes>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-release-plugin</artifactId>
<version>2.5.2</version>
</plugin>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>build-helper-maven-plugin</artifactId>
</plugin>
</plugins>
</build>
</project>
13 changes: 11 additions & 2 deletions src/main/java/com/microsoft/azure/storage/blob/BlobURL.java
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,8 @@
import com.microsoft.azure.storage.blob.models.*;
import com.microsoft.rest.v2.http.HttpPipeline;
import io.reactivex.Single;
import io.reactivex.SingleSource;
import io.reactivex.functions.Function;

import java.net.MalformedURLException;
import java.net.URL;
Expand Down Expand Up @@ -192,6 +194,9 @@ public Single<BlobsAbortCopyFromURLResponse> abortCopyFromURL(
* information, see the
* <a href="https://docs.microsoft.com/rest/api/storageservices/get-blob">Azure Docs</a>.
*
* Please consider using the {@link RetryReader} in conjunction with this method to achieve more reliable downloads
* that are resilient to transient network failures.
*
* @param range
* {@link BlobRange}
* @param accessConditions
Expand All @@ -201,7 +206,7 @@ public Single<BlobsAbortCopyFromURLResponse> abortCopyFromURL(
* @return
* Emits the successful response.
*/
public Single<BlobsDownloadResponse> download(
public Single<DownloadResponse> download(
BlobRange range, BlobAccessConditions accessConditions, boolean rangeGetContentMD5) {

Boolean getMD5 = rangeGetContentMD5 ? rangeGetContentMD5 : null;
Expand All @@ -216,7 +221,11 @@ public Single<BlobsDownloadResponse> download(
accessConditions.getHttpAccessConditions().getIfUnmodifiedSince(),
accessConditions.getHttpAccessConditions().getIfMatch().toString(),
accessConditions.getHttpAccessConditions().getIfNoneMatch().toString(),
null));
null))
.map(response ->
new DownloadResponse(response.statusCode(), response.headers(),
response.rawHeaders(), response.body()));

}

/**
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,17 @@
/*
* Copyright Microsoft Corporation
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.microsoft.azure.storage.blob;

import com.microsoft.azure.storage.blob.models.BlockBlobsCommitBlockListResponse;
Expand Down Expand Up @@ -34,6 +48,17 @@ private CommonRestResponse() {
commitBlockListResponse = null;
}

/**
* @return
* The status code for the response
*/
public int statusCode() {
if (uploadBlobResponse != null) {
return uploadBlobResponse.statusCode();
}
return commitBlockListResponse.statusCode();
}

/**
* @return
* An HTTP Etag for the blob at the time of the request.
Expand All @@ -49,7 +74,7 @@ public String eTag() {
* @return
* The time when the blob was last modified.
*/
public OffsetDateTime lastModifiedTime() {
public OffsetDateTime lastModified() {
if (uploadBlobResponse != null) {
return uploadBlobResponse.headers().lastModified();
}
Expand All @@ -73,9 +98,9 @@ public String requestId() {
*/
public OffsetDateTime date() {
if (uploadBlobResponse != null) {
return uploadBlobResponse.headers().dateProperty();
return uploadBlobResponse.headers().date();
}
return commitBlockListResponse.headers().dateProperty();
return commitBlockListResponse.headers().date();
}

/**
Expand Down
17 changes: 1 addition & 16 deletions src/main/java/com/microsoft/azure/storage/blob/Constants.java
Original file line number Diff line number Diff line change
Expand Up @@ -121,7 +121,7 @@ static final class HeaderConstants {
/**
* Specifies the value to use for UserAgent header.
*/
static final String USER_AGENT_VERSION = "0.0.0";
static final String USER_AGENT_VERSION = "10.0.1-Preview";

private HeaderConstants() {
// Private to prevent construction.
Expand All @@ -143,21 +143,6 @@ private HeaderConstants() {
*/
static final int MB = 1024 * KB;

/**
* Specifies the maximum number of bytes that can be sent in a call to PutBlock.
*/
public static final int MAX_BLOCK_SIZE = 100 * MB;

/**
* Specifies the maximum number of blocks allowed in a block blob.
*/
public static final int MAX_BLOCKS = 50000;

/**
* Specifies the maximum number of bytes that may be put in a single upload operation.
*/
public static final int MAX_PUT_BLOB_BYTES = 256 * MB;

/**
* An empty {@code String} to use for comparison.
*/
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,94 @@
/*
* Copyright Microsoft Corporation
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package com.microsoft.azure.storage.blob;

import com.microsoft.azure.storage.blob.models.BlobsDownloadHeaders;
import com.microsoft.rest.v2.RestResponse;
import io.reactivex.Flowable;
import io.reactivex.Single;
import io.reactivex.internal.functions.Functions;

import java.io.Closeable;
import java.nio.ByteBuffer;
import java.util.Map;

/**
* {@code DownloadResponse} wraps the protocol-layer response from {@link BlobURL#download(BlobRange,
* BlobAccessConditions, boolean)} to help provide information for retrying.
*/
public class DownloadResponse extends RestResponse<BlobsDownloadHeaders, Flowable<ByteBuffer>> implements Closeable {
private BlobURL blobURL;

private RetryReader.HTTPGetterInfo info;

DownloadResponse(int statusCode, BlobsDownloadHeaders blobsDownloadHeaders, Map<String, String> rawHeaders,
Flowable<ByteBuffer> byteBufferFlowable) {
super(statusCode, blobsDownloadHeaders, rawHeaders, byteBufferFlowable);
}

/**
* Constructs a new {@link RetryReader} stream for reliably reading data if desired (if {@code options != null} and
* {@code options.maxRetryRequests > 0}. If retries are enabled, if a connection fails while reading, the {@code
* RetryReader} will make additional requests to reestablish a connection and continue reading.
*
* @param options
* {@link RetryReaderOptions}
* @return A {@code Flowable} which emits the data as {@code ByteBuffer}s.
*/
public Flowable<ByteBuffer> body(RetryReaderOptions options) {
options = options == null ? new RetryReaderOptions() : options;
if (options.maxRetryRequests == 0) {
return super.body();
}
return new RetryReader(Single.just(this), this.info, options,
newInfo -> this.blobURL.download(new BlobRange(newInfo.offset, newInfo.count), new BlobAccessConditions(
new HTTPAccessConditions(null, null, info.eTag, null),
null, null, null),
false));
}

@Override
public int statusCode() {
return super.statusCode();
}

@Override
public BlobsDownloadHeaders headers() {
return super.headers();
}

@Override
public Map<String, String> rawHeaders() {
return super.rawHeaders();
}

/**
* Equivalent to calling {@link #body(RetryReaderOptions)} with {@code null}.
*/
@Override
public Flowable<ByteBuffer> body() {
return this.body(null);
}

/**
* Disposes of the connection associated with this stream response.
*/
@Override
public void close() {
// Taken from BlobsDownloadResponse.
body().subscribe(Functions.emptyConsumer(), Functions.<Throwable>emptyConsumer()).dispose();
}
}
Loading

0 comments on commit 6a5ffc4

Please sign in to comment.