Iqss/6489 - Direct uploads to S3 using presigned URLs

doc/release-notes/6485-multiple-stores.md

@@ -29,8 +29,4 @@ Any additional S3 options you have set will need to be replaced as well, followi
 
 Once these options are set, restarting the glassfish service is all that is needed to complete the change.  
 
-<<<<<<< HEAD
 Note that the "\-Ddataverse.files.directory", if defined, continues to control where temporary files are stored (in the /temp subdir of that directory), independent of the location of any 'file' store defined above.
-=======
-Note that the "\-Ddataverse.files.directory", if defined, continues to control where temporary files are stored (in the /temp subdir of that directory), independent of the location of any 'file' store defined above.
->>>>>>> branch 'IQSS/6485' of https://github.com/TexasDigitalLibrary/dataverse.git

doc/release-notes/6489-release-notes.md

@@ -0,0 +1,17 @@
+# S3 Direct Upload support
+
+S3 stores can now optionally be configured to support direct upload of files, as one option for supporting upload of larger files.
+
+General information about this capability can be found in the <a href="http://guides.dataverse.org/en/latest/developers/big-data-support.html">Big Data Support Guide</a> with specific information about how to enable it in the <a href="http://guides.dataverse.org/en/latest/installation/config.html">Configuration Guide</a> - File Storage section.
+
+**Upgrade Information:** 
+
+Direct upload to S3 is enabled per store by one new jvm option:
+
+    ./asadmin create-jvm-options "\-Ddataverse.files.<id>.upload-redirect=true"
+
+The existing :MaxFileUploadSizeInBytes property and ```dataverse.files.<id>.url-expiration-minutes``` jvm option for the same store also apply to direct upload.
+
+Direct upload via the Dataverse web interface is transparent to the user and handled automatically by the browser. Some minor differences in file upload exist: directly uploaded files are not unzipped and Dataverse does not scan their content to help in assigning a MIME type. Ingest of tabular files and metadata extraction from FITS files will occur, but can be turned off for files above a specified size limit through the new dataverse.files.<id>.ingestsizelimit jvm option.
+
+API calls to support direct upload also exist, and, if direct upload is enabled for a store in Dataverse, the latest DVUploader (v1.0.8) provides a'-directupload' flag that enables its use. 

doc/sphinx-guides/source/developers/big-data-support.rst

@@ -6,7 +6,52 @@ Big data support is highly experimental. Eventually this content will move to th
 .. contents:: |toctitle|
         :local:
 
-Various components need to be installed and configured for big data support.
+Various components need to be installed and/or configured for big data support.
+
+S3 Direct Upload and Download
+-----------------------------
+
+A lightweight option for supporting file sizes beyond a few gigabytes - a size that can cause performance issues when uploaded through the Dataverse server itself - is to configure an S3 store to provide direct upload and download via 'pre-signed URLs'. When these options are configured, file uploads and downloads are made directly to and from a configured S3 store using secure (https) connections that enforce Dataverse's access controls. (The upload and download URLs are signed with a unique key that only allows access for a short time period and Dataverse will only generate such a URL if the user has permission to upload/download the specific file in question.)
+
+This option can handle files >40GB and could be appropriate for files up to a TB. Other options can scale farther, but this option has the advantages that it is simple to configure and does not require any user training - uploads and downloads are done via the same interface as normal uploads to Dataverse.
+
+To configure these options, an administrator must set two JVM options for the Dataverse server using the same process as for other configuration options:
+
+``./asadmin create-jvm-options "-Ddataverse.files.<id>.download-redirect=true"``
+``./asadmin create-jvm-options "-Ddataverse.files.<id>.upload-redirect=true"``
+
+
+With multiple stores configured, it is possible to configure one S3 store with direct upload and/or download to support large files (in general or for specific dataverses) while configuring only direct download, or no direct access for another store.  
+
+It is also possible to set file upload size limits per store. See the :MaxFileUploadSizeInBytes setting described in the :doc:`/installation/config` guide.
+
+At present, one potential drawback for direct-upload is that files are only partially 'ingested', tabular and FITS files are processed, but zip files are not unzipped, and the file contents are not inspected to evaluate their mimetype. This could be appropriate for large files, or it may be useful to completely turn off ingest processing for performance reasons (ingest processing requires a copy of the file to be retrieved by Dataverse from the S3 store). A store using direct upload can be configured to disable all ingest processing for files above a given size limit:
+
+``./asadmin create-jvm-options "-Ddataverse.files.<id>.ingestsizelimit=<size in bytes>"``
+
+
+**IMPORTANT:** One additional step that is required to enable direct download to work with previewers is to allow cross site (CORS) requests on your S3 store. 
+The example below shows how to enable the minimum needed CORS rules on a bucket using the AWS CLI command line tool. Note that you may need to add more methods and/or locations, if you also need to support certain previewers and external tools. 
+
+``aws s3api put-bucket-cors --bucket <BUCKET_NAME> --cors-configuration file://cors.json``
+
+with the contents of the file cors.json as follows:
+
+.. code-block:: json
+
+        {
+          "CORSRules": [
+             {
+                "AllowedOrigins": ["https://<DATAVERSE SERVER>"],
+                "AllowedHeaders": ["*"],
+                "AllowedMethods": ["PUT", "GET"]
+             }
+          ]
+        }
+
+Alternatively, you can enable CORS using the AWS S3 web interface, using json-encoded rules as in the example above. 
+
+Since the direct upload mechanism creates the final file rather than an intermediate temporary file, user actions, such as neither saving or canceling an upload session before closing the browser page, can leave an abandoned file in the store. The direct upload mechanism attempts to use S3 Tags to aid in identifying/removing such files. Upon upload, files are given a "dv-status":"temp" tag which is removed when the dataset changes are saved and the new file(s) are added in Dataverse. Note that not all S3 implementations support Tags: Minio does not. WIth such stores, direct upload works, but Tags are not used. 
 
 Data Capture Module (DCM)
 -------------------------

doc/sphinx-guides/source/installation/config.rst

@@ -244,6 +244,8 @@ If you wish to change which store is used by default, you'll need to delete the
 
   ./asadmin $ASADMIN_OPTS delete-jvm-options "-Ddataverse.files.storage-driver-id=file"
   ./asadmin $ASADMIN_OPTS create-jvm-options "-Ddataverse.files.storage-driver-id=<id>"
+
+  It is also possible to set maximum file upload size limits per store. See the :ref:`:MaxFileUploadSizeInBytes` setting below.
 
 File Storage
 ++++++++++++
@@ -505,7 +507,9 @@ JVM Option                                   Value               Description
 dataverse.files.storage-driver-id            <id>                Enable <id> as the default storage driver.                          ``file``
 dataverse.files.<id>.bucket-name             <?>                 The bucket name. See above.                                         (none)
 dataverse.files.<id>.download-redirect       ``true``/``false``  Enable direct download or proxy through Dataverse.                  ``false``
-dataverse.files.<id>.url-expiration-minutes  <?>                 If direct downloads: time until links expire. Optional.             60
+dataverse.files.<id>.upload-redirect         ``true``/``false``  Enable direct upload of files added to a dataset  to the S3 store.  ``false``
+dataverse.files.<id>.ingestsizelimit         <size in bytes>     Maximum size of directupload files that should be ingested          (none)
+dataverse.files.<id>.url-expiration-minutes  <?>                 If direct uploads/downloads: time until links expire. Optional.     60
 dataverse.files.<id>.custom-endpoint-url     <?>                 Use custom S3 endpoint. Needs URL either with or without protocol.  (none)
 dataverse.files.<id>.custom-endpoint-region  <?>                 Only used when using custom endpoint. Optional.                     ``dataverse``
 dataverse.files.<id>.path-style-access       ``true``/``false``  Use path style buckets instead of subdomains. Optional.             ``false``
@@ -1474,7 +1478,14 @@ Alongside the ``:StatusMessageHeader`` you need to add StatusMessageText for the
 :MaxFileUploadSizeInBytes
 +++++++++++++++++++++++++
 
-Set `MaxFileUploadSizeInBytes` to "2147483648", for example, to limit the size of files uploaded to 2 GB.
+This setting controls the maximum size of uploaded files. 
+- To have one limit for all stores, set `MaxFileUploadSizeInBytes` to "2147483648", for example, to limit the size of files uploaded to 2 GB:
+
+``curl -X PUT -d 2147483648 http://localhost:8080/api/admin/settings/:MaxFileUploadSizeInBytes``
+
+- To have limits per store with an optional default, use a serialized json object for the value of `MaxFileUploadSizeInBytes` with an entry per store, as in the following example, which maintains a 2 GB default and adds higher limits for stores with ids "fileOne" and "s3".
+
+``curl -X PUT -d '{"default":"2147483648","fileOne":"4000000000","s3":"8000000000"}' http://localhost:8080/api/admin/settings/:MaxFileUploadSizeInBytes``
 
 Notes:
 
@@ -1484,7 +1495,7 @@ Notes:
 
 - For larger file upload sizes, you may need to configure your reverse proxy timeout. If using apache2 (httpd) with Shibboleth, add a timeout to the ProxyPass defined in etc/httpd/conf.d/ssl.conf (which is described in the :doc:`/installation/shibboleth` setup).
 
-``curl -X PUT -d 2147483648 http://localhost:8080/api/admin/settings/:MaxFileUploadSizeInBytes``
+
 
 :ZipDownloadLimit
 +++++++++++++++++

src/main/java/edu/harvard/iq/dataverse/DataFileServiceBean.java

@@ -1565,7 +1565,6 @@ public void finalizeFileDelete(Long dataFileId, String storageLocation) throws I
             throw new IOException("Attempted to permanently delete a physical file still associated with an existing DvObject "
                     + "(id: " + dataFileId + ", location: " + storageLocation);
         }
-        logger.info("deleting: " + storageLocation);
         StorageIO<DvObject> directStorageAccess = DataAccess.getDirectStorageIO(storageLocation);
         directStorageAccess.delete();
     }

src/main/java/edu/harvard/iq/dataverse/DatasetPage.java

@@ -1774,7 +1774,6 @@ public void updateOwnerDataverse() {
     private String init(boolean initFull) {
 
         //System.out.println("_YE_OLDE_QUERY_COUNTER_");  // for debug purposes
-        this.maxFileUploadSizeInBytes = systemConfig.getMaxFileUploadSize();
         setDataverseSiteUrl(systemConfig.getDataverseSiteUrl());
 
         guestbookResponse = new GuestbookResponse();
@@ -1821,7 +1820,9 @@ private String init(boolean initFull) {
                 // Set Working Version and Dataset by DatasaetVersion Id
                 //retrieveDatasetVersionResponse = datasetVersionService.retrieveDatasetVersionByVersionId(versionId);
 
-            } 
+            }
+            this.maxFileUploadSizeInBytes = systemConfig.getMaxFileUploadSizeForStore(dataset.getOwner().getEffectiveStorageDriverId());
+
 
             if (retrieveDatasetVersionResponse == null) {
                 return permissionsWrapper.notFound();
@@ -2981,16 +2982,6 @@ public void setLinkingDataverseErrorMessage(String linkingDataverseErrorMessage)
         this.linkingDataverseErrorMessage = linkingDataverseErrorMessage;
     }
 
-    UIInput selectedLinkingDataverseMenu;
-
-    public UIInput getSelectedDataverseMenu() {
-        return selectedLinkingDataverseMenu;
-    }
-
-    public void setSelectedDataverseMenu(UIInput selectedDataverseMenu) {
-        this.selectedLinkingDataverseMenu = selectedDataverseMenu;
-    }
-
     private Boolean saveLink(Dataverse dataverse){
         boolean retVal = true;
         if (readOnly) {

src/main/java/edu/harvard/iq/dataverse/Dataverse.java

@@ -32,6 +32,8 @@
 import javax.validation.constraints.NotNull;
 import javax.validation.constraints.Pattern;
 import javax.validation.constraints.Size;
+
+import org.apache.commons.lang.StringUtils;
 import org.hibernate.validator.constraints.NotBlank;
 import org.hibernate.validator.constraints.NotEmpty;
 
@@ -149,7 +151,7 @@ public String getIndexableCategoryName() {
 
     private String affiliation;
 
-    private String storageDriver="";
+    private String storageDriver=null;
 
 	// Note: We can't have "Remove" here, as there are role assignments that refer
     //       to this role. So, adding it would mean violating a forign key contstraint.
@@ -762,7 +764,7 @@ public boolean isAncestorOf( DvObject other ) {
 
 	public String getEffectiveStorageDriverId() {
 		String id = storageDriver;
-		if(id == null) {
+		if(StringUtils.isBlank(id)) {
 			if(this.getOwner() != null) {
 				id = this.getOwner().getEffectiveStorageDriverId(); 
 			} else {
@@ -774,10 +776,17 @@ public String getEffectiveStorageDriverId() {
 
 
 	public String getStorageDriverId() {
+		if(storageDriver==null) {
+			return DataAccess.UNDEFINED_STORAGE_DRIVER_IDENTIFIER;
+		}
 		return storageDriver;
 	}
 
 	public void setStorageDriverId(String storageDriver) {
-		this.storageDriver = storageDriver;
+		if(storageDriver!=null&&storageDriver.equals(DataAccess.UNDEFINED_STORAGE_DRIVER_IDENTIFIER)) {
+			this.storageDriver=null;
+		} else {
+		  this.storageDriver = storageDriver;
+		}
 	}
 }

src/main/java/edu/harvard/iq/dataverse/DataversePage.java

@@ -1214,19 +1214,24 @@ public Set<Entry<String, String>> getStorageDriverOptions() {
     	HashMap<String, String> drivers =new HashMap<String, String>();
     	drivers.putAll(DataAccess.getStorageDriverLabels());
     	//Add an entry for the default (inherited from an ancestor or the system default)
-    	drivers.put(getDefaultStorageDriverLabel(), "");
+    	drivers.put(getDefaultStorageDriverLabel(), DataAccess.UNDEFINED_STORAGE_DRIVER_IDENTIFIER);
     	return drivers.entrySet();
     }
 
     public String getDefaultStorageDriverLabel() {
     	String storageDriverId = DataAccess.DEFAULT_STORAGE_DRIVER_IDENTIFIER;
     	Dataverse parent = dataverse.getOwner();
+    	boolean fromAncestor=false;
     	if(parent != null) {
     		storageDriverId = parent.getEffectiveStorageDriverId();
-    	}
-    	boolean fromAncestor=false;
-   		if(!storageDriverId.equals(DataAccess.DEFAULT_STORAGE_DRIVER_IDENTIFIER)) {
-    			fromAncestor = true;
+    		//recurse dataverse chain to root and if any have a storagedriver set, fromAncestor is true
+    	    while(parent!=null) {
+    	    	if(!parent.getStorageDriverId().equals(DataAccess.UNDEFINED_STORAGE_DRIVER_IDENTIFIER)) {
+    	    		fromAncestor=true;
+    	    		break;
+    	    	}
+    	    	parent=parent.getOwner();
+    	    }
     	}
    		String label = DataAccess.getStorageDriverLabelFor(storageDriverId);
    		if(fromAncestor) {