9002 allow direct upload setting

doc/release-notes/4.20-release-notes.md

@@ -90,10 +90,16 @@ Also note that the :MaxFileUploadSizeInBytes property has a new option to provid
 
 ### Direct S3 Upload Changes
 
-Direct upload to S3 is enabled per store by one new jvm option:
+Direct upload to S3 in UI and API is enabled per store by one new jvm option:
 
     ./asadmin create-jvm-options "\-Ddataverse.files.<id>.upload-redirect=true"
-
+
+This option makes direct upload the default in the UI. In the API, you can use either: direct upload or upload via Dataverse upload. Direct upload to S3 in API only is enabled per store by this new jvm option:
+
+    ./asadmin create-jvm-options "\-Ddataverse.files.<id>.api-direct-upload=true"
+
+That option leaves via Dataverse upload by default in UI, but makes both: uploads via Dataverse and direct uploads possible via API.
+
 The existing :MaxFileUploadSizeInBytes property and ```dataverse.files.<id>.url-expiration-minutes``` jvm option for the same store also apply to direct upload.
 
 Direct upload via the Dataverse web interface is transparent to the user and handled automatically by the browser. Some minor differences in file upload exist: directly uploaded files are not unzipped and Dataverse does not scan their content to help in assigning a MIME type. Ingest of tabular files and metadata extraction from FITS files will occur, but can be turned off for files above a specified size limit through the new dataverse.files.<id>.ingestsizelimit jvm option.
@@ -126,7 +132,8 @@ We made changes to the JSON Export in this release (Issue 6650, PR #6669). If yo
 ## New JVM Options for file storage drivers
 
 - The JVM option dataverse.files.file.directory=<your directory> controls where temporary files are stored (in the /temp subdir of the defined directory), independent of the location of any 'file' store defined above.
-- The JVM option dataverse.files.<id>.upload-redirect enables direct upload of files added to a dataset to the S3 bucket. (S3 stores only!)
+- The JVM option dataverse.files.<id>.upload-redirect enables direct upload of files added to a dataset in the S3 bucket. (S3 stores only!)
+- The JVM option dataverse.files.<id>.api-direct-upload enables direct upload of files added to a dataset in any storage. (Via API only and when the uploading tool has direct access to the relevant storage used; i.e., upload the file first and register it via API!)
 - The JVM option dataverse.files.<id>.MaxFileUploadSizeInBytes controls the maximum size of file uploads allowed for the given file store. 
 - The JVM option dataverse.files.<id>.ingestsizelimit controls the maximum size of files for which ingest will be attempted, for the given file store. 
 

doc/sphinx-guides/source/installation/config.rst

@@ -545,7 +545,8 @@ List of S3 Storage Options
     dataverse.files.<id>.label                   <?>                 **Required** label to be shown in the UI for this storage                   (none)
     dataverse.files.<id>.bucket-name             <?>                 The bucket name. See above.                                                 (none)
     dataverse.files.<id>.download-redirect       ``true``/``false``  Enable direct download or proxy through Dataverse.                          ``false``
-    dataverse.files.<id>.upload-redirect         ``true``/``false``  Enable direct upload of files added to a dataset  to the S3 store.          ``false``
+    dataverse.files.<id>.upload-redirect         ``true``/``false``  Enable direct upload of files added to a dataset in the S3 store.           ``false``
+    dataverse.files.<id>.api-direct-upload       ``true``/``false``  Enable direct upload of files added to a dataset via API only.              ``false``
     dataverse.files.<id>.ingestsizelimit         <size in bytes>     Maximum size of directupload files that should be ingested                  (none)
     dataverse.files.<id>.url-expiration-minutes  <?>                 If direct uploads/downloads: time until links expire. Optional.             60
     dataverse.files.<id>.min-part-size           <?>                 Multipart direct uploads will occur for files larger than this. Optional.   ``1024**3``

src/main/java/edu/harvard/iq/dataverse/dataaccess/StorageIO.java

@@ -605,7 +605,8 @@ public static String getDriverPrefix(String driverId) {
     }
 
     public static boolean isDirectUploadEnabled(String driverId) {
-        return Boolean.parseBoolean(System.getProperty("dataverse.files." + driverId + ".upload-redirect"));
+        return Boolean.parseBoolean(System.getProperty("dataverse.files." + driverId + ".upload-redirect")) ||
+            Boolean.parseBoolean(System.getProperty("dataverse.files." + driverId + ".api-direct-upload"));
     }
 
     //Check that storageIdentifier is consistent with store's config

src/main/java/edu/harvard/iq/dataverse/util/FileUtil.java

@@ -1748,7 +1748,8 @@ public static boolean isPackageFile(DataFile dataFile) {
 
     public static S3AccessIO getS3AccessForDirectUpload(Dataset dataset) {
     	String driverId = dataset.getEffectiveStorageDriverId();
-    	boolean directEnabled = Boolean.getBoolean("dataverse.files." + driverId + ".upload-redirect");
+    	boolean directEnabled = Boolean.parseBoolean(System.getProperty("dataverse.files." + driverId + ".upload-redirect")) ||
+            Boolean.parseBoolean(System.getProperty("dataverse.files." + driverId + ".api-direct-upload"));
     	//Should only be requested when it is allowed, but we'll log a warning otherwise
     	if(!directEnabled) {
     		logger.warning("Direct upload not supported for files in this dataset: " + dataset.getId());

src/main/java/edu/harvard/iq/dataverse/util/SystemConfig.java

@@ -1112,7 +1112,8 @@ public boolean isDatafileValidationOnPublishEnabled() {
     }
 
 	public boolean directUploadEnabled(DvObjectContainer container) {
-    	return Boolean.getBoolean("dataverse.files." + container.getEffectiveStorageDriverId() + ".upload-redirect");
+        // this method is used in UI only, therfore "dataverse.files." + driverId + ".api-direct-upload" is not used here
+    	return Boolean.parseBoolean(System.getProperty("dataverse.files." + container.getEffectiveStorageDriverId() + ".upload-redirect"));
 	}
 
 	public String getDataCiteRestApiUrlString() {