Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Rename property key alluxio.underfs.s3.threads.max #18670

Merged
merged 1 commit into from
Aug 8, 2024
Merged

Rename property key alluxio.underfs.s3.threads.max #18670

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 8 additions & 7 deletions core/common/src/main/java/alluxio/conf/PropertyKey.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1459,12 +1459,13 @@ public String toString() {
.setConsistencyCheckLevel(ConsistencyCheckLevel.ENFORCE)
.setScope(Scope.SERVER)
.build();
public static final PropertyKey UNDERFS_S3_THREADS_MAX =
intBuilder(Name.UNDERFS_S3_THREADS_MAX)
.setDefaultValue(40)
.setDescription("The maximum number of threads to use for communicating with S3 and "
+ "the maximum number of concurrent connections to S3. Includes both threads "
+ "for data upload and metadata operations. This number should be at least as "
public static final PropertyKey UNDERFS_S3_CONNECTIONS_MAX =
intBuilder(Name.UNDERFS_S3_CONNECTIONS_MAX)
.setAlias("alluxio.underfs.s3.threads.max")
.setDefaultValue(1024)
.setDescription("The maximum number of concurrent connections to communicate with S3. "
+ "This value includes both connections for data upload and metadata operations. "
+ "This number should be at least as "
+ "large as the max admin threads plus max upload threads.")
.setConsistencyCheckLevel(ConsistencyCheckLevel.WARN)
.setScope(Scope.SERVER)
Expand Down Expand Up @@ -8104,7 +8105,7 @@ public static final class Name {
public static final String UNDERFS_S3_PROXY_HOST = "alluxio.underfs.s3.proxy.host";
public static final String UNDERFS_S3_PROXY_PORT = "alluxio.underfs.s3.proxy.port";
public static final String UNDERFS_S3_REGION = "alluxio.underfs.s3.region";
public static final String UNDERFS_S3_THREADS_MAX = "alluxio.underfs.s3.threads.max";
public static final String UNDERFS_S3_CONNECTIONS_MAX = "alluxio.underfs.s3.connections.max";
public static final String UNDERFS_S3_UPLOAD_THREADS_MAX =
"alluxio.underfs.s3.upload.threads.max";
public static final String KODO_ENDPOINT = "alluxio.underfs.kodo.endpoint";
Expand Down
10 changes: 5 additions & 5 deletions underfs/s3a/src/main/java/alluxio/underfs/s3a/S3AUnderFileSystem.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -221,14 +221,14 @@ public static S3AUnderFileSystem createInstance(AlluxioURI uri,
int numAdminThreads = conf.getInt(PropertyKey.UNDERFS_S3_ADMIN_THREADS_MAX);
int numTransferThreads =
conf.getInt(PropertyKey.UNDERFS_S3_UPLOAD_THREADS_MAX);
int numThreads = conf.getInt(PropertyKey.UNDERFS_S3_THREADS_MAX);
if (numThreads < numAdminThreads + numTransferThreads) {
int numConns = conf.getInt(PropertyKey.UNDERFS_S3_CONNECTIONS_MAX);
if (numConns < numAdminThreads + numTransferThreads) {
LOG.warn("Configured s3 max threads ({}) is less than # admin threads ({}) plus transfer "
+ "threads ({}). Using admin threads + transfer threads as max threads instead.",
numThreads, numAdminThreads, numTransferThreads);
numThreads = numAdminThreads + numTransferThreads;
numConns, numAdminThreads, numTransferThreads);
numConns = numAdminThreads + numTransferThreads;
}
clientConf.setMaxConnections(numThreads);
clientConf.setMaxConnections(numConns);

// Set client request timeout for all requests since multipart copy is used,
// and copy parts can only be set with the client configuration.
Expand Down
Loading