Add AlluxioCosException #18419
Closed
Add AlluxioCosException #18419
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
As mentioned in Alluxio#8991, some data structures about state of index in the master are stored in memory, which could cause memory issues. To know if this is happening, some metrics are added to compute the size of these structures. These already existed: Master.FilesPinned Master.FilesToBePersisted And these are added: Master.TTLBuckets Master.TTLInodes Master.ReplicationLimitedFiles For an example of how these new metrics work, can try something like: In Alluxio-site.properties set ```alluxio.master.ttl.checker.interval.ms=60000``` Then run ``` ./bin/alluxio runTests -Dalluxio.user.file.create.ttl=1m -Dalluxio.user.file.create.ttl.action=DELETE -Dalluxio.user.file.replication.max=1``` And watch ```watch -n 1 "curl 127.0.0.1:19999/metrics/json/ | grep -i -A 4 -e Master.TTL -e Master.ReplicationLimitedFiles"``` pr-link: Alluxio#16818 change-id: cid-960d695f986eb6ee8b09c881869f383a98f9a499
### What changes are proposed in this pull request? Point to new contributor tasks since old one is abandoned. ### Why are the changes needed? Please clarify why the changes are needed. For instance, 1. If you propose a new API, clarify the use case for a new API. 2. If you fix a bug, describe the bug. ### Does this PR introduce any user facing changes? Please list the user-facing changes introduced by your change, including 1. change in user-facing APIs 2. addition or removal of property keys 3. webui pr-link: Alluxio#16820 change-id: cid-818ec3cfb6fff04b8643a90e4b73e9455e9a19d7
### What changes are proposed in this pull request? Refactor s3 low level output stream and support OSS and OBS. ### Why are the changes needed? 1. extract the generic logic to `ObjectLowLevelOutputStream` to make it easy to support new object storage. 2. Support streaming uploads for OBS and OSS. 3. fix the bug that empty files cannot be persisted to UFS. 4. specify MD5 when upload parts. ### Does this PR introduce any user facing changes? `alluxio.underfs.oss.intermediate.upload.clean.age`: clean incomplete multi abort age for OSS. `alluxio.underfs.oss.streaming.upload.enabled`: Whether to enable stream upload for OSS. `alluxio.underfs.oss.streaming.upload.partition.size`: straming upload partition size for OSS. `alluxio.underfs.oss.streaming.upload.threads`: thread pool size for OSS streaming upload. `alluxio.underfs.obs.intermediate.upload.clean.age`: clean incomplete multi abort age for obs. `alluxio.underfs.obs.streaming.upload.enabled`: Whether to enable stream upload for OBS. `alluxio.underfs.obs.streaming.upload.partition.size`: straming upload partition size for OBS. `alluxio.underfs.obs.streaming.upload.threads`: thread pool size for OBS streaming upload. pr-link: Alluxio#16122 change-id: cid-91f6e2b5ec6b79d2175e71754654b59e650c0c32
### What changes are proposed in this pull request? Please outline the changes and how this PR fixes the issue. ### Why are the changes needed? Please clarify why the changes are needed. For instance, 1. If you propose a new API, clarify the use case for a new API. 2. If you fix a bug, describe the bug. ### Does this PR introduce any user facing changes? Please list the user-facing changes introduced by your change, including 1. change in user-facing APIs 2. addition or removal of property keys 3. webui pr-link: Alluxio#16824 change-id: cid-975e463d895496294e2172d43d6b287fb37565c5
### What changes are proposed in this pull request? Please outline the changes and how this PR fixes the issue. ### Why are the changes needed? Please clarify why the changes are needed. For instance, 1. If you propose a new API, clarify the use case for a new API. 2. If you fix a bug, describe the bug. ### Does this PR introduce any user facing changes? Please list the user-facing changes introduced by your change, including 1. change in user-facing APIs 2. addition or removal of property keys 3. webui pr-link: Alluxio#16825 change-id: cid-71f088a46766a572f64ef76b1ec01a207645a425
### What changes are proposed in this pull request? Fix debug log in FileSystemContext#reinit(). ### Why are the changes needed? The argument is wrong. ### Does this PR introduce any user facing changes? No. pr-link: Alluxio#16826 change-id: cid-5453a328dff317614d5e529a6a82c413241194fa
### What changes are proposed in this pull request? Fix Alluxio#16821 Developers can execute cross compare with `build-docker.sh` etc. `./build-docker.sh buildx linux/arm64` ### Why are the changes needed? Nope. ### Does this PR introduce any user facing changes? Nope, just for developers. pr-link: Alluxio#16822 change-id: cid-736afb945b565d6756fa671e257a67d1db5e408a
Signed-off-by: Huang Hua <[email protected]> ### What changes are proposed in this pull request? Added examples for Alluxio Native API, Hadoop Compatible File System and HDFS ### Why are the changes needed? These examples show how to use the Alluxio native API, Hadoop compatible File System and HDFS programming in Java. Developers can use them as a quick start. ### Does this PR introduce any user facing changes? No. pr-link: Alluxio#16647 change-id: cid-fea930ad1488b244ba824f6f2c0c162552794b80
relative links within the codebase do not render to usable links when publishing to our documentation website. `[description](/path/to/file)` will render as `https://docs.alluxio.io/path/to/file` replace with the full github url to the corresponding file in master branch instead pr-link: Alluxio#16834 change-id: cid-6ad346b38e0e2ff0c8018105930fae64261cd58b
### What changes are proposed in this pull request? Some wording in cn client-metric table ### Why are the changes needed? Clarified some metric descriptions. ### Does this PR introduce any user facing changes? No pr-link: Alluxio#16799 change-id: cid-a2e3aab8f7e92eb23380729e845902477fb47c06
### What changes are proposed in this pull request? The PR adds the two-choice random cache eviction policy. The algorithm selects two random page IDs and evicts the one least recently used. ### Why are the changes needed? From some evaluation (https://danluu.com/2choices-eviction/), the two-choice random policy has competitive performance compared with LRU. ### Does this PR introduce any user facing changes? Users can configure the new cache eviction policy. pr-link: Alluxio#16828 change-id: cid-f9a140208b3a6938ddc70766abfb3d9f90c722f0
### What changes are proposed in this pull request? Implemented a RpcServerStandbyGrpcService which runs gRPC server on both standby and primary masters. ### Why are the changes needed? We are going to implement a feature which allows workers registering to all masters to speed up the master failover process. This requires standby masters to enable gRPC servers so that the works can rpc. This PR made these changes. ### Does this PR introduce any user facing changes? N/A pr-link: Alluxio#16839 change-id: cid-743160dea3f42872555b48d9a62125b2740962a3
### What changes are proposed in this pull request? The PR cherry-picks @beinan's implementation of local cache invalidation to the master branch. pr-link: Alluxio#16841 change-id: cid-5191eaf15bb8bf730db7653670ad4e6241ba31c6
### What changes are proposed in this pull request? Co-Author-By: Chen Liang <[email protected]> This change adds a scheduled thread in LocalCacheManager, to periodically check the page create time, and delete pages that have passed the configurable time threshold. ### Why are the changes needed? Certain use cases require the cached data to be removed beyond certain longevity. This is mainly for compliance considerations. ### Does this PR introduce any user facing changes? No pr-link: Alluxio#16843 change-id: cid-49289cc488680f44b39386a5f364d6d7449cd11a
Add update notify to master This issue introduced by Alluxio#13514 pr-link: Alluxio#16764 change-id: cid-36f2010233c3af95f28cf9cb7ac0a0a1963c86fe
Fix Alluxio#16733 pr-link: Alluxio#16702 change-id: cid-500283d523f5cc5835cb889b85cf7804842a4af6
### What changes are proposed in this pull request? Fix the flaky test in the two-choice random cache eviction policy ### Why are the changes needed? As the test is flaky, it will influence other PRs. ### Does this PR introduce any user facing changes? No. pr-link: Alluxio#16850 change-id: cid-71dd4af6c446f88be4187d11d7b9c6b6713ec677
### What changes are proposed in this pull request? Optimize rename directory performance on object store by merging delete object requests. ### Why are the changes needed? In the previous implementation, if there were 10,000 objects in a directory, then there were 10,000 API calls for copying and deleting object. this PR merges the API for deleting object so that there are only 10,000 API calls for copying and 10 API calls for deleting objects in bulk. In our performance tests, when moving a directory with 400,000 objects, we were able to achieve a 30% performance improvement. ### Does this PR introduce any user facing changes? No user facing changes. pr-link: Alluxio#16527 change-id: cid-a0af2782b8d538daaee63a06b2e5c0a6108835be
Page Not Found is reported on the first time to access such page, or when the page cache is evicted. This is normal and not an error. Signed-off-by: Huang Hua <[email protected]> ### What changes are proposed in this pull request? Change the log level from error to debug. ### Why are the changes needed? When client reads some file for the first time, worker has the following error messages: ``` 2023-02-08 15:07:43,839 ERROR LocalCacheManager - Failed to get existing page PageId{FileId=1478269e1f194060bc522a21dce77eeee147367afad6d4011ae65543e9b0256b, PageIndex=0} from pageStore alluxio.exception.PageNotFoundException: /Volumes/ramdisk/LOCAL/1048576/221/1478269e1f194060bc522a21dce77eeee147367afad6d4011ae65543e9b0256b/0 at alluxio.client.file.cache.store.LocalPageStore.get(LocalPageStore.java:118) at alluxio.client.file.cache.LocalCacheManager.getPage(LocalCacheManager.java:703) at alluxio.client.file.cache.LocalCacheManager.get(LocalCacheManager.java:483) at alluxio.client.file.cache.NoExceptionCacheManager.get(NoExceptionCacheManager.java:105) at alluxio.worker.dora.PagedFileReader.read(PagedFileReader.java:117) at alluxio.worker.dora.PagedFileReader.transferTo(PagedFileReader.java:177) at alluxio.worker.grpc.FileReadHandler$DataReader.getDataBuffer(FileReadHandler.java:472) at alluxio.worker.grpc.FileReadHandler$DataReader.runInternal(FileReadHandler.java:335) at alluxio.worker.grpc.FileReadHandler$DataReader.run(FileReadHandler.java:284) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266) at java.util.concurrent.FutureTask.run(FutureTask.java) at alluxio.worker.grpc.GrpcExecutors$ImpersonateThreadPoolExecutor.lambda$execute$0(GrpcExecutors.java:159) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) ``` This is not an error. This is a normal case. If this happens when some cache is evicted/removed, worker/client will re-fetch the data again. ### Does this PR introduce any user facing changes? No. pr-link: Alluxio#16856 change-id: cid-64eb4cdbfc35f9a3d6d3700075e0181fb02929ae
### What changes are proposed in this pull request? Support delete file from alluxio only and reset its directChildrenLoadedState of parent. ### Why are the changes needed? ### Does this PR introduce any user facing changes? ```Console $ bin/alluxio fs rm -R --alluxioOnly --syncParentNextTime=true /test/bb $ bin/alluxio fs ls /test/ drwxr-xr-x mbl staff 0 PERSISTED 05-30-2022 14:45:24:142 DIR /test/bb $ bin/alluxio fs rm -R --alluxioOnly --syncParentNextTime=false /test/bb $ bin/alluxio fs ls /test/ # /test/bb will not be loaded because metadata sync on /test is disabled ``` pr-link: Alluxio#15647 change-id: cid-95acf6dd9c2ee820821e382a2a7332486c865caa
### What changes are proposed in this pull request? Rearchitecturing the s3 service ### Why are the changes needed? Rearchitecturing the s3 service ### Does this PR introduce any user facing changes? N/A pr-link: Alluxio#16654 change-id: cid-2b39b3f50b165cad6aece868f75d8e6218de0cfe
### What changes are proposed in this pull request? Make version service return unavailable on standby masters ### Why are the changes needed? In this Alluxio#16839 PR, we adds the capability to run grpc services on standby masters. However, if one uses an old alluxio client and connects to the new master with standby master enabled, it will get some errors. This PR is used to address the issue to make the change backward compatible. ### Does this PR introduce any user facing changes? N/A pr-link: Alluxio#16854 change-id: cid-6c82bce1b1d6cb2649658ec09f0f8ed4e243917a
Bumps hadoop-common from 3.2.1 to 3.2.4. [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) - `@dependabot use these labels` will set the current labels as the default for future PRs for this repo and language - `@dependabot use these reviewers` will set the current reviewers as the default for future PRs for this repo and language - `@dependabot use these assignees` will set the current assignees as the default for future PRs for this repo and language - `@dependabot use this milestone` will set the current milestone as the default for future PRs for this repo and language You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/Alluxio/alluxio/network/alerts). </details> pr-link: Alluxio#16830 change-id: cid-c9e528db52ed4d3bf78ca8ea210d525150454ca4
### What changes are proposed in this pull request? Replace ManagementFactoryHelper with ManagementFactory for java11 ### Why are the changes needed? Java 11 doesn't support to directly call ManagementFactoryHelper, so it is necessary to find an alternative one in java11. ### Does this PR introduce any user facing changes? NA pr-link: Alluxio#16874 change-id: cid-1e60172569c14a862603b23f013870cca605b3ff
### What changes are proposed in this pull request? See title. Adds https://github.com/git-commit-id/git-commit-id-maven-plugin v.4.0.5 as a dependency. Latest version (5.0.0) requires Java 11. ### Why are the changes needed? Updating `pom.xml` to support reproducible Maven builds. See https://maven.apache.org/guides/mini/guide-reproducible-builds.html. ### Does this PR introduce any user facing changes? Adds a `git.properties` file to output directories containing commit into (can be configurable). ex: ``` #Generated by Git-Commit-Id-Plugin git.build.version=2.10.0-SNAPSHOT git.commit.id.abbrev=622dfb4 git.commit.id.full=622dfb4ca246020efaeb169b18012bd394170e9a git.commit.time=2023-02-13T21\:47\:53Z ``` pr-link: Alluxio#16881 change-id: cid-4e90a78a4d1068b5e4cb7620c06b374c96504022
### What changes are proposed in this pull request? reduce useless async cache request when read from remote worker ### Why are the changes needed? It is useless to trigger async cache when the remote worker has already cached block. ### Does this PR introduce any user facing changes? No user facing changes. pr-link: Alluxio#16313 change-id: cid-ad4f236cf143346085b59b24122a3dc6870281e7
Fix the security issue introduced by following link. The newest version of hadoop fix this issue. - http://horus.oa.com/advisory/HOSA-pzst-tfwm6u4cy - http://horus.oa.com/advisory/HOSA-n612-2d07um3px - http://horus.oa.com/advisory/HOSA-yn2x-x9qulrobf - http://horus.oa.com/advisory/HOSA-jatn-nizpv9w2u I remove the hadoop.version property to keep consistent with parent module. pr-link: Alluxio#16844 change-id: cid-b9e60f612d667b0ae3daea1314593a284b65b622
What changes are proposed in this pull request? Update cn version of Upgrade doc. Why are the changes needed? There is no corresponding Chinese documentation for upgrade. Does this PR introduce any user facing changes? More Chinese users can access Alluxio documentation more easily. pr-link: Alluxio#16840 change-id: cid-be17d3c207fbbaa3211f6e3e7611a2e15741b66f
### What changes are proposed in this pull request? Fix Worker.ActiveClients is negative when load from ufs ### Why are the changes needed? Please clarify why the changes are needed. For instance, createUfsBlockReader will invoke closeUfsBlock then invoke commitBlock then this metric decrease. But forget to increase firstly . So it need to increase firstly. ### Does this PR introduce any user facing changes? Please list the user-facing changes introduced by your change, including no. pr-link: Alluxio#16784 change-id: cid-13d0875dc42336cc612dfed121bba8572164ed60
Add dockerfile with java 17 base and java 8 added for compilation, mirroring the java 11 one pr-link: Alluxio#16885 change-id: cid-aa63908b5ef3ef2ef83a022677ec58706d29730d
### What changes are proposed in this pull request? Delegate `getFileBlockLocation` to external file system in `LocalCacheFileSystem`. ### Why are the changes needed? Otherwise, `LocalCacheFileSystem` inherits the default behavior of `org.apache.hadoop.fs.FileSystem` which returns `localhost` only. ### Does this PR introduce any user facing changes? No. pr-link: Alluxio#17672 change-id: cid-eb545dbd8ed42001d074fecfb9c8d6b118a559c1
### What changes are proposed in this pull request? Support dynamic enable/disable audit log ### Why are the changes needed? Now, if we disable auditlog before master startup, we cannot support enable it dynamically, but it can be disable if it is enabled at startup time, it is tricky. And it is not expensive to start an object. <img width="842" alt="image" src="https://github.com/Alluxio/alluxio/assets/17329931/df16dbfd-7307-4542-8fff-9f1fa556b66c"> ### Does this PR introduce any user facing changes? No pr-link: Alluxio#17489 change-id: cid-49454d429d6875157b090b1ca767552743006560
### What changes are proposed in this pull request? Now we unpin and set min replication to 0 if we failed due to IO related reasons. (such as UFS not available, file does not exist etc). ### Why are the changes needed? Previously if a replication job fails, it would be rescheduled again and again, causing Denial of Service. ### Does this PR introduce any user facing changes? Yes, the user would expect certain files unpinned or adjusted if it becomes unavailable. pr-link: Alluxio#17865 change-id: cid-a2cde6da1e7a95fb361157307bd026303c0164ad
Cherry-pick of existing commit. orig-pr: Alluxio#17413 orig-commit: Alluxio/alluxio@16cbf9c orig-commit-author: elega <[email protected]> pr-link: Alluxio#17858 change-id: cid-8bfb171d1660c6da96977129b62b6b6640625f4c
### What changes are proposed in this pull request? Add the alluxio hadoop3 jar client to the artifacts ### Why are the changes needed? Some users may want to use hadoop3 ### Does this PR introduce any user facing changes? no pr-link: Alluxio#18090 change-id: cid-7945e9bbce1480c5fdc7c45dc618649ec2ad5e29
### What changes are proposed in this pull request? Fix a deadlock. ### Why are the changes needed? On the Alluxio client, there is a monitor lock of `FileSystemContext` to sync individual client-to-master operations. One application thread can successfully hold the monitor lock (`alluxio.client.file.FileSystemContext`) in order to get worker list, but blocked by another “lock” `FileSystemContextReinitializer` (waiting for latch on on-going RPCs down to zero) for acquiring the block master client to really connect to master process (waiting for other callers involving `FileSystemContextReinitializer` to finish). On the other hand, another heartbeat thread "config-hash-master-heartbeat-0" is awaking periodically to sync with the master process to fetch the latest configuration. This thread detected the conf update and thus entered `FileSystemContextReinitializer` (bumping latch) but was blocked by waiting for the monitor lock of `alluxio.client.file.FileSystemContext` in order to get the master address. This PR moves `getMasterAddress` outside `reinit` block to avoid holding the `Reinitializer` object and wait for the monitor object of `FileSystemContext`. ### Does this PR introduce any user facing changes? No pr-link: Alluxio#18109 change-id: cid-0fab540f7dbef3661acb44e7472425add163b019
### What changes are proposed in this pull request? ref Alluxio@8cbc490 ### Why are the changes needed? Fix potentially failing test for different versions ### Does this PR introduce any user facing changes? no pr-link: Alluxio#18091 change-id: cid-3cc882f8e3679a3c60f5046cc07b6e8b0b208bc7
Fix Alluxio#17461 pr-link: Alluxio#17474 change-id: cid-6c1c5f50c36a0eac8a35e4f176290c0b7ae3e2fe
### What changes are proposed in this pull request? Support read bytebuffer for non ByteBufferReadable input stream ### Why are the changes needed? Without this changes, a non ByteBufferReadable input stream can throw exception when the bytebuffer apis are called. ### Does this PR introduce any user facing changes? No pr-link: Alluxio#17982 change-id: cid-1363dbbeca327cbb7b4d39cea1afbb8c51905483
### What changes are proposed in this pull request? Retry requiring a buffer when loading a block throws OutOfMemoryError. ### Why are the changes needed? Failure to require buffer may result in the following exception. ``` alluxio.exception.BlockAlreadyExistsException: Temp blockId 50331677 is not available, because it already exists ``` pr-link: Alluxio#17735 change-id: cid-e87da40ce5c060d7426f95c270fd40539e188e23
### What changes are proposed in this pull request? Check for more results when the filtered result is null when loading data. ### Why are the changes needed? Fix Alluxio#18043. pr-link: Alluxio#18133 change-id: cid-a05d66db230971ba6585b732c7bb2990ba02f7f7
Fix Alluxio#17889 pr-link: Alluxio#17910 change-id: cid-4fa6c6aff69785d58e35cf1e3d45cafb6ff922ee
### What changes are proposed in this pull request? Before: Web UI displays only files ending with ".log" in name. After: Web UI displays logs files like `.log.XXX`, which are rolled over log files. Also the displayed files include `.out` and captured `txt` and `json` files like metrics and jstack. ### Why are the changes needed? Better observability for admin. ### Does this PR introduce any user facing changes? Please list the user-facing changes introduced by your change, including 1. change in user-facing APIs 2. addition or removal of property keys 3. webui pr-link: Alluxio#17983 change-id: cid-714bcde45116ed84016d6292e365389e39da1b90
### What changes are proposed in this pull request? Support unmount and mount to another mount point within the same lock ### Why are the changes needed? Support replace ufs for a mount point. The operation is atomic so no user operations should fail due to observing the middle state. ### Does this PR introduce any user facing changes? A new rpc field and cmd options are added. pr-link: Alluxio#17984 change-id: cid-d96b4df7d24ab1082e46e6d5a72563534714a48f
### What changes are proposed in this pull request? Add cache rejected metrics. ### Why are the changes needed? Add cache rejected metrics. ### Does this PR introduce any user facing changes? Add metrics `Worker_CacheRejectedBlocks`. pr-link: Alluxio#18103 change-id: cid-b92ed474cadbc7c52091e51c1431cd1c95ff448b
provide more information about worker state pr-link: Alluxio#18185 change-id: cid-b972079ff380e4ac32fa2e013b3034931f94786a
### What changes are proposed in this pull request? Support creating directories and files with default permissions based on configuration propertykey in HDFS API. ### Why are the changes needed? Alluxio Hdfs api hasn't the corresponding implementation of Mkdir without permission parameter. If it does not carry permission, the umask property in the configuration item shall prevail. ### Does this PR introduce any user facing changes? pr-link: Alluxio#18253 change-id: cid-6ddd2243bac00ebbdbdff1e731036c0d3d6228c8
### What changes are proposed in this pull request? Using the user of the OSS bucket to represent the user of the object loaded from OSS. Add the mapping way from OSS username to the custom username. Add the default permission mode for existing files loaded from OSS. ### Why are the changes needed? the object loaded from OSS ufs will be null user, null group, and 777 by default. ### Does this PR introduce any user facing changes? pr-link: Alluxio#18262 change-id: cid-7e56861c9def695876cde32d1e0aa453b512470d
### What changes are proposed in this pull request? Check cluster ufs version in getClusterConf. ### Why are the changes needed? Fix Alluxio#18221. pr-link: Alluxio#18222 change-id: cid-2c67758e30cac31a84f41d47b50c0ed958c88ffb
### What changes are proposed in this pull request? - Don't have to inherit the HdfsUnderFileSystemFactory CosNUnderFileSystemFactory directly implement UnderFileSystemFactory interface ### Why are the changes needed? - Possibly to resolve package conflicts, Alluxio#17024 removed HdfsUnderFileSystemFactory from COSN UFS jar, resulting in inability to use COSN interface - However, CosNUnderFileSystemFactory inherits from HdfsUnderFileSystemFactory,the ServiceLoader.load method loads CosNUnderFileSystemFactory and first searches for its parent class, so removing HdfsUnderFileSystemFactory directly will result in an error. ``` failed to load jar alluxio-underfs-hadoop-cosn-3.1.0-5.8.5-2.9.3.jar NoClassdDefFoundError :alluxio/underfs/hdfs/HdfsUnderFileSystemFactory ``` pr-link: Alluxio#18143 change-id: cid-4a80f6cdeae5b9bdb9e956c36838403ee6ce7c46
Fix bug involved by Alluxio#17024 After Alluxio#18143, `HdfsUnderFileSystemFactory` is no longer included in COSN jar therefore no need to exclude. pr-link: Alluxio#18303 change-id: cid-6931be7291f52728022e4555d1c7183857948b9f
### What changes are proposed in this pull request? Please outline the changes and how this PR fixes the issue. modify the calculation method for cacheMissPercentage metric, ensuring it comprehensively accounts for the influence of job worker read operations. Alluxio#16945 ### Why are the changes needed? Because of the incorrect calculation results of this metric, it may produce exception value. ### Does this PR introduce any user facing changes? No pr-link: Alluxio#18208 change-id: cid-d24a6f324f7148cab4a30fbbf819510a1a314ebc
### What changes are proposed in this pull request? Bump Hadoop version from `3.3.1` to `3.3.4`. ### Why are the changes needed? Fix hadoop CVE-2021-37404. ### Does this PR introduce any user facing changes? Hadoop version bump. pr-link: Alluxio#17002 change-id: cid-fd12eec84b42efd3112c3c71039702dca92ca775
### What changes are proposed in this pull request? The v1 and v2 versions of proxy use uniform judgment conditions. pr-link: Alluxio#18371 change-id: cid-1842a0b96b5233f9a456e1aee7dbd95e2bd2d5e0
### What changes are proposed in this pull request? As issue Alluxio#15795 mentioned. When output any warnings, the cli of judging HA_ENABLED will be invalid. This PR fixs it. ### Why are the changes needed? Bug fix. ### Does this PR introduce any user facing changes? No. pr-link: Alluxio#18334 change-id: cid-c0d595d263788c5e64e6ce4508e2ee120b527b74
### What changes are proposed in this pull request? Datalake formats such as Iceberg requires frequent changes to certain files, it is better not to cache them at all to avoid frequent invalidations. ### Why are the changes needed? for correct functionality using iceberg ### Does this PR introduce any user facing changes? one user property that is by default empty, alluxio.user.file.direct.access pr-link: Alluxio#18326 change-id: cid-25a1ee3c2876a4a4126727113342c24d40b06126
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes are proposed in this pull request?
Add AlluxioCosException as PR #18388
Why are the changes needed?
Please clarify why the changes are needed. For instance,
Does this PR introduce any user facing changes?
Please list the user-facing changes introduced by your change, including