Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Merge with latest main branch #18665

Merged
merged 5 commits into from
Jul 29, 2024
Merged

Merge with latest main branch #18665

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
40ebdd0
Fix metrics when delete non existing page
jja725 Jun 7, 2024
19aa8f0
Update version to 313
alluxio-bot Jun 7, 2024
efed93c
Update version to 314-SNAPSHOT
alluxio-bot Jun 7, 2024
05d2537
Add back shaded netty within grpc
jja725 Jul 1, 2024
6d99656
Install ucx on dockerfile jdk8
LuQQiu Jul 25, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion assembly/client/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
<parent>
<groupId>org.alluxio</groupId>
<artifactId>alluxio-assembly</artifactId>
<version>313-SNAPSHOT</version>
<version>314-SNAPSHOT</version>
</parent>
<artifactId>alluxio-assembly-client</artifactId>
<packaging>jar</packaging>
Expand Down
2 changes: 1 addition & 1 deletion assembly/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
<parent>
<groupId>org.alluxio</groupId>
<artifactId>alluxio-parent</artifactId>
<version>313-SNAPSHOT</version>
<version>314-SNAPSHOT</version>
</parent>
<artifactId>alluxio-assembly</artifactId>
<packaging>pom</packaging>
Expand Down
2 changes: 1 addition & 1 deletion assembly/server/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
<parent>
<groupId>org.alluxio</groupId>
<artifactId>alluxio-assembly</artifactId>
<version>313-SNAPSHOT</version>
<version>314-SNAPSHOT</version>
</parent>
<artifactId>alluxio-assembly-server</artifactId>
<packaging>jar</packaging>
Expand Down
2 changes: 1 addition & 1 deletion common/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
<parent>
<groupId>org.alluxio</groupId>
<artifactId>alluxio-parent</artifactId>
<version>313-SNAPSHOT</version>
<version>314-SNAPSHOT</version>
</parent>
<artifactId>alluxio-common</artifactId>
<packaging>pom</packaging>
Expand Down
2 changes: 1 addition & 1 deletion common/transport/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
<parent>
<groupId>org.alluxio</groupId>
<artifactId>alluxio-common</artifactId>
<version>313-SNAPSHOT</version>
<version>314-SNAPSHOT</version>
</parent>
<artifactId>alluxio-core-transport</artifactId>
<packaging>jar</packaging>
Expand Down
11 changes: 11 additions & 0 deletions dev/github/Dockerfile-jdk11
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -174,3 +174,14 @@ RUN ARCH=$(dpkg --print-architecture) && \
wget --quiet https://releases.hashicorp.com/terraform/1.0.1/terraform_1.0.1_linux_${ARCH}.zip && \
unzip -o ./terraform_1.0.1_linux_${ARCH}.zip -d /usr/local/bin/ && \
rm terraform_1.0.1_linux_${ARCH}.zip
# UCX for RDMA
RUN wget https://github.com/openucx/ucx/releases/download/v1.16.0/ucx-1.16.0.tar.gz && \
tar xzf ucx-1.16.0.tar.gz && \
cd ucx-1.16.0 && \
mkdir build && \
cd build && \
../configure --prefix=/usr/local --without-go && \
make -j4 && \
make install && \
echo "/usr/local/lib" | tee /etc/ld.so.conf.d/ucx.conf && \
ldconfig \
11 changes: 11 additions & 0 deletions dev/github/Dockerfile-jdk17
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -179,3 +179,14 @@ RUN ARCH=$(dpkg --print-architecture) && \
wget --quiet https://releases.hashicorp.com/terraform/1.0.1/terraform_1.0.1_linux_${ARCH}.zip && \
unzip -o ./terraform_1.0.1_linux_${ARCH}.zip -d /usr/local/bin/ && \
rm terraform_1.0.1_linux_${ARCH}.zip
# UCX for RDMA
RUN wget https://github.com/openucx/ucx/releases/download/v1.16.0/ucx-1.16.0.tar.gz && \
tar xzf ucx-1.16.0.tar.gz && \
cd ucx-1.16.0 && \
mkdir build && \
cd build && \
../configure --prefix=/usr/local --without-go && \
make -j4 && \
make install && \
echo "/usr/local/lib" | tee /etc/ld.so.conf.d/ucx.conf && \
ldconfig \
11 changes: 11 additions & 0 deletions dev/github/Dockerfile-jdk8
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,3 +57,14 @@ RUN ARCH=$(dpkg --print-architecture) && \
wget --quiet https://releases.hashicorp.com/terraform/1.0.1/terraform_1.0.1_linux_${ARCH}.zip && \
unzip -o ./terraform_1.0.1_linux_${ARCH}.zip -d /usr/local/bin/ && \
rm terraform_1.0.1_linux_${ARCH}.zip
# UCX for RDMA
RUN wget https://github.com/openucx/ucx/releases/download/v1.16.0/ucx-1.16.0.tar.gz && \
tar xzf ucx-1.16.0.tar.gz && \
cd ucx-1.16.0 && \
mkdir build && \
cd build && \
../configure --prefix=/usr/local --without-go && \
make -j4 && \
make install && \
echo "/usr/local/lib" | tee /etc/ld.so.conf.d/ucx.conf && \
ldconfig \
4 changes: 2 additions & 2 deletions docs/_config.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,10 +7,10 @@ kramdown:
# The full version string
# Used to match the version string portion of file paths, URLs, and dependency versions
# e.g. client jar path,
ALLUXIO_VERSION_STRING: 313-SNAPSHOT
ALLUXIO_VERSION_STRING: 314-SNAPSHOT
# We must inline the version string (e.g., "1.4.0-SNAPSHOT") rather than using the macro of Alluxio version.
# Otherwise the macro name remains in the output.
ALLUXIO_CLIENT_JAR_PATH: /<PATH_TO_ALLUXIO>/client/alluxio-313-SNAPSHOT-client.jar
ALLUXIO_CLIENT_JAR_PATH: /<PATH_TO_ALLUXIO>/client/alluxio-314-SNAPSHOT-client.jar
# For release branches, this should be the Alluxio version in the form of <major>.<minor>, e.g. both 1.5.0 and 1.5.0-RC1 should use "1.5"
# For master branch, this should be "edge"
# This should be used to reference versioned pages that are not relative to the docs/ directory
Expand Down
2 changes: 1 addition & 1 deletion dora/core/client/fs/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
<parent>
<groupId>org.alluxio</groupId>
<artifactId>alluxio-core-client</artifactId>
<version>313-SNAPSHOT</version>
<version>314-SNAPSHOT</version>
</parent>
<artifactId>alluxio-core-client-fs</artifactId>
<packaging>jar</packaging>
Expand Down
46 changes: 45 additions & 1 deletion dora/core/client/fs/src/main/java/alluxio/client/file/cache/LocalCacheManager.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -751,6 +751,50 @@ public boolean append(PageId pageId, int appendAt, byte[] page, CacheContext cac
return put(pageId, page, cacheContext);
}

/**
* delete the specified page. Be cautious that this method will return true if the page
* does not exist since the page already gone.
*
* @param info pageInfo
* @param isTemporary whether is it temporary or not
* @return whether the page is deleted successfully or not
*/
public boolean deletePageIfExists(PageInfo info, boolean isTemporary) {
if (mState.get() != READ_WRITE) {
Metrics.DELETE_NOT_READY_ERRORS.inc();
Metrics.DELETE_ERRORS.inc();
return false;
}
boolean ok = true;
ReadWriteLock pageLock = getPageLock(info.getPageId());
try (LockResource r = new LockResource(pageLock.writeLock())) {
try (LockResource r1 = new LockResource(mPageMetaStore.getLock().writeLock())) {
try {
mPageMetaStore.removePage(info.getPageId(), isTemporary);
} catch (PageNotFoundException e) {
Metrics.DELETE_NON_EXISTING_PAGE_ERRORS.inc();
Metrics.DELETE_ERRORS.inc();
// pass through to delete the page from page store
}
}
try {
info.getLocalCacheDir().getPageStore().delete(info.getPageId(), isTemporary);
} catch (IOException e) {
LOG.error("Failed to delete page {} (isTemporary: {}) from pageStore.",
info.getPageId(), isTemporary, e);
ok = false;
Metrics.DELETE_STORE_DELETE_ERRORS.inc();
Metrics.DELETE_ERRORS.inc();
} catch (PageNotFoundException e) {
Metrics.DELETE_NON_EXISTING_PAGE_ERRORS.inc();
Metrics.DELETE_ERRORS.inc();
ok = true;
}
LOG.debug("delete({}) exits, success: {}", info.getPageId(), ok);
return ok;
}
}

/**
* Restores a page store at the configured location, updating meta store accordingly.
* If restore process fails, cleanup the location and create a new page store.
Expand Down Expand Up @@ -921,7 +965,7 @@ public void invalidate(Predicate<PageInfo> predicate) {
PageInfo pageInfo = pageInfoOpt.get();
boolean isPageDeleted = false;
if (predicate.test(pageInfo)) {
isPageDeleted = delete(pageInfo.getPageId());
isPageDeleted = deletePageIfExists(pageInfo, false);
}
if (isPageDeleted) {
MetricsSystem.meter(MetricKey.CLIENT_CACHE_PAGES_INVALIDATED.getName()).mark();
Expand Down
2 changes: 1 addition & 1 deletion dora/core/client/hdfs/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
<parent>
<groupId>org.alluxio</groupId>
<artifactId>alluxio-core-client</artifactId>
<version>313-SNAPSHOT</version>
<version>314-SNAPSHOT</version>
</parent>
<artifactId>alluxio-core-client-hdfs</artifactId>
<packaging>jar</packaging>
Expand Down
2 changes: 1 addition & 1 deletion dora/core/client/hdfs3/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
<parent>
<groupId>org.alluxio</groupId>
<artifactId>alluxio-core-client</artifactId>
<version>313-SNAPSHOT</version>
<version>314-SNAPSHOT</version>
</parent>
<artifactId>alluxio-core-client-hdfs3</artifactId>
<packaging>jar</packaging>
Expand Down
2 changes: 1 addition & 1 deletion dora/core/client/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
<parent>
<groupId>org.alluxio</groupId>
<artifactId>alluxio-core</artifactId>
<version>313-SNAPSHOT</version>
<version>314-SNAPSHOT</version>
</parent>
<artifactId>alluxio-core-client</artifactId>
<packaging>pom</packaging>
Expand Down
2 changes: 1 addition & 1 deletion dora/core/common/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
<parent>
<groupId>org.alluxio</groupId>
<artifactId>alluxio-core</artifactId>
<version>313-SNAPSHOT</version>
<version>314-SNAPSHOT</version>
</parent>
<artifactId>alluxio-core-common</artifactId>
<packaging>jar</packaging>
Expand Down
2 changes: 1 addition & 1 deletion dora/core/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
<parent>
<groupId>org.alluxio</groupId>
<artifactId>alluxio-dora</artifactId>
<version>313-SNAPSHOT</version>
<version>314-SNAPSHOT</version>
</parent>
<artifactId>alluxio-core</artifactId>
<packaging>pom</packaging>
Expand Down
2 changes: 1 addition & 1 deletion dora/core/server/common/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
<parent>
<groupId>org.alluxio</groupId>
<artifactId>alluxio-core-server</artifactId>
<version>313-SNAPSHOT</version>
<version>314-SNAPSHOT</version>
</parent>
<artifactId>alluxio-core-server-common</artifactId>
<packaging>jar</packaging>
Expand Down
2 changes: 1 addition & 1 deletion dora/core/server/master/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
<parent>
<groupId>org.alluxio</groupId>
<artifactId>alluxio-core-server</artifactId>
<version>313-SNAPSHOT</version>
<version>314-SNAPSHOT</version>
</parent>
<artifactId>alluxio-core-server-master</artifactId>
<packaging>jar</packaging>
Expand Down
2 changes: 1 addition & 1 deletion dora/core/server/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
<parent>
<groupId>org.alluxio</groupId>
<artifactId>alluxio-core</artifactId>
<version>313-SNAPSHOT</version>
<version>314-SNAPSHOT</version>
</parent>
<artifactId>alluxio-core-server</artifactId>
<packaging>pom</packaging>
Expand Down
2 changes: 1 addition & 1 deletion dora/core/server/proxy/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
<parent>
<groupId>org.alluxio</groupId>
<artifactId>alluxio-core-server</artifactId>
<version>313-SNAPSHOT</version>
<version>314-SNAPSHOT</version>
</parent>
<artifactId>alluxio-core-server-proxy</artifactId>
<packaging>jar</packaging>
Expand Down
2 changes: 1 addition & 1 deletion dora/core/server/worker/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
<parent>
<groupId>org.alluxio</groupId>
<artifactId>alluxio-core-server</artifactId>
<version>313-SNAPSHOT</version>
<version>314-SNAPSHOT</version>
</parent>
<artifactId>alluxio-core-server-worker</artifactId>
<packaging>jar</packaging>
Expand Down
2 changes: 1 addition & 1 deletion dora/examples/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
<parent>
<groupId>org.alluxio</groupId>
<artifactId>alluxio-dora</artifactId>
<version>313-SNAPSHOT</version>
<version>314-SNAPSHOT</version>
</parent>
<artifactId>alluxio-examples</artifactId>
<packaging>jar</packaging>
Expand Down
2 changes: 1 addition & 1 deletion dora/integration/fuse/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
<parent>
<groupId>org.alluxio</groupId>
<artifactId>alluxio-integration</artifactId>
<version>313-SNAPSHOT</version>
<version>314-SNAPSHOT</version>
</parent>
<artifactId>alluxio-integration-fuse</artifactId>
<name>Alluxio Integration - FUSE</name>
Expand Down
2 changes: 1 addition & 1 deletion dora/integration/jnifuse/fs/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
<parent>
<groupId>org.alluxio</groupId>
<artifactId>alluxio-integration-jnifuse</artifactId>
<version>313-SNAPSHOT</version>
<version>314-SNAPSHOT</version>
</parent>

<artifactId>alluxio-integration-jnifuse-fs</artifactId>
Expand Down
2 changes: 1 addition & 1 deletion dora/integration/jnifuse/native/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
<parent>
<groupId>org.alluxio</groupId>
<artifactId>alluxio-integration-jnifuse</artifactId>
<version>313-SNAPSHOT</version>
<version>314-SNAPSHOT</version>
</parent>

<artifactId>alluxio-integration-jnifuse-native</artifactId>
Expand Down
2 changes: 1 addition & 1 deletion dora/integration/jnifuse/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
<parent>
<groupId>org.alluxio</groupId>
<artifactId>alluxio-integration</artifactId>
<version>313-SNAPSHOT</version>
<version>314-SNAPSHOT</version>
</parent>
<artifactId>alluxio-integration-jnifuse</artifactId>
<packaging>pom</packaging>
Expand Down
2 changes: 1 addition & 1 deletion dora/integration/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
<parent>
<groupId>org.alluxio</groupId>
<artifactId>alluxio-dora</artifactId>
<version>313-SNAPSHOT</version>
<version>314-SNAPSHOT</version>
</parent>
<artifactId>alluxio-integration</artifactId>
<packaging>pom</packaging>
Expand Down
2 changes: 1 addition & 1 deletion dora/integration/tools/hms/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
<parent>
<groupId>org.alluxio</groupId>
<artifactId>alluxio-integration-tools</artifactId>
<version>313-SNAPSHOT</version>
<version>314-SNAPSHOT</version>
</parent>

<artifactId>alluxio-integration-tools-hms</artifactId>
Expand Down
2 changes: 1 addition & 1 deletion dora/integration/tools/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
<parent>
<groupId>org.alluxio</groupId>
<artifactId>alluxio-integration</artifactId>
<version>313-SNAPSHOT</version>
<version>314-SNAPSHOT</version>
</parent>
<artifactId>alluxio-integration-tools</artifactId>
<packaging>pom</packaging>
Expand Down
2 changes: 1 addition & 1 deletion dora/integration/tools/validation/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
<parent>
<groupId>org.alluxio</groupId>
<artifactId>alluxio-integration-tools</artifactId>
<version>313-SNAPSHOT</version>
<version>314-SNAPSHOT</version>
</parent>
<artifactId>alluxio-integration-tools-validation</artifactId>
<packaging>jar</packaging>
Expand Down
2 changes: 1 addition & 1 deletion dora/job/client/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
<parent>
<groupId>org.alluxio</groupId>
<artifactId>alluxio-job</artifactId>
<version>313-SNAPSHOT</version>
<version>314-SNAPSHOT</version>
</parent>
<artifactId>alluxio-job-client</artifactId>
<packaging>jar</packaging>
Expand Down
2 changes: 1 addition & 1 deletion dora/job/common/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
<parent>
<groupId>org.alluxio</groupId>
<artifactId>alluxio-job</artifactId>
<version>313-SNAPSHOT</version>
<version>314-SNAPSHOT</version>
</parent>
<artifactId>alluxio-job-common</artifactId>
<packaging>jar</packaging>
Expand Down
2 changes: 1 addition & 1 deletion dora/job/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
<parent>
<groupId>org.alluxio</groupId>
<artifactId>alluxio-dora</artifactId>
<version>313-SNAPSHOT</version>
<version>314-SNAPSHOT</version>
</parent>
<artifactId>alluxio-job</artifactId>
<packaging>pom</packaging>
Expand Down
2 changes: 1 addition & 1 deletion dora/job/server/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
<parent>
<groupId>org.alluxio</groupId>
<artifactId>alluxio-job</artifactId>
<version>313-SNAPSHOT</version>
<version>314-SNAPSHOT</version>
</parent>
<artifactId>alluxio-job-server</artifactId>
<packaging>jar</packaging>
Expand Down
2 changes: 1 addition & 1 deletion dora/microbench/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
<parent>
<groupId>org.alluxio</groupId>
<artifactId>alluxio-dora</artifactId>
<version>313-SNAPSHOT</version>
<version>314-SNAPSHOT</version>
</parent>
<artifactId>alluxio-microbench</artifactId>
<packaging>jar</packaging>
Expand Down
2 changes: 1 addition & 1 deletion dora/minicluster/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
<parent>
<groupId>org.alluxio</groupId>
<artifactId>alluxio-dora</artifactId>
<version>313-SNAPSHOT</version>
<version>314-SNAPSHOT</version>
</parent>

<artifactId>alluxio-minicluster</artifactId>
Expand Down
Loading
Loading