Add project structure and template for native components #18164

dbw9580 · 2023-09-18T07:35:33Z

What changes are proposed in this pull request?

Prepare project structure for native components.
Add a maven archetype for Rust-based native components.
The native module is behind a profile native-components and is deactivated by default.

Why are the changes needed?

Add support for native components

Does this PR introduce any user facing changes?

No.

Signed-off-by: Bowen Ding <[email protected]>

jja725 · 2023-09-18T21:56:34Z

This is a great first step for introducing Rust in Alluxio!

… into feat/alluxio-native-components

Xenorith · 2023-09-19T14:40:58Z

native/rust/README.md

+
+## Building the modules
+
+To build Rust sources, you will need the Rust toolchains. The recommended way is to use


it would be good to update the alluxio-maven image that is used as the build environment to include the Rust toolchains. the dockerfile for the image is located in dev/github/Dockerfile-jdk8. there are also dockerfiles for jdk11 and jdk17; it would be nice to update those too but they are currently not in use.

I added rust toolchain to the dockerfile:

alluxio/dev/github/Dockerfile-jdk8

Lines 40 to 41 in 9205036

# Rust toolchains for native components

RUN apt-get install -y rust-all

but it does not seem to work:

2023-09-25T17:54:35.0392570Z 17:54:35.038 [INFO] 17:54:35.037 [ERROR] -> [Help 1] 2023-09-25T17:54:35.0398031Z 17:54:35.039 [INFO] org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.questdb:rust-maven-plugin:1.1.1:build (release-build) on project alluxio-rust-example: Rust's `cargo` not found in PATH=/usr/local/openjdk-8/bin:/usr/local/go/bin:/usr/local/openjdk-8/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin. 2023-09-25T17:54:35.0608292Z 17:54:35.060 [INFO] 2023-09-25T17:54:35.0620795Z 17:54:35.061 [INFO] See https://www.rust-lang.org/tools/install

looks like the rust toolchain is not properly installed. Does it require re-building the image so that github CI can pick up the change?

yes requires rebuilding the image and subsequently bumping up the image version in dev/github/run_docker.sh

Xenorith · 2023-09-19T14:42:41Z

native/rust/README.md

+by `artifactId`. The archetype generation command will also edit the pom file of the
+parent module, which is `native/rust`, to add the new submodule to the module list.
+
+> Note: The archetype plugin modifies the pom file in a way that is not compatible with


it may be possible to override the plugin settings for the rust module by explicitly declaring them in the pom? just a guess, i'm not confident that this is the way to do so, but i imagine there must be some way to achieve it

I digged into archetype's source code but unfortunately was unable to find a configuration option to stop it from editing the parent pom file.

One workaround is to generate the submodule outside the directory of a parent module, and then move the generated module back into native/rust, and manually add the module to the list. It's still a lot of hassle and kind of defeats the purpose of using a archetype.

pom.xml

Xenorith · 2023-09-19T14:44:48Z

native/rust/archetype/src/main/resources/archetype-resources/pom.xml

+    <version>${project.version}</version>
+  </parent>
+
+  <artifactId>\${artifactId}</artifactId>


are the /$ intentional in this file?

yes indeed. there are two passes of string interpolation:

the first is by maven resource plugin, when copying the resource files from the source directory to the output directory for packaging. Things like ${project.version} should be kept in sync with the whole project and should not be customized when a user generates a submodule.

the second is by the archetype plugin, when a user actually uses the archetype to generate a submodule. Things like ${artifactId} are meant to be different for different modules, so must be customized.

The escaping is used to make sure templates for the second interpolation are not eagerly processed by the resource plugin.

Xenorith · 2023-09-19T14:47:51Z

pom.xml

@@ -493,6 +494,11 @@
        <artifactId>jets3t</artifactId>
        <version>0.8.1</version>
      </dependency>
+      <dependency>


double check that the new dependencies are compatible with apache 2.0 license and not introducing any forbidden dependencies as described in https://www.apache.org/legal/resolved.html#category-x

New dependencies;

JNA (net.java.dev.jna:jna-jpms): LGPL and Apache 2.0 dual licensed.

Maven Archetype Plugin: Apache 2.0 as it's part of the Maven project

Maven plugin for Rust(org.questdb:rust-maven-plugin): Apache 2.0.

zuston · 2023-09-20T06:44:55Z

Do you have any roadmap of rust in Alluxio ? I'm interested

beinan

Great contribution, we can start to integrate something really fun with native code now

dbw9580 · 2023-09-25T18:00:13Z

pom.xml

+    <profile>
+      <id>native-components</id>
+      <modules>
+        <module>native</module>


I decided to hide the module behind a profile because before the infrastructure for native components is fully sorted out, we don't want to block other people's builds

Xenorith · 2023-09-26T07:32:26Z

dev/github/Dockerfile-jdk11

@@ -153,3 +153,5 @@ RUN ARCH=$(dpkg --print-architecture) && \
    wget --quiet https://releases.hashicorp.com/terraform/1.0.1/terraform_1.0.1_linux_${ARCH}.zip && \
    unzip -o ./terraform_1.0.1_linux_${ARCH}.zip -d /usr/local/bin/ && \
    rm terraform_1.0.1_linux_${ARCH}.zip
+# Rust toolchains for native components
+RUN apt-get install -y rust-all


append rust-all to the apt-get install on line 141. reduces the number of image layers that are cached. similar for other dockerfiles

Ok. But let me remove this change from this PR and move them into a dedicated PR that updates the docker images.

dbw9580 · 2023-09-26T08:36:06Z

alluxio-bot, merge this please

dbw9580 added 5 commits September 10, 2023 00:04

add native module

a270337

Merge branch 'dora' into feat/alluxio-native-components

c68ea2c

merge main

b421c81

update

0aead8f

remove example module

1fd2360

Signed-off-by: Bowen Ding <[email protected]>

dbw9580 requested a review from beinan September 18, 2023 07:37

dbw9580 added 2 commits September 19, 2023 18:23

add doc

35e0813

Merge remote-tracking branch 'dbw9580/feat/alluxio-native-components'…

5dd66df

… into feat/alluxio-native-components

dbw9580 requested a review from Xenorith September 19, 2023 14:37

Xenorith reviewed Sep 19, 2023

View reviewed changes

beinan approved these changes Sep 21, 2023

View reviewed changes

add rust toolchain to docker image and enabled integration tests

9205036

dbw9580 commented Sep 25, 2023

View reviewed changes

disable building native modules by default on CI

6821212

Xenorith reviewed Sep 26, 2023

View reviewed changes

dbw9580 added 2 commits September 26, 2023 15:42

undo changes to dockerfiles

69aa043

tweak docs

54a1b79

Xenorith approved these changes Sep 26, 2023

View reviewed changes

dbw9580 added type-feature This issue is a feature request POM Change area-build Build (maven, tarball) and tests area-native Non-Java native components labels Sep 26, 2023

alluxio-bot merged commit dc0849a into Alluxio:main Sep 26, 2023
12 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add project structure and template for native components #18164

Add project structure and template for native components #18164

dbw9580 commented Sep 18, 2023 •

edited

Loading

jja725 commented Sep 18, 2023

Xenorith Sep 19, 2023

dbw9580 Sep 25, 2023

Xenorith Sep 26, 2023

Xenorith Sep 19, 2023

dbw9580 Sep 25, 2023

Xenorith Sep 19, 2023

dbw9580 Sep 25, 2023

Xenorith Sep 19, 2023

dbw9580 Sep 25, 2023

zuston commented Sep 20, 2023

beinan left a comment

dbw9580 Sep 25, 2023

Xenorith Sep 26, 2023

dbw9580 Sep 26, 2023

Xenorith Sep 26, 2023

dbw9580 commented Sep 26, 2023


		## Building the modules

		To build Rust sources, you will need the Rust toolchains. The recommended way is to use

	# Rust toolchains for native components
	RUN apt-get install -y rust-all

Add project structure and template for native components #18164

Add project structure and template for native components #18164

Conversation

dbw9580 commented Sep 18, 2023 • edited Loading

What changes are proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user facing changes?

jja725 commented Sep 18, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zuston commented Sep 20, 2023

beinan left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dbw9580 commented Sep 26, 2023

dbw9580 commented Sep 18, 2023 •

edited

Loading