Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[improve][misc] Include native epoll library for Netty for arm64 #22319

Merged
merged 1 commit into from
Mar 21, 2024
Merged

[improve][misc] Include native epoll library for Netty for arm64 #22319

merged 1 commit into from
Mar 21, 2024

Conversation

lhotari
Copy link
Member

@lhotari lhotari commented Mar 21, 2024

Motivation

When running Pulsar in Docker on MacOS Apple Silicon M3, I get this error message:

Caused by: java.io.FileNotFoundException: META-INF/native/libnetty_transport_native_epoll_aarch_64.so
    at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:186) ~[io.netty-netty-common-4.1.100.Final.jar:4.1.100.Final]
    at io.netty.channel.epoll.Native.loadNativeLibrary(Native.java:323) ~[io.netty-netty-transport-classes-epoll-4.1.100.Final.jar:4.1.100.Final]
    at io.netty.channel.epoll.Native.<clinit>(Native.java:85) ~[io.netty-netty-transport-classes-epoll-4.1.100.Final.jar:4.1.100.Final]
    at io.netty.channel.epoll.Epoll.<clinit>(Epoll.java:40) ~[io.netty-netty-transport-classes-epoll-4.1.100.Final.jar:4.1.100.Final]
    at org.apache.pulsar.common.util.netty.EventLoopUtil.newEventLoopGroup(EventLoopUtil.java:59) ~[org.apache.pulsar-pulsar-common-3.0.3.jar:3.0.3]
    at org.apache.pulsar.broker.PulsarService.<init>(PulsarService.java:337) ~[org.apache.pulsar-pulsar-broker-3.0.3.jar:3.0.3]
    at org.apache.pulsar.PulsarBrokerStarter$BrokerStarter.<init>(PulsarBrokerStarter.java:204) ~[org.apache.pulsar-pulsar-broker-3.0.3.jar:3.0.3]
    at org.apache.pulsar.PulsarBrokerStarter.main(PulsarBrokerStarter.java:333) ~[org.apache.pulsar-pulsar-broker-3.0.3.jar:3.0.3]
    Suppressed: java.lang.UnsatisfiedLinkError: no netty_transport_native_epoll_aarch_64 in java.library.path: /usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib
        at java.lang.ClassLoader.loadLibrary(ClassLoader.java:2434) ~[?:?]
        at java.lang.Runtime.loadLibrary0(Runtime.java:818) ~[?:?]
        at java.lang.System.loadLibrary(System.java:1993) ~[?:?]
        at io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38) ~[io.netty-netty-common-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:396) ~[io.netty-netty-common-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:161) ~[io.netty-netty-common-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.channel.epoll.Native.loadNativeLibrary(Native.java:323) ~[io.netty-netty-transport-classes-epoll-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.channel.epoll.Native.<clinit>(Native.java:85) ~[io.netty-netty-transport-classes-epoll-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.channel.epoll.Epoll.<clinit>(Epoll.java:40) ~[io.netty-netty-transport-classes-epoll-4.1.100.Final.jar:4.1.100.Final]
        at org.apache.pulsar.common.util.netty.EventLoopUtil.newEventLoopGroup(EventLoopUtil.java:59) ~[org.apache.pulsar-pulsar-common-3.0.3.jar:3.0.3]
        at org.apache.pulsar.broker.PulsarService.<init>(PulsarService.java:337) ~[org.apache.pulsar-pulsar-broker-3.0.3.jar:3.0.3]
        at org.apache.pulsar.PulsarBrokerStarter$BrokerStarter.<init>(PulsarBrokerStarter.java:204) ~[org.apache.pulsar-pulsar-broker-3.0.3.jar:3.0.3]
        at org.apache.pulsar.PulsarBrokerStarter.main(PulsarBrokerStarter.java:333) ~[org.apache.pulsar-pulsar-broker-3.0.3.jar:3.0.3]
        Suppressed: java.lang.UnsatisfiedLinkError: no netty_transport_native_epoll_aarch_64 in java.library.path: /usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib
            at java.lang.ClassLoader.loadLibrary(ClassLoader.java:2434) ~[?:?]
            at java.lang.Runtime.loadLibrary0(Runtime.java:818) ~[?:?]
            at java.lang.System.loadLibrary(System.java:1993) ~[?:?]
            at io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38) ~[io.netty-netty-common-4.1.100.Final.jar:4.1.100.Final]
            at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
            at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[?:?]
            at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
            at java.lang.reflect.Method.invoke(Method.java:568) ~[?:?]
            at io.netty.util.internal.NativeLibraryLoader$1.run(NativeLibraryLoader.java:430) ~[io.netty-netty-common-4.1.100.Final.jar:4.1.100.Final]
            at java.security.AccessController.doPrivileged(AccessController.java:318) ~[?:?]
            at io.netty.util.internal.NativeLibraryLoader.loadLibraryByHelper(NativeLibraryLoader.java:422) ~[io.netty-netty-common-4.1.100.Final.jar:4.1.100.Final]
            at io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:388) ~[io.netty-netty-common-4.1.100.Final.jar:4.1.100.Final]
            at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:161) ~[io.netty-netty-common-4.1.100.Final.jar:4.1.100.Final]
            at io.netty.channel.epoll.Native.loadNativeLibrary(Native.java:323) ~[io.netty-netty-transport-classes-epoll-4.1.100.Final.jar:4.1.100.Final]
            at io.netty.channel.epoll.Native.<clinit>(Native.java:85) ~[io.netty-netty-transport-classes-epoll-4.1.100.Final.jar:4.1.100.Final]
            at io.netty.channel.epoll.Epoll.<clinit>(Epoll.java:40) ~[io.netty-netty-transport-classes-epoll-4.1.100.Final.jar:4.1.100.Final]
            at org.apache.pulsar.common.util.netty.EventLoopUtil.newEventLoopGroup(EventLoopUtil.java:59) ~[org.apache.pulsar-pulsar-common-3.0.3.jar:3.0.3]
            at org.apache.pulsar.broker.PulsarService.<init>(PulsarService.java:337) ~[org.apache.pulsar-pulsar-broker-3.0.3.jar:3.0.3]
            at org.apache.pulsar.PulsarBrokerStarter$BrokerStarter.<init>(PulsarBrokerStarter.java:204) ~[org.apache.pulsar-pulsar-broker-3.0.3.jar:3.0.3]
            at org.apache.pulsar.PulsarBrokerStarter.main(PulsarBrokerStarter.java:333) ~[org.apache.pulsar-pulsar-broker-3.0.3.jar:3.0.3]

Modifications

  • include native epoll library for Netty for arm64 (aarch64) platforms

Documentation

  • doc
  • doc-required
  • doc-not-needed
  • doc-complete

@lhotari lhotari requested a review from Technoboy- March 21, 2024 09:13
@lhotari lhotari self-assigned this Mar 21, 2024
@github-actions github-actions bot added the doc-not-needed Your PR changes do not impact docs label Mar 21, 2024
@lhotari lhotari merged commit 24e9437 into apache:master Mar 21, 2024
50 of 53 checks passed
lhotari added a commit that referenced this pull request Mar 27, 2024
@lhotari
[improve][misc] Include native epoll library for Netty for arm64 (#22319
ce9ce60 
)

(cherry picked from commit 24e9437)
lhotari added a commit that referenced this pull request Mar 27, 2024
@lhotari
[improve][misc] Include native epoll library for Netty for arm64 (#22319
22b724f 
)

(cherry picked from commit 24e9437)
nodece added a commit to nodece/pulsar that referenced this pull request Apr 29, 2024
@nodece
[improve][misc] Include native epoll library for Netty for arm64
be5daf3 
From apache#22319

Signed-off-by: Zixuan Liu <[email protected]>
nodece added a commit to ascentstream/pulsar that referenced this pull request May 10, 2024
@nodece @lhotari
@merlimat @michaeljmarshall @tisonkun @zymap @RobertIndie
[feat][build] Add support for ARM image (#12)
67a2d52 
* [Dockerfile] Enable retries for apt-get when building Pulsar docker image (apache#14513)

- also reduce default timeout to 30 seconds
- prevents issues where apt repository doesn't respond

(cherry picked from commit d3f6fe3)

* PIP-155: Removed Python 2 support (apache#15376)

* Remove Pulsar Client Build for Python 2.7

* Remove outdated homebrew files (source of truth is upstream homebrew)

* Remove Python 2.7 build references; print error in some cases

* Update python client tests to run with python client for python 3.5m

* PIP-155: Removed Python 2 support

* Fixed invocation in pulsar-build image

* Fixed clang-format-10 indent differences

* Fixed script invocation with wrong python

* We don't need to rebuild the manylinux image each time

* Fixed image name

* Reverted back to use newer protobuf

* Fixed image name

* Fixed missing python3 in centos:7 image

* Use python3 for gtest-parallel

* Show bash commands in docker-tests.sh

* Fixed gh action issue with git directory permissions

* Fixed python to 3

* Fixed custom_logger_test.py

* Fixed path in run_python_instance_tests.sh

* Function runtime should use python3

* Fixed function runtime test python expectation

* Fixed presto worker launcher

* Fixed notes on how to format C++ code

Co-authored-by: Michael Marshall <[email protected]>

(cherry picked from commit 2b2e0c5)
Signed-off-by: Zixuan Liu <[email protected]>

* [improve][docker] Switch to Temurin JDK (apache#17129)

Signed-off-by: Zixuan Liu <[email protected]>

(cherry picked from commit 4378856)
Signed-off-by: Zixuan Liu <[email protected]>

* [refactor][ci] Build the docker image with docker-maven-plugin (apache#17148)

(cherry picked from commit a68b58d)
Signed-off-by: Zixuan Liu <[email protected]>

* [feat][build] Support ARM64-based docker images (apache#17733)

(cherry picked from commit 9a2aeb2)
Signed-off-by: Zixuan Liu <[email protected]>

* PIP-209: Removed C++/Python clients from main repo (apache#17881)

* PIP-209: Removed C++/Python clients from main repo

* Removed python directory from Docekrfile

* Fixed python client version argument scoping

* Fixed handling of pulsar.functions.serde

(cherry picked from commit f3c547b)
Signed-off-by: Zixuan Liu <[email protected]>

* [improve][build] Avoid building image multiple times (apache#17208)

Signed-off-by: Zixuan Liu <[email protected]>
(cherry picked from commit 79a97a9)

* [improve] Allow to build and push multi-arch Docker images (apache#19432)

Co-authored-by: Lari Hotari <[email protected]>
Co-authored-by: Yong Zhang <[email protected]>
Co-authored-by: Zixuan Liu <[email protected]>
Co-authored-by: tison <[email protected]>

(cherry picked from commit 4190e40)
Signed-off-by: Zixuan Liu <[email protected]>

* [fix][build] Fix publish image script (apache#20305)

Signed-off-by: Zixuan Liu <[email protected]>

(cherry picked from commit 94c7bf3)
Signed-off-by: Zixuan Liu <[email protected]>

* [fix][build] Fix the pulsar-all image may use the wrong upstream image (apache#20435)

Signed-off-by: Zike Yang <[email protected]>
Co-authored-by: Lari Hotari <[email protected]>

(cherry picked from commit d7f3558)
Signed-off-by: Zixuan Liu <[email protected]>

* [feat][build] Adapt to Python client to be compatible with ARM arch

Signed-off-by: Zixuan Liu <[email protected]>

* [fix][build] Configure git-commit-id-plugin to skip git describe (apache#20550)

(cherry picked from commit 05f7e62)

* [improve][misc] Include native epoll library for Netty for arm64

From apache#22319

Signed-off-by: Zixuan Liu <[email protected]>

* [fix][misc] Rename all shaded Netty native libraries

From apache#22415

Signed-off-by: Zixuan Liu <[email protected]>

* [cleanup][build] Cleanup -Ddocker.nocache=true

Signed-off-by: Zixuan Liu <[email protected]>

* [fix][build] Fix ubuntu mirror

Signed-off-by: Zixuan Liu <[email protected]>

* [fix][build] Fix license

Signed-off-by: Zixuan Liu <[email protected]>

* [fix][build] Downgrade docker-maven to 0.43.3

Signed-off-by: Zixuan Liu <[email protected]>

---------

Signed-off-by: Zixuan Liu <[email protected]>
Signed-off-by: Zike Yang <[email protected]>
Co-authored-by: Lari Hotari <[email protected]>
Co-authored-by: Matteo Merli <[email protected]>
Co-authored-by: Michael Marshall <[email protected]>
Co-authored-by: tison <[email protected]>
Co-authored-by: Yong Zhang <[email protected]>
Co-authored-by: Zike Yang <[email protected]>
Co-authored-by: Lari Hotari <[email protected]>
nodece added a commit to nodece/pulsar that referenced this pull request May 11, 2024
@nodece @lhotari
@merlimat @michaeljmarshall @tisonkun @zymap @RobertIndie
[feat][build] Add support for ARM image (#12)
07c3930 
* [Dockerfile] Enable retries for apt-get when building Pulsar docker image (apache#14513)

- also reduce default timeout to 30 seconds
- prevents issues where apt repository doesn't respond

(cherry picked from commit d3f6fe3)

* PIP-155: Removed Python 2 support (apache#15376)

* Remove Pulsar Client Build for Python 2.7

* Remove outdated homebrew files (source of truth is upstream homebrew)

* Remove Python 2.7 build references; print error in some cases

* Update python client tests to run with python client for python 3.5m

* PIP-155: Removed Python 2 support

* Fixed invocation in pulsar-build image

* Fixed clang-format-10 indent differences

* Fixed script invocation with wrong python

* We don't need to rebuild the manylinux image each time

* Fixed image name

* Reverted back to use newer protobuf

* Fixed image name

* Fixed missing python3 in centos:7 image

* Use python3 for gtest-parallel

* Show bash commands in docker-tests.sh

* Fixed gh action issue with git directory permissions

* Fixed python to 3

* Fixed custom_logger_test.py

* Fixed path in run_python_instance_tests.sh

* Function runtime should use python3

* Fixed function runtime test python expectation

* Fixed presto worker launcher

* Fixed notes on how to format C++ code

Co-authored-by: Michael Marshall <[email protected]>

(cherry picked from commit 2b2e0c5)
Signed-off-by: Zixuan Liu <[email protected]>

* [improve][docker] Switch to Temurin JDK (apache#17129)

Signed-off-by: Zixuan Liu <[email protected]>

(cherry picked from commit 4378856)
Signed-off-by: Zixuan Liu <[email protected]>

* [refactor][ci] Build the docker image with docker-maven-plugin (apache#17148)

(cherry picked from commit a68b58d)
Signed-off-by: Zixuan Liu <[email protected]>

* [feat][build] Support ARM64-based docker images (apache#17733)

(cherry picked from commit 9a2aeb2)
Signed-off-by: Zixuan Liu <[email protected]>

* PIP-209: Removed C++/Python clients from main repo (apache#17881)

* PIP-209: Removed C++/Python clients from main repo

* Removed python directory from Docekrfile

* Fixed python client version argument scoping

* Fixed handling of pulsar.functions.serde

(cherry picked from commit f3c547b)
Signed-off-by: Zixuan Liu <[email protected]>

* [improve][build] Avoid building image multiple times (apache#17208)

Signed-off-by: Zixuan Liu <[email protected]>
(cherry picked from commit 79a97a9)

* [improve] Allow to build and push multi-arch Docker images (apache#19432)

Co-authored-by: Lari Hotari <[email protected]>
Co-authored-by: Yong Zhang <[email protected]>
Co-authored-by: Zixuan Liu <[email protected]>
Co-authored-by: tison <[email protected]>

(cherry picked from commit 4190e40)
Signed-off-by: Zixuan Liu <[email protected]>

* [fix][build] Fix publish image script (apache#20305)

Signed-off-by: Zixuan Liu <[email protected]>

(cherry picked from commit 94c7bf3)
Signed-off-by: Zixuan Liu <[email protected]>

* [fix][build] Fix the pulsar-all image may use the wrong upstream image (apache#20435)

Signed-off-by: Zike Yang <[email protected]>
Co-authored-by: Lari Hotari <[email protected]>

(cherry picked from commit d7f3558)
Signed-off-by: Zixuan Liu <[email protected]>

* [feat][build] Adapt to Python client to be compatible with ARM arch

Signed-off-by: Zixuan Liu <[email protected]>

* [fix][build] Configure git-commit-id-plugin to skip git describe (apache#20550)

(cherry picked from commit 05f7e62)

* [improve][misc] Include native epoll library for Netty for arm64

From apache#22319

Signed-off-by: Zixuan Liu <[email protected]>

* [fix][misc] Rename all shaded Netty native libraries

From apache#22415

Signed-off-by: Zixuan Liu <[email protected]>

* [cleanup][build] Cleanup -Ddocker.nocache=true

Signed-off-by: Zixuan Liu <[email protected]>

* [fix][build] Fix ubuntu mirror

Signed-off-by: Zixuan Liu <[email protected]>

* [fix][build] Fix license

Signed-off-by: Zixuan Liu <[email protected]>

* [fix][build] Downgrade docker-maven to 0.43.3

Signed-off-by: Zixuan Liu <[email protected]>

---------

Signed-off-by: Zixuan Liu <[email protected]>
Signed-off-by: Zike Yang <[email protected]>
Co-authored-by: Lari Hotari <[email protected]>
Co-authored-by: Matteo Merli <[email protected]>
Co-authored-by: Michael Marshall <[email protected]>
Co-authored-by: tison <[email protected]>
Co-authored-by: Yong Zhang <[email protected]>
Co-authored-by: Zike Yang <[email protected]>
Co-authored-by: Lari Hotari <[email protected]>
lhotari added a commit that referenced this pull request Jun 13, 2024
@lhotari
[improve][misc] Include native epoll library for Netty for arm64 (#22319
69ecbcd 
)

(cherry picked from commit 24e9437)
(cherry picked from commit 22b724f)

# Conflicts:
#	distribution/server/src/assemble/LICENSE.bin.txt
#	distribution/shell/src/assemble/LICENSE.bin.txt
#	pulsar-sql/presto-distribution/LICENSE
nikhil-ctds pushed a commit to datastax/pulsar that referenced this pull request Jun 17, 2024
@lhotari @nikhil-ctds
[improve][misc] Include native epoll library for Netty for arm64 (apa…
a7bcb02 
…che#22319)

(cherry picked from commit 24e9437)
(cherry picked from commit 22b724f)

# Conflicts:
#	distribution/server/src/assemble/LICENSE.bin.txt
#	distribution/shell/src/assemble/LICENSE.bin.txt
#	pulsar-sql/presto-distribution/LICENSE
(cherry picked from commit 69ecbcd)
nikhil-ctds pushed a commit to datastax/pulsar that referenced this pull request Jun 25, 2024
@lhotari @nikhil-ctds
[improve][misc] Include native epoll library for Netty for arm64 (apa…
de47988 
…che#22319)

(cherry picked from commit 24e9437)
(cherry picked from commit 22b724f)

# Conflicts:
#	distribution/server/src/assemble/LICENSE.bin.txt
#	distribution/shell/src/assemble/LICENSE.bin.txt
#	pulsar-sql/presto-distribution/LICENSE
(cherry picked from commit 69ecbcd)
srinath-ctds pushed a commit to datastax/pulsar that referenced this pull request Jul 1, 2024
@lhotari @srinath-ctds
[improve][misc] Include native epoll library for Netty for arm64 (apa…
aa39152 
…che#22319)

(cherry picked from commit 24e9437)
(cherry picked from commit 22b724f)

# Conflicts:
#	distribution/server/src/assemble/LICENSE.bin.txt
#	distribution/shell/src/assemble/LICENSE.bin.txt
#	pulsar-sql/presto-distribution/LICENSE
(cherry picked from commit 69ecbcd)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@nicoloboschi nicoloboschi nicoloboschi approved these changes

@Technoboy- Technoboy- Awaiting requested review from Technoboy-

@eolivelli eolivelli Awaiting requested review from eolivelli

@tisonkun tisonkun Awaiting requested review from tisonkun

Assignees

@lhotari lhotari

Labels
cherry-picked/branch-3.0 cherry-picked/branch-3.1 cherry-picked/branch-3.2 doc-not-needed Your PR changes do not impact docs ready-to-test release/3.0.4 release/3.1.4 release/3.2.2
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@lhotari @nicoloboschi