Custom spark 3.5.0 image built with spark connect giving serialization error with rapids libs #10611

kkdatta · 2024-03-19T13:40:21Z

kkdatta
Mar 19, 2024

I am builidng a custom spark docker image to use spark connect in kubernetes. the base image of my docker image is nvidia/cuda:11.8.0-devel-ubuntu22.04 . I am installing jdk 17 and the spark libs are of spark 3.5.0. i am using the rapids-4-spark_2.12-24.02.0.jar.
While executing any stage .The error thrown is
java.lang.ClassCastException: cannot assign instance of scala.collection.immutable.List$SerializationProxy to field org.apache.spark.rdd.RDD.dependencies_ of type scala.collection.Seq in instance of org.apache.spark.rdd.MapPartitionsRDD at java.base/java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2096) at java.base/java.io.ObjectStreamClass$FieldReflector.checkObjectFieldValueTypes(ObjectStreamClass.java:2060) at java.base/java.io.ObjectStreamClass.checkObjFieldValueTypes(ObjectStreamClass.java:1347) at java.base/java.io.ObjectInputStream$FieldValues.defaultCheckFieldValues(ObjectInputStream.java:2679) at java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2486) at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2257) at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1733) at java.base/java.io.ObjectInputStream$FieldValues.<init>(ObjectInputStream.java:2606) at java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2457) at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2257) at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1733) at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:509) at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:467) at scala.collection.immutable.List$SerializationProxy.readObject(List.scala:527) at jdk.internal.reflect.GeneratedMethodAccessor5.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:568) at java.base/java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1100) at java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2423) at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2257) at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1733) at java.base/java.io.ObjectInputStream$FieldValues.<init>(ObjectInputStream.java:2606) at java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2412) at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2257) at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1733) at java.base/java.io.ObjectInputStream$FieldValues.<init>(ObjectInputStream.java:2606) at java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2457) at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2257) at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1733) at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:509) at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:467)

I found this #3704 But looks like the config spark.rapids.force.caller.classloader is not avaiable in newer versions rapids-4-spark_2.12-24.02.0.jar? The necessary spark confs are set as expected
The custom spark image built is working fine if the rapids are removed.

The docker file for reference

#
# Copyright (c) 2020-2023, NVIDIA CORPORATION. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

FROM nvidia/cuda:11.8.0-devel-ubuntu22.04

ARG DEBIAN_FRONTEND="noninteractive"


ARG EXECUTOR_UID=9999
ARG EXECUTOR_GID=9999
RUN addgroup --system --gid ${EXECUTOR_GID} executor && \
    adduser --system --gid ${EXECUTOR_GID} --home /home/executor --uid ${EXECUTOR_UID} --disabled-password executor

ARG spark_uid=$EXECUTOR_UID


RUN apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/3bf863cc.pub

ENV DEBIAN_FRONTEND="noninteractive"

RUN apt-get update && apt-get install -y wget

RUN apt-get install -y curl ca-certificates



ENV JAVA_VERSION=17
ENV TEMURIN_DOWNLOAD_URL=https://api.adoptium.net/v3/binary/latest/${JAVA_VERSION}/ga/linux/x64/jdk/hotspot/normal/eclipse

RUN wget -O jdk.tar.gz ${TEMURIN_DOWNLOAD_URL} \
    && mkdir /opt/java \
    && tar -xzf jdk.tar.gz -C /opt/java --strip-components=1 \
    && rm jdk.tar.gz

ENV JAVA_HOME=/opt/java
ENV PATH=${PATH}:${JAVA_HOME}/bin

# Before building the docker image, first either download Apache Spark 3.1+ from
# http://spark.apache.org/downloads.html or build and make a Spark distribution following the
# instructions in http://spark.apache.org/docs/3.1.2/building-spark.html (see
# https://nvidia.github.io/spark-rapids/docs/download.html for other supported versions).  If this
# docker file is being used in the context of building your images from a Spark distribution, the
# docker build command should be invoked from the top level directory of the Spark
# distribution. E.g.: docker build -t spark:3.1.2 -f kubernetes/dockerfiles/spark/Dockerfile .


RUN set -ex && \
    ln -s /lib /lib64 && \
    apt install -y bash tini libc6 libpam-modules krb5-user libnss3 procps net-tools logrotate && \
    mkdir -p /opt/spark && \
    mkdir -p /opt/spark/jars && \
    mkdir -p /opt/spark/examples && \
    mkdir -p /opt/spark/work-dir && \
    mkdir -p /opt/sparkRapidsPlugin && \
    touch /opt/spark/RELEASE && \
    rm /bin/sh && \
    ln -sv /bin/bash /bin/sh && \
    echo "auth required pam_wheel.so use_uid" >> /etc/pam.d/su && \
    chgrp root /etc/passwd && chmod ug+rw /etc/passwd


COPY spark/jars /opt/spark/jars
COPY spark/bin /opt/spark/bin
COPY spark/sbin /opt/spark/sbin
COPY spark/kubernetes/dockerfiles/spark/entrypoint.sh /opt/
COPY spark/kubernetes/dockerfiles/spark/decom.sh /opt/
COPY spark/examples /opt/spark/examples
COPY spark/kubernetes/tests /opt/spark/tests
COPY spark/data /opt/spark/data
COPY hadoop-azure-3.3.6.jar /opt/spark/jars/
COPY azure-storage-7.0.1.jar /opt/spark/jars/
COPY delta-core_2.12-2.4.0.jar /opt/spark/jars/
COPY delta-storage-3.1.0.jar /opt/spark/jars/
COPY hadoop-common-3.3.6.jar /opt/spark/jars/
COPY spark-hadoop-cloud_2.12-3.5.0.jar /opt/spark/jars/
COPY hadoop-azure-datalake-3.3.6.jar /opt/spark/jars/
COPY hadoop-datajoin-3.3.6.jar /opt/spark/jars/
COPY hadoop-extras-3.3.6.jar /opt/spark/jars/
COPY hadoop-resourceestimator-3.3.6.jar /opt/spark/jars/
COPY woodstox-core-6.6.1.jar /opt/spark/jars/
COPY stax2-api-4.2.2.jar /opt/spark/jars/
COPY commons-configuration2-2.9.0.jar /opt/spark/jars/
COPY re2j-1.7.jar /opt/spark/jars/
COPY azure-data-lake-store-sdk-2.3.10.jar /opt/spark/jars/
COPY hadoop-client-3.3.6.jar /opt/spark/jars/
COPY jdom2-2.0.6.1.jar /opt/spark/jars/
COPY ojalgo-53.3.0.jar /opt/spark/jars/
COPY org.jacoco.agent-0.8.11.jar /opt/spark/jars/
COPY wildfly-openssl-2.2.5.Final.jar /opt/spark/jars/
COPY rapids-4-spark_2.12-*.jar /opt/spark/jars

#COPY rapids-4-spark_2.12-*.jar /opt/sparkRapidsPlugin
COPY getGpusResources.sh /opt/sparkRapidsPlugin
RUN chmod 777 /opt/sparkRapidsPlugin
RUN chmod 777 /opt/sparkRapidsPlugin/getGpusResources.sh
RUN chown executor:executor /opt/sparkRapidsPlugin
RUN chmod 777 /opt/sparkRapidsPlugin
RUN chmod 777 /opt/sparkRapidsPlugin/getGpusResources.sh



RUN mkdir /opt/spark/python
# TODO: Investigate running both pip and pip3 via virtualenvs
RUN apt install -y python3 python3-pip && \
    pip install --upgrade pip setuptools && \
    rm -r /root/.cache && rm -rf /var/cache/apt/* && \
    ln -s /usr/bin/python3 /usr/bin/python

COPY spark/python/pyspark /opt/spark/python/pyspark
COPY spark/python/lib /opt/spark/python/lib

ENV SPARK_HOME /opt/spark

WORKDIR /opt/spark/work-dir

RUN chown executor:executor /opt/spark/work-dir
RUN chmod a+x /opt/decom.sh
#Sparkconnect: Config default log dir
RUN mkdir -p /opt/spark/logs && chmod 777 /opt/spark/logs
#Sparkconnect : Config python and pyspark
RUN ln -sv /usr/bin/python3 /usr/local/bin/python3

ENTRYPOINT [ "/opt/entrypoint.sh" ]


# Specify the User that the actual main process will run as
USER ${spark_uid}
# Sparkconnect:include user's local bin for pip install
ENV PATH=/home/executor/.local/bin:$PATH

Will be helpful if someone explains what is going wrong here.
Thanks

Answered by tgravescs

Mar 19, 2024

we haven't tested with Spark connect but we have seen some a similar issue with our jar that has multiple shims have classloader issues.

I would suggest rebuilding spark-rapids for just the single spark release (3.5.0) that you are using to see if it fixes this issue:
https://github.com/NVIDIA/spark-rapids/blob/branch-24.02/CONTRIBUTING.md#building-a-distribution-for-a-single-spark-release

Please let us know if this works for you.

View full answer

tgravescs · 2024-03-19T15:06:08Z

tgravescs
Mar 19, 2024
Maintainer

we haven't tested with Spark connect but we have seen some a similar issue with our jar that has multiple shims have classloader issues.

I would suggest rebuilding spark-rapids for just the single spark release (3.5.0) that you are using to see if it fixes this issue:
https://github.com/NVIDIA/spark-rapids/blob/branch-24.02/CONTRIBUTING.md#building-a-distribution-for-a-single-spark-release

Please let us know if this works for you.

3 replies

kkdatta Mar 19, 2024
Author

Thanks @tgravescs will try and let u know.
I dont think it is anything to do with sparkconnect but will test that too.
For reference , this is the following logs at the startup. I dont see anything unexpected here too.

++ id -u
+ myuid=9999
++ id -g
+ mygid=9999
+ set +e
++ getent passwd 9999
+ uidentry=executor:x:9999:9999::/home/executor:/usr/sbin/nologin
+ set -e
+ '[' -z executor:x:9999:9999::/home/executor:/usr/sbin/nologin ']'
+ '[' -z /opt/java ']'
+ SPARK_CLASSPATH=':/opt/spark/jars/*'
+ env
+ grep SPARK_JAVA_OPT_
+ sort -t_ -k4 -n
+ sed 's/[^=]*=\(.*\)/\1/g'
++ command -v readarray
+ '[' readarray ']'
+ readarray -t SPARK_EXECUTOR_JAVA_OPTS
+ '[' -n '' ']'
+ '[' -z ']'
+ '[' -z ']'
+ '[' -n '' ']'
+ '[' -z ']'
+ '[' -z x ']'
+ SPARK_CLASSPATH='/opt/spark/conf::/opt/spark/jars/*'
+ SPARK_CLASSPATH='/opt/spark/conf::/opt/spark/jars/*:/opt/spark/work-dir'
+ case "$1" in
+ shift 1
+ CMD=(${JAVA_HOME}/bin/java "${SPARK_EXECUTOR_JAVA_OPTS[@]}" -Xms$SPARK_EXECUTOR_MEMORY -Xmx$SPARK_EXECUTOR_MEMORY -cp "$SPARK_CLASSPATH:$SPARK_DIST_CLASSPATH" org.apache.spark.scheduler.cluster.k8s.KubernetesExecutorBackend --driver-url $SPARK_DRIVER_URL --executor-id $SPARK_EXECUTOR_ID --cores $SPARK_EXECUTOR_CORES --app-id $SPARK_APPLICATION_ID --hostname $SPARK_EXECUTOR_POD_IP --resourceProfileId $SPARK_RESOURCE_PROFILE_ID --podName $SPARK_EXECUTOR_POD_NAME)
+ exec /usr/bin/tini -s -- /opt/java/bin/java -Djava.net.preferIPv6Addresses=false -XX:+IgnoreUnrecognizedVMOptions --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/sun.nio.cs=ALL-UNNAMED --add-opens=java.base/sun.security.action=ALL-UNNAMED --add-opens=java.base/sun.util.calendar=ALL-UNNAMED --add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED -Djdk.reflect.useDirectMethodHandle=false -Dspark.driver.port=37063 -Xms153600m -Xmx153600m -cp '/opt/spark/conf::/opt/spark/jars/*:/opt/spark/work-dir:' org.apache.spark.scheduler.cluster.k8s.KubernetesExecutorBackend --driver-url spark://[email protected]:37063 --executor-id 1 --cores 20 --app-id spark-0c8d00d58d1f48b08d7c125ab69feb77 --hostname 172.16.4.105 --resourceProfileId 0 --podName spark-connect-server-ceb90d8e56d734ef-exec-1
Using Spark's default log4j profile: org/apache/spark/log4j2-defaults.properties
24/03/19 13:12:45 INFO KubernetesExecutorBackend: Started daemon with process name: 15@spark-connect-server-ceb90d8e56d734ef-exec-1
24/03/19 13:12:45 INFO SignalUtils: Registering signal handler for TERM
24/03/19 13:12:45 INFO SignalUtils: Registering signal handler for HUP
24/03/19 13:12:45 INFO SignalUtils: Registering signal handler for INT
24/03/19 13:12:45 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
24/03/19 13:12:45 INFO SecurityManager: Changing view acls to: executor
24/03/19 13:12:45 INFO SecurityManager: Changing modify acls to: executor
24/03/19 13:12:45 INFO SecurityManager: Changing view acls groups to: 
24/03/19 13:12:45 INFO SecurityManager: Changing modify acls groups to: 
24/03/19 13:12:45 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: executor; groups with view permissions: EMPTY; users with modify permissions: executor; groups with modify permissions: EMPTY
24/03/19 13:12:45 INFO TransportClientFactory: Successfully created connection to /172.16.4.252:37063 after 41 ms (0 ms spent in bootstraps)
24/03/19 13:12:46 INFO SecurityManager: Changing view acls to: executor
24/03/19 13:12:46 INFO SecurityManager: Changing modify acls to: executor
24/03/19 13:12:46 INFO SecurityManager: Changing view acls groups to: 
24/03/19 13:12:46 INFO SecurityManager: Changing modify acls groups to: 
24/03/19 13:12:46 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: executor; groups with view permissions: EMPTY; users with modify permissions: executor; groups with modify permissions: EMPTY
24/03/19 13:12:46 INFO TransportClientFactory: Successfully created connection to /172.16.4.252:37063 after 1 ms (0 ms spent in bootstraps)
24/03/19 13:12:46 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-8e6d22de-6149-48ee-8330-e1d4c198e7ae
24/03/19 13:12:46 INFO MemoryStore: MemoryStore started with capacity 89.8 GiB
24/03/19 13:12:46 INFO CoarseGrainedExecutorBackend: Connecting to driver: spark://[email protected]:37063
24/03/19 13:12:46 INFO ResourceDiscoveryScriptPlugin: Discovering resources for gpu with script: /opt/sparkRapidsPlugin/getGpusResources.sh
24/03/19 13:12:46 INFO ResourceUtils: ==============================================================
24/03/19 13:12:46 INFO ResourceUtils: Custom resources for spark.executor:
gpu -> [name: gpu, addresses: 0]
24/03/19 13:12:46 INFO ResourceUtils: ==============================================================
24/03/19 13:12:46 INFO CoarseGrainedExecutorBackend: Successfully registered with driver
24/03/19 13:12:46 INFO Executor: Starting executor ID 1 on host 172.16.4.105
24/03/19 13:12:46 INFO Executor: OS info Linux, 5.15.0-1057-azure, amd64
24/03/19 13:12:46 INFO Executor: Java version 17.0.10
24/03/19 13:12:46 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 34275.
24/03/19 13:12:46 INFO NettyBlockTransferService: Server created on 172.16.4.105:34275
24/03/19 13:12:46 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
24/03/19 13:12:46 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(1, 172.16.4.105, 34275, None)
24/03/19 13:12:46 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(1, 172.16.4.105, 34275, None)
24/03/19 13:12:46 INFO BlockManager: Initialized BlockManager: BlockManagerId(1, 172.16.4.105, 34275, None)
24/03/19 13:12:46 INFO Executor: Starting executor with user classpath (userClassPathFirst = false): ''
24/03/19 13:12:46 INFO Executor: Created or updated repl class loader org.apache.spark.util.MutableURLClassLoader@2258f7d6 for default.
24/03/19 13:12:46 INFO Executor: Fetching spark://172.16.4.252:37063/files/org.apache.spark_spark-connect_2.12-3.5.0.jar with timestamp 1710853927506
24/03/19 13:12:46 INFO TransportClientFactory: Successfully created connection to /172.16.4.252:37063 after 5 ms (0 ms spent in bootstraps)
24/03/19 13:12:46 INFO Utils: Fetching spark://172.16.4.252:37063/files/org.apache.spark_spark-connect_2.12-3.5.0.jar to /opt/spark/work-dir/./fetchFileTemp8369460045948077799.tmp
24/03/19 13:12:46 INFO Executor: Fetching spark://172.16.4.252:37063/files/org.spark-project.spark_unused-1.0.0.jar with timestamp 1710853927506
24/03/19 13:12:46 INFO Utils: Fetching spark://172.16.4.252:37063/files/org.spark-project.spark_unused-1.0.0.jar to /opt/spark/work-dir/./fetchFileTemp7016878810528022801.tmp
24/03/19 13:12:46 INFO Executor: Fetching spark://172.16.4.252:37063/jars/org.apache.spark_spark-connect_2.12-3.5.0.jar with timestamp 1710853927506
24/03/19 13:12:46 INFO Utils: Fetching spark://172.16.4.252:37063/jars/org.apache.spark_spark-connect_2.12-3.5.0.jar to /opt/spark/work-dir/./fetchFileTemp8863728185612930748.tmp
24/03/19 13:12:46 INFO Utils: /opt/spark/work-dir/./fetchFileTemp8863728185612930748.tmp has been previously copied to /opt/spark/work-dir/./org.apache.spark_spark-connect_2.12-3.5.0.jar
24/03/19 13:12:46 INFO Executor: Adding file:/opt/spark/work-dir/./org.apache.spark_spark-connect_2.12-3.5.0.jar to class loader default
24/03/19 13:12:46 INFO Executor: Fetching spark://172.16.4.252:37063/jars/org.spark-project.spark_unused-1.0.0.jar with timestamp 1710853927506
24/03/19 13:12:46 INFO Utils: Fetching spark://172.16.4.252:37063/jars/org.spark-project.spark_unused-1.0.0.jar to /opt/spark/work-dir/./fetchFileTemp2854440234807949995.tmp
24/03/19 13:12:46 INFO Utils: /opt/spark/work-dir/./fetchFileTemp2854440234807949995.tmp has been previously copied to /opt/spark/work-dir/./org.spark-project.spark_unused-1.0.0.jar
24/03/19 13:12:46 INFO Executor: Adding file:/opt/spark/work-dir/./org.spark-project.spark_unused-1.0.0.jar to class loader default
24/03/19 13:12:46 INFO ShimLoader: Loading shim for Spark version: 3.5.0
24/03/19 13:12:46 INFO ShimLoader: Complete Spark build info: 3.5.0, https://github.com/apache/spark, HEAD, ce5ddad990373636e94071e7cef2f31021add07b, 2023-09-09T01:53:20Z
24/03/19 13:12:46 INFO ShimLoader: Scala version: version 2.12.18
24/03/19 13:12:46 INFO ShimLoader: findURLClassLoader found a URLClassLoader org.apache.spark.util.MutableURLClassLoader@2258f7d6
24/03/19 13:12:46 INFO ShimLoader: Updating spark classloader org.apache.spark.util.MutableURLClassLoader@2258f7d6 with the URLs: jar:file:/opt/spark/jars/rapids-4-spark_2.12-24.02.0.jar!/spark3xx-common/, jar:file:/opt/spark/jars/rapids-4-spark_2.12-24.02.0.jar!/spark350/
24/03/19 13:12:46 INFO ShimLoader: Spark classLoader org.apache.spark.util.MutableURLClassLoader@2258f7d6 updated successfully
24/03/19 13:12:46 INFO ShimLoader: Updating spark classloader org.apache.spark.util.MutableURLClassLoader@2258f7d6 with the URLs: jar:file:/opt/spark/jars/rapids-4-spark_2.12-24.02.0.jar!/spark3xx-common/, jar:file:/opt/spark/jars/rapids-4-spark_2.12-24.02.0.jar!/spark350/
24/03/19 13:12:46 INFO ShimLoader: Spark classLoader org.apache.spark.util.MutableURLClassLoader@2258f7d6 updated successfully
24/03/19 13:12:50 INFO RapidsPluginUtils: RAPIDS Accelerator build: {date=2024-03-06T15:31:17Z, cudf_version=24.02.1, version=24.02.0, user=, branch=HEAD, url=https://github.com/NVIDIA/spark-rapids.git, revision=4b866f591e5079255ad88d39560211fa1053f191}
24/03/19 13:12:50 INFO RapidsPluginUtils: RAPIDS Accelerator JNI build: {date=2024-02-28T05:34:16Z, version=24.02.1, user=, branch=HEAD, url=https://github.com/NVIDIA/spark-rapids-jni.git, revision=298fe9ff0a5d704dd2b7f3f0dd2df50c5637c52e}
24/03/19 13:12:50 INFO RapidsPluginUtils: cudf build: {date=2024-02-28T05:34:16Z, version=24.02.1, user=, branch=HEAD, url=https://github.com/rapidsai/cudf.git, revision=dd34fdbe35e68ba56a2183f11ed822ddaa6c927b}
24/03/19 13:12:50 WARN RapidsPluginUtils: RAPIDS Accelerator 24.02.0 using cudf 24.02.1.
24/03/19 13:12:50 INFO RapidsPluginUtils: Estimated number of cores is 20
24/03/19 13:12:50 INFO RapidsExecutorPlugin: RAPIDS Accelerator build: {date=2024-03-06T15:31:17Z, cudf_version=24.02.1, version=24.02.0, user=, branch=HEAD, url=https://github.com/NVIDIA/spark-rapids.git, revision=4b866f591e5079255ad88d39560211fa1053f191}
24/03/19 13:12:50 INFO RapidsExecutorPlugin: cudf build: {date=2024-02-28T05:34:16Z, version=24.02.1, user=, branch=HEAD, url=https://github.com/rapidsai/cudf.git, revision=dd34fdbe35e68ba56a2183f11ed822ddaa6c927b}
24/03/19 13:12:50 INFO RapidsExecutorPlugin: Initializing memory from Executor Plugin
24/03/19 13:12:50 INFO GpuDeviceManager: Initializing RMM ASYNC pool size = 79970.125 MB on gpuId 0
24/03/19 13:12:50 INFO GpuDeviceManager: Using per-thread default stream
24/03/19 13:12:50 INFO ShimDiskBlockManager: Created local directory at /tmp/blockmgr-50e9c3ef-5e91-406f-bdd1-78d39f4fcc22
24/03/19 13:12:50 INFO RapidsBufferCatalog: Installing GPU memory handler for spill
24/03/19 13:12:50 INFO ExecutorPluginContainer: Initialized executor component for plugin com.nvidia.spark.SQLPlugin.
24/03/19 13:16:34 INFO CoarseGrainedExecutorBackend: Got assigned task 0
24/03/19 13:16:34 INFO Executor: Starting executor with user classpath (userClassPathFirst = false): ''
24/03/19 13:16:34 INFO Executor: Using REPL class URI: spark://172.16.4.252:37063/artifacts/b4c1fb58-aa6d-4f14-bcf1-aa5115736b26/classes/
24/03/19 13:16:34 INFO Executor: Created or updated repl class loader org.apache.spark.executor.ExecutorClassLoader@2c69145c for b4c1fb58-aa6d-4f14-bcf1-aa5115736b26.
24/03/19 13:16:34 INFO Executor: Running task 0.0 in stage 0.0 (TID 0)
24/03/19 13:16:34 INFO TorrentBroadcast: Started reading broadcast variable 0 with 1 pieces (estimated total size 4.0 MiB)
24/03/19 13:16:34 INFO TransportClientFactory: Successfully created connection to /172.16.4.252:43921 after 1 ms (0 ms spent in bootstraps)
24/03/19 13:16:34 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 41.7 KiB, free 89.8 GiB)
24/03/19 13:16:34 INFO TorrentBroadcast: Reading broadcast variable 0 took 65 ms
24/03/19 13:16:34 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 112.7 KiB, free 89.8 GiB)
24/03/19 13:16:35 INFO Executor: Finished task 0.0 in stage 0.0 (TID 0). 2548 bytes result sent to driver
24/03/19 13:16:36 INFO CoarseGrainedExecutorBackend: Got assigned task 1
24/03/19 13:16:36 INFO Executor: Running task 0.0 in stage 1.0 (TID 1)
24/03/19 13:16:36 INFO TorrentBroadcast: Started reading broadcast variable 2 with 1 pieces (estimated total size 4.0 MiB)
24/03/19 13:16:36 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 12.3 KiB, free 89.8 GiB)
24/03/19 13:16:36 INFO TorrentBroadcast: Reading broadcast variable 2 took 16 ms
24/03/19 13:16:36 INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 27.1 KiB, free 89.8 GiB)
24/03/19 13:16:36 INFO TransportClientFactory: Found inactive connection to /172.16.4.252:37063, creating a new one.
24/03/19 13:16:36 INFO TransportClientFactory: Successfully created connection to /172.16.4.252:37063 after 1 ms (0 ms spent in bootstraps)```

kkdatta Mar 20, 2024
Author

Hi @tgravescs ,
The solution worked. Thanks a lot

tgravescs Mar 20, 2024
Maintainer

thanks for confirming, we will investigate other solutions under the 9375 issue that Gera linked to so that hopefully in the future you don't have to build it yourself.

gerashegalov · 2024-03-20T18:44:32Z

gerashegalov
Mar 20, 2024
Collaborator

Related to #9375

6 replies

KevinLu233 Jun 18, 2024

in my env, some error like this:

24/06/18 06:39:41 INFO Executor: Running task 0.1 in stage 1.0 (TID 1)
24/06/18 06:39:41 INFO TorrentBroadcast: Started reading broadcast variable 1 with 1 pieces (estimated total size 4.0 MiB)
24/06/18 06:39:41 INFO TransportClientFactory: Successfully created connection to ip-10-12-4-15.tj.agoralab.co/10.12.4.15:32865 after 3 ms (0 ms spent in bootstraps)
24/06/18 06:39:41 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 10.5 KiB, free 4.1 GiB)
24/06/18 06:39:41 INFO TorrentBroadcast: Reading broadcast variable 1 took 108 ms
24/06/18 06:39:42 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 29.5 KiB, free 4.1 GiB)
24/06/18 06:39:42 INFO MapOutputTrackerWorker: Don't have map outputs for shuffle 0, fetching them
24/06/18 06:39:42 INFO MapOutputTrackerWorker: Doing the fetch; tracker endpoint = NettyRpcEndpointRef(spark://[email protected]:31385)
24/06/18 06:39:42 INFO MapOutputTrackerWorker: Got the map output locations
24/06/18 06:39:43 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
24/06/18 06:39:43 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 16 ms
24/06/18 06:39:43 INFO TransportClientFactory: Successfully created connection to ip-10-12-4-15.tj.agoralab.co/10.12.4.15:31385 after 2 ms (0 ms spent in bootstraps)
24/06/18 06:39:43 INFO CodeGenerator: Code generated in 329.683748 ms
24/06/18 06:39:59 ERROR Executor: Exception in task 0.1 in stage 1.0 (TID 1)
java.lang.NullPointerException
at com.nvidia.spark.rapids.RapidsBufferCatalog.addTable(RapidsBufferCatalog.scala:407)
at com.nvidia.spark.rapids.RapidsBufferCatalog.addTable(RapidsBufferCatalog.scala:381)
at com.nvidia.spark.rapids.RapidsBufferCatalog.$anonfun$addBatch$2(RapidsBufferCatalog.scala:359)
at com.nvidia.spark.rapids.Arm$.closeOnExcept(Arm.scala:98)
at com.nvidia.spark.rapids.RapidsBufferCatalog.addBatch(RapidsBufferCatalog.scala:358)
at com.nvidia.spark.rapids.RapidsBufferCatalog$.addBatch(RapidsBufferCatalog.scala:932)
at com.nvidia.spark.rapids.SpillableColumnarBatch$.$anonfun$addBatch$1(SpillableColumnarBatch.scala:319)
at com.nvidia.spark.rapids.Arm$.withResource(Arm.scala:30)
at com.nvidia.spark.rapids.SpillableColumnarBatch$.addBatch(SpillableColumnarBatch.scala:302)
at com.nvidia.spark.rapids.SpillableColumnarBatch$.apply(SpillableColumnarBatch.scala:255)
at com.nvidia.spark.rapids.ColumnarToRowIterator.$anonfun$loadNextBatch$1(GpuColumnarToRowExec.scala:261)
at com.nvidia.spark.rapids.ColumnarToRowIterator.$anonfun$loadNextBatch$1$adapted(GpuColumnarToRowExec.scala:260)
at scala.Option.foreach(Option.scala:407)
at com.nvidia.spark.rapids.ColumnarToRowIterator.loadNextBatch(GpuColumnarToRowExec.scala:260)
at com.nvidia.spark.rapids.ColumnarToRowIterator.hasNext(GpuColumnarToRowExec.scala:304)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
at org.apache.spark.sql.execution.SparkPlan.$anonfun$getByteArrayRdd$1(SparkPlan.scala:388)
at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:890)
at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:890)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:364)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:328)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:93)
at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:161)
at org.apache.spark.scheduler.Task.run(Task.scala:141)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:620)
at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64)
at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:623)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
24/06/18 06:39:59 INFO YarnCoarseGrainedExecutorBackend: Got assigned task 2
24/06/18 06:39:59 INFO Executor: Running task 0.2 in stage 1.0 (TID 2)
24/06/18 06:39:59 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
24/06/18 06:39:59 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
24/06/18 06:39:59 ERROR Executor: Exception in task 0.2 in stage 1.0 (TID 2)
java.lang.NullPointerException
at com.nvidia.spark.rapids.RapidsBufferCatalog.addTable(RapidsBufferCatalog.scala:407)
at com.nvidia.spark.rapids.RapidsBufferCatalog.addTable(RapidsBufferCatalog.scala:381)
at com.nvidia.spark.rapids.RapidsBufferCatalog.$anonfun$addBatch$2(RapidsBufferCatalog.scala:359)
at com.nvidia.spark.rapids.Arm$.closeOnExcept(Arm.scala:98)
at com.nvidia.spark.rapids.RapidsBufferCatalog.addBatch(RapidsBufferCatalog.scala:358)
at com.nvidia.spark.rapids.RapidsBufferCatalog$.addBatch(RapidsBufferCatalog.scala:932)
at com.nvidia.spark.rapids.SpillableColumnarBatch$.$anonfun$addBatch$1(SpillableColumnarBatch.scala:319)
at com.nvidia.spark.rapids.Arm$.withResource(Arm.scala:30)
at com.nvidia.spark.rapids.SpillableColumnarBatch$.addBatch(SpillableColumnarBatch.scala:302)
at com.nvidia.spark.rapids.SpillableColumnarBatch$.apply(SpillableColumnarBatch.scala:255)
at com.nvidia.spark.rapids.ColumnarToRowIterator.$anonfun$loadNextBatch$1(GpuColumnarToRowExec.scala:261)
at com.nvidia.spark.rapids.ColumnarToRowIterator.$anonfun$loadNextBatch$1$adapted(GpuColumnarToRowExec.scala:260)
at scala.Option.foreach(Option.scala:407)
at com.nvidia.spark.rapids.ColumnarToRowIterator.loadNextBatch(GpuColumnarToRowExec.scala:260)
at com.nvidia.spark.rapids.ColumnarToRowIterator.hasNext(GpuColumnarToRowExec.scala:304)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
at org.apache.spark.sql.execution.SparkPlan.$anonfun$getByteArrayRdd$1(SparkPlan.scala:388)
at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:890)
at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:890)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:364)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:328)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:93)
at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:161)
at org.apache.spark.scheduler.Task.run(Task.scala:141)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:620)
at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64)
at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:623)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
24/06/18 06:39:59 INFO YarnCoarseGrainedExecutorBackend: Got assigned task 3

jlowe Jun 18, 2024
Maintainer

@KevinLu233 can you file an issue with details such as Spark version and environment being used, RAPIDS Accelerator version, etc.? This is an NPE rather than a class cast/loading problem and appears to be a different issue.

jlowe Jun 18, 2024
Maintainer

can you file an issue with details

Ah I see there already is one at #11065

KevinLu233 Jun 19, 2024

Yeh , I can find NPE functions when I used Java17 to exec

24/06/19 09:03:34 WARN TaskSetManager: Lost task 8.0 in stage 0.0 (TID 1) (ip-10-12-4-15.tj.agoralab.co executor 1): java.lang.NullPointerException: Cannot invoke "com.nvidia.spark.rapids.PerfIOConf.s3PerfEnabled()" because the return value of "com.nvidia.spark.rapids.PerfIO$.s3ApiConf()" is null
at com.nvidia.spark.rapids.PerfIO$.readParquetFooterBuffer(PerfIO.scala:83)
at com.nvidia.spark.rapids.GpuParquetFileFilterHandler.readFooterBuffer(GpuParquetScan.scala:539)
at com.nvidia.spark.rapids.GpuParquetFileFilterHandler.$anonfun$getFooterBuffer$5(GpuParquetScan.scala:521)
at scala.Option.getOrElse(Option.scala:189)
at com.nvidia.spark.rapids.GpuParquetFileFilterHandler.getFooterBuffer(GpuParquetScan.scala:519)
at com.nvidia.spark.rapids.GpuParquetFileFilterHandler.readAndFilterFooter(GpuParquetScan.scala:607)
at com.nvidia.spark.rapids.GpuParquetFileFilterHandler.$anonfun$filterBlocks$1(GpuParquetScan.scala:690)
at com.nvidia.spark.rapids.Arm$.withResource(Arm.scala:30)
at com.nvidia.spark.rapids.GpuParquetFileFilterHandler.filterBlocks(GpuParquetScan.scala:676)
at com.nvidia.spark.rapids.GpuParquetMultiFilePartitionReaderFactory.com$nvidia$spark$rapids$GpuParquetMultiFilePartitionReaderFactory$$filterBlocksForCoalescingReader(GpuParquetScan.scala:1166)
at com.nvidia.spark.rapids.GpuParquetMultiFilePartitionReaderFactory.$anonfun$buildBaseColumnarReaderForCoalescing$4(GpuParquetScan.scala:1242)
at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198)
at scala.collection.TraversableLike.map(TraversableLike.scala:286)
at scala.collection.TraversableLike.map$(TraversableLike.scala:279)
at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:198)

KevinLu233 Jun 19, 2024

I have resolved this issue. Spark-connect is now running correctly. Thank you all for your help.

spark conf:
spark.rapids.sql.concurrentGpuTasks=128
spark.executor.resource.gpu.amount=1
spark.executor.memory=15G
spark.rapids.sql.shuffle.spillThreads=12
spark.rapids.memory.pinnedPool.size=8G
spark.executor.memoryOverhead=2G
spark.plugins=com.nvidia.spark.SQLPlugin
spark.executor.extraJavaOptions='-Dai.rapids.cudf.prefer-pinned=true'
spark.executor.resource.gpu.discoveryScript=/usr/hdp/3.1.4.0-315/spark3/getGpusResources.sh
spark.sql.files.maxPartitionBytes=512m
spark.plugins=com.nvidia.spark.SQLPlugin
spark.kryo.registrator=com.nvidia.spark.rapids.GpuKryoRegistrator
spark.rapids.sql.incompatibleDateFormats.enabled=true
spark.rapids.sql.castStringToTimestamp.enabled=true
spark.rapids.sql.hasExtendedYearValues=false

old and error steps:

mvn package -pl dist -am -Dbuildver=351 -DallowConventionalDistJar=true -DskipTests
./spark-3.5.1-bin-hadoop3/bin/spark-submit --class org.apache.spark.sql.connect.service.SparkConnectServer --driver-cores=5 --driver-memory=15g --executor-cores=3 --executor-memory=8g --jars=./rapids-4-spark_2.12-24.06.0-SNAPSHOT-cuda11.jar,./spark-connect_2.12-3.5.1.jar --master=yarn --name=${HOSTNAME} --queue=dumper --conf spark.yarn.appMasterEnv.JAVA_HOME=/usr/lib/jvm/java-1.17.0-openjdk-amd64/ --conf spark.executorEnv.JAVA_HOME=/usr/lib/jvm/java-1.17.0-openjdk-amd64

steps：

mvn package -pl dist -am -Dbuildver=351 -DallowConventionalDistJar=true -DskipTests
cp rapids-4-spark_2.12-24.06.0-SNAPSHOT-cuda11.jar spark-3.5.1-bin-hadoop3/jars/
./spark-3.5.1-bin-hadoop3/bin/spark-submit --class org.apache.spark.sql.connect.service.SparkConnectServer --driver-cores=5 --driver-memory=15g --executor-cores=3 --executor-memory=8g --jars=./spark-connect_2.12-3.5.1.jar --master=yarn --name=${HOSTNAME} --queue=dumper --conf spark.yarn.appMasterEnv.JAVA_HOME=/usr/lib/jvm/java-1.17.0-openjdk-amd64/ --conf spark.executorEnv.JAVA_HOME=/usr/lib/jvm/java-1.17.0-openjdk-amd64

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Custom spark 3.5.0 image built with spark connect giving serialization error with rapids libs #10611

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 2 comments 9 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

Custom spark 3.5.0 image built with spark connect giving serialization error with rapids libs #10611

kkdatta Mar 19, 2024

Replies: 2 comments · 9 replies

tgravescs Mar 19, 2024 Maintainer

kkdatta Mar 19, 2024 Author

kkdatta Mar 20, 2024 Author

tgravescs Mar 20, 2024 Maintainer

gerashegalov Mar 20, 2024 Collaborator

KevinLu233 Jun 18, 2024

jlowe Jun 18, 2024 Maintainer

jlowe Jun 18, 2024 Maintainer

KevinLu233 Jun 19, 2024

KevinLu233 Jun 19, 2024

kkdatta
Mar 19, 2024

Replies: 2 comments 9 replies

tgravescs
Mar 19, 2024
Maintainer

kkdatta Mar 19, 2024
Author

kkdatta Mar 20, 2024
Author

tgravescs Mar 20, 2024
Maintainer

gerashegalov
Mar 20, 2024
Collaborator

jlowe Jun 18, 2024
Maintainer

jlowe Jun 18, 2024
Maintainer