End to End xtable to Trino query with source Hudi and target Iceberg and Delta Lake with HMS and S3 #457

alberttwong · 2024-06-03T17:29:18Z

alberttwong
Jun 3, 2024

Using openjdk 11, spark 3.4, hive 2.3.10, hadoop 2.10.2

Here is the PR - xtable demo on s3 with documentation. #459 and https://github.com/alberttwong/incubator-xtable/tree/main/demo-s3

------- HISTORY

Look the xtable docker demo at https://xtable.apache.org/docs/demo/docker

list of versions to use with hudi 0.14.1 at https://github.com/apache/hudi/blob/release-0.14.1/pom.xml

added minio container images to provide an object store
changed HMS image to use the Starburst HMS image because Starburst has the S3 libraries already built in to the image.
built a custom spark 3.4 container image based on JDK 11 with hadoop 2.10.2 and hive 2.3.10 (can't use 2.3.1 due to hive 2.3.1 bug) installed. Available at https://hub.docker.com/r/atwong/openjdk-11-spark-3.4-hive-2.3.10-hadoop-2.10.2 if you dont' want to build it.
git clone hudi and compile mvn with JDK 8 so you can get the hudi-hive-sync jars (you can skip this through hudi-hive-sync-bundle on mvnrepository.com)
adding missing libraries to run run_sync_tool.sh. https://mvnrepository.com/artifact/com.amazonaws/aws-java-sdk-s3, https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-aws, https://mvnrepository.com/artifact/com.esotericsoftware/kryo-shaded/4.0.2, https://mvnrepository.com/artifact/org.apache.parquet/parquet-avro, https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-client
modifications to iceberg, hudi and delta Trino catalog configurations to support S3 bucket lookups
added core-site.xml to inject parameters to xtable and modified /etc/hadoop/core-site.xml to jnject parameters to hudi-hive-sync tool
Modified pyspark demo script to include S3 configs

What works so far

if I create the hudi 0.14.1 files in S3 with hms sync in the spark config, I can skip the registering hudi files to HMS
able to use xtable to sync from hudi to iceberg to delta
able to query hudi, iceberg and delta

What doesn't work. How far am I in the process

created hudi 0.14.1 files in S3
stuck on registering hudi files in S3 to metastore.

Running

./run_sync_tool.sh  \
--metastore-uris 'thrift://hive-metastore:9083' \
--partitioned-by city \
--base-path 's3a://warehouse/people' \
--database hudi_db \
--table people \
--sync-mode hms

pyspark

from pyspark.sql.types import *
from pyspark import SparkContext
from pyspark.sql import SparkSession
import os

# initialize the bucket
database_name = "hudi_db"
table_name = "people"
base_path = "s3a://warehouse/"

records = [
   (1, 'John', 25, 'NYC', '2023-09-28 00:00:00'),
   (2, 'Emily', 30, 'SFO', '2023-09-28 00:00:00'),
   (3, 'Michael', 35, 'ORD', '2023-09-28 00:00:00'),
   (4, 'Andrew', 40, 'NYC', '2023-10-28 00:00:00'),
   (5, 'Bob', 28, 'SEA', '2023-09-23 00:00:00'),
   (6, 'Charlie', 31, 'DFW', '2023-08-29 00:00:00')
]

schema = StructType([
   StructField("id", IntegerType(), True),
   StructField("name", StringType(), True),
   StructField("age", IntegerType(), True),
   StructField("city", StringType(), True),
   StructField("create_ts", StringType(), True)
])

spark = SparkSession.builder.getOrCreate()

df = spark.createDataFrame(records, schema)

hudi_options = {
#   'hoodie.datasource.hive_sync.enable': 'true',
#   'hoodie.datasource.hive_sync.mode': 'hms',
#   'hoodie.datasource.hive_sync.database': database_name,
#   'hoodie.datasource.hive_sync.table': table_name,
#   'hoodie.datasource.hive_sync.metastore.uris': 'thrift://hive-metastore:9083',
   'hoodie.table.name': table_name,
   'hoodie.datasource.write.partitionpath.field': 'city',
   'hoodie.datasource.write.hive_style_partitioning': 'true'

}

(
   df.write
   .format("hudi")
   .options(**hudi_options)
   .mode('overwrite')
   .save(f"{base_path}/{table_name}")
)

commands modified from the docker demo

pyspark --packages com.amazonaws:aws-java-sdk-s3:1.12.661,org.apache.hadoop:hadoop-aws:3.3.1,org.apache.hudi:hudi-spark3.4-bundle_2.12:0.14.1   --conf "spark.serializer=org.apache.spark.serializer.KryoSerializer"   --conf "spark.sql.catalog.spark_catalog=org.apache.spark.sql.hudi.catalog.HoodieCatalog"   --conf "spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension" 

export AWS_SECRET_ACCESS_KEY=password
export AWS_ACCESS_KEY_ID=admin
export ENDPOINT=http://minio:9000
export AWS_REGION=us-east-1
java -jar xtable-utilities-0.1.0-SNAPSHOT-bundled.jar --datasetConfig xtable_hudi.yaml -p core-site.xml


cd /opt/hudi/hudi-sync/hudi-hive-sync

./run_sync_tool.sh  \
--metastore-uris 'thrift://hive-metastore:9083' \
--partitioned-by city \
--base-path 's3a://warehouse/people' \
--database hudi_db \
--table people \
--sync-mode hms 


spark-sql --packages com.amazonaws:aws-java-sdk-s3:1.12.661,org.apache.hadoop:hadoop-aws:3.3.1,org.apache.spark:spark-hadoop-cloud_2.12:3.2.0,io.delta:delta-core_2.12:2.4.0 \
--conf "spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension" \
--conf "spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog" \
--conf "spark.sql.catalogImplementation=hive"

CREATE SCHEMA delta_db LOCATION 's3a://warehouse/';

CREATE TABLE delta_db.people USING DELTA LOCATION 's3a://warehouse/people';

spark-sql --packages com.amazonaws:aws-java-sdk-s3:1.12.661,org.apache.hadoop:hadoop-aws:3.3.1,org.apache.iceberg:iceberg-spark-runtime-3.4_2.12:1.5.2 \
--conf "spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions" \
--conf "spark.sql.catalog.spark_catalog=org.apache.iceberg.spark.SparkSessionCatalog" \
--conf "spark.sql.catalog.spark_catalog.type=hive" \
--conf "spark.sql.catalog.hive_prod=org.apache.iceberg.spark.SparkCatalog" \
--conf "spark.sql.catalog.hive_prod.type=hive"

CREATE SCHEMA iceberg_db;

CALL hive_prod.system.register_table(
   table => 'hive_prod.iceberg_db.people',
   metadata_file => 's3a://warehouse/people/metadata/v2.metadata.json'
);

select * from hudi.hudi_db.people;
select * from delta.delta_db.people;
select * from iceberg.iceberg_db.people;

for Hadoop 2.10.2. Here is the modified pyspark to create hudi files with AWS hadoop 2.10.2

pyspark --packages com.amazonaws:aws-java-sdk-s3:1.11.271,org.apache.hadoop:hadoop-client:2.10.2,org.apache.hadoop:hadoop-aws:2.10.2,org.apache.hudi:hudi-spark3.4-bundle_2.12:0.14.1   --conf "spark.serializer=org.apache.spark.serializer.KryoSerializer"   --conf "spark.sql.catalog.spark_catalog=org.apache.spark.sql.hudi.catalog.HoodieCatalog"   --conf "spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension" --conf "spark.hadoop.fs.s3a.aws.credentials.provider=org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider"

export AWS_SECRET_ACCESS_KEY=password
export AWS_ACCESS_KEY_ID=admin
export ENDPOINT=http://minio:9000
export AWS_REGION=us-east-1
cd /opt/xtable/jars/; java -jar xtable-utilities-0.1.0-SNAPSHOT-bundled.jar --datasetConfig xtable_hudi.yaml -p core-site.xml


pyspark --packages org.apache.parquet:parquet-avro:1.10.1,com.esotericsoftware:kryo-shaded:4.0.2

pyspark --packages org.apache.hudi:hudi-hive-sync:0.14.1,com.amazonaws:aws-java-sdk-s3:1.11.271,org.apache.hadoop:hadoop-client:2.10.2,org.apache.hadoop:hadoop-aws:2.10.2,org.apache.parquet:parquet-avro:1.10.1,com.esotericsoftware:kryo-shaded:4.0.2

pyspark --packages org.apache.hudi:hudi-hive-sync-bundle:0.14.1,com.amazonaws:aws-java-sdk-s3:1.11.271,org.apache.hadoop:hadoop-client:2.10.2,org.apache.hadoop:hadoop-aws:2.10.2

rm -Rf /opt/hudi/packaging/;  cp -R /opt/xtable/jars/packaging/ /opt/hudi/


vi /opt/hudi/hudi-sync/hudi-hive-sync/run_sync_tool.sh
~/.ivy2/jars/*:

vi /hadoop/etc/hadoop/core-site.xml 

cd /opt/hudi/hudi-sync/hudi-hive-sync
./run_sync_tool.sh  \
--metastore-uris 'thrift://hive-metastore:9083' \
--partitioned-by city \
--base-path 's3a://warehouse/people' \
--database hudi_db \
--table people \
--sync-mode hms 

vi /etc/trino/catalog

spark-sql --packages com.amazonaws:aws-java-sdk-s3:1.11.271,org.apache.hadoop:hadoop-aws:2.10.2,io.delta:delta-core_2.12:2.4.0 \
--conf "spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension" \
--conf "spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog" \
--conf "spark.sql.catalogImplementation=hive" \
--conf "spark.hadoop.fs.s3a.aws.credentials.provider=org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider"

CREATE SCHEMA delta_db LOCATION 's3a://warehouse/';

CREATE TABLE delta_db.people USING DELTA LOCATION 's3a://warehouse/people';

spark-sql --packages com.amazonaws:aws-java-sdk-s3:1.11.271,org.apache.hadoop:hadoop-aws:2.10.2,org.apache.iceberg:iceberg-spark-runtime-3.4_2.12:1.5.2 \
--conf "spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions" \
--conf "spark.sql.catalog.spark_catalog=org.apache.iceberg.spark.SparkSessionCatalog" \
--conf "spark.sql.catalog.spark_catalog.type=hive" \
--conf "spark.sql.catalog.hive_prod=org.apache.iceberg.spark.SparkCatalog" \
--conf "spark.sql.catalog.hive_prod.type=hive" \
--conf "spark.hadoop.fs.s3a.aws.credentials.provider=org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider"

CREATE SCHEMA iceberg_db;

CALL hive_prod.system.register_table(
   table => 'hive_prod.iceberg_db.people',
   metadata_file => 's3a://warehouse/people/metadata/v2.metadata.json'
);

select * from hudi.hudi_db.people;
select * from delta.delta_db.people;
select * from iceberg.iceberg_db.people;

alberttwong · 2024-06-03T17:44:07Z

alberttwong
Jun 3, 2024
Author

even with these modifications I'm running into

root@spark:/opt/hudi/hudi-sync/hudi-hive-sync# ./run_sync_tool.sh  --metastore-uris 'thrift://hive-metastore:9083' --partitioned-by city --base-path 's3a://warehouse/people' --database hudi_db --table people --sync-mode hms 
setting hadoop conf dir
Running Command : java -cp /hive/lib/hive-metastore-3.1.3.jar::/hive/lib/hive-service-3.1.3.jar::/hive/lib/hive-exec-3.1.3.jar::/hive/lib/hive-jdbc-3.1.3.jar:/hive/lib/hive-jdbc-handler-3.1.3.jar::/hive/lib/jackson-annotations-2.12.0.jar:/hive/lib/jackson-core-2.12.0.jar:/hive/lib/jackson-core-asl-1.9.13.jar:/hive/lib/jackson-databind-2.12.0.jar:/hive/lib/jackson-dataformat-smile-2.12.0.jar:/hive/lib/jackson-mapper-asl-1.9.13.jar:/hive/lib/jackson-module-scala_2.11-2.12.0.jar::/hadoop/share/hadoop/common/*:/hadoop/share/hadoop/mapreduce/*:/hadoop/share/hadoop/hdfs/*:/hadoop/share/hadoop/common/lib/*:/hadoop/share/hadoop/hdfs/lib/*:/root/.ivy2/jars/*:/hadoop/etc/hadoop:/opt/hudi/hudi-sync/hudi-hive-sync/../../packaging/hudi-hive-sync-bundle/target/hudi-hive-sync-bundle-1.0.0-SNAPSHOT.jar org.apache.hudi.hive.HiveSyncTool --metastore-uris thrift://hive-metastore:9083 --partitioned-by city --base-path s3a://warehouse/people --database hudi_db --table people --sync-mode hms
2024-06-03 17:15:25,270 INFO  [main] conf.HiveConf (HiveConf.java:findConfigFile(187)) - Found configuration file null
2024-06-03 17:15:25,444 WARN  [main] util.NativeCodeLoader (NativeCodeLoader.java:<clinit>(60)) - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2024-06-03 17:15:25,550 INFO  [main] impl.MetricsConfig (MetricsConfig.java:loadFirst(120)) - Loaded properties from hadoop-metrics2.properties
2024-06-03 17:15:25,581 INFO  [main] impl.MetricsSystemImpl (MetricsSystemImpl.java:startTimer(378)) - Scheduled Metric snapshot period at 10 second(s).
2024-06-03 17:15:25,581 INFO  [main] impl.MetricsSystemImpl (MetricsSystemImpl.java:start(191)) - s3a-file-system metrics system started
2024-06-03 17:15:26,025 INFO  [main] table.HoodieTableMetaClient (HoodieTableMetaClient.java:<init>(148)) - Loading HoodieTableMetaClient from s3a://warehouse/people
2024-06-03 17:15:26,120 INFO  [main] table.HoodieTableConfig (HoodieTableConfig.java:<init>(309)) - Loading table properties from s3a://warehouse/people/.hoodie/hoodie.properties
2024-06-03 17:15:26,140 INFO  [main] table.HoodieTableMetaClient (HoodieTableMetaClient.java:<init>(169)) - Finished Loading Table of type COPY_ON_WRITE(version=1) from s3a://warehouse/people
2024-06-03 17:15:26,140 INFO  [main] table.HoodieTableMetaClient (HoodieTableMetaClient.java:<init>(171)) - Loading Active commit timeline for s3a://warehouse/people
2024-06-03 17:15:26,159 INFO  [main] timeline.HoodieActiveTimeline (HoodieActiveTimeline.java:<init>(177)) - Loaded instants upto : Option{val=[20240603170053432__commit__COMPLETED]}
2024-06-03 17:15:26,229 ERROR [main] utils.MetaStoreUtils (MetaStoreUtils.java:logAndThrowMetaException(166)) - Got exception: java.net.URISyntaxException Illegal character in hostname at index 35: thrift://demo-hive-metastore-1.demo_default:9083
java.net.URISyntaxException: Illegal character in hostname at index 35: thrift://demo-hive-metastore-1.demo_default:9083

1 reply

alberttwong Jun 4, 2024
Author

thrift://demo-hive-metastore-1.demo_default:9083 error is due to docker compose creating a default network with underscores in it. You need to add a defined custom docker compose network

networks:
  default:
     name: lakehouse

alberttwong · 2024-06-03T18:12:08Z

alberttwong
Jun 3, 2024
Author

new error

Running Command : java -cp /hive/lib/hive-metastore-3.1.3.jar::/hive/lib/hive-service-3.1.3.jar::/hive/lib/hive-exec-3.1.3.jar::/hive/lib/hive-jdbc-3.1.3.jar:/hive/lib/hive-jdbc-handler-3.1.3.jar::/hive/lib/jackson-annotations-2.12.0.jar:/hive/lib/jackson-core-2.12.0.jar:/hive/lib/jackson-core-asl-1.9.13.jar:/hive/lib/jackson-databind-2.12.0.jar:/hive/lib/jackson-dataformat-smile-2.12.0.jar:/hive/lib/jackson-mapper-asl-1.9.13.jar:/hive/lib/jackson-module-scala_2.11-2.12.0.jar::/hadoop/share/hadoop/common/*:/hadoop/share/hadoop/mapreduce/*:/hadoop/share/hadoop/hdfs/*:/hadoop/share/hadoop/common/lib/*:/hadoop/share/hadoop/hdfs/lib/*:/root/.ivy2/jars/*:/hadoop/etc/hadoop:/opt/hudi/hudi-sync/hudi-hive-sync/../../packaging/hudi-hive-sync-bundle/target/hudi-hive-sync-bundle-1.0.0-SNAPSHOT.jar org.apache.hudi.hive.HiveSyncTool --metastore-uris thrift://hive-metastore:9083 --partitioned-by city --base-path s3a://warehouse/people --database hudi_db --table people --sync-mode hms
2024-06-03 18:09:34,124 INFO  [main] conf.HiveConf (HiveConf.java:findConfigFile(187)) - Found configuration file null
2024-06-03 18:09:34,314 WARN  [main] util.NativeCodeLoader (NativeCodeLoader.java:<clinit>(60)) - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2024-06-03 18:09:34,426 INFO  [main] impl.MetricsConfig (MetricsConfig.java:loadFirst(120)) - Loaded properties from hadoop-metrics2.properties
2024-06-03 18:09:34,471 INFO  [main] impl.MetricsSystemImpl (MetricsSystemImpl.java:startTimer(378)) - Scheduled Metric snapshot period at 10 second(s).
2024-06-03 18:09:34,471 INFO  [main] impl.MetricsSystemImpl (MetricsSystemImpl.java:start(191)) - s3a-file-system metrics system started
2024-06-03 18:09:34,850 INFO  [main] table.HoodieTableMetaClient (HoodieTableMetaClient.java:<init>(148)) - Loading HoodieTableMetaClient from s3a://warehouse/people
2024-06-03 18:09:34,941 INFO  [main] table.HoodieTableConfig (HoodieTableConfig.java:<init>(309)) - Loading table properties from s3a://warehouse/people/.hoodie/hoodie.properties
2024-06-03 18:09:34,959 INFO  [main] table.HoodieTableMetaClient (HoodieTableMetaClient.java:<init>(169)) - Finished Loading Table of type COPY_ON_WRITE(version=1) from s3a://warehouse/people
2024-06-03 18:09:34,959 INFO  [main] table.HoodieTableMetaClient (HoodieTableMetaClient.java:<init>(171)) - Loading Active commit timeline for s3a://warehouse/people
2024-06-03 18:09:34,985 INFO  [main] timeline.HoodieActiveTimeline (HoodieActiveTimeline.java:<init>(177)) - Loaded instants upto : Option{val=[20240603180921712__commit__COMPLETED]}
2024-06-03 18:09:35,037 INFO  [main] metastore.HiveMetaStoreClient (HiveMetaStoreClient.java:open(441)) - Trying to connect to metastore with URI thrift://demo-hive-metastore-1.lakehouse:9083
2024-06-03 18:09:35,049 INFO  [main] metastore.HiveMetaStoreClient (HiveMetaStoreClient.java:open(517)) - Opened a connection to metastore, current connections: 1
2024-06-03 18:09:35,065 INFO  [main] metastore.HiveMetaStoreClient (HiveMetaStoreClient.java:open(570)) - Connected to metastore.
2024-06-03 18:09:35,065 INFO  [main] metastore.RetryingMetaStoreClient (RetryingMetaStoreClient.java:<init>(97)) - RetryingMetaStoreClient proxy=class org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient ugi=root (auth:SIMPLE) retries=1 delay=1 lifetime=0
2024-06-03 18:09:35,132 INFO  [main] hive.HiveSyncTool (HiveSyncTool.java:syncHoodieTable(161)) - Syncing target hoodie table with hive table(hudi_db.people). Hive metastore URL from HiveConf:thrift://hive-metastore:9083). Hive metastore URL from HiveSyncConfig:thrift://hive-metastore:9083, basePath :s3a://warehouse/people
2024-06-03 18:09:35,132 INFO  [main] hive.HiveSyncTool (HiveSyncTool.java:syncHoodieTable(223)) - Trying to sync hoodie table people with base path s3a://warehouse/people of type COPY_ON_WRITE
2024-06-03 18:09:35,718 INFO  [main] metastore.HiveMetaStoreClient (HiveMetaStoreClient.java:close(600)) - Closed a connection to metastore, current connections: 0
Exception in thread "main" org.apache.hudi.exception.HoodieException: Got runtime exception when hive syncing people
        at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:170)
        at org.apache.hudi.hive.HiveSyncTool.main(HiveSyncTool.java:480)
Caused by: org.apache.hudi.sync.common.HoodieSyncException: Failed to read schema from storage.
        at org.apache.hudi.sync.common.HoodieSyncClient.getStorageSchema(HoodieSyncClient.java:113)
        at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:246)
        at org.apache.hudi.hive.HiveSyncTool.doSync(HiveSyncTool.java:179)
        at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:167)
        ... 1 more
Caused by: java.lang.IllegalArgumentException: Completion time should not be empty
        at org.apache.hudi.common.util.ValidationUtils.checkArgument(ValidationUtils.java:42)
        at org.apache.hudi.common.table.timeline.HoodieInstant.getCompleteFileName(HoodieInstant.java:262)
        at org.apache.hudi.common.table.timeline.HoodieInstant.getFileName(HoodieInstant.java:296)
        at org.apache.hudi.common.table.timeline.HoodieActiveTimeline.getInstantFileName(HoodieActiveTimeline.java:344)
        at org.apache.hudi.common.table.timeline.HoodieActiveTimeline.getInstantDetails(HoodieActiveTimeline.java:351)
        at org.apache.hudi.common.table.timeline.HoodieActiveTimeline.lambda$getCommitMetadataStream$4(HoodieActiveTimeline.java:392)
        at java.base/java.util.stream.ReferencePipeline$3$1.accept(Unknown Source)
        at java.base/java.util.stream.SortedOps$SizedRefSortingSink.end(Unknown Source)
        at java.base/java.util.stream.AbstractPipeline.copyIntoWithCancel(Unknown Source)
        at java.base/java.util.stream.AbstractPipeline.copyInto(Unknown Source)
        at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source)
        at java.base/java.util.stream.FindOps$FindOp.evaluateSequential(Unknown Source)
        at java.base/java.util.stream.AbstractPipeline.evaluate(Unknown Source)
        at java.base/java.util.stream.ReferencePipeline.findFirst(Unknown Source)
        at org.apache.hudi.common.table.timeline.HoodieActiveTimeline.getLastCommitMetadataWithValidSchema(HoodieActiveTimeline.java:364)
        at org.apache.hudi.common.table.TableSchemaResolver.getLatestCommitMetadataWithValidSchema(TableSchemaResolver.java:419)
        at org.apache.hudi.common.table.TableSchemaResolver.getTableSchemaFromLatestCommitMetadata(TableSchemaResolver.java:209)
        at org.apache.hudi.common.table.TableSchemaResolver.getTableAvroSchemaInternal(TableSchemaResolver.java:183)
        at org.apache.hudi.common.table.TableSchemaResolver.getTableAvroSchema(TableSchemaResolver.java:136)
        at org.apache.hudi.common.table.ParquetTableSchemaResolver.getTableParquetSchema(ParquetTableSchemaResolver.java:63)
        at org.apache.hudi.sync.common.HoodieSyncClient.getStorageSchema(HoodieSyncClient.java:111)
        ... 4 more
2024-06-03 18:09:35,726 INFO  [shutdown-hook-0] impl.MetricsSystemImpl (MetricsSystemImpl.java:stop(210)) - Stopping s3a-file-system metrics system...
2024-06-03 18:09:35,727 INFO  [shutdown-hook-0] impl.MetricsSystemImpl (MetricsSystemImpl.java:stop(216)) - s3a-file-system metrics system stopped.
2024-06-03 18:09:35,728 INFO  [shutdown-hook-0] impl.MetricsSystemImpl (MetricsSystemImpl.java:shutdown(611)) - s3a-file-system metrics system shutdown complete.

3 replies

alberttwong Jun 3, 2024
Author

apache/hudi#10424

alberttwong Jun 3, 2024
Author

switching to hudi 0.14.1 branch and recompile and then try again with new sync jars

alberttwong Jun 4, 2024
Author

using hudi 0.14.1 compiled jars works

alberttwong · 2024-06-04T04:59:12Z

alberttwong
Jun 4, 2024
Author

new issue. I couldn't get any combination of hive or hadoop to work so I was browsing the hudi pom.xml and it says to use hive 2.3.1 and hadoop 2.10.2. As a result, I rebuilt my spark image. With the new image....

first, load up some libraries into ~/.ivy2/jars. To make it easy on myself, I'm using pyspark packages to download and then just run pyspark and then exit().

pyspark --packages org.apache.hadoop:hadoop-aws:2.10.2,com.amazonaws:aws-java-sdk-s3:1.11.271,org.apache.hadoop:hadoop-client:2.10.2

modify run_sync_tool.sh to use the jars that I loaded

java -cp ~/.ivy2/jars/*:$HUDI_HIVE_UBER_JAR:${HADOOP_HIVE_JARS}:${HADOOP_CONF_DIR} org.apache.hudi.hive.HiveSyncTool "$@"

running

root@spark:/opt/hudi/hudi-sync/hudi-hive-sync# ./run_sync_tool.sh  --metastore-uris 'thrift://hive-metastore:9083' --partitioned-by city --base-path 's3a://warehouse/people' --database hudi_db --table people --sync-mode hms 
setting hadoop conf dir
Running Command : java -cp /hive/lib/hive-metastore-2.3.1.jar::/hive/lib/hive-service-2.3.1.jar::/hive/lib/hive-exec-2.3.1.jar::/hive/lib/hive-jdbc-2.3.1.jar:/hive/lib/hive-jdbc-handler-2.3.1.jar::/hive/lib/jackson-annotations-2.6.0.jar:/hive/lib/jackson-core-2.6.5.jar:/hive/lib/jackson-databind-2.6.5.jar:/hive/lib/jackson-dataformat-smile-2.4.6.jar:/hive/lib/jackson-datatype-guava-2.4.6.jar:/hive/lib/jackson-datatype-joda-2.4.6.jar:/hive/lib/jackson-jaxrs-1.9.13.jar:/hive/lib/jackson-jaxrs-base-2.4.6.jar:/hive/lib/jackson-jaxrs-json-provider-2.4.6.jar:/hive/lib/jackson-jaxrs-smile-provider-2.4.6.jar:/hive/lib/jackson-module-jaxb-annotations-2.4.6.jar:/hive/lib/jackson-xc-1.9.13.jar::/hadoop/share/hadoop/common/*:/hadoop/share/hadoop/mapreduce/*:/hadoop/share/hadoop/hdfs/*:/hadoop/share/hadoop/common/lib/*:/hadoop/share/hadoop/hdfs/lib/*:/hadoop/etc/hadoop:/opt/hudi/hudi-sync/hudi-hive-sync/../../packaging/hudi-hive-sync-bundle/target/hudi-hive-sync-bundle-0.14.1.jar org.apache.hudi.hive.HiveSyncTool --metastore-uris thrift://hive-metastore:9083 --partitioned-by city --base-path s3a://warehouse/people --database hudi_db --table people --sync-mode hms
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/root/.ivy2/jars/org.slf4j_slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/hadoop-2.10.2/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Reload4jLoggerFactory]
2024-06-04 04:58:06,464 INFO  [main] conf.HiveConf (HiveConf.java:findConfigFile(181)) - Found configuration file null
2024-06-04 04:58:06,666 INFO  [main] table.HoodieTableMetaClient (HoodieTableMetaClient.java:<init>(133)) - Loading HoodieTableMetaClient from s3a://warehouse/people
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.hadoop.security.authentication.util.KerberosUtil (file:/root/.ivy2/jars/org.apache.hadoop_hadoop-auth-2.10.2.jar) to method sun.security.krb5.Config.getInstance()
WARNING: Please consider reporting this to the maintainers of org.apache.hadoop.security.authentication.util.KerberosUtil
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
2024-06-04 04:58:06,695 WARN  [main] util.NativeCodeLoader (NativeCodeLoader.java:<clinit>(62)) - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2024-06-04 04:58:08,458 INFO  [main] Configuration.deprecation (Configuration.java:logDeprecation(1294)) - fs.s3a.server-side-encryption-key is deprecated. Instead, use fs.s3a.server-side-encryption.key
Exception in thread "main" org.apache.hudi.hive.HoodieHiveSyncException: Got runtime exception when hive syncing
        at org.apache.hudi.hive.HiveSyncTool.initSyncClient(HiveSyncTool.java:131)
        at org.apache.hudi.hive.HiveSyncTool.<init>(HiveSyncTool.java:120)
        at org.apache.hudi.hive.HiveSyncTool.main(HiveSyncTool.java:491)
Caused by: org.apache.hudi.exception.HoodieIOException: Could not check if s3a://warehouse/people is a valid table
        at org.apache.hudi.exception.TableNotFoundException.checkTableValidity(TableNotFoundException.java:59)
        at org.apache.hudi.common.table.HoodieTableMetaClient.<init>(HoodieTableMetaClient.java:140)
        at org.apache.hudi.common.table.HoodieTableMetaClient.newMetaClient(HoodieTableMetaClient.java:692)
        at org.apache.hudi.common.table.HoodieTableMetaClient.access$000(HoodieTableMetaClient.java:85)
        at org.apache.hudi.common.table.HoodieTableMetaClient$Builder.build(HoodieTableMetaClient.java:774)
        at org.apache.hudi.sync.common.HoodieSyncClient.<init>(HoodieSyncClient.java:67)
        at org.apache.hudi.hive.HoodieHiveSyncClient.<init>(HoodieHiveSyncClient.java:82)
        at org.apache.hudi.hive.HiveSyncTool.initSyncClient(HiveSyncTool.java:126)
        ... 2 more
Caused by: java.nio.file.AccessDeniedException: s3a://warehouse/people/.hoodie: getFileStatus on s3a://warehouse/people/.hoodie: com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: 4JTD71XMGK1VCPNG; S3 Extended Request ID: Mr9xlNlq6PjN2KV/++VSz9HJ0Ddm2wrxu9hLQtgi7Ilr7UqkIPKnvu6AQjgmAc6JFp6/Cd4TE/DTcxeTXyfCPQ==), S3 Extended Request ID: Mr9xlNlq6PjN2KV/++VSz9HJ0Ddm2wrxu9hLQtgi7Ilr7UqkIPKnvu6AQjgmAc6JFp6/Cd4TE/DTcxeTXyfCPQ==

tried to export a bunch of env but it didn't work.

export AWS_SECRET_ACCESS_KEY=password
export AWS_ACCESS_KEY_ID=admin
export ENDPOINT=http://minio:9000
export AWS_REGION=us-east-1

5 replies

alberttwong Jun 4, 2024
Author

@sagarlakshmipathy do you have a working example of run_sync_tool.sh with Hudi in S3?

alberttwong Jun 4, 2024
Author

modify vi /hadoop/etc/hadoop/core-site.xml

with

    <property>
        <name>fs.defaultFS</name>
        <value>s3a://minio:9000</value>
    </property>


    <!-- Minio properties -->
    <property>
        <name>fs.s3a.connection.ssl.enabled</name>
        <value>false</value>
    </property>

    <property>
        <name>fs.s3a.endpoint</name>
        <value>http://minio:9000</value>
    </property>

    <property>
        <name>fs.s3a.access.key</name>
        <value>admin</value>
    </property>

    <property>
        <name>fs.s3a.secret.key</name>
        <value>password</value>
    </property>

    <property>
        <name>fs.s3a.path.style.access</name>
        <value>true</value>
    </property>

    <property>
        <name>fs.s3a.impl</name>
        <value>org.apache.hadoop.fs.s3a.S3AFileSystem</value>
    </property>

alberttwong Jun 4, 2024
Author

@dipankarmazumdar do you have a working example of run_sync_tool.sh with Hudi in S3?

sagarlakshmipathy Jun 4, 2024

@alberttwong I was only able to get run_sync_tool to work with the version that exists in newer EMR clusters.

alberttwong Jun 4, 2024
Author

@sagarlakshmipathy well... now we got another method. I got it working end to end on my local docker compose.

alberttwong · 2024-06-04T05:22:59Z

alberttwong
Jun 4, 2024
Author

Which reminds me... I wish I can just load all the jars from https://mvnrepository.com/artifact/org.apache.hudi/hudi-hive-sync/0.14.1 and then just run the sync tool but this doesn't work.

1 reply

alberttwong Jun 4, 2024
Author

You still need a hadoop and hive installation but don't need to compile hudi if you download the hudi-hive-sync-bundle from mvnrepository

pyspark --packages org.apache.hudi:hudi-hive-sync-bundle:0.14.1,com.amazonaws:aws-java-sdk-s3:1.11.271,org.apache.hadoop:hadoop-client:2.10.2,org.apache.hadoop:hadoop-aws:2.10.2
pyspark --packages org.apache.parquet:parquet-avro:1.10.1,com.esotericsoftware:kryo-shaded:4.0.2

alberttwong · 2024-06-04T05:41:50Z

alberttwong
Jun 4, 2024
Author

new error

root@spark:/opt/hudi/hudi-sync/hudi-hive-sync# ./run_sync_tool.sh  --metastore-uris 'thrift://hive-metastore:9083' --partitioned-by city --base-path 's3a://warehouse/people' --database hudi_db --table people --sync-mode hms 
setting hadoop conf dir
Running Command : java -cp /hive/lib/hive-metastore-2.3.1.jar::/hive/lib/hive-service-2.3.1.jar::/hive/lib/hive-exec-2.3.1.jar::/hive/lib/hive-jdbc-2.3.1.jar:/hive/lib/hive-jdbc-handler-2.3.1.jar::/hive/lib/jackson-annotations-2.6.0.jar:/hive/lib/jackson-core-2.6.5.jar:/hive/lib/jackson-databind-2.6.5.jar:/hive/lib/jackson-dataformat-smile-2.4.6.jar:/hive/lib/jackson-datatype-guava-2.4.6.jar:/hive/lib/jackson-datatype-joda-2.4.6.jar:/hive/lib/jackson-jaxrs-1.9.13.jar:/hive/lib/jackson-jaxrs-base-2.4.6.jar:/hive/lib/jackson-jaxrs-json-provider-2.4.6.jar:/hive/lib/jackson-jaxrs-smile-provider-2.4.6.jar:/hive/lib/jackson-module-jaxb-annotations-2.4.6.jar:/hive/lib/jackson-xc-1.9.13.jar::/hadoop/share/hadoop/common/*:/hadoop/share/hadoop/mapreduce/*:/hadoop/share/hadoop/hdfs/*:/hadoop/share/hadoop/common/lib/*:/hadoop/share/hadoop/hdfs/lib/*:/hadoop/etc/hadoop:/opt/hudi/hudi-sync/hudi-hive-sync/../../packaging/hudi-hive-sync-bundle/target/hudi-hive-sync-bundle-0.14.1.jar org.apache.hudi.hive.HiveSyncTool --metastore-uris thrift://hive-metastore:9083 --partitioned-by city --base-path s3a://warehouse/people --database hudi_db --table people --sync-mode hms
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/root/.ivy2/jars/org.slf4j_slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/hadoop-2.10.2/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Reload4jLoggerFactory]
2024-06-04 05:37:20,273 INFO  [main] conf.HiveConf (HiveConf.java:findConfigFile(181)) - Found configuration file null
2024-06-04 05:37:20,481 INFO  [main] table.HoodieTableMetaClient (HoodieTableMetaClient.java:<init>(133)) - Loading HoodieTableMetaClient from s3a://warehouse/people
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.hadoop.security.authentication.util.KerberosUtil (file:/root/.ivy2/jars/org.apache.hadoop_hadoop-auth-2.10.2.jar) to method sun.security.krb5.Config.getInstance()
WARNING: Please consider reporting this to the maintainers of org.apache.hadoop.security.authentication.util.KerberosUtil
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
2024-06-04 05:37:20,519 WARN  [main] util.NativeCodeLoader (NativeCodeLoader.java:<clinit>(62)) - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2024-06-04 05:37:21,050 INFO  [main] Configuration.deprecation (Configuration.java:logDeprecation(1294)) - fs.s3a.server-side-encryption-key is deprecated. Instead, use fs.s3a.server-side-encryption.key
2024-06-04 05:37:21,083 INFO  [main] table.HoodieTableConfig (HoodieTableConfig.java:<init>(276)) - Loading table properties from s3a://warehouse/people/.hoodie/hoodie.properties
2024-06-04 05:37:21,094 INFO  [main] table.HoodieTableMetaClient (HoodieTableMetaClient.java:<init>(152)) - Finished Loading Table of type COPY_ON_WRITE(version=1, baseFileFormat=PARQUET) from s3a://warehouse/people
2024-06-04 05:37:21,094 INFO  [main] table.HoodieTableMetaClient (HoodieTableMetaClient.java:<init>(155)) - Loading Active commit timeline for s3a://warehouse/people
2024-06-04 05:37:21,117 INFO  [main] timeline.HoodieActiveTimeline (HoodieActiveTimeline.java:<init>(172)) - Loaded instants upto : Option{val=[20240604051151813__commit__COMPLETED__20240604051158033]}
2024-06-04 05:37:21,289 ERROR [main] hive.log (MetaStoreUtils.java:logAndThrowMetaException(1380)) - Got exception: java.lang.ClassCastException class [Ljava.lang.Object; cannot be cast to class [Ljava.net.URI; ([Ljava.lang.Object; and [Ljava.net.URI; are in module java.base of loader 'bootstrap')
java.lang.ClassCastException: class [Ljava.lang.Object; cannot be cast to class [Ljava.net.URI; ([Ljava.lang.Object; and [Ljava.net.URI; are in module java.base of loader 'bootstrap')

related? https://issues.apache.org/jira/browse/SPARK-29245

1 reply

alberttwong Jun 4, 2024
Author

fixed by moving to Hive 2.3.10

alberttwong · 2024-06-04T05:58:02Z

alberttwong
Jun 4, 2024
Author

root@spark:/opt/hudi/hudi-sync/hudi-hive-sync# ./run_sync_tool.sh  \
> --metastore-uris 'thrift://hive-metastore:9083' \
> --partitioned-by city \
> --base-path 's3a://warehouse/people' \
> --database hudi_db \
> --table people \
> --sync-mode hms 
setting hadoop conf dir
Running Command : java -cp /hive/lib/hive-metastore-2.3.10.jar::/hive/lib/hive-service-2.3.10.jar::/hive/lib/hive-exec-2.3.10.jar::/hive/lib/hive-jdbc-2.3.10.jar:/hive/lib/hive-jdbc-handler-2.3.10.jar::/hive/lib/jackson-annotations-2.12.0.jar:/hive/lib/jackson-core-2.12.0.jar:/hive/lib/jackson-core-asl-1.9.13.jar:/hive/lib/jackson-databind-2.12.0.jar:/hive/lib/jackson-dataformat-smile-2.12.0.jar:/hive/lib/jackson-datatype-guava-2.12.0.jar:/hive/lib/jackson-datatype-joda-2.12.0.jar:/hive/lib/jackson-jaxrs-1.9.13.jar:/hive/lib/jackson-jaxrs-base-2.12.0.jar:/hive/lib/jackson-jaxrs-json-provider-2.12.0.jar:/hive/lib/jackson-jaxrs-smile-provider-2.12.0.jar:/hive/lib/jackson-mapper-asl-1.9.13.jar:/hive/lib/jackson-module-jaxb-annotations-2.12.0.jar:/hive/lib/jackson-module-scala_2.11-2.12.0.jar:/hive/lib/jackson-xc-1.9.13.jar::/hadoop/share/hadoop/common/*:/hadoop/share/hadoop/mapreduce/*:/hadoop/share/hadoop/hdfs/*:/hadoop/share/hadoop/common/lib/*:/hadoop/share/hadoop/hdfs/lib/*:/hadoop/etc/hadoop:/opt/hudi/hudi-sync/hudi-hive-sync/../../packaging/hudi-hive-sync-bundle/target/hudi-hive-sync-bundle-0.14.1.jar org.apache.hudi.hive.HiveSyncTool --metastore-uris thrift://hive-metastore:9083 --partitioned-by city --base-path s3a://warehouse/people --database hudi_db --table people --sync-mode hms
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/root/.ivy2/jars/org.slf4j_slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/hadoop-2.10.2/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Reload4jLoggerFactory]
2024-06-04 05:55:43,753 INFO  [main] conf.HiveConf (HiveConf.java:findConfigFile(187)) - Found configuration file null
2024-06-04 05:55:43,970 INFO  [main] table.HoodieTableMetaClient (HoodieTableMetaClient.java:<init>(133)) - Loading HoodieTableMetaClient from s3a://warehouse/people
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.hadoop.security.authentication.util.KerberosUtil (file:/root/.ivy2/jars/org.apache.hadoop_hadoop-auth-2.10.2.jar) to method sun.security.krb5.Config.getInstance()
WARNING: Please consider reporting this to the maintainers of org.apache.hadoop.security.authentication.util.KerberosUtil
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
2024-06-04 05:55:44,006 WARN  [main] util.NativeCodeLoader (NativeCodeLoader.java:<clinit>(62)) - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2024-06-04 05:55:44,452 INFO  [main] Configuration.deprecation (Configuration.java:logDeprecation(1294)) - fs.s3a.server-side-encryption-key is deprecated. Instead, use fs.s3a.server-side-encryption.key
2024-06-04 05:55:44,490 INFO  [main] table.HoodieTableConfig (HoodieTableConfig.java:<init>(276)) - Loading table properties from s3a://warehouse/people/.hoodie/hoodie.properties
2024-06-04 05:55:44,501 INFO  [main] table.HoodieTableMetaClient (HoodieTableMetaClient.java:<init>(152)) - Finished Loading Table of type COPY_ON_WRITE(version=1, baseFileFormat=PARQUET) from s3a://warehouse/people
2024-06-04 05:55:44,501 INFO  [main] table.HoodieTableMetaClient (HoodieTableMetaClient.java:<init>(155)) - Loading Active commit timeline for s3a://warehouse/people
2024-06-04 05:55:44,528 INFO  [main] timeline.HoodieActiveTimeline (HoodieActiveTimeline.java:<init>(172)) - Loaded instants upto : Option{val=[20240604055504659__commit__COMPLETED__20240604055511943]}
2024-06-04 05:55:44,567 INFO  [main] hive.metastore (HiveMetaStoreClient.java:open(408)) - Trying to connect to metastore with URI thrift://hive-metastore:9083
2024-06-04 05:55:44,578 INFO  [main] hive.metastore (HiveMetaStoreClient.java:open(493)) - Opened a connection to metastore, current connections: 1
2024-06-04 05:55:44,594 INFO  [main] hive.metastore (HiveMetaStoreClient.java:open(546)) - Connected to metastore.
2024-06-04 05:55:44,630 INFO  [main] hive.HiveSyncTool (HiveSyncTool.java:syncHoodieTable(162)) - Syncing target hoodie table with hive table(hudi_db.people). Hive metastore URL from HiveConf:thrift://hive-metastore:9083). Hive metastore URL from HiveSyncConfig:thrift://hive-metastore:9083, basePath :s3a://warehouse/people
2024-06-04 05:55:44,630 INFO  [main] hive.HiveSyncTool (HiveSyncTool.java:syncHoodieTable(224)) - Trying to sync hoodie table people with base path s3a://warehouse/people of type COPY_ON_WRITE
2024-06-04 05:55:45,497 INFO  [main] table.TableSchemaResolver (TableSchemaResolver.java:readSchemaFromParquetBaseFile(329)) - Reading schema from s3a://warehouse/people/city=NYC/efa9b6b0-8526-4ea5-9cc2-e68c28c00a4e-0_5-13-0_20240604055504659.parquet
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
2024-06-04 05:55:45,615 INFO  [main] hive.metastore (HiveMetaStoreClient.java:close(576)) - Closed a connection to metastore, current connections: 0
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/parquet/schema/LogicalTypeAnnotation

1 reply

alberttwong Jun 4, 2024
Author

pyspark --packages org.apache.parquet:parquet-avro:1.10.1

alberttwong · 2024-06-04T06:11:59Z

alberttwong
Jun 4, 2024
Author

root@spark:/opt/hudi/hudi-sync/hudi-hive-sync# ./run_sync_tool.sh  --metastore-uris 'thrift://hive-metastore:9083' --partitioned-by city --base-path 's3a://warehouse/people' --database hudi_db --table people --sync-mode hms 
setting hadoop conf dir
Running Command : java -cp /hive/lib/hive-metastore-2.3.10.jar::/hive/lib/hive-service-2.3.10.jar::/hive/lib/hive-exec-2.3.10.jar::/hive/lib/hive-jdbc-2.3.10.jar:/hive/lib/hive-jdbc-handler-2.3.10.jar::/hive/lib/jackson-annotations-2.12.0.jar:/hive/lib/jackson-core-2.12.0.jar:/hive/lib/jackson-core-asl-1.9.13.jar:/hive/lib/jackson-databind-2.12.0.jar:/hive/lib/jackson-dataformat-smile-2.12.0.jar:/hive/lib/jackson-datatype-guava-2.12.0.jar:/hive/lib/jackson-datatype-joda-2.12.0.jar:/hive/lib/jackson-jaxrs-1.9.13.jar:/hive/lib/jackson-jaxrs-base-2.12.0.jar:/hive/lib/jackson-jaxrs-json-provider-2.12.0.jar:/hive/lib/jackson-jaxrs-smile-provider-2.12.0.jar:/hive/lib/jackson-mapper-asl-1.9.13.jar:/hive/lib/jackson-module-jaxb-annotations-2.12.0.jar:/hive/lib/jackson-module-scala_2.11-2.12.0.jar:/hive/lib/jackson-xc-1.9.13.jar::/hadoop/share/hadoop/common/*:/hadoop/share/hadoop/mapreduce/*:/hadoop/share/hadoop/hdfs/*:/hadoop/share/hadoop/common/lib/*:/hadoop/share/hadoop/hdfs/lib/*:/hadoop/etc/hadoop:/opt/hudi/hudi-sync/hudi-hive-sync/../../packaging/hudi-hive-sync-bundle/target/hudi-hive-sync-bundle-0.14.1.jar org.apache.hudi.hive.HiveSyncTool --metastore-uris thrift://hive-metastore:9083 --partitioned-by city --base-path s3a://warehouse/people --database hudi_db --table people --sync-mode hms
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/root/.ivy2/jars/org.slf4j_slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/hadoop-2.10.2/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Reload4jLoggerFactory]
2024-06-04 06:02:28,459 INFO  [main] conf.HiveConf (HiveConf.java:findConfigFile(187)) - Found configuration file null
2024-06-04 06:02:28,648 INFO  [main] table.HoodieTableMetaClient (HoodieTableMetaClient.java:<init>(133)) - Loading HoodieTableMetaClient from s3a://warehouse/people
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.hadoop.security.authentication.util.KerberosUtil (file:/root/.ivy2/jars/org.apache.hadoop_hadoop-auth-2.10.2.jar) to method sun.security.krb5.Config.getInstance()
WARNING: Please consider reporting this to the maintainers of org.apache.hadoop.security.authentication.util.KerberosUtil
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
2024-06-04 06:02:28,680 WARN  [main] util.NativeCodeLoader (NativeCodeLoader.java:<clinit>(62)) - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2024-06-04 06:02:29,109 INFO  [main] Configuration.deprecation (Configuration.java:logDeprecation(1294)) - fs.s3a.server-side-encryption-key is deprecated. Instead, use fs.s3a.server-side-encryption.key
2024-06-04 06:02:29,148 INFO  [main] table.HoodieTableConfig (HoodieTableConfig.java:<init>(276)) - Loading table properties from s3a://warehouse/people/.hoodie/hoodie.properties
2024-06-04 06:02:29,183 INFO  [main] table.HoodieTableMetaClient (HoodieTableMetaClient.java:<init>(152)) - Finished Loading Table of type COPY_ON_WRITE(version=1, baseFileFormat=PARQUET) from s3a://warehouse/people
2024-06-04 06:02:29,184 INFO  [main] table.HoodieTableMetaClient (HoodieTableMetaClient.java:<init>(155)) - Loading Active commit timeline for s3a://warehouse/people
2024-06-04 06:02:29,206 INFO  [main] timeline.HoodieActiveTimeline (HoodieActiveTimeline.java:<init>(172)) - Loaded instants upto : Option{val=[20240604055504659__commit__COMPLETED__20240604055511943]}
2024-06-04 06:02:29,237 INFO  [main] hive.metastore (HiveMetaStoreClient.java:open(408)) - Trying to connect to metastore with URI thrift://hive-metastore:9083
2024-06-04 06:02:29,246 INFO  [main] hive.metastore (HiveMetaStoreClient.java:open(493)) - Opened a connection to metastore, current connections: 1
2024-06-04 06:02:29,263 INFO  [main] hive.metastore (HiveMetaStoreClient.java:open(546)) - Connected to metastore.
2024-06-04 06:02:29,293 INFO  [main] hive.HiveSyncTool (HiveSyncTool.java:syncHoodieTable(162)) - Syncing target hoodie table with hive table(hudi_db.people). Hive metastore URL from HiveConf:thrift://hive-metastore:9083). Hive metastore URL from HiveSyncConfig:thrift://hive-metastore:9083, basePath :s3a://warehouse/people
2024-06-04 06:02:29,293 INFO  [main] hive.HiveSyncTool (HiveSyncTool.java:syncHoodieTable(224)) - Trying to sync hoodie table people with base path s3a://warehouse/people of type COPY_ON_WRITE
2024-06-04 06:02:29,458 INFO  [main] table.TableSchemaResolver (TableSchemaResolver.java:readSchemaFromParquetBaseFile(329)) - Reading schema from s3a://warehouse/people/city=NYC/efa9b6b0-8526-4ea5-9cc2-e68c28c00a4e-0_5-13-0_20240604055504659.parquet
2024-06-04 06:02:29,531 INFO  [main] hive.metastore (HiveMetaStoreClient.java:close(576)) - Closed a connection to metastore, current connections: 0
Exception in thread "main" java.lang.NoClassDefFoundError: com/esotericsoftware/kryo/KryoSerializable

1 reply

alberttwong Jun 4, 2024
Author

pyspark --packages org.apache.parquet:parquet-avro:1.10.1,com.esotericsoftware:kryo-shaded:4.0.2

alberttwong · 2024-06-04T06:16:55Z

alberttwong
Jun 4, 2024
Author

working

root@spark:/opt/hudi/hudi-sync/hudi-hive-sync# ./run_sync_tool.sh  --metastore-uris 'thrift://hive-metastore:9083' --partitioned-by city --base-path 's3a://warehouse/people' --database hudi_db --table people --sync-mode hms 
setting hadoop conf dir
Running Command : java -cp /hive/lib/hive-metastore-2.3.10.jar::/hive/lib/hive-service-2.3.10.jar::/hive/lib/hive-exec-2.3.10.jar::/hive/lib/hive-jdbc-2.3.10.jar:/hive/lib/hive-jdbc-handler-2.3.10.jar::/hive/lib/jackson-annotations-2.12.0.jar:/hive/lib/jackson-core-2.12.0.jar:/hive/lib/jackson-core-asl-1.9.13.jar:/hive/lib/jackson-databind-2.12.0.jar:/hive/lib/jackson-dataformat-smile-2.12.0.jar:/hive/lib/jackson-datatype-guava-2.12.0.jar:/hive/lib/jackson-datatype-joda-2.12.0.jar:/hive/lib/jackson-jaxrs-1.9.13.jar:/hive/lib/jackson-jaxrs-base-2.12.0.jar:/hive/lib/jackson-jaxrs-json-provider-2.12.0.jar:/hive/lib/jackson-jaxrs-smile-provider-2.12.0.jar:/hive/lib/jackson-mapper-asl-1.9.13.jar:/hive/lib/jackson-module-jaxb-annotations-2.12.0.jar:/hive/lib/jackson-module-scala_2.11-2.12.0.jar:/hive/lib/jackson-xc-1.9.13.jar::/hadoop/share/hadoop/common/*:/hadoop/share/hadoop/mapreduce/*:/hadoop/share/hadoop/hdfs/*:/hadoop/share/hadoop/common/lib/*:/hadoop/share/hadoop/hdfs/lib/*:/hadoop/etc/hadoop:/opt/hudi/hudi-sync/hudi-hive-sync/../../packaging/hudi-hive-sync-bundle/target/hudi-hive-sync-bundle-0.14.1.jar org.apache.hudi.hive.HiveSyncTool --metastore-uris thrift://hive-metastore:9083 --partitioned-by city --base-path s3a://warehouse/people --database hudi_db --table people --sync-mode hms
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/root/.ivy2/jars/org.slf4j_slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/hadoop-2.10.2/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Reload4jLoggerFactory]
2024-06-04 06:10:10,759 INFO  [main] conf.HiveConf (HiveConf.java:findConfigFile(187)) - Found configuration file null
2024-06-04 06:10:10,948 INFO  [main] table.HoodieTableMetaClient (HoodieTableMetaClient.java:<init>(133)) - Loading HoodieTableMetaClient from s3a://warehouse/people
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.hadoop.security.authentication.util.KerberosUtil (file:/root/.ivy2/jars/org.apache.hadoop_hadoop-auth-2.10.2.jar) to method sun.security.krb5.Config.getInstance()
WARNING: Please consider reporting this to the maintainers of org.apache.hadoop.security.authentication.util.KerberosUtil
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
2024-06-04 06:10:10,978 WARN  [main] util.NativeCodeLoader (NativeCodeLoader.java:<clinit>(62)) - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2024-06-04 06:10:11,349 INFO  [main] Configuration.deprecation (Configuration.java:logDeprecation(1294)) - fs.s3a.server-side-encryption-key is deprecated. Instead, use fs.s3a.server-side-encryption.key
2024-06-04 06:10:11,383 INFO  [main] table.HoodieTableConfig (HoodieTableConfig.java:<init>(276)) - Loading table properties from s3a://warehouse/people/.hoodie/hoodie.properties
2024-06-04 06:10:11,393 INFO  [main] table.HoodieTableMetaClient (HoodieTableMetaClient.java:<init>(152)) - Finished Loading Table of type COPY_ON_WRITE(version=1, baseFileFormat=PARQUET) from s3a://warehouse/people
2024-06-04 06:10:11,393 INFO  [main] table.HoodieTableMetaClient (HoodieTableMetaClient.java:<init>(155)) - Loading Active commit timeline for s3a://warehouse/people
2024-06-04 06:10:11,411 INFO  [main] timeline.HoodieActiveTimeline (HoodieActiveTimeline.java:<init>(172)) - Loaded instants upto : Option{val=[20240604055504659__commit__COMPLETED__20240604055511943]}
2024-06-04 06:10:11,440 INFO  [main] hive.metastore (HiveMetaStoreClient.java:open(408)) - Trying to connect to metastore with URI thrift://hive-metastore:9083
2024-06-04 06:10:11,447 INFO  [main] hive.metastore (HiveMetaStoreClient.java:open(493)) - Opened a connection to metastore, current connections: 1
2024-06-04 06:10:11,460 INFO  [main] hive.metastore (HiveMetaStoreClient.java:open(546)) - Connected to metastore.
2024-06-04 06:10:11,488 INFO  [main] hive.HiveSyncTool (HiveSyncTool.java:syncHoodieTable(162)) - Syncing target hoodie table with hive table(hudi_db.people). Hive metastore URL from HiveConf:thrift://hive-metastore:9083). Hive metastore URL from HiveSyncConfig:thrift://hive-metastore:9083, basePath :s3a://warehouse/people
2024-06-04 06:10:11,488 INFO  [main] hive.HiveSyncTool (HiveSyncTool.java:syncHoodieTable(224)) - Trying to sync hoodie table people with base path s3a://warehouse/people of type COPY_ON_WRITE
2024-06-04 06:10:11,650 INFO  [main] table.TableSchemaResolver (TableSchemaResolver.java:readSchemaFromParquetBaseFile(329)) - Reading schema from s3a://warehouse/people/city=NYC/efa9b6b0-8526-4ea5-9cc2-e68c28c00a4e-0_5-13-0_20240604055504659.parquet
2024-06-04 06:10:11,734 INFO  [main] hive.HiveSyncTool (HiveSyncTool.java:syncFirstTime(321)) - Sync table people for the first time.
2024-06-04 06:10:11,922 INFO  [main] hive.HiveSyncTool (HiveSyncTool.java:syncHoodieTable(265)) - Last commit time synced was found to be null, last commit completion time is found to be null
2024-06-04 06:10:11,924 INFO  [main] hive.HiveSyncTool (HiveSyncTool.java:syncHoodieTable(279)) - Sync all partitions given the last commit time synced is empty or before the start of the active timeline. Listing all partitions in s3a://warehouse/people, file system: S3AFileSystem{uri=s3a://warehouse, workingDir=s3a://warehouse/user/root, inputPolicy=normal, partSize=104857600, enableMultiObjectsDelete=true, maxKeys=5000, readAhead=65536, blockSize=33554432, multiPartThreshold=2147483647, serverSideEncryptionAlgorithm='NONE', metastore=NullMetadataStore, authoritative=false, boundedExecutor=BlockingThreadPoolExecutorService{SemaphoredDelegatingExecutor{permitCount=25, available=25, waiting=0}, activeCount=0}, unboundedExecutor=java.util.concurrent.ThreadPoolExecutor@1989e8c6[Running, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0], DirectoryMarkerRetention{policy='delete'}, statistics {447477 bytes read, 0 bytes written, 8 read ops, 0 large read ops, 0 write ops}, metrics {{Context=S3AFileSystem} {FileSystemId=7da91a6a-ff72-40fa-9060-b226dc240803-warehouse} {fsURI=s3a://warehouse/people/.hoodie} {files_created=0} {files_copied=0} {files_copied_bytes=0} {files_deleted=0} {fake_directories_deleted=0} {directories_created=0} {directories_deleted=0} {ignored_errors=0} {op_copy_from_local_file=0} {op_exists=0} {op_get_file_status=7} {op_glob_status=0} {op_is_directory=0} {op_is_file=0} {op_list_files=0} {op_list_located_status=0} {op_list_status=1} {op_mkdirs=0} {op_rename=0} {object_copy_requests=0} {object_delete_requests=0} {object_list_requests=3} {object_continue_list_requests=0} {object_metadata_requests=7} {object_multipart_aborted=0} {object_put_bytes=0} {object_put_requests=0} {object_put_requests_completed=0} {stream_write_failures=0} {stream_write_block_uploads=0} {stream_write_block_uploads_committed=0} {stream_write_block_uploads_aborted=0} {stream_write_total_time=0} {stream_write_total_data=0} {s3guard_metadatastore_put_path_request=7} {s3guard_metadatastore_initialization=0} {object_put_requests_active=0} {object_put_bytes_pending=0} {stream_write_block_uploads_active=0} {stream_write_block_uploads_pending=0} {stream_write_block_uploads_data_pending=0} {S3guard_metadatastore_put_path_latencyNumOps=0} {S3guard_metadatastore_put_path_latency50thPercentileLatency=0} {S3guard_metadatastore_put_path_latency75thPercentileLatency=0} {S3guard_metadatastore_put_path_latency90thPercentileLatency=0} {S3guard_metadatastore_put_path_latency95thPercentileLatency=0} {S3guard_metadatastore_put_path_latency99thPercentileLatency=0} {stream_read_fully_operations=0} {stream_opened=5} {stream_bytes_skipped_on_seek=0} {stream_closed=5} {stream_bytes_backwards_on_seek=434155} {stream_bytes_read=447477} {stream_read_operations_incomplete=9} {stream_bytes_discarded_in_abort=0} {stream_close_operations=5} {stream_read_operations=896} {stream_aborted=0} {stream_forward_seek_operations=0} {stream_backward_seek_operations=1} {stream_seek_operations=1} {stream_bytes_read_in_close=8} {stream_read_exceptions=0} }}
2024-06-04 06:10:11,985 INFO  [main] table.HoodieTableConfig (HoodieTableConfig.java:<init>(276)) - Loading table properties from s3a://warehouse/people/.hoodie/hoodie.properties
2024-06-04 06:10:12,021 INFO  [main] hive.HiveSyncTool (HiveSyncTool.java:syncPartitions(459)) - New Partitions [city=DFW, city=NYC, city=ORD, city=SEA, city=SFO]
2024-06-04 06:10:12,021 INFO  [main] ddl.HMSDDLExecutor (HMSDDLExecutor.java:addPartitionsToTable(195)) - Adding partitions 5 to table people
2024-06-04 06:10:12,210 INFO  [main] ddl.HMSDDLExecutor (HMSDDLExecutor.java:addPartitionsToTable(213)) - HMSDDLExecutor add a batch partitions done: 5
2024-06-04 06:10:12,282 INFO  [main] hive.HiveSyncTool (HiveSyncTool.java:syncHoodieTable(298)) - Sync complete for people
2024-06-04 06:10:12,284 INFO  [main] hive.metastore (HiveMetaStoreClient.java:close(576)) - Closed a connection to metastore, current connections: 0

trino

trino@trino:/home/data$ cat /etc/trino/catalog/hudi.properties
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
connector.name=hudi
hive.metastore.uri=thrift://hive-metastore:9083
hive.s3.aws-access-key=admin
hive.s3.aws-secret-key=password
hive.s3.endpoint=http://minio:9000
trino@trino:/home/data$ trino
trino> select * from hudi.hudi_db.people;
 _hoodie_commit_time | _hoodie_commit_seqno  |  _hoodie_record_key   | _hoodie_partition_path |                            _hoodie_file_name                            | id>
---------------------+-----------------------+-----------------------+------------------------+-------------------------------------------------------------------------+--->
 20240604055504659   | 20240604055504659_5_4 | 20240604055504659_5_0 | city=NYC               | efa9b6b0-8526-4ea5-9cc2-e68c28c00a4e-0_5-13-0_20240604055504659.parquet |  4>
 20240604055504659   | 20240604055504659_2_2 | 20240604055504659_2_0 | city=SFO               | 5de0c8d2-c2e3-45fe-b0ee-4761cd584285-0_2-10-0_20240604055504659.parquet |  2>
 20240604055504659   | 20240604055504659_7_5 | 20240604055504659_7_0 | city=DFW               | 88a5ee19-1ddc-462e-ac12-825f6eab4d26-0_7-15-0_20240604055504659.parquet |  6>
 20240604055504659   | 20240604055504659_6_3 | 20240604055504659_6_0 | city=SEA               | b5c70a11-c6df-427f-93b9-ab0478a1c4f8-0_6-14-0_20240604055504659.parquet |  5>
 20240604055504659   | 20240604055504659_3_6 | 20240604055504659_3_0 | city=ORD               | c3ff98d0-c782-48ff-a26d-8225b0825409-0_3-11-0_20240604055504659.parquet |  3>
 20240604055504659   | 20240604055504659_1_1 | 20240604055504659_1_0 | city=NYC               | c471db89-5642-437a-9e38-333ef0ad7678-0_1-9-0_20240604055504659.parquet  |  1>
(6 rows)

Query 20240604_061504_00000_ni9iw, FINISHED, 1 node
Splits: 6 total, 6 done (100.00%)
2.52 [6 rows, 2.49MB] [2 rows/s, 1014KB/s]

~/.ivy2/jars

root@spark:/opt/hudi/hudi-sync/hudi-hive-sync# ls ~/.ivy2/jars
aopalliance_aopalliance-1.0.jar                                     org.apache.commons_commons-compress-1.8.1.jar
asm_asm-3.2.jar                                                     org.apache.commons_commons-lang3-3.4.jar
ch.qos.reload4j_reload4j-1.2.18.3.jar                               org.apache.commons_commons-math3-3.1.1.jar
com.amazonaws_aws-java-sdk-bundle-1.11.271.jar                      org.apache.curator_curator-client-2.13.0.jar
com.amazonaws_aws-java-sdk-core-1.11.271.jar                        org.apache.curator_curator-framework-2.13.0.jar
com.amazonaws_aws-java-sdk-kms-1.11.271.jar                         org.apache.curator_curator-recipes-2.13.0.jar
com.amazonaws_aws-java-sdk-s3-1.11.271.jar                          org.apache.directory.api_api-asn1-api-1.0.0-M20.jar
com.amazonaws_jmespath-java-1.11.271.jar                            org.apache.directory.api_api-util-1.0.0-M20.jar
com.esotericsoftware_kryo-shaded-4.0.2.jar                          org.apache.directory.server_apacheds-i18n-2.0.0-M15.jar
com.esotericsoftware_minlog-1.3.0.jar                               org.apache.directory.server_apacheds-kerberos-codec-2.0.0-M15.jar
com.fasterxml.jackson.core_jackson-annotations-2.6.0.jar            org.apache.geronimo.specs_geronimo-jcache_1.0_spec-1.0-alpha-1.jar
com.fasterxml.jackson.core_jackson-core-2.6.7.jar                   org.apache.hadoop_hadoop-annotations-2.10.2.jar
com.fasterxml.jackson.core_jackson-databind-2.6.7.1.jar             org.apache.hadoop_hadoop-auth-2.10.2.jar
com.fasterxml.jackson.dataformat_jackson-dataformat-cbor-2.6.7.jar  org.apache.hadoop_hadoop-aws-2.10.2.jar
com.fasterxml.woodstox_woodstox-core-5.3.0.jar                      org.apache.hadoop_hadoop-client-2.10.2.jar
com.github.spotbugs_spotbugs-annotations-3.1.9.jar                  org.apache.hadoop_hadoop-common-2.10.2.jar
com.github.stephenc.jcip_jcip-annotations-1.0-1.jar                 org.apache.hadoop_hadoop-hdfs-client-2.10.2.jar
com.google.code.findbugs_jsr305-3.0.2.jar                           org.apache.hadoop_hadoop-mapreduce-client-app-2.10.2.jar
com.google.code.gson_gson-2.2.4.jar                                 org.apache.hadoop_hadoop-mapreduce-client-common-2.10.2.jar
com.google.guava_guava-11.0.2.jar                                   org.apache.hadoop_hadoop-mapreduce-client-core-2.10.2.jar
com.google.inject_guice-3.0.jar                                     org.apache.hadoop_hadoop-mapreduce-client-jobclient-2.10.2.jar
com.google.protobuf_protobuf-java-2.5.0.jar                         org.apache.hadoop_hadoop-mapreduce-client-shuffle-2.10.2.jar
com.microsoft.sqlserver_mssql-jdbc-6.2.1.jre7.jar                   org.apache.hadoop_hadoop-yarn-api-2.10.2.jar
commons-beanutils_commons-beanutils-1.9.4.jar                       org.apache.hadoop_hadoop-yarn-client-2.10.2.jar
commons-cli_commons-cli-1.2.jar                                     org.apache.hadoop_hadoop-yarn-common-2.10.2.jar
commons-codec_commons-codec-1.10.jar                                org.apache.hadoop_hadoop-yarn-registry-2.10.2.jar
commons-codec_commons-codec-1.9.jar                                 org.apache.hadoop_hadoop-yarn-server-common-2.10.2.jar
commons-collections_commons-collections-3.2.2.jar                   org.apache.htrace_htrace-core4-4.1.0-incubating.jar
commons-configuration_commons-configuration-1.6.jar                 org.apache.httpcomponents_httpclient-4.5.13.jar
commons-digester_commons-digester-1.8.jar                           org.apache.httpcomponents_httpclient-4.5.2.jar
commons-io_commons-io-2.5.jar                                       org.apache.httpcomponents_httpcore-4.4.13.jar
commons-lang_commons-lang-2.6.jar                                   org.apache.httpcomponents_httpcore-4.4.4.jar
commons-logging_commons-logging-1.1.3.jar                           org.apache.hudi_hudi-spark3.4-bundle_2.12-0.14.1.jar
commons-net_commons-net-3.1.jar                                     org.apache.parquet_parquet-avro-1.10.1.jar
commons-pool_commons-pool-1.6.jar                                   org.apache.parquet_parquet-column-1.10.1.jar
com.nimbusds_nimbus-jose-jwt-7.9.jar                                org.apache.parquet_parquet-common-1.10.1.jar
com.squareup.okhttp_okhttp-2.7.5.jar                                org.apache.parquet_parquet-encoding-1.10.1.jar
com.squareup.okio_okio-1.6.0.jar                                    org.apache.parquet_parquet-format-2.4.0.jar
com.sun.jersey.contribs_jersey-guice-1.9.jar                        org.apache.parquet_parquet-hadoop-1.10.1.jar
com.sun.jersey_jersey-client-1.9.jar                                org.apache.parquet_parquet-jackson-1.10.1.jar
com.sun.jersey_jersey-core-1.9.jar                                  org.apache.yetus_audience-annotations-0.5.0.jar
com.sun.jersey_jersey-json-1.9.jar                                  org.apache.zookeeper_zookeeper-3.4.14.jar
com.sun.jersey_jersey-server-1.9.jar                                org.codehaus.jackson_jackson-core-asl-1.9.13.jar
com.sun.xml.bind_jaxb-impl-2.2.3-1.jar                              org.codehaus.jackson_jackson-jaxrs-1.9.13.jar
com.thoughtworks.paranamer_paranamer-2.3.jar                        org.codehaus.jackson_jackson-mapper-asl-1.9.13.jar
com.thoughtworks.paranamer_paranamer-2.7.jar                        org.codehaus.jackson_jackson-xc-1.9.13.jar
com.zaxxer_HikariCP-java7-2.4.12.jar                                org.codehaus.jettison_jettison-1.1.jar
io.netty_netty-3.10.6.Final.jar                                     org.codehaus.woodstox_stax2-api-4.2.1.jar
it.unimi.dsi_fastutil-7.0.13.jar                                    org.ehcache_ehcache-3.3.1.jar
javax.activation_activation-1.1.jar                                 org.fusesource.leveldbjni_leveldbjni-all-1.8.jar
javax.inject_javax.inject-1.jar                                     org.mortbay.jetty_jetty-sslengine-6.1.26.jar
javax.servlet.jsp_jsp-api-2.1.jar                                   org.mortbay.jetty_jetty-util-6.1.26.jar
javax.servlet_servlet-api-2.5.jar                                   org.objenesis_objenesis-2.5.1.jar
javax.xml.bind_jaxb-api-2.2.2.jar                                   org.slf4j_slf4j-api-1.7.22.jar
javax.xml.stream_stax-api-1.0-2.jar                                 org.slf4j_slf4j-api-1.7.36.jar
jline_jline-0.9.94.jar                                              org.slf4j_slf4j-reload4j-1.7.36.jar
joda-time_joda-time-2.8.1.jar                                       org.sonatype.sisu.inject_cglib-2.2.1-v20090111.jar
log4j_log4j-1.2.17.jar                                              org.tukaani_xz-1.5.jar
net.minidev_json-smart-1.3.3.jar                                    org.xerial.snappy_snappy-java-1.0.5.jar
org.apache.avro_avro-1.7.7.jar                                      org.xerial.snappy_snappy-java-1.1.2.6.jar
org.apache.avro_avro-1.8.2.jar                                      software.amazon.ion_ion-java-1.0.2.jar
org.apache.commons_commons-compress-1.21.jar                        xmlenc_xmlenc-0.52.jar

0 replies

alberttwong · 2024-06-04T07:07:06Z

alberttwong
Jun 4, 2024
Author

faster and easier way....

don't need to compile the hudi project

use my spark docker image.
get the libraries

pyspark --packages org.apache.hudi:hudi-hive-sync-bundle:0.14.1,com.amazonaws:aws-java-sdk-s3:1.11.271,org.apache.hadoop:hadoop-client:2.10.2,org.apache.hadoop:hadoop-aws:2.10.2
pyspark --packages org.apache.parquet:parquet-avro:1.10.1,com.esotericsoftware:kryo-shaded:4.0.2

add ~/.ivy2/jars/*: classpath infront of the run_sync_tool java statement`
add your s3 settings in /hadoop/etc/hadoop/core-site.xml
run the sync tool

./run_sync_tool.sh
--metastore-uris 'thrift://hive-metastore:9083'
--partitioned-by city
--base-path 's3a://warehouse/people'
--database hudi_db
--table people
--sync-mode hms

1 reply

sagarlakshmipathy Jun 4, 2024

this is awesome!

End to End xtable to Trino query with source Hudi and target Iceberg and Delta Lake with HMS and S3 #457

alberttwong Jun 3, 2024

Replies: 9 comments · 14 replies

alberttwong Jun 3, 2024 Author

alberttwong Jun 4, 2024 Author

alberttwong Jun 3, 2024 Author

alberttwong Jun 3, 2024 Author

alberttwong Jun 3, 2024 Author

alberttwong Jun 4, 2024 Author

alberttwong Jun 4, 2024 Author

alberttwong Jun 4, 2024 Author

alberttwong Jun 4, 2024 Author

alberttwong Jun 4, 2024 Author

sagarlakshmipathy Jun 4, 2024

alberttwong Jun 4, 2024 Author

alberttwong Jun 4, 2024 Author

alberttwong Jun 4, 2024 Author

alberttwong Jun 4, 2024 Author

alberttwong Jun 4, 2024 Author

alberttwong Jun 4, 2024 Author

alberttwong Jun 4, 2024 Author

alberttwong Jun 4, 2024 Author

alberttwong Jun 4, 2024 Author

alberttwong Jun 4, 2024 Author

alberttwong Jun 4, 2024 Author

sagarlakshmipathy Jun 4, 2024

alberttwong
Jun 3, 2024

Replies: 9 comments 14 replies

alberttwong
Jun 3, 2024
Author

alberttwong Jun 4, 2024
Author

alberttwong
Jun 3, 2024
Author

alberttwong Jun 3, 2024
Author

alberttwong Jun 3, 2024
Author

alberttwong Jun 4, 2024
Author

alberttwong
Jun 4, 2024
Author

alberttwong Jun 4, 2024
Author

alberttwong Jun 4, 2024
Author

alberttwong Jun 4, 2024
Author

alberttwong Jun 4, 2024
Author

alberttwong
Jun 4, 2024
Author

alberttwong Jun 4, 2024
Author

alberttwong
Jun 4, 2024
Author

alberttwong Jun 4, 2024
Author

alberttwong
Jun 4, 2024
Author

alberttwong Jun 4, 2024
Author

alberttwong
Jun 4, 2024
Author

alberttwong Jun 4, 2024
Author

alberttwong
Jun 4, 2024
Author

alberttwong
Jun 4, 2024
Author