[HUDI-4568] Shade dropwizard metrics-core in hudi-aws-bundle

[codope]

Change Logs

• Shade metrics-core in hudi-aws-bundle
• Remove duplicate includes in other bundles

Impact

Without this change, if Hudi metrics is turned on and metrics report type is Cloudwatch, then write client initialization fails. Stacktrace in HUDI-4568. The reason is that metrics-core is shaded in hudi-spark-bundle but not in hudi-aws-bundle but this dependency is used in hudi-aws for which we get NoMethodFoundError.

Risk level: none | low | medium | high

High

Run below script (with metrics and hive turned on). Without this fix, the write will fail with NoMethodFoundError

./bin/pyspark \
  --jars /home/hadoop/hudi-spark3.2-bundle_2.12-0.13.0-SNAPSHOT.jar,/home/hadoop/hudi-aws-bundle-0.13.0-SNAPSHOT.jar \
  --conf "spark.serializer=org.apache.spark.serializer.KryoSerializer"   \
  --conf "spark.sql.catalog.spark_catalog=org.apache.spark.sql.hudi.catalog.HoodieCatalog" \
  --conf "spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension"

sc.setLogLevel("WARN")
dataGen = sc._jvm.org.apache.hudi.QuickstartUtils.DataGenerator()
inserts = sc._jvm.org.apache.hudi.QuickstartUtils.convertToStringList(
    dataGen.generateInserts(10)
)
from pyspark.sql.functions import expr

df = spark.read.json(spark.sparkContext.parallelize(inserts, 10)).withColumn(
    "part", expr("'foo'")
)
tableName = "test_hudi_pyspark2"
basePath = f"/tmp/{tableName}"
hudi_options = {
    "hoodie.table.name": tableName,
    "hoodie.datasource.write.recordkey.field": "uuid",
    "hoodie.datasource.write.partitionpath.field": "part",
    "hoodie.datasource.write.table.name": tableName,
    "hoodie.datasource.write.operation": "upsert",
    "hoodie.datasource.write.precombine.field": "ts",
    "hoodie.upsert.shuffle.parallelism": 2,
    "hoodie.insert.shuffle.parallelism": 2,
    "hoodie.datasource.hive_sync.database": "default",
    "hoodie.datasource.hive_sync.table": tableName,
    "hoodie.datasource.hive_sync.mode": "hms",
    "hoodie.datasource.hive_sync.enable": "true",
    "hoodie.datasource.hive_sync.partition_fields": "part",
    "hoodie.datasource.hive_sync.partition_extractor_class": "org.apache.hudi.hive.MultiPartKeysValueExtractor",
    "hoodie.metrics.on": "true",
    "hoodie.metrics.reporter.type": "CLOUDWATCH"

}
df.write.format("hudi").options(**hudi_options).mode("overwrite").save(basePath)

Contributor's checklist

• Read through contributor's guide
• Change Logs and Impact were stated clearly
• Adequate tests were added if applicable
• CI passed

[codope]

Duplicate. Already included in another line.

[hudi-bot]

CI report:

• d63590c Azure: SUCCESS

Bot commands

@hudi-bot supports the following commands:

• @hudi-bot run azure re-run the last Azure build

[rahil-c]

Tested on recent AWS EMR with this pr, passing the spark bundle first and then aws bundle to spark-shell, and was able to get past the NoSuchMethodError seen below

java.lang.NoSuchMethodError: org.apache.hudi.aws.cloudwatch.CloudWatchReporter.forRegistry(Lorg/apache/hudi/com/codahale/metrics/MetricRegistry;)Lorg/apache/hudi/aws/cloudwatch/CloudWatchReporter$Builder;
  at org.apache.hudi.metrics.cloudwatch.CloudWatchMetricsReporter.createCloudWatchReporter(CloudWatchMetricsReporter.java:57)
  at org.apache.hudi.metrics.cloudwatch.CloudWatchMetricsReporter.<init>(CloudWatchMetricsReporter.java:47)
  at org.apache.hudi.metrics.MetricsReporterFactory.createReporter(MetricsReporterFactory.java:82)
  at org.apache.hudi.metrics.Metrics.<init>(Metrics.java:50)
  at org.apache.hudi.metrics.Metrics.init(Metrics.java:96)
  at org.apache.hudi.metrics.HoodieMetrics.<init>(HoodieMetrics.java:61)
  at org.apache.hudi.client.BaseHoodieWriteClient.<init>(BaseHoodieWriteClient.java:178)
  at org.apache.hudi.client.SparkRDDWriteClient.<init>(SparkRDDWriteClient.java:95)
  at org.apache.hudi.client.SparkRDDWriteClient.<init>(SparkRDDWriteClient.java:79)
  at org.apache.hudi.DataSourceUtils.createHoodieClient(DataSourceUtils.java:193)
  at org.apache.hudi.HoodieSparkSqlWriter$.$anonfun$write$13(HoodieSparkSqlWriter.scala:311)
  at scala.Option.getOrElse(Option.scala:189)
  at org.apache.hudi.HoodieSparkSqlWriter$.write(HoodieSparkSqlWriter.scala:310)
  at org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:183)
  at org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:45)