WIP: Spark action to compute the partition stats #73
Conversation
| Broadcast<Table> tableBroadcast = sparkContext.broadcast(serializableTable); | ||
| int numShufflePartitions = spark.sessionState().conf().numShufflePartitions(); | ||
|
|
||
| return manifestBeanDS(table, null, numShufflePartitions) |
There was a problem hiding this comment.
Hi @aokolnychyi,
I am working on this PR to compute partition stats via spark action. (DRAFT)
I have a problem with Spark encoding with beans.
In BaseSparkAction, I have added partitionEntryDS similar to contentFileDS.
ReadManifestForPartitionStats.call() properly creates a new object filled with contents PartitionEntryBean.
But when I collect the partitionEntryDS, I will have empty PartitionEntryBean objects.
So, I am guessing codegen is using default constructor (empty objects) instead of the actual constructor.
When I remove default constructor from the bean class, I get the below error.
File 'generated.java', Line 24, Column 8: No applicable constructor/method found for zero actual parameters; candidates are: "org.apache.iceberg.spark.actions.PartitionEntryBean(org.apache.iceberg.FileContent, int, org.apache.iceberg.StructLike, long, long, long, long)"
Do you have any suggestions for me? I am not aware of Spark in depth and not finding docs or blog explaining this.
There was a problem hiding this comment.
I tried having simple two siring members for PartitionEntryBean. But still I get the result as null after I collect partitionEntryDS. It does make a new object with proper contents. But when I collect, it is null 🤔
There was a problem hiding this comment.
never mind. I figured it out. getter setter naming conventions has to be standard for Scala codegen to pick it up. I used iceberg conventions. So, it was unable to detect it.
No description provided.