-
Notifications
You must be signed in to change notification settings - Fork 28.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-43995][SPARK-43996][CONNECT] Add support for UDFRegistration to the Connect Scala Client #41953
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you also modify the client's compatibility tests to include the two new classes you added?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Took a cursory look, and seems fine. cc @xinrong-meng too who worked about here.
Merged to master. |
…o the Connect Scala Client ### What changes were proposed in this pull request? This PR adds support to register a scala UDF from the scala/jvm client. The following APIs are implemented in `UDFRegistration`: - `def register(name: String, udf: UserDefinedFunction): UserDefinedFunction` - `def register[RT: TypeTag, A1: TypeTag ...](name: String, func: (A1, ...) => RT): UserDefinedFunction` for 0 to 22 arguments. The following API is implemented in `functions`: - `def call_udf(udfName: String, cols: Column*): Column` Note: This PR is stacked on apache#41959. ### Why are the changes needed? To reach parity with classic Spark. ### Does this PR introduce _any_ user-facing change? Yes. spark.udf.register() is added as shown below: ```scala class A(x: Int) { def get = x * 100 } val myUdf = udf((x: Int) => new A(x).get) spark.udf.register("dummyUdf", myUdf) spark.sql("select dummyUdf(id) from range(5)").as[Long].collect() ``` The output: ```scala Array[Long] = Array(0L, 100L, 200L, 300L, 400L) ```` ### How was this patch tested? New tests in `ReplE2ESuite`. Closes apache#41953 from vicennial/SPARK-43995. Authored-by: vicennial <[email protected]> Signed-off-by: Hyukjin Kwon <[email protected]>
…o the Connect Scala Client ### What changes were proposed in this pull request? This PR adds support to register a scala UDF from the scala/jvm client. The following APIs are implemented in `UDFRegistration`: - `def register(name: String, udf: UserDefinedFunction): UserDefinedFunction` - `def register[RT: TypeTag, A1: TypeTag ...](name: String, func: (A1, ...) => RT): UserDefinedFunction` for 0 to 22 arguments. The following API is implemented in `functions`: - `def call_udf(udfName: String, cols: Column*): Column` Note: This PR is stacked on apache#41959. ### Why are the changes needed? To reach parity with classic Spark. ### Does this PR introduce _any_ user-facing change? Yes. spark.udf.register() is added as shown below: ```scala class A(x: Int) { def get = x * 100 } val myUdf = udf((x: Int) => new A(x).get) spark.udf.register("dummyUdf", myUdf) spark.sql("select dummyUdf(id) from range(5)").as[Long].collect() ``` The output: ```scala Array[Long] = Array(0L, 100L, 200L, 300L, 400L) ```` ### How was this patch tested? New tests in `ReplE2ESuite`. Closes apache#41953 from vicennial/SPARK-43995. Authored-by: vicennial <[email protected]> Signed-off-by: Hyukjin Kwon <[email protected]>
What changes were proposed in this pull request?
This PR adds support to register a scala UDF from the scala/jvm client.
The following APIs are implemented in
UDFRegistration
:def register(name: String, udf: UserDefinedFunction): UserDefinedFunction
def register[RT: TypeTag, A1: TypeTag ...](name: String, func: (A1, ...) => RT): UserDefinedFunction
for 0 to 22 arguments.The following API is implemented in
functions
:def call_udf(udfName: String, cols: Column*): Column
Note: This PR is stacked on #41959.
Why are the changes needed?
To reach parity with classic Spark.
Does this PR introduce any user-facing change?
Yes. spark.udf.register() is added as shown below:
The output:
How was this patch tested?
New tests in
ReplE2ESuite
.