[SPARK-20544][SPARKR] R wrapper for input_file_name #17818
Conversation
|
Test build #76344 has finished for PR 17818 at commit
|
|
Test build #76357 has finished for PR 17818 at commit
|
zero323
force-pushed
the
SPARK-20544
branch
3 times, most recently
from
8457f01
to
2dd17dc
Compare
|
Test build #76358 has finished for PR 17818 at commit
|
|
Test build #76359 has finished for PR 17818 at commit
|
|
hmm, not clear why AppVeyor failed. you could trigger it again by closing and re-opening this PR |
R/pkg/R/functions.R
Outdated
|
|
||
| #' input_file_name | ||
| #' | ||
| #' Creates a string column for the file name of the current Spark task. |
There was a problem hiding this comment.
I actually find this description in Scala API quite a bit confusing - what is "Spark task" and how it has "file name"?
| path <- tempfile(pattern = "input_file_name_test", fileext = ".txt") | ||
| write.table(iris[1:50, ], path, row.names = FALSE, col.names = FALSE) | ||
|
|
||
| df <- read.text(path) |
There was a problem hiding this comment.
does it work with df <- read.json(jsonPath)?
if yes, consider adding to the test for column functions
(again, this is regarding: #17817 to consolidate/skip tests)
There was a problem hiding this comment.
Should work with any file input as far as I remember. I'd skip collecting but there have been some issues with PySpark in the past.
Look I'll have to rebase it anyway but thank you so much for the hint. I've been meaning to ask if there is some equivalent of Jenkins helpers. |
|
Test build #76375 has finished for PR 17818 at commit
|
R/pkg/R/functions.R
Outdated
| #' | ||
| #' @rdname input_file_name | ||
| #' @name input_file_name | ||
| #' @aliases input_file_name,missing-method |
There was a problem hiding this comment.
actually, could you add @family normal_funcs here? I missed this earlier and in the other PR.
|
Test build #76397 has finished for PR 17818 at commit
|
zero323
changed the title
|
|
||
| #' input_file_name | ||
| #' | ||
| #' Creates a string column with the input file name for a given row |
There was a problem hiding this comment.
this actually makes a lot more sense...
|
merged to master |
What changes were proposed in this pull request?
Adds wrapper for
o.a.s.sql.functions.input_file_nameHow was this patch tested?
Existing unit tests, additional unit tests,
check-cran.sh.