-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix(sdk): Allow keyword-only arguments in pipeline function signature #4544
Merged
google-oss-robot
merged 11 commits into
kubeflow:master
from
Udiknedormin:fix/kwargs_in_pipeline_func
Jan 30, 2021
Merged
Changes from 5 commits
Commits
Show all changes
11 commits
Select commit
Hold shift + click to select a range
da72cf7
add test for keyword-only arguments in pipeline func
Udiknedormin 4270bb8
fix: kwargs-only argument for pipeline func
Udiknedormin d04ad60
test: kwargs generate same yaml as args
Udiknedormin 2fa11d9
Merge branch 'master' into fix/kwargs_in_pipeline_func
Udiknedormin 60667b0
Merge branch 'master' into fix/kwargs_in_pipeline_func
Udiknedormin c991216
Merge branch 'master' of github.com:kubeflow/pipelines into fix/kwarg…
Udiknedormin 9c3e983
remove whole metadata
Udiknedormin cdea000
assert -> self.assertEqual
Udiknedormin eef61fb
programmatic example --> fixed example
Udiknedormin 83e8b84
same name for both
Udiknedormin b1f2593
Merge branch 'master' into fix/kwargs_in_pipeline_func
Ark-kun File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -11,6 +11,7 @@ | |
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
# See the License for the specific language governing permissions and | ||
# limitations under the License. | ||
from typing import List | ||
|
||
import kfp | ||
import kfp.compiler as compiler | ||
|
@@ -1043,3 +1044,45 @@ def some_pipeline(): | |
parameter_arguments_json = template['metadata']['annotations']['pipelines.kubeflow.org/arguments.parameters'] | ||
parameter_arguments = json.loads(parameter_arguments_json) | ||
self.assertEqual(set(parameter_arguments.keys()), {'Input 1'}) | ||
|
||
def test_keyword_only_argument_for_pipeline_func(self): | ||
def some_pipeline(casual_argument: str, *, keyword_only_argument: str): | ||
pass | ||
kfp.compiler.Compiler()._create_workflow(some_pipeline) | ||
|
||
def test_keyword_only_argument_for_pipeline_func_identity(self): | ||
test_data_dir = os.path.join(os.path.dirname(__file__), 'testdata') | ||
sys.path.append(test_data_dir) | ||
import basic | ||
|
||
pipeline_func_arg = basic.save_most_frequent_word | ||
Udiknedormin marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
# clone name and description | ||
@dsl.pipeline( | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Perhaps There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It should have the same name and description as |
||
name = pipeline_func_arg._component_human_name, | ||
description = pipeline_func_arg._component_description | ||
) | ||
def pipeline_func_kwarg(*args, **kwargs): | ||
return basic.save_most_frequent_word(*args, **kwargs) | ||
|
||
# clone signature, but changing all arguments to keyword-only | ||
import inspect | ||
sig = inspect.signature(pipeline_func_arg) | ||
new_parameters = [ | ||
param.replace(kind = inspect.Parameter.KEYWORD_ONLY) | ||
for param in sig.parameters.values() | ||
] | ||
new_sig = sig.replace(parameters = new_parameters) | ||
pipeline_func_kwarg.__signature__ = new_sig | ||
|
||
pipeline_yaml_arg = kfp.compiler.Compiler()._create_workflow(pipeline_func_arg) | ||
pipeline_yaml_kwarg = kfp.compiler.Compiler()._create_workflow(pipeline_func_kwarg) | ||
|
||
# the yamls may differ in creation time, remove it | ||
def remove_creation_time(yaml) -> None: | ||
del yaml['metadata']['annotations']['pipelines.kubeflow.org/pipeline_compilation_time'] | ||
Udiknedormin marked this conversation as resolved.
Show resolved
Hide resolved
|
||
remove_creation_time(pipeline_yaml_arg) | ||
remove_creation_time(pipeline_yaml_kwarg) | ||
|
||
# compare | ||
assert pipeline_yaml_arg == pipeline_yaml_kwarg | ||
Udiknedormin marked this conversation as resolved.
Show resolved
Hide resolved
|
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe we can put all arguments in a dictionary and then just use signature.bind which will sort out what goes where?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think it would work with position-only arguments:
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Position-only arguments won't work with kwargs in signature.bind:
So it seems to me that the easiest approach is to only bound keyword-only arguments to kwargs, and the rest arguments to args.