Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tekton incorrectly maps pipeline parameters and task outputs with the same name #5988

Closed
typeness opened this issue Jan 12, 2023 · 5 comments · Fixed by #6004
Closed

Tekton incorrectly maps pipeline parameters and task outputs with the same name #5988

typeness opened this issue Jan 12, 2023 · 5 comments · Fixed by #6004
Labels
kind/bug Categorizes issue or PR as related to a bug.

Comments

@typeness
Copy link

Expected Behavior

The pipeline run should correctly map pipeline parameters and task outputs with the same name.

Regression: This scenario works correctly on v0.36.

Actual Behavior

The print-msg-2 task has the start, step, end parameters which are pipeline parameters with the same name.

    - name: print-msg-2
      params:
      - name: end
        value: $(params.end)
      - name: start
        value: $(params.start)
      - name: step
        value: $(params.step)
      taskSpec:
        metadata:
          annotations:
            pipelines.kubeflow.org/component_spec_digest: '{"name": "Print msg", "outputs":
              [], "version": "Print msg@sha256=ed3c7f41a60c7f07d32b7a3fac26e50b0231d7c92ea25c722a9f887d0d8fd084"}'
          labels:
            pipelines.kubeflow.org/cache_enabled: "true"
        params:
        - name: end
          type: string
        - name: start
          type: string
        - name: step
          type: string

The print-msg task has the start, step, end parameters which are outputs from other tasks with the same name

   - name: print-msg
      params:
      - name: end
        value: $(tasks.end.results.Output)
      - name: start
        value: $(tasks.start.results.Output)
      - name: step
        value: $(tasks.step.results.Output)
      taskSpec:
        metadata:
          annotations:
            pipelines.kubeflow.org/component_spec_digest: '{"name": "Print msg", "outputs":
              [], "version": "Print msg@sha256=ed3c7f41a60c7f07d32b7a3fac26e50b0231d7c92ea25c722a9f887d0d8fd084"}'
          labels:
            pipelines.kubeflow.org/cache_enabled: "true"
        params:
        - name: end
          type: string
        - name: start
          type: string
        - name: step
          type: string

In runtime both print-msg and print-msg-2 tasks parameters are incorrectly mapped to task outputs

    - name: print-msg
      params:
      - name: end
        value: $(tasks.end.results.Output)
      - name: start
        value: $(tasks.start.results.Output)
      - name: step
        value: $(tasks.step.results.Output)
    - name: print-msg-2
      params:
      - name: end
        value: $(tasks.end.results.Output)
      - name: start
        value: $(tasks.start.results.Output)
      - name: step
        value: $(tasks.step.results.Output)

Steps to Reproduce the Problem

  1. Run the following pipeline in Tekton 0.41
apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"tekton.dev/v1beta1","kind":"PipelineRun","metadata":{"annotations":{"pipelines.kubeflow.org/big_data_passing_format":"$(workspaces.$TASK_NAME.path)/artifacts/$ORIG_PR_NAME/$TASKRUN_NAME/$TASK_PARAM_NAME","pipelines.kubeflow.org/pipeline_spec":"{\"inputs\": [{\"default\": \"asdfsdf\", \"name\": \"start\", \"optional\": true, \"type\": \"String\"}, {\"default\": \"bsdfsd\", \"name\": \"step\", \"optional\": true, \"type\": \"String\"}, {\"default\": \"c\", \"name\": \"end\", \"optional\": true, \"type\": \"String\"}], \"name\": \"pipeline2\"}","sidecar.istio.io/inject":"false","tekton.dev/artifact_bucket":"mlpipeline","tekton.dev/artifact_endpoint":"minio-service.kubeflow:9000","tekton.dev/artifact_endpoint_scheme":"http://","tekton.dev/artifact_items":"{\"end\": [[\"Output\", \"$(results.Output.path)\"]], \"print-msg\": [], \"print-msg-2\": [], \"start\": [[\"Output\", \"$(results.Output.path)\"]], \"step\": [[\"Output\", \"$(results.Output.path)\"]]}","tekton.dev/input_artifacts":"{\"print-msg\": [{\"name\": \"end\", \"parent_task\": \"end\"}, {\"name\": \"start\", \"parent_task\": \"start\"}, {\"name\": \"step\", \"parent_task\": \"step\"}]}","tekton.dev/output_artifacts":"{\"end\": [{\"key\": \"artifacts/$PIPELINERUN/end/Output.tgz\", \"name\": \"end\", \"path\": \"/tmp/outputs/Output/data\"}], \"start\": [{\"key\": \"artifacts/$PIPELINERUN/start/Output.tgz\", \"name\": \"start\", \"path\": \"/tmp/outputs/Output/data\"}], \"step\": [{\"key\": \"artifacts/$PIPELINERUN/step/Output.tgz\", \"name\": \"step\", \"path\": \"/tmp/outputs/Output/data\"}]}","tekton.dev/template":""},"labels":{"pipelines.kubeflow.org/generation":"","pipelines.kubeflow.org/pipelinename":""},"name":"pipeline2","namespace":"default"},"spec":{"params":[{"name":"end","value":"c"},{"name":"start","value":"asdfsdf"},{"name":"step","value":"bsdfsd"}],"pipelineSpec":{"params":[{"default":"c","name":"end"},{"default":"asdfsdf","name":"start"},{"default":"bsdfsd","name":"step"}],"tasks":[{"name":"start","taskSpec":{"metadata":{"annotations":{"pipelines.kubeflow.org/component_spec_digest":"{\"name\": \"Start\", \"outputs\": [{\"name\": \"Output\", \"type\": \"String\"}], \"version\": \"Start@sha256=40dfec809092674e3d2ef2d51f9babefeb8c06b026f1ae1bc82611c88b6d6f0c\"}"},"labels":{"pipelines.kubeflow.org/cache_enabled":"true"}},"results":[{"description":"/tmp/outputs/Output/data","name":"Output","type":"string"}],"steps":[{"args":["----output-paths","$(results.Output.path)"],"command":["sh","-ec","program_path=$(mktemp)\nprintf \"%s\" \"$0\" \u003e \"$program_path\"\npython3 -u \"$program_path\" \"$@\"\n","def start():\n    return '2'\n\ndef _serialize_str(str_value: str) -\u003e str:\n    if not isinstance(str_value, str):\n        raise TypeError('Value \"{}\" has type \"{}\" instead of str.'.format(\n            str(str_value), str(type(str_value))))\n    return str_value\n\nimport argparse\n_parser = argparse.ArgumentParser(prog='Start', description='')\n_parser.add_argument(\"----output-paths\", dest=\"_output_paths\", type=str, nargs=1)\n_parsed_args = vars(_parser.parse_args())\n_output_files = _parsed_args.pop(\"_output_paths\", [])\n\n_outputs = start(**_parsed_args)\n\n_outputs = [_outputs]\n\n_output_serializers = [\n    _serialize_str,\n\n]\n\nimport os\nfor idx, output_file in enumerate(_output_files):\n    try:\n        os.makedirs(os.path.dirname(output_file))\n    except OSError:\n        pass\n    with open(output_file, 'w') as f:\n        f.write(_output_serializers[idx](_outputs[idx]))\n"],"image":"python:alpine3.6","name":"main"}]}},{"name":"step","taskSpec":{"metadata":{"annotations":{"pipelines.kubeflow.org/component_spec_digest":"{\"name\": \"Step\", \"outputs\": [{\"name\": \"Output\", \"type\": \"String\"}], \"version\": \"Step@sha256=ac1b6409fa045ecde871cb0a4f56a6e2553b573242050fb1bd8f101f56b0e262\"}"},"labels":{"pipelines.kubeflow.org/cache_enabled":"true"}},"results":[{"description":"/tmp/outputs/Output/data","name":"Output","type":"string"}],"steps":[{"args":["----output-paths","$(results.Output.path)"],"command":["sh","-ec","program_path=$(mktemp)\nprintf \"%s\" \"$0\" \u003e \"$program_path\"\npython3 -u \"$program_path\" \"$@\"\n","def step():\n    return '3'\n\ndef _serialize_str(str_value: str) -\u003e str:\n    if not isinstance(str_value, str):\n        raise TypeError('Value \"{}\" has type \"{}\" instead of str.'.format(\n            str(str_value), str(type(str_value))))\n    return str_value\n\nimport argparse\n_parser = argparse.ArgumentParser(prog='Step', description='')\n_parser.add_argument(\"----output-paths\", dest=\"_output_paths\", type=str, nargs=1)\n_parsed_args = vars(_parser.parse_args())\n_output_files = _parsed_args.pop(\"_output_paths\", [])\n\n_outputs = step(**_parsed_args)\n\n_outputs = [_outputs]\n\n_output_serializers = [\n    _serialize_str,\n\n]\n\nimport os\nfor idx, output_file in enumerate(_output_files):\n    try:\n        os.makedirs(os.path.dirname(output_file))\n    except OSError:\n        pass\n    with open(output_file, 'w') as f:\n        f.write(_output_serializers[idx](_outputs[idx]))\n"],"image":"python:alpine3.6","name":"main"}]}},{"name":"end","taskSpec":{"metadata":{"annotations":{"pipelines.kubeflow.org/component_spec_digest":"{\"name\": \"End\", \"outputs\": [{\"name\": \"Output\", \"type\": \"String\"}], \"version\": \"End@sha256=fc4ae432a3c7fecc61c191f63601c267dac41344940769319845daff355642d2\"}"},"labels":{"pipelines.kubeflow.org/cache_enabled":"true"}},"results":[{"description":"/tmp/outputs/Output/data","name":"Output","type":"string"}],"steps":[{"args":["----output-paths","$(results.Output.path)"],"command":["sh","-ec","program_path=$(mktemp)\nprintf \"%s\" \"$0\" \u003e \"$program_path\"\npython3 -u \"$program_path\" \"$@\"\n","def end():\n    return '8'\n\ndef _serialize_str(str_value: str) -\u003e str:\n    if not isinstance(str_value, str):\n        raise TypeError('Value \"{}\" has type \"{}\" instead of str.'.format(\n            str(str_value), str(type(str_value))))\n    return str_value\n\nimport argparse\n_parser = argparse.ArgumentParser(prog='End', description='')\n_parser.add_argument(\"----output-paths\", dest=\"_output_paths\", type=str, nargs=1)\n_parsed_args = vars(_parser.parse_args())\n_output_files = _parsed_args.pop(\"_output_paths\", [])\n\n_outputs = end(**_parsed_args)\n\n_outputs = [_outputs]\n\n_output_serializers = [\n    _serialize_str,\n\n]\n\nimport os\nfor idx, output_file in enumerate(_output_files):\n    try:\n        os.makedirs(os.path.dirname(output_file))\n    except OSError:\n        pass\n    with open(output_file, 'w') as f:\n        f.write(_output_serializers[idx](_outputs[idx]))\n"],"image":"python:alpine3.6","name":"main"}]}},{"name":"print-msg","params":[{"name":"end","value":"$(tasks.end.results.Output)"},{"name":"start","value":"$(tasks.start.results.Output)"},{"name":"step","value":"$(tasks.step.results.Output)"}],"taskSpec":{"metadata":{"annotations":{"pipelines.kubeflow.org/component_spec_digest":"{\"name\": \"Print msg\", \"outputs\": [], \"version\": \"Print msg@sha256=ed3c7f41a60c7f07d32b7a3fac26e50b0231d7c92ea25c722a9f887d0d8fd084\"}"},"labels":{"pipelines.kubeflow.org/cache_enabled":"true"}},"params":[{"name":"end"},{"name":"start"},{"name":"step"}],"steps":[{"args":["--msg1","$(inputs.params.start)","--msg2","$(inputs.params.step)","--msg3","$(inputs.params.end)"],"command":["sh","-ec","program_path=$(mktemp)\nprintf \"%s\" \"$0\" \u003e \"$program_path\"\npython3 -u \"$program_path\" \"$@\"\n","def print_msg(msg1, msg2, msg3):\n    \"\"\"Print a message.\"\"\"\n    print(msg1)\n    print(msg2)\n    print(msg3)\n\nimport argparse\n_parser = argparse.ArgumentParser(prog='Print msg', description='Print a message.')\n_parser.add_argument(\"--msg1\", dest=\"msg1\", type=str, required=True, default=argparse.SUPPRESS)\n_parser.add_argument(\"--msg2\", dest=\"msg2\", type=str, required=True, default=argparse.SUPPRESS)\n_parser.add_argument(\"--msg3\", dest=\"msg3\", type=str, required=True, default=argparse.SUPPRESS)\n_parsed_args = vars(_parser.parse_args())\n\n_outputs = print_msg(**_parsed_args)\n"],"image":"python:3.7","name":"main"}]}},{"name":"print-msg-2","params":[{"name":"end","value":"$(params.end)"},{"name":"start","value":"$(params.start)"},{"name":"step","value":"$(params.step)"}],"taskSpec":{"metadata":{"annotations":{"pipelines.kubeflow.org/component_spec_digest":"{\"name\": \"Print msg\", \"outputs\": [], \"version\": \"Print msg@sha256=ed3c7f41a60c7f07d32b7a3fac26e50b0231d7c92ea25c722a9f887d0d8fd084\"}"},"labels":{"pipelines.kubeflow.org/cache_enabled":"true"}},"params":[{"name":"end"},{"name":"start"},{"name":"step"}],"steps":[{"args":["--msg1","$(inputs.params.start)","--msg2","$(inputs.params.step)","--msg3","$(inputs.params.end)"],"command":["sh","-ec","program_path=$(mktemp)\nprintf \"%s\" \"$0\" \u003e \"$program_path\"\npython3 -u \"$program_path\" \"$@\"\n","def print_msg(msg1, msg2, msg3):\n    \"\"\"Print a message.\"\"\"\n    print(msg1)\n    print(msg2)\n    print(msg3)\n\nimport argparse\n_parser = argparse.ArgumentParser(prog='Print msg', description='Print a message.')\n_parser.add_argument(\"--msg1\", dest=\"msg1\", type=str, required=True, default=argparse.SUPPRESS)\n_parser.add_argument(\"--msg2\", dest=\"msg2\", type=str, required=True, default=argparse.SUPPRESS)\n_parser.add_argument(\"--msg3\", dest=\"msg3\", type=str, required=True, default=argparse.SUPPRESS)\n_parsed_args = vars(_parser.parse_args())\n\n_outputs = print_msg(**_parsed_args)\n"],"image":"python:3.7","name":"main"}]}}]}}}
    pipelines.kubeflow.org/big_data_passing_format: $(workspaces.$TASK_NAME.path)/artifacts/$ORIG_PR_NAME/$TASKRUN_NAME/$TASK_PARAM_NAME
    pipelines.kubeflow.org/pipeline_spec: '{"inputs": [{"default": "asdfsdf", "name":
      "start", "optional": true, "type": "String"}, {"default": "bsdfsd", "name":
      "step", "optional": true, "type": "String"}, {"default": "c", "name": "end",
      "optional": true, "type": "String"}], "name": "pipeline2"}'
    sidecar.istio.io/inject: "false"
    tekton.dev/artifact_bucket: mlpipeline
    tekton.dev/artifact_endpoint: minio-service.kubeflow:9000
    tekton.dev/artifact_endpoint_scheme: http://
    tekton.dev/artifact_items: '{"end": [["Output", "$(results.Output.path)"]], "print-msg":
      [], "print-msg-2": [], "start": [["Output", "$(results.Output.path)"]], "step":
      [["Output", "$(results.Output.path)"]]}'
    tekton.dev/input_artifacts: '{"print-msg": [{"name": "end", "parent_task": "end"},
      {"name": "start", "parent_task": "start"}, {"name": "step", "parent_task": "step"}]}'
    tekton.dev/output_artifacts: '{"end": [{"key": "artifacts/$PIPELINERUN/end/Output.tgz",
      "name": "end", "path": "/tmp/outputs/Output/data"}], "start": [{"key": "artifacts/$PIPELINERUN/start/Output.tgz",
      "name": "start", "path": "/tmp/outputs/Output/data"}], "step": [{"key": "artifacts/$PIPELINERUN/step/Output.tgz",
      "name": "step", "path": "/tmp/outputs/Output/data"}]}'
    tekton.dev/template: ""
  creationTimestamp: "2023-01-12T13:34:01Z"
  generation: 1
  labels:
    pipelines.kubeflow.org/generation: ""
    pipelines.kubeflow.org/pipelinename: ""
    tekton.dev/pipeline: pipeline2
  name: pipeline2
  namespace: default
  resourceVersion: "1393490959"
  uid: 95bff33c-040e-4476-8e19-ffc094dddc09
spec:
  params:
  - name: end
    value: c
  - name: start
    value: asdfsdf
  - name: step
    value: bsdfsd
  pipelineSpec:
    params:
    - default: c
      name: end
      type: string
    - default: asdfsdf
      name: start
      type: string
    - default: bsdfsd
      name: step
      type: string
    tasks:
    - name: start
      taskSpec:
        metadata:
          annotations:
            pipelines.kubeflow.org/component_spec_digest: '{"name": "Start", "outputs":
              [{"name": "Output", "type": "String"}], "version": "Start@sha256=40dfec809092674e3d2ef2d51f9babefeb8c06b026f1ae1bc82611c88b6d6f0c"}'
          labels:
            pipelines.kubeflow.org/cache_enabled: "true"
        results:
        - description: /tmp/outputs/Output/data
          name: Output
          type: string
        spec: null
        steps:
        - args:
          - '----output-paths'
          - $(results.Output.path)
          command:
          - sh
          - -ec
          - |
            program_path=$(mktemp)
            printf "%s" "$0" > "$program_path"
            python3 -u "$program_path" "$@"
          - |
            def start():
                return '2'

            def _serialize_str(str_value: str) -> str:
                if not isinstance(str_value, str):
                    raise TypeError('Value "{}" has type "{}" instead of str.'.format(
                        str(str_value), str(type(str_value))))
                return str_value

            import argparse
            _parser = argparse.ArgumentParser(prog='Start', description='')
            _parser.add_argument("----output-paths", dest="_output_paths", type=str, nargs=1)
            _parsed_args = vars(_parser.parse_args())
            _output_files = _parsed_args.pop("_output_paths", [])

            _outputs = start(**_parsed_args)

            _outputs = [_outputs]

            _output_serializers = [
                _serialize_str,

            ]

            import os
            for idx, output_file in enumerate(_output_files):
                try:
                    os.makedirs(os.path.dirname(output_file))
                except OSError:
                    pass
                with open(output_file, 'w') as f:
                    f.write(_output_serializers[idx](_outputs[idx]))
          image: python:alpine3.6
          name: main
          resources: {}
    - name: step
      taskSpec:
        metadata:
          annotations:
            pipelines.kubeflow.org/component_spec_digest: '{"name": "Step", "outputs":
              [{"name": "Output", "type": "String"}], "version": "Step@sha256=ac1b6409fa045ecde871cb0a4f56a6e2553b573242050fb1bd8f101f56b0e262"}'
          labels:
            pipelines.kubeflow.org/cache_enabled: "true"
        results:
        - description: /tmp/outputs/Output/data
          name: Output
          type: string
        spec: null
        steps:
        - args:
          - '----output-paths'
          - $(results.Output.path)
          command:
          - sh
          - -ec
          - |
            program_path=$(mktemp)
            printf "%s" "$0" > "$program_path"
            python3 -u "$program_path" "$@"
          - |
            def step():
                return '3'

            def _serialize_str(str_value: str) -> str:
                if not isinstance(str_value, str):
                    raise TypeError('Value "{}" has type "{}" instead of str.'.format(
                        str(str_value), str(type(str_value))))
                return str_value

            import argparse
            _parser = argparse.ArgumentParser(prog='Step', description='')
            _parser.add_argument("----output-paths", dest="_output_paths", type=str, nargs=1)
            _parsed_args = vars(_parser.parse_args())
            _output_files = _parsed_args.pop("_output_paths", [])

            _outputs = step(**_parsed_args)

            _outputs = [_outputs]

            _output_serializers = [
                _serialize_str,

            ]

            import os
            for idx, output_file in enumerate(_output_files):
                try:
                    os.makedirs(os.path.dirname(output_file))
                except OSError:
                    pass
                with open(output_file, 'w') as f:
                    f.write(_output_serializers[idx](_outputs[idx]))
          image: python:alpine3.6
          name: main
          resources: {}
    - name: end
      taskSpec:
        metadata:
          annotations:
            pipelines.kubeflow.org/component_spec_digest: '{"name": "End", "outputs":
              [{"name": "Output", "type": "String"}], "version": "End@sha256=fc4ae432a3c7fecc61c191f63601c267dac41344940769319845daff355642d2"}'
          labels:
            pipelines.kubeflow.org/cache_enabled: "true"
        results:
        - description: /tmp/outputs/Output/data
          name: Output
          type: string
        spec: null
        steps:
        - args:
          - '----output-paths'
          - $(results.Output.path)
          command:
          - sh
          - -ec
          - |
            program_path=$(mktemp)
            printf "%s" "$0" > "$program_path"
            python3 -u "$program_path" "$@"
          - |
            def end():
                return '8'

            def _serialize_str(str_value: str) -> str:
                if not isinstance(str_value, str):
                    raise TypeError('Value "{}" has type "{}" instead of str.'.format(
                        str(str_value), str(type(str_value))))
                return str_value

            import argparse
            _parser = argparse.ArgumentParser(prog='End', description='')
            _parser.add_argument("----output-paths", dest="_output_paths", type=str, nargs=1)
            _parsed_args = vars(_parser.parse_args())
            _output_files = _parsed_args.pop("_output_paths", [])

            _outputs = end(**_parsed_args)

            _outputs = [_outputs]

            _output_serializers = [
                _serialize_str,

            ]

            import os
            for idx, output_file in enumerate(_output_files):
                try:
                    os.makedirs(os.path.dirname(output_file))
                except OSError:
                    pass
                with open(output_file, 'w') as f:
                    f.write(_output_serializers[idx](_outputs[idx]))
          image: python:alpine3.6
          name: main
          resources: {}
    - name: print-msg
      params:
      - name: end
        value: $(tasks.end.results.Output)
      - name: start
        value: $(tasks.start.results.Output)
      - name: step
        value: $(tasks.step.results.Output)
      taskSpec:
        metadata:
          annotations:
            pipelines.kubeflow.org/component_spec_digest: '{"name": "Print msg", "outputs":
              [], "version": "Print msg@sha256=ed3c7f41a60c7f07d32b7a3fac26e50b0231d7c92ea25c722a9f887d0d8fd084"}'
          labels:
            pipelines.kubeflow.org/cache_enabled: "true"
        params:
        - name: end
          type: string
        - name: start
          type: string
        - name: step
          type: string
        spec: null
        steps:
        - args:
          - --msg1
          - $(inputs.params.start)
          - --msg2
          - $(inputs.params.step)
          - --msg3
          - $(inputs.params.end)
          command:
          - sh
          - -ec
          - |
            program_path=$(mktemp)
            printf "%s" "$0" > "$program_path"
            python3 -u "$program_path" "$@"
          - |
            def print_msg(msg1, msg2, msg3):
                """Print a message."""
                print(msg1)
                print(msg2)
                print(msg3)

            import argparse
            _parser = argparse.ArgumentParser(prog='Print msg', description='Print a message.')
            _parser.add_argument("--msg1", dest="msg1", type=str, required=True, default=argparse.SUPPRESS)
            _parser.add_argument("--msg2", dest="msg2", type=str, required=True, default=argparse.SUPPRESS)
            _parser.add_argument("--msg3", dest="msg3", type=str, required=True, default=argparse.SUPPRESS)
            _parsed_args = vars(_parser.parse_args())

            _outputs = print_msg(**_parsed_args)
          image: python:3.7
          name: main
          resources: {}
    - name: print-msg-2
      params:
      - name: end
        value: $(params.end)
      - name: start
        value: $(params.start)
      - name: step
        value: $(params.step)
      taskSpec:
        metadata:
          annotations:
            pipelines.kubeflow.org/component_spec_digest: '{"name": "Print msg", "outputs":
              [], "version": "Print msg@sha256=ed3c7f41a60c7f07d32b7a3fac26e50b0231d7c92ea25c722a9f887d0d8fd084"}'
          labels:
            pipelines.kubeflow.org/cache_enabled: "true"
        params:
        - name: end
          type: string
        - name: start
          type: string
        - name: step
          type: string
        spec: null
        steps:
        - args:
          - --msg1
          - $(inputs.params.start)
          - --msg2
          - $(inputs.params.step)
          - --msg3
          - $(inputs.params.end)
          command:
          - sh
          - -ec
          - |
            program_path=$(mktemp)
            printf "%s" "$0" > "$program_path"
            python3 -u "$program_path" "$@"
          - |
            def print_msg(msg1, msg2, msg3):
                """Print a message."""
                print(msg1)
                print(msg2)
                print(msg3)

            import argparse
            _parser = argparse.ArgumentParser(prog='Print msg', description='Print a message.')
            _parser.add_argument("--msg1", dest="msg1", type=str, required=True, default=argparse.SUPPRESS)
            _parser.add_argument("--msg2", dest="msg2", type=str, required=True, default=argparse.SUPPRESS)
            _parser.add_argument("--msg3", dest="msg3", type=str, required=True, default=argparse.SUPPRESS)
            _parsed_args = vars(_parser.parse_args())

            _outputs = print_msg(**_parsed_args)
          image: python:3.7
          name: main
          resources: {}
  serviceAccountName: default
  timeout: 24h0m0s
status:
  annotations:
    ws-pipelines.ibm.com/payment-account/runtimes: '[]'
    ws-pipelines.ibm.com/payment-account/usage: '[]'
  completionTime: "2023-01-12T13:34:01Z"
  conditions:
  - lastTransitionTime: "2023-01-12T13:34:01Z"
    message: task "end" referenced by result was not successful
    reason: InvalidTaskResultReference
    status: "False"
    type: Succeeded
  pipelineSpec:
    params:
    - default: c
      name: end
      type: string
    - default: asdfsdf
      name: start
      type: string
    - default: bsdfsd
      name: step
      type: string
    tasks:
    - name: start
      taskSpec:
        metadata:
          annotations:
            pipelines.kubeflow.org/component_spec_digest: '{"name": "Start", "outputs":
              [{"name": "Output", "type": "String"}], "version": "Start@sha256=40dfec809092674e3d2ef2d51f9babefeb8c06b026f1ae1bc82611c88b6d6f0c"}'
          labels:
            pipelines.kubeflow.org/cache_enabled: "true"
        results:
        - description: /tmp/outputs/Output/data
          name: Output
          type: string
        spec: null
        steps:
        - args:
          - '----output-paths'
          - $(results.Output.path)
          command:
          - sh
          - -ec
          - |
            program_path=$(mktemp)
            printf "%s" "$0" > "$program_path"
            python3 -u "$program_path" "$@"
          - |
            def start():
                return '2'

            def _serialize_str(str_value: str) -> str:
                if not isinstance(str_value, str):
                    raise TypeError('Value "{}" has type "{}" instead of str.'.format(
                        str(str_value), str(type(str_value))))
                return str_value

            import argparse
            _parser = argparse.ArgumentParser(prog='Start', description='')
            _parser.add_argument("----output-paths", dest="_output_paths", type=str, nargs=1)
            _parsed_args = vars(_parser.parse_args())
            _output_files = _parsed_args.pop("_output_paths", [])

            _outputs = start(**_parsed_args)

            _outputs = [_outputs]

            _output_serializers = [
                _serialize_str,

            ]

            import os
            for idx, output_file in enumerate(_output_files):
                try:
                    os.makedirs(os.path.dirname(output_file))
                except OSError:
                    pass
                with open(output_file, 'w') as f:
                    f.write(_output_serializers[idx](_outputs[idx]))
          image: python:alpine3.6
          name: main
          resources: {}
    - name: step
      taskSpec:
        metadata:
          annotations:
            pipelines.kubeflow.org/component_spec_digest: '{"name": "Step", "outputs":
              [{"name": "Output", "type": "String"}], "version": "Step@sha256=ac1b6409fa045ecde871cb0a4f56a6e2553b573242050fb1bd8f101f56b0e262"}'
          labels:
            pipelines.kubeflow.org/cache_enabled: "true"
        results:
        - description: /tmp/outputs/Output/data
          name: Output
          type: string
        spec: null
        steps:
        - args:
          - '----output-paths'
          - $(results.Output.path)
          command:
          - sh
          - -ec
          - |
            program_path=$(mktemp)
            printf "%s" "$0" > "$program_path"
            python3 -u "$program_path" "$@"
          - |
            def step():
                return '3'

            def _serialize_str(str_value: str) -> str:
                if not isinstance(str_value, str):
                    raise TypeError('Value "{}" has type "{}" instead of str.'.format(
                        str(str_value), str(type(str_value))))
                return str_value

            import argparse
            _parser = argparse.ArgumentParser(prog='Step', description='')
            _parser.add_argument("----output-paths", dest="_output_paths", type=str, nargs=1)
            _parsed_args = vars(_parser.parse_args())
            _output_files = _parsed_args.pop("_output_paths", [])

            _outputs = step(**_parsed_args)

            _outputs = [_outputs]

            _output_serializers = [
                _serialize_str,

            ]

            import os
            for idx, output_file in enumerate(_output_files):
                try:
                    os.makedirs(os.path.dirname(output_file))
                except OSError:
                    pass
                with open(output_file, 'w') as f:
                    f.write(_output_serializers[idx](_outputs[idx]))
          image: python:alpine3.6
          name: main
          resources: {}
    - name: end
      taskSpec:
        metadata:
          annotations:
            pipelines.kubeflow.org/component_spec_digest: '{"name": "End", "outputs":
              [{"name": "Output", "type": "String"}], "version": "End@sha256=fc4ae432a3c7fecc61c191f63601c267dac41344940769319845daff355642d2"}'
          labels:
            pipelines.kubeflow.org/cache_enabled: "true"
        results:
        - description: /tmp/outputs/Output/data
          name: Output
          type: string
        spec: null
        steps:
        - args:
          - '----output-paths'
          - $(results.Output.path)
          command:
          - sh
          - -ec
          - |
            program_path=$(mktemp)
            printf "%s" "$0" > "$program_path"
            python3 -u "$program_path" "$@"
          - |
            def end():
                return '8'

            def _serialize_str(str_value: str) -> str:
                if not isinstance(str_value, str):
                    raise TypeError('Value "{}" has type "{}" instead of str.'.format(
                        str(str_value), str(type(str_value))))
                return str_value

            import argparse
            _parser = argparse.ArgumentParser(prog='End', description='')
            _parser.add_argument("----output-paths", dest="_output_paths", type=str, nargs=1)
            _parsed_args = vars(_parser.parse_args())
            _output_files = _parsed_args.pop("_output_paths", [])

            _outputs = end(**_parsed_args)

            _outputs = [_outputs]

            _output_serializers = [
                _serialize_str,

            ]

            import os
            for idx, output_file in enumerate(_output_files):
                try:
                    os.makedirs(os.path.dirname(output_file))
                except OSError:
                    pass
                with open(output_file, 'w') as f:
                    f.write(_output_serializers[idx](_outputs[idx]))
          image: python:alpine3.6
          name: main
          resources: {}
    - name: print-msg
      params:
      - name: end
        value: $(tasks.end.results.Output)
      - name: start
        value: $(tasks.start.results.Output)
      - name: step
        value: $(tasks.step.results.Output)
      taskSpec:
        metadata:
          annotations:
            pipelines.kubeflow.org/component_spec_digest: '{"name": "Print msg", "outputs":
              [], "version": "Print msg@sha256=ed3c7f41a60c7f07d32b7a3fac26e50b0231d7c92ea25c722a9f887d0d8fd084"}'
          labels:
            pipelines.kubeflow.org/cache_enabled: "true"
        params:
        - name: end
          type: string
        - name: start
          type: string
        - name: step
          type: string
        spec: null
        steps:
        - args:
          - --msg1
          - $(inputs.params.start)
          - --msg2
          - $(inputs.params.step)
          - --msg3
          - $(inputs.params.end)
          command:
          - sh
          - -ec
          - |
            program_path=$(mktemp)
            printf "%s" "$0" > "$program_path"
            python3 -u "$program_path" "$@"
          - |
            def print_msg(msg1, msg2, msg3):
                """Print a message."""
                print(msg1)
                print(msg2)
                print(msg3)

            import argparse
            _parser = argparse.ArgumentParser(prog='Print msg', description='Print a message.')
            _parser.add_argument("--msg1", dest="msg1", type=str, required=True, default=argparse.SUPPRESS)
            _parser.add_argument("--msg2", dest="msg2", type=str, required=True, default=argparse.SUPPRESS)
            _parser.add_argument("--msg3", dest="msg3", type=str, required=True, default=argparse.SUPPRESS)
            _parsed_args = vars(_parser.parse_args())

            _outputs = print_msg(**_parsed_args)
          image: python:3.7
          name: main
          resources: {}
    - name: print-msg-2
      params:
      - name: end
        value: $(tasks.end.results.Output)
      - name: start
        value: $(tasks.start.results.Output)
      - name: step
        value: $(tasks.step.results.Output)
      taskSpec:
        metadata:
          annotations:
            pipelines.kubeflow.org/component_spec_digest: '{"name": "Print msg", "outputs":
              [], "version": "Print msg@sha256=ed3c7f41a60c7f07d32b7a3fac26e50b0231d7c92ea25c722a9f887d0d8fd084"}'
          labels:
            pipelines.kubeflow.org/cache_enabled: "true"
        params:
        - name: end
          type: string
        - name: start
          type: string
        - name: step
          type: string
        spec: null
        steps:
        - args:
          - --msg1
          - $(inputs.params.start)
          - --msg2
          - $(inputs.params.step)
          - --msg3
          - $(inputs.params.end)
          command:
          - sh
          - -ec
          - |
            program_path=$(mktemp)
            printf "%s" "$0" > "$program_path"
            python3 -u "$program_path" "$@"
          - |
            def print_msg(msg1, msg2, msg3):
                """Print a message."""
                print(msg1)
                print(msg2)
                print(msg3)

            import argparse
            _parser = argparse.ArgumentParser(prog='Print msg', description='Print a message.')
            _parser.add_argument("--msg1", dest="msg1", type=str, required=True, default=argparse.SUPPRESS)
            _parser.add_argument("--msg2", dest="msg2", type=str, required=True, default=argparse.SUPPRESS)
            _parser.add_argument("--msg3", dest="msg3", type=str, required=True, default=argparse.SUPPRESS)
            _parsed_args = vars(_parser.parse_args())

            _outputs = print_msg(**_parsed_args)
          image: python:3.7
          name: main
          resources: {}
  startTime: "2023-01-12T13:34:01Z"

Additional Info

  • Kubernetes version:

Client Version: v1.26.0

  • Tekton Pipeline version:
    0.41
@typeness typeness added the kind/bug Categorizes issue or PR as related to a bug. label Jan 12, 2023
@chengjoey
Copy link
Member

chengjoey commented Jan 17, 2023

hi @typeness ,i think your pipelinerun yml that provided should add runAfter in task print-msg and print-msg-2
runAfter:

runAfter:
    - end
    - start
    - step

after modified, the pipelinerun yml should be like this:

apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
  labels:
    pipelines.kubeflow.org/generation: ""
    pipelines.kubeflow.org/pipelinename: ""
    tekton.dev/pipeline: pipeline2
  name: pipeline2
  namespace: default
spec:
  params:
  - name: end
    value: c
  - name: start
    value: asdfsdf
  - name: step
    value: bsdfsd
  pipelineSpec:
    params:
    - default: c
      name: end
      type: string
    - default: asdfsdf
      name: start
      type: string
    - default: bsdfsd
      name: step
      type: string
    tasks:
    - name: start
      taskSpec:
        results:
        - description: /tmp/outputs/Output/data
          name: Output
          type: string
        steps:
        - args:
          - '----output-paths'
          - $(results.Output.path)
          command:
          - sh
          - -ec
          - |
            program_path=$(mktemp)
            printf "%s" "$0" > "$program_path"
            python3 -u "$program_path" "$@"
          - |
            def start():
                return '2'

            def _serialize_str(str_value: str) -> str:
                if not isinstance(str_value, str):
                    raise TypeError('Value "{}" has type "{}" instead of str.'.format(
                        str(str_value), str(type(str_value))))
                return str_value

            import argparse
            _parser = argparse.ArgumentParser(prog='Start', description='')
            _parser.add_argument("----output-paths", dest="_output_paths", type=str, nargs=1)
            _parsed_args = vars(_parser.parse_args())
            _output_files = _parsed_args.pop("_output_paths", [])

            _outputs = start(**_parsed_args)

            _outputs = [_outputs]

            _output_serializers = [
                _serialize_str,

            ]

            import os
            for idx, output_file in enumerate(_output_files):
                try:
                    os.makedirs(os.path.dirname(output_file))
                except OSError:
                    pass
                with open(output_file, 'w') as f:
                    f.write(_output_serializers[idx](_outputs[idx]))
          image: python:alpine3.6
          name: main
          resources: {}
    - name: step
      taskSpec:
        results:
        - description: /tmp/outputs/Output/data
          name: Output
          type: string
        steps:
        - args:
          - '----output-paths'
          - $(results.Output.path)
          command:
          - sh
          - -ec
          - |
            program_path=$(mktemp)
            printf "%s" "$0" > "$program_path"
            python3 -u "$program_path" "$@"
          - |
            def step():
                return '3'

            def _serialize_str(str_value: str) -> str:
                if not isinstance(str_value, str):
                    raise TypeError('Value "{}" has type "{}" instead of str.'.format(
                        str(str_value), str(type(str_value))))
                return str_value

            import argparse
            _parser = argparse.ArgumentParser(prog='Step', description='')
            _parser.add_argument("----output-paths", dest="_output_paths", type=str, nargs=1)
            _parsed_args = vars(_parser.parse_args())
            _output_files = _parsed_args.pop("_output_paths", [])

            _outputs = step(**_parsed_args)

            _outputs = [_outputs]

            _output_serializers = [
                _serialize_str,

            ]

            import os
            for idx, output_file in enumerate(_output_files):
                try:
                    os.makedirs(os.path.dirname(output_file))
                except OSError:
                    pass
                with open(output_file, 'w') as f:
                    f.write(_output_serializers[idx](_outputs[idx]))
          image: python:alpine3.6
          name: main
          resources: {}
    - name: end
      taskSpec:
        results:
        - description: /tmp/outputs/Output/data
          name: Output
          type: string
        steps:
        - args:
          - '----output-paths'
          - $(results.Output.path)
          command:
          - sh
          - -ec
          - |
            program_path=$(mktemp)
            printf "%s" "$0" > "$program_path"
            python3 -u "$program_path" "$@"
          - |
            def end():
                return '8'

            def _serialize_str(str_value: str) -> str:
                if not isinstance(str_value, str):
                    raise TypeError('Value "{}" has type "{}" instead of str.'.format(
                        str(str_value), str(type(str_value))))
                return str_value

            import argparse
            _parser = argparse.ArgumentParser(prog='End', description='')
            _parser.add_argument("----output-paths", dest="_output_paths", type=str, nargs=1)
            _parsed_args = vars(_parser.parse_args())
            _output_files = _parsed_args.pop("_output_paths", [])

            _outputs = end(**_parsed_args)

            _outputs = [_outputs]

            _output_serializers = [
                _serialize_str,

            ]

            import os
            for idx, output_file in enumerate(_output_files):
                try:
                    os.makedirs(os.path.dirname(output_file))
                except OSError:
                    pass
                with open(output_file, 'w') as f:
                    f.write(_output_serializers[idx](_outputs[idx]))
          image: python:alpine3.6
          name: main
          resources: {}
    - name: print-msg
      runAfter:
        - end
        - start
        - step
      params:
      - name: end
        value: $(tasks.end.results.Output)
      - name: start
        value: $(tasks.start.results.Output)
      - name: step
        value: $(tasks.step.results.Output)
      taskSpec:
        params:
        - name: end
          type: string
        - name: start
          type: string
        - name: step
          type: string
        steps:
        - args:
          - --msg1
          - $(inputs.params.start)
          - --msg2
          - $(inputs.params.step)
          - --msg3
          - $(inputs.params.end)
          command:
          - sh
          - -ec
          - |
            program_path=$(mktemp)
            printf "%s" "$0" > "$program_path"
            python3 -u "$program_path" "$@"
          - |
            def print_msg(msg1, msg2, msg3):
                """Print a message."""
                print(msg1)
                print(msg2)
                print(msg3)

            import argparse
            _parser = argparse.ArgumentParser(prog='Print msg', description='Print a message.')
            _parser.add_argument("--msg1", dest="msg1", type=str, required=True, default=argparse.SUPPRESS)
            _parser.add_argument("--msg2", dest="msg2", type=str, required=True, default=argparse.SUPPRESS)
            _parser.add_argument("--msg3", dest="msg3", type=str, required=True, default=argparse.SUPPRESS)
            _parsed_args = vars(_parser.parse_args())

            _outputs = print_msg(**_parsed_args)
          image: python:3.7
          name: main
          resources: {}
    - name: print-msg-2
      runAfter:
        - end
        - start
        - step
      params:
      - name: end
        value: $(params.end)
      - name: start
        value: $(params.start)
      - name: step
        value: $(params.step)
      taskSpec:
        params:
        - name: end
          type: string
        - name: start
          type: string
        - name: step
          type: string
        steps:
        - args:
          - --msg1
          - $(inputs.params.start)
          - --msg2
          - $(inputs.params.step)
          - --msg3
          - $(inputs.params.end)
          command:
          - sh
          - -ec
          - |
            program_path=$(mktemp)
            printf "%s" "$0" > "$program_path"
            python3 -u "$program_path" "$@"
          - |
            def print_msg(msg1, msg2, msg3):
                """Print a message."""
                print(msg1)
                print(msg2)
                print(msg3)

            import argparse
            _parser = argparse.ArgumentParser(prog='Print msg', description='Print a message.')
            _parser.add_argument("--msg1", dest="msg1", type=str, required=True, default=argparse.SUPPRESS)
            _parser.add_argument("--msg2", dest="msg2", type=str, required=True, default=argparse.SUPPRESS)
            _parser.add_argument("--msg3", dest="msg3", type=str, required=True, default=argparse.SUPPRESS)
            _parsed_args = vars(_parser.parse_args())

            _outputs = print_msg(**_parsed_args)
          image: python:3.7
          name: main

after executed, will get the result:
image

If you move the task print-msg-2 before print-msg, will get the result:
image

@typeness
Copy link
Author

Hello @chengjoey, thanks for your comment. As I understand you provided a workaround but still consider this a valid issue. The pipeline used to work in Tekton 0.36

@chengjoey
Copy link
Member

chengjoey commented Jan 18, 2023

Hello @chengjoey, thanks for your comment. As I understand you provided a workaround but still consider this a valid issue. The pipeline used to work in Tekton 0.36

In Tekton 0.36, if you set enable-api-fields as alpha, will also get incorrectly parameters. I agree with your opinion, this is a valid issue, because print-msg and pring-msg-2 are two parallel tasks,changing the order of definitions should not get in different parameters, i'm willing to try to fix it

@rafalbigaj
Copy link
Contributor

Thanks @chengjoey for quick turn around. Is it at all possible to backport this fix to 0.41.x?

@chengjoey
Copy link
Member

Thanks @chengjoey for quick turn around. Is it at all possible to backport this fix to 0.41.x?

Yes, the cherry-pick has been opened on #6032

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Categorizes issue or PR as related to a bug.
Projects
Status: Done
Development

Successfully merging a pull request may close this issue.

3 participants