Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix execution summary for optional parameters #3286

Merged
merged 2 commits into from
May 8, 2024

Conversation

starhel
Copy link
Contributor

@starhel starhel commented May 7, 2024

Description

If an optional parameter has its own next_in_enumeration method, generating a summary ends in an error if the same task has been run both with a value of None and with a proper value for the parameter.

Motivation and Context

import luigi


class A(luigi.ExternalTask):
    def output(self):
        return luigi.LocalTarget("a.txt")


class RecB(luigi.Task):
    step = luigi.OptionalIntParameter(default=None)

    def requires(self):
        step = 3 if self.step is None else self.step
        if step == 0:
            yield self.clone(A)
        else:
            yield self.clone(step=step-1)

    def run(self):
        with self.output().open("w") as f:
            f.write("1")


    def output(self):
        return luigi.LocalTarget(f"b_{self.step}.txt")


luigi.build([RecB()])
Traceback (most recent call last):
  File "/home/astachlewski/tmp/luigi_bug_recursive_graph/bug.py", line 28, in <module>
    luigi.build([RecB()])
  File "/home/astachlewski/tmp/luigi_bug_recursive_graph/venv/lib/python3.10/site-packages/luigi/interface.py", line 243, in build
    luigi_run_result = _schedule_and_run(tasks, worker_scheduler_factory, override_defaults=env_params)
  File "/home/astachlewski/tmp/luigi_bug_recursive_graph/venv/lib/python3.10/site-packages/luigi/interface.py", line 178, in _schedule_and_run
    luigi_run_result = LuigiRunResult(worker, success)
  File "/home/astachlewski/tmp/luigi_bug_recursive_graph/venv/lib/python3.10/site-packages/luigi/execution_summary.py", line 79, in __init__
    self.summary_text = _summary_wrap(_summary_format(summary_dict, worker))
  File "/home/astachlewski/tmp/luigi_bug_recursive_graph/venv/lib/python3.10/site-packages/luigi/execution_summary.py", line 414, in _summary_format
    str_output += '{0}\n'.format(_get_str(group_tasks[status], status in _PENDING_SUB_STATUSES))
  File "/home/astachlewski/tmp/luigi_bug_recursive_graph/venv/lib/python3.10/site-packages/luigi/execution_summary.py", line 196, in _get_str
    first, last = _ranging_attributes(attributes, param_class)
  File "/home/astachlewski/tmp/luigi_bug_recursive_graph/venv/lib/python3.10/site-packages/luigi/execution_summary.py", line 261, in _ranging_attributes
    next_attributes = {param_class.next_in_enumeration(attribute) for attribute in attributes}
  File "/home/astachlewski/tmp/luigi_bug_recursive_graph/venv/lib/python3.10/site-packages/luigi/execution_summary.py", line 261, in <setcomp>
    next_attributes = {param_class.next_in_enumeration(attribute) for attribute in attributes}
  File "/home/astachlewski/tmp/luigi_bug_recursive_graph/venv/lib/python3.10/site-packages/luigi/parameter.py", line 686, in next_in_enumeration
    return value + 1
TypeError: unsupported operand type(s) for +: 'NoneType' and 'int'

Have you tested this? If so, how?

Automated tests as well as running the above code.

@starhel starhel requested review from dlstadther and a team as code owners May 7, 2024 19:21
@starhel
Copy link
Contributor Author

starhel commented May 7, 2024

I've also fixed some broken test for jsonschema (as order of validation is changed in newest version).

@dlstadther dlstadther merged commit 318a81d into spotify:master May 8, 2024
42 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants