Fine grain output verbosity

[plannigan]

What's the problem this feature will solve?

There are multiple open issues that are requesting more control over the output displayed by pytest. --verbose turns everything up, but it would be nice to be able to just increase or decrease the verbosity for specific aspects of the output.

Open issues that I've found that fit into this problem space (there may be others):

• Separate terminal verbosity from diff verbosity #6723
• Add distinct option to show full (assertion) diffs always #3962
• parametrize: truncate long parameters in output in -v mode, still display full value in -vv #5285
• Add option to force "short summary" despite high verbosity #6292

Describe the solution you'd like

I think a way that could resolve all of these is to add an abstraction layer on top config.option.verbose. Rough pseudo-code example:

class OutputVerbosity:
  def __init__(self, config_options):
     self._options = config_options
     self.verbose = self._options.verbose

  def verbosity_for(self, output_type):
     return getattr(self._options, output_type, self.verbose)

For general cases, the verbose attribute could be used. For special cases, an output type name could be passed in retrieve the more fine grained setting. If the specific config is not set, it will default to the normal verbose level. Once this is in place, it would be easier to add more cases in the future if they come up.

Implementation

• I should be able to work on an actual implementation. I'd likely start with only implementing diff verbosity since the PR might get large if I attempted to address all of the identified cases at once.
• That being said, I think it makes sense to decide on a naming convention for these configuration fields. verbosity_{OUTPUT_TYPE} seems reasonable to me.
• I'm leaning towards not providing CLI access to these configuration fields. They feel like they wouldn't be used regularly and it should be easy to add them after the fact if needed.

[plannigan]

Started working on this and unsurprisingly realized that how config.options.verbose is used currently is a bit complicated. So I took the time to see all the places that it is used. The following was based off 6c2feb75d (current head of main):

General Summary

The vast majority of uses of the verbose setting are within _pytest.terminal. In addition to using it directly, TerminalReporter also re-exposes it as a property. The pytest code base does not use the property outside of the TerminalReporter class.

• TerminalReporter is exposed to
  plugins as part of the pytest_terminal_summary() hook.
• The Config object is also exposed to plugins: Either directly in a hook (example) or indirectly (example)

Uses of config.options.verbose outside of TerminalReporter

• cacheprovider.pytest_report_header(): >0 display cachedir path (always displayed if not using default chachedir path)
• assertions.truncate._should_truncate_item(): truncate full explanation if < 2 or NOT running in CI
• assertions.uti.assertrepr_compare() (accessed add passed down to other function)
  • > 1: call saferepr_unlimited() for left & right expressions
  • <= 1: call saferepr for left & right expressions (maxsize depends on operator string)
  • passed into _compare_eq_any() and _notin_text() (which continue to pass down to other compare functions)
  • _diff_text(): <1 trim long sequences of leading or trailing identical characters
  • _compare_eq_cls(): <2 omit identical items
    • recurses into _compare_eq_any()
  • _compare_eq_sequence(): (passed in, but unused)
  • _compare_eq_set(): (passed in, but unused)
  • _compare_eq_dict():
    • < 2: omit identical items
    • else: list common items
  • _compare_eq_iterable(): <=0 (and NOT running in CI) don't diff pretty print of objects
• logging.pytest_runtestloop(): value forced to 1 if ((--log-cli-level is provided or log_cli is set) AND terminalreporter plugin exists)
  • I haven't looked into the deeper implications of this

Uses of config.getoption("verbose")

• cacheprovider.pytest_report_collectionfinish(): >= 0 (and active) return report status
• logging.pytest_runtestloop(): < 1
  • See config.options.verbose usage
• nodes.Node._repr_failure_py(): > 1 don't truncate locals for exception info representation.
• python.importtestmodule(): < 2 filter traceback information
• stepwise.pytest_report_collectionfinish(): >= 0 return report status
• asserstions.rewrite._get_maxsize_for_saferepr():
  • >= 2: no max size
  • >= 1: DEFAULT_REPR_MAX_SIZE * 10
  • else: DEFAULT_REPR_MAX_SIZE

Uses of config.options.verbose in TerminalReporter

• pytest_runtest_logreport():
  • <2 - possibly trimmed skipped reason
  • 2>= full skipped reason
• pytest_collection():
  • if atty terminal and >= 0 write "collecting..."
  • if >=1 write "collecting..."
• report_collect(): <0 exit early instead of writing summary count line
• pytest_collection_finish(): if --collect-only passed and the session has item, > -1 write empty line before printing collected items
• _printcollecteditems(): (only called if --collect-only)
  • < 0:
    • if < -1 write out summary of items (filename & count)
    • else write out each collected item. (filename & item name)
  • else: tree view of each item
    • >= 1: include docstring

Uses of the TerminalReporter.verbosity property

A read-only property exposed by TerminalReporter that exposes config.options.verbose

• _locationline():
  • < 2: relative path from nodeid (if fspath is non-empty string)
  • >= 2: same as above, but add " <- {bestrelpath(...)}" if fspath doesn't match node id
• pytest_runtest_logfinish(): <= 0 show line ending progress info (if showing progress info)
• pytest_runtest_logreport(): <= 0 write just the result letter instead of calling _locationline()
• pytest_sessionstart(): > 0 include "-- {executable name}" in first line of session header (or debug or pastebin)
• showfspath: return (verbosity >= 0) if self._showfspath is not set
  • only used in pytest_runtest_logstart() and only if showlongtestinfo is False
• showheader: return (verbosity >= 0)
  • only used in pytest_sessionstart()
• showlongtestinfo: return (verbosity > 0)
  • only used in pytest_runtest_logstart()
• summary_stats():
  • < -1: don't write anything
  • >= 0: include full width seperator in addition to stats message

Others

• PYTEST_DEBUG - When set, pytest will print tracing and debug information.
• --no-header & --no-summary: explicitly disable to named terminal output component.

[plannigan]

That reference is helpful to understand what things accessing verbosity and how. But it's not the easiest to understand the full behavior. So I went through and documented the behavior by verbosity level. (This helped me get a better understanding of specific aspects that can be controlled.)

-2

• assertions
  • truncate full explanation (if not in CI)
  • abbreviate left & right repr
  • string compares - trim long sequences of leading or trailing identical characters
  • data object compares - omit identical items
  • dict compares - omit identical items
  • iterable compares - diff pretty print of objects (if in CI)
• filter tracebacks
• rewrite maxsize = DEFAULT_REPR_MAX_SIZE
• (if --collect-only) write filenames and count of items in file
• write result letter instead of item location

-1

• assertions
  • truncate full explanation (if not in CI)
  • abbreviate left & right repr
  • string compares - trim long sequences of leading or trailing identical characters
  • data object compares - omit identical items
  • dict compares - omit identical items
  • iterable compares - diff pretty print of objects (if in CI)
• filter tracebacks
• rewrite maxsize = DEFAULT_REPR_MAX_SIZE
• (if --collect-only) write each item (filename and item name)
• write result letter instead of item location
• write summary stats

0

• assertions
  • truncate full explanation (if not in CI)
  • abbreviate left & right repr
  • string compares - trim long sequences of leading or trailing identical characters
  • data object compares - omit identical items
  • dict compares - omit identical items
  • iterable compares - diff pretty print of objects (if in CI)
• write cache report (if active)
• write stepwise report (if active)
• filter tracebacks
• rewrite maxsize = DEFAULT_REPR_MAX_SIZE
• (if atty terminal) write "collecting..."
• write collection report
• (if --collect-only) write tree view of each item
• write result letter instead of item location
• write line ending progress info (if showing progress info)
• write file path for module once before result letters for test cases in module
• write summary stats with full width separator

1

• write cache dir
• write executable path
• assertions
  • truncate full explanation (if not in CI)
  • abbreviate left & right repr
  • string compares - trim long sequences of leading or trailing identical characters
  • data object compares - omit identical items
  • dict compares - omit identical items
  • iterable compares - diff pretty print of objects
• write cache report (if active)
• write stepwise report (if active)
• filter tracebacks
• rewrite maxsize = DEFAULT_REPR_MAX_SIZE * 10
• write "collecting..."
• write collection report
• (if --collect-only) write tree view of each item with docstring
• write item location for each item
• write trimmed (if necessary) skip reason
• write summary stats with full width separator

2

• write cache dir
• write executable path
• assertions
  • full left & right repr
  • string compares - no trimming
  • data object compares - all items listed
  • dict compares - all items listed
  • iterable compares - diff pretty print of objects
• write cache report (if active)
• write stepwise report (if active)
• no rewrite maxsize
• write "collecting..."
• write collection report
• (if --collect-only) write tree view of each item with docstring
• write item location for each item
• write full skip reason
• write summary stats with full width separator

[plannigan]

I completely missed assertions.util on the first pass (oops). I updated the previous comments with the missing details.

[The-Compiler]

Should this really have been closed, since (from what I can gather) only a small part of the proposal is implemented so far?

[nicoddemus]

True!