feat: extend component run info

[tstadel]

Related Issues

• fixes logging component run info via logging extra in addition to tracing + adds additional information (e.g. input and output lengths).

Background:
Currently there is little to no information about a component's run invocation in our logs. We get a log Running component writer without any further information. Additional information is only available via tracing. Traces are not always available or suitable in certain situations as for example user-facing logs.

Proposed Changes:

• emit component run info via logging extra in addition to tracing
• add input lengths
• add output types
• add output lengths

How did you test it?

Notes for the reviewer

Checklist

• I have read the contributors guidelines and the code of conduct
• I have updated the related issue with new insights and changes
• I added unit tests and updated the docstrings
• I've used one of the conventional commit types for my PR title: fix:, feat:, build:, chore:, ci:, docs:, style:, refactor:, perf:, test:.
• I documented my code
• I ran pre-commit hooks and fixed any issue

[silvanocerza]

Are inputs and outputs lengths really necessary? Which use case would stop people from calculating it themselves? 🤔

[tstadel]

Are inputs and outputs lengths really necessary? Which use case would stop people from calculating it themselves? 🤔

@silvanocerza Not sure what you mean with "calculating it themselves" exactly.
Input and output lengths can vary heavily througout the pipeline and additonally can depend heavily on the actual input.
For example take the following pipeline in which you

• start with one file (e.g. a pdf) -> 1 path
• split the pdf into 1 image per page -> n_pages images
• run OCR using an llm per page -> n_pages documents
• build strides of 3 pages -> n_pages-2 strides
• extract some information for each stride using llm -> n_pages-2 information
• join all information -> n_information
• write the information to document_store

To actually monitor progress and to decide whether an invocation takes too long or is within bounds you need to know input and output lengths of the individual components. OCR might be quick for 5 pages, but take an hour for 1k pages.

[coveralls]

Pull Request Test Coverage Report for Build 11178179903

Details

• 0 of 0 changed or added relevant lines in 0 files are covered.
• 17 unchanged lines in 1 file lost coverage.
• Overall coverage increased (+0.01%) to 90.312%

---

Files with Coverage Reduction	New Missed Lines	%
core/pipeline/pipeline.py	17	81.63%

Totals
Change from base Build 11144701295:	0.01%
Covered Lines:	7467
Relevant Lines:	8268

---

💛 - Coveralls

[shadeMe]

I'm afraid this change falls outside the purview of the the Pipeline class. The current tracing hooks expose the inputs of each component run invocation, and that's the extent of the class' involvement - It's up to the consumer of the tracing instrumentation to house the logic that reasons about those inputs.

We can additionally log the parameters that we currently passing to the tracing instrumentation (and add the output types to that), but logic to infer lengths doesn't belong here.

[tstadel]

I'm afraid this change falls outside the purview of the the Pipeline class. The current tracing hooks expose the inputs of each component run invocation, and that's the extent of the class' involvement - It's up to the consumer of the tracing instrumentation to house the logic that reasons about those inputs.

We can additionally log the parameters that we currently passing to the tracing instrumentation (and add the output types to that), but logic to infer lengths doesn't belong here.

@shadeMe So how would I log the number of input and output items (which I believe is an important number for debugging/monitoring pipelines)?

[shadeMe]

@shadeMe So how would I log the number of input and output items (which I believe is an important number for debugging/monitoring pipelines)?

Not sure what your constraints are, but perhaps by creating a custom tracer that inspects the inputs and output and writes their characteristics to to the logs? The haystack.tracing module should have the required scaffolding.

On second thought, such a tracer would be a useful tool for Haystack users too.

[tstadel]

@shadeMe So how would I log the number of input and output items (which I believe is an important number for debugging/monitoring pipelines)?

Not sure what your constraints are, but perhaps by creating a custom tracer that inspects the inputs and output and writes their characteristics to to the logs? The haystack.tracing module should have the required scaffolding.

On second thought, such a tracer would be a useful tool for Haystack users too.

@shadeMe If the LoggingTracer is the way to go, do you think adding the extra to the existing log lines in this PR makes still sense?

[shadeMe]

@shadeMe If the LoggingTracer is the way to go, do you think adding the extra to the existing log lines in this PR makes still sense?

I don't see an issue with logging the input and output types, if only for completeness' sake.

[silvanocerza]

LoggingTracer introduced with #8447 should solve the problem this PR is trying to solve. Closing.