feat: Add ByteStream metadata and other metadata to Documents created by HTMLToDocument

[awinml]

Related Issues

• fixes HTMLToDocument to add ByteStream metadata to Document  #6275
• fixes a part of 2.x File converters: allow passing metadata (on Document creation) #6131

Proposed Changes:

• Adds ByteStream metadata to Document metadata on creation of documents.
• Adds an optional meta parameter in the run() method, that allows the users to pass additional metadata to the Documents.

How did you test it?

Unit tests, Integration tests and manual verification.

Notes for the reviewer

Usage examples showcasing the various methods for adding metadata to the documents can be viewed in this Colab Notebook. It also has an example utilizing HTMLToDocument with LinkContentFetcher.

Checklist

• I have read the contributors guidelines and the code of conduct
• I have updated the related issue with new insights and changes
• I added unit tests and updated the docstrings
• I've used one of the conventional commit types for my PR title: fix:, feat:, build:, chore:, ci:, docs:, style:, refactor:, perf:, test:.
• I documented my code
• I ran pre-commit hooks and fixed any issue

[TuanaCelik]

Hey @awinml thanks so much for the contribution 🚀
I'll be back in a 1.5 weeks but in the meantime let's see if someone from our team will review this PR.

[vblagoje]

Hey @awinml , this looks like a good contribution; thank you! I'm wondering if you spoke with someone who asked you to add progress bars - as far as I know, we don't add them (at least for now) and having only this component with progress bars would introduce inconsistencies. If you added progress bars on your own initiative, would you please update this PR to remove them?

[awinml]

Thanks! @vblagoje

I saw that TextFileToDocument has a similar implementation for progress bars. I had also added it to MarkdownToDocument (#6159), hence thought this would be a good addition to HTMLToDocument.

I understand how this would introduce inconsistencies with the other file_converters, so I'll update the PR to remove it for now.

[agnieszka-m]

Documentation-wise, looks good!

[vblagoje]

Is there a reason these unit tests are marked as integration? If not, would you please update the integration tests into unit tests by removing the existing markers. In future, tests will be classified as unit tests by default unless marked as integration tests.

Additionally, consider using truthiness checks (e.g., if some_variable:) instead of is not None checks - if that's applicable. Seems like the code would work with both, but I just wanted to double check with you. And they are more compact and easier to read.

[awinml]

@vblagoje, You're right! I've made the updates you suggested. I have marked the tests as unit tests and replaced the is not None checks with truthiness checks as per your recommendation.

[vblagoje]

Ok, looks good, let's 🚢 this one @awinml