feat: Add SentenceTransformersDiversityRanker

[awinml]

Related Issues

• fixes DiversityRanker #7094

Proposed Changes:

Adds SentenceTransformersDiversityRanker.

The Diversity Ranker orders documents in such a way as to maximize the overall diversity of the given documents. The ranker leverages sentence-transformer models to calculate semantic embeddings for each document and the query.

The ranker first calculates embeddings for each document and the query. It starts by selecting the document that is semantically closest to the query. Then, for each remaining document, it selects the one that, on average, is least similar to the already selected documents. This process continues until all documents are selected, resulting in a list where each subsequent document contributes the most to the overall diversity of the selected set.

How did you test it?

Tests have been added in test_sentence_transformers_diversity.py.

Checklist

• I have read the contributors guidelines and the code of conduct
• I have updated the related issue with new insights and changes
• I added unit tests and updated the docstrings
• I've used one of the conventional commit types for my PR title: fix:, feat:, build:, chore:, ci:, docs:, style:, refactor:, perf:, test:.
• I documented my code
• I ran pre-commit hooks and fixed any issue

[coveralls]

Pull Request Test Coverage Report for Build 8232016781

Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

• For more information on this, see Tracking coverage changes with pull request builds.
• To avoid this issue with future PRs, see these Recommended CI Configurations.
• For a quick fix, rebase this PR at GitHub. Your next report should be accurate.

Details

• 0 of 0 changed or added relevant lines in 0 files are covered.
• No unchanged relevant lines lost coverage.
• Overall coverage increased (+0.1%) to 90.176%

---

Totals
Change from base Build 8190601051:	0.1%
Covered Lines:	5370
Relevant Lines:	5955

---

💛 - Coveralls

[sjrl]

Thanks for the work on this, I did a quick look over and had two comments that I wanted to bring up.

[vblagoje]

@sjrl can you please take over, I'm overloaded with some other unrelated items 🙏

[sjrl]

Yup I can review this more later this week.

[awinml]

@sjrl All the tests, except those running the component on real data, have been converted to unit tests. Mocks have been used for the sentence-transformer model. An additional test for warm_up() has also been added. These improvements now bring the test coverage for this component to 100%.

[sjrl]

Thanks a lot @awinml this looks great!