Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add SentenceTransformersRanker with pre-trained Cross-Encoder #1209

Merged
merged 6 commits into from
Jul 7, 2021

Conversation

julian-risch
Copy link
Member

@julian-risch julian-risch commented Jun 18, 2021

In contrast to FARMRanker, SentenceTransformerRanker uses the logit as similarity score and not the classifier's probability of label "1"
see example here: https://www.sbert.net/docs/pretrained-models/ce-msmarco.html#usage-with-transformer

I tested with a subset of the nq_dev dataset. Here are the results of a pipeline with ElasticsearchRetriever and SentenceTransformerRanker with "cross-encoder/ms-marco-MiniLM-L-12-v2" as model:

EvalRetriever
-----------------
has_answer recall@2: 0.7200 (18/25)
no_answer recall@2:  1.00 (25/25) (no_answer samples are always treated as correctly retrieved)
has_answer mean_reciprocal_rank@2: 0.6200
no_answer mean_reciprocal_rank@2:  1.0000 (no_answer samples are always treated as correctly retrieved at rank 1)
recall@2: 0.8600 (43 / 50)
mean_reciprocal_rank@2: 0.8100

Retriever (Speed)
---------------
No indexing performed via Retriever.run()
Queries Performed: 50
Query time: 0.3390099899999086s
0.0067801997999981725 seconds per query

EvalRanker
-----------------
has_answer recall@2: 0.7600 (19/25)
no_answer recall@2:  1.00 (25/25) (no_answer samples are always treated as correctly retrieved)
has_answer mean_reciprocal_rank@2: 0.6600
no_answer mean_reciprocal_rank@2:  1.0000 (no_answer samples are always treated as correctly retrieved at rank 1)
recall@2: 0.8800 (44 / 50)
mean_reciprocal_rank@2: 0.8300

Ranker (Speed)
---------------
Queries Performed: 50
Query time: 161.3018365920002s
3.226036731840004 seconds per query

Limitations: documentation on the website has not been updated. It might be unclear/confusing for users at the moment whether to use FARMRanker or SentenceTransformerRanker.

closes #1129

@julian-risch julian-risch marked this pull request as ready for review July 2, 2021 12:49
@julian-risch julian-risch requested a review from tholor July 2, 2021 12:49
@julian-risch julian-risch changed the title WIP: Add SentenceTransformersRanker with pre-trained Cross-Encoder Add SentenceTransformersRanker with pre-trained Cross-Encoder Jul 5, 2021
Copy link
Member

@tholor tholor left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the separation makes sense (at least for now). We might combine them later on and rather add an arg similarity_type to differentiate between the two different approaches.

Please add some more documentation (see comments) and a basic test case for both rankers that ensures the expected scores / sorting of some dummy docs (FARM + sentencetransformers).

@julian-risch julian-risch requested a review from tholor July 7, 2021 13:38
@julian-risch
Copy link
Member Author

@tholor Thank you for your feedback! I made the requested changes.

Copy link
Member

@tholor tholor left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking good! Thx for the changes.

I added a minor sentence to the docstring (we should keep in mind that some user might not know what Re-ranking is and explain at least very briefly the "value / use case".

Also added the import to the init so that we can just import from haystack.ranker import SentenceTransformersRanker similar to our other building blocks.

@julian-risch julian-risch merged commit dbb9efb into master Jul 7, 2021
@julian-risch julian-risch deleted the transformers_ranker branch July 7, 2021 15:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Check Re-ranking Performance
2 participants