Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add "no answer" aggregation to Transformersreader #259

Merged
merged 4 commits into from
Aug 6, 2020

Conversation

Timoeller
Copy link
Contributor

@Timoeller Timoeller commented Jul 23, 2020

Linked to #257 and #258

Right now the Transformersreader just gives back "no answers" for each document it predicts on.

We need to aggregate those "no answers" and their scores, similar to what the FARMReader is doing.

The Transformersreader pipeline does not seem to have an option to

  1. modify the "no answer" bias
  2. always return the "no answer" (no matter how unlikely it is)

Because of these limitations we need workarounds.

@Timoeller Timoeller requested review from tholor and removed request for tholor July 23, 2020 20:07
@Timoeller
Copy link
Contributor Author

I added the aggregation functionality.
The aggregation only works correctly when "no answer" is most likely across all documents. In all other cases the "no answer" is not correctly sorted into our list of predictions. This happens because transformers not always returns the "no answer"

@Timoeller Timoeller changed the title WIP Add "no answer" aggregation to Transformersreader Add "no answer" aggregation to Transformersreader Jul 29, 2020
@Timoeller Timoeller requested a review from tholor July 29, 2020 15:06
@Timoeller Timoeller merged commit d9e8b52 into master Aug 6, 2020
@julian-risch julian-risch deleted the aggregate_noanswer_transformersreader branch November 15, 2021 07:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant