Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support perfect retriever evaluation in Reader nodes #1848

Closed
tstadel opened this issue Dec 6, 2021 · 0 comments · Fixed by #1962
Closed

Support perfect retriever evaluation in Reader nodes #1848

tstadel opened this issue Dec 6, 2021 · 0 comments · Fixed by #1962
Assignees
Labels
topic:eval type:feature New feature or request

Comments

@tstadel
Copy link
Member

tstadel commented Dec 6, 2021

According to #1719 in pipeline.eval(), we want to evaluate reader nodes besides the usual pipeline evaluation as if there was a perfect retriever sitting in front of them. Needs to be done:

  • extend pipeline.eval() to receive a parameter that enables perfect retriever evaluation
  • extend Reader.run() to receive the parameter enabling perfect retriever evaluation and the labels
  • implement a second predict pass that operates on perfect documents from the labels
  • store result of second predict pass in result dict
  • ensure that second predict result is stored in _debug output
  • extract second predict result in pipeline.eval() and write it to dataframe using the node_input column from Support more than one answer/document output per node in pipeline.eval() #1850 to distinguish it from default results
  • extend pipeline.print_eval_report to show metrics of multiple node_inputs if provided
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
topic:eval type:feature New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants