introduce node_input param

[tstadel]

Proposed changes:

• introduce node_input parameter to EvaluationResult.calculate_metrics() and EvaluationResult.wrong_examples()
• introduce node_input as eval dataframe column with default value 'prediction'
• (node_input value for 'perfect retriever' would be 'label')

Status (please check what you already did):

• First draft (up for discussions & feedback)
• Final code
• Added tests
• Updated documentation

Closes #1850

[tstadel]

@julian-risch @tholor
What do you think about the 'node_input' parameter? For me it sounds more intuitive than anything containing 'open / closed domain' for choosing different inputs we evaluated on. However, it might be harder to refer to it if we're talking about 'open / closed domain' in the docs.

[julian-risch]

I like the idea of that parameter. Maybe we can find names for the parameter values that explain both what happens (as of now with pipeline and labels) and when to use it (open / closed domain eval).
However, let's still consider alternatives. For example, we could store separate EvalResults for open and closed domain eval. In that case no change would be necessary in the calculate_metrics() and wrong_examples() method, right? Only in the pipeline.eval() method and in the generation of the eval report.

[tholor]

How would the signature of pipeline.eval() look like this with this parameter design?
Would we, by default, always create both predictions in the dataframe (from prediction and from label)?
I am just thinking where else this node_input param might appear and if it sounds intuitive in these places 🤔

---

For example, we could store separate EvalResults for open and closed domain eval. In that case no change would be necessary in the calculate_metrics() and wrong_examples() method, right? Only in the pipeline.eval() method and in the generation of the eval report.

I think there should be at least the option to run pipeline.eval() only once and have all results you need available. Returning two dataframes in this case sounds a bit unintuitive to me.

[tstadel]

How would the signature of pipeline.eval() look like this with this parameter design?
Would we, by default, always create both predictions in the dataframe (from prediction and from label)?
I am just thinking where else this node_input param might appear and if it sounds intuitive in these places

Yes, node_input might not be what we want to pass to pipeline.eval(). Here we might rather pass a flag, that enables label node_input evaluation. prediction only would be the default. So node_input is designed to be an advanced param, that you don't need in the most common way (call pipeline.eval() and pass the result to pipeline.print_eval_report(), if there is 'label' node_input it'll be used, otherwise not)

[julian-risch]

Looks good to me (as already discussed via call). 👍