DC SDK - load pipeline from deepset cloud

[ArzelaAscoIi]

Proposed changes:
Deepset cloud SDK - load, run and evaluate pipelines from deepset cloud. We already have the possibility to load an existing pipeline from a yaml configuration. As another origin we can now load pipelines from deepset cloud.

Usage
Set environment variables:

DEEPSET_CLOUD_API_ENDPOINT=<future-deepset-cloud-url>
DEEPSET_CLOUD_API_KEY=<my-dc-api-key>

and run the following lines:

test_query = Pipeline.load_from_dc(pipeline_config_name="document_retrieval_1",pipeline_name="query")

Status (please check what you already did):

• First draft (up for discussions & feedback)
• Final code
• Added tests
• Updated documentation

closes #2002

[tstadel]

Hey @ArzelaAscoIi,
sorry for spoiling your PR. I should have made a pull request into your branch.
I introduced the concept of a pipeline_definition which is part of the pipeline_config (formerly known as yaml). pipeline_definition addresses exactly one pipeline.

I would suggest to keep the separation between pipeline_config and pipeline_definition in the load_from_dc method by specifying something like a pipeline_config_name or pipeline_config_id and the pipeline_name Then it feels much the same as load_from_yaml.
To emphasize the convention of having always one query and one indexing pipeline, we should mention this in the docstrings or even set the default of pipeline_name to "query". However this is just some API sugar. Under the hood Pipeline.load_from_config already takes the first pipeline if no pipeline_name has been specified and this should always be the query pipeline in DC.

[tholor]

Looking good. Left a few minor comments - mainly around documentation.

One bigger conceptual question is probably whether to stick in the future with the "two pipelines in one YAML" design. I see more and more cases where it would make more sense to define a Pipeline as the combination of 1x index and 1x query "pipeline/flow/graph".
In that case, you would always load both here, and maybe we split pipeline.run() into pipeline.run() and pipeline.index()?
Probably out of scope for this issue, but we should discuss and decide on this in the next weeks.

[CLAassistant]

All committers have signed the CLA.

[tstadel]

Good to go!