[Weaviate] Exit the while loop when we query less documents than available

[masci]

fixes #2517

When a query returns less documents than available in the store (e.g. when there are more documents than QUERY_MAXIMUM_RESULTS (see Weaviate docs), the while loop paginating the results never exits. This is because the offset increases as we fetch more documents, but after QUERY_MAXIMUM_RESULTS docs, the query starts returning an empty list, the offset stays the same and we get stuck.

Proposed changes:

• While paginating results, we exit the loop if the query returns an empty list of documents.

Status (please check what you already did):

• First draft (up for discussions & feedback)
• Final code
• Enable actions on your fork
• Added tests
• Updated documentation

[ZanSara]

Changes look good 🙂

I wonder if a piece is missing though. Although the original issue is solved (get_all_documents work regardless of QUERY_MAXIMUM_RESULTS), still it would be confusing for users not to be able to query more than 10k docs no matter what. We could:

• Add a very loud warning if get_documents_count is above QUERY_MAXIMUM_RESULTS, or
• Add a very loud warning if a user tries to write more than QUERY_MAXIMUM_RESULTS into Weaviate.

In both cases, the user is at least aware that this might be an issue: I have the impression most Haystack users are not aware of this specific Weaviate quirk and would benefit a hint.

I think this is a small change worth adding here. Let me know if you prefer a separate PR for it, in this case I will approve this one.

[masci]

We could:

• Add a very loud warning if get_documents_count is above QUERY_MAXIMUM_RESULTS, or
• Add a very loud warning if a user tries to write more than QUERY_MAXIMUM_RESULTS into Weaviate.

I went with the former because it looks less obtrusive.

[ZanSara]

All good! I left a couple of small comments but nothing important 😊