sensor_service.list_value() doesn't return all the keys macthing the prefix

[anirudhbagri]

SUMMARY

Inside a PollingSensor, I am trying to fetch all the key from a datastore which starts with a prefix 'entity_' like this:

self.sensor_service.list_value(prefix="entity_")

However, if the datastore contains several (or at least more than 100, that I could test) the method doesn't return all the keys.
Is there a pagination here? If yes, how can we get the entire list in one go? Or at least fetch the next set of keys?

STACKSTORM VERSION

st2 3.3.0, on Python 3.6.9

OS, environment, install method

Post what OS you are running this on, along with any other relevant information/

Linux

Steps to reproduce the problem

Use any sensor, store 100+ keys with the same prefix in the datastore.
Next, try to fetch all the keys stored with the same prefix.

Expected Results

I expect to get a list of all the keys that have the common prefix or have a way to get the entire list.

Actual Results

The code mentioned didn't return all the keys that were expected.

Thanks!

[jamesdreid]

Ran into the same issue and had to implement a separate call to the API endpoint to get around it. The sensor code uses the ST2 client to get the keys and the ST2 client supports an option to request all the key values (-1) but this is not exposed to the method in the sensor code.
Duplicate of #5097

[anirudhbagri]

Can you share more details on the workaround? What is the separate call to API endpoint?

[jamesdreid]

I cannot share the actual code as it was part of a custom sensor we wrote for an internal project, but we basically used Requests to hit the ST2 API endpoint to get the keys (exactly the same way the st2 client gets this data):

https://api.stackstorm.com/api/v1/keys/#/key_value_pair_controller.get_all

With the API endpoint, you can pass the "limit=-1" parameter in the URL to have the API server pass all the values from the keystore and overcome the API server limit set to 100.

The test URL we have in Postman that would work like you need would be as follows:

https://ST2-API-ENDPOINT-URL/api/v1/keys?prefix=entity_&limit=-1

[anirudhbagri]

That's great! Thank you :)