Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix st2client @key=file_path.json notation for specifying complex objects in a file #5140

Merged
merged 6 commits into from
Feb 12, 2021

Conversation

Kami
Copy link
Member

@Kami Kami commented Feb 7, 2021

I tried to use st2 run @param=file_path.json notation to specify complex JSON parameter value in a file.

It turns out it's not working correctly. Using --debug flag it shows it's not correctly converting bytes to unicode which means JSON serialization fails (actual value hasb"" prefix which indicates that code is incorrectly calling str() on the value somewhere instead of using bytes.decode()).

st2 --debug run scalyr.timeseries_queries @queries=1.json
...
ERROR: Object of type 'bytes' is not JSON serializable
....
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2client/utils/httpclient.py", line 98, in post
    response = requests.post(self.root + url, json.dumps(data), **kwargs)
  File "/usr/lib/python3.6/json/__init__.py", line 231, in dumps
    return _default_encoder.encode(obj)
  File "/usr/lib/python3.6/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib/python3.6/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/usr/lib/python3.6/json/encoder.py", line 180, in default
    o.__class__.__name__)
TypeError: Object of type 'bytes' is not JSON serializable

This pull request fixes that.

TODO

  • Add tests
  • Add tests and verify it works correctly with http runner (that one may expect bytes, but unlikely since the current code doesn't handle that anyway)

Previously it was broken, because it didn't correctly convert bytes to
string which means all data would have b"" prefix which would break JSON
and other parsing.

This bug was likely introduced when adding support for Python 3.
@pull-request-size pull-request-size bot added the size/XS PR that changes 0-9 lines. Quick fix/merge. label Feb 7, 2021
@blag blag added the python3 label Feb 8, 2021
@Kami
Copy link
Member Author

Kami commented Feb 8, 2021

I've added tests for it (32d3c8d) and verified the functionality was indeed never meant to be used with binary data, but only string data.

In theory, we could probably also make it support binary data, but this would likely also require changes in the http layer (client, API) and I don't think anyone requests something like that before.

@pull-request-size pull-request-size bot added size/M PR that changes 30-99 lines. Good size to review. and removed size/XS PR that changes 0-9 lines. Quick fix/merge. labels Feb 8, 2021
Copy link
Member

@arm4b arm4b left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's a good find.
I bet we have more dark rarely used corners like that.

Copy link
Member

@cognifloyd cognifloyd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. I haven't looked at Travis to see why it's failing.

@blag
Copy link
Contributor

blag commented Feb 11, 2021

I can't restart the Travis-CI job for some HTTP 403 reason. I tried signing out of Travis and back in, same result. @Kami Can you try restarting the Travis build?

@Kami Kami added this to the 3.4.0 milestone Feb 11, 2021
@Kami
Copy link
Member Author

Kami commented Feb 11, 2021

Lint check is failing and it doesn't appear to be related to any of my changes:

 ************* Module st2common.models.db.execution
st2common/st2common/models/db/execution.py:101:39: E1101: Instance of 'ActionExecutionDB' has no 'id' member (no-member)
st2common/st2common/models/db/execution.py:101:39: E1101: Instance of 'ActionExecutionDB' has no 'id' member (no-member)
st2common/st2common/models/db/execution.py:101:39: E1101: Instance of 'ActionExecutionDB' has no 'id' member (no-member)
st2common/st2common/models/db/execution.py:101:39: E1101: Instance of 'ActionExecutionDB' has no 'id' member (no-member)
st2common/st2common/models/db/execution.py:101:39: E1101: Instance of 'ActionExecutionDB' has no 'id' member (no-member)
st2common/st2common/models/db/execution.py:101:39: E1101: Instance of 'ActionExecutionDB' has no 'id' member (no-member)

It seems more pylint related - e.g. our custom pylint plugin not running or similar.

@Kami Kami merged commit 1c5d07d into master Feb 12, 2021
@Kami Kami deleted the fix_st2client_at_notation branch February 12, 2021 11:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
component:st2client python3 size/M PR that changes 30-99 lines. Good size to review.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants