Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

zstandard compression for parameters and results #5995

Open
wants to merge 70 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
70 commits
Select commit Hold shift + click to select a range
e6680e4
initial zstandard compression for parameters and results; also remove…
guzzijones Jun 22, 2023
4ffe612
fix test_executions_fixtures.py tests
guzzijones Jun 23, 2023
10d0be6
index liveaction
guzzijones Jun 23, 2023
d9acc79
fix action_chain unit testing
guzzijones Jun 23, 2023
155a2b2
fix execution1.yaml fixture for rule enforcement testing
guzzijones Jun 23, 2023
d68660b
black reformat, fix log format string
guzzijones Jun 23, 2023
b1083f6
fix test_garbage_collector.py integration test
guzzijones Jun 23, 2023
354108b
black formatting
guzzijones Jun 23, 2023
5e3779b
black fixes for contrib
guzzijones Jun 23, 2023
5be68a1
fix lint errors
guzzijones Jun 23, 2023
8cad290
fix migration test for 3.5
guzzijones Jul 3, 2023
c65b90b
black format v35 unit test
guzzijones Jul 3, 2023
3a9d22d
set actual byte size in result_size field instead of compressed size
guzzijones Jul 3, 2023
9aeb080
setting maxDiff to see errors
guzzijones Jul 3, 2023
3903313
remove traceback
guzzijones Jul 3, 2023
1efd870
change log entry
guzzijones Jul 3, 2023
b9e8c1c
use errors list instead of original execution.errors
guzzijones Jul 3, 2023
61e976d
test each error individually
guzzijones Jul 3, 2023
18d5343
benchmarks and integration test
guzzijones Jul 5, 2023
33ae861
black fixes
guzzijones Jul 5, 2023
c6a07a2
fix lint error
guzzijones Jul 5, 2023
7d8a4a8
fix output in integration test
guzzijones Jul 5, 2023
84bd96d
add migration for liveaction
guzzijones Jul 5, 2023
bcc5b0e
remove sym link migration
guzzijones Jul 5, 2023
3123fac
add migration script for execution db liveaction
guzzijones Jul 5, 2023
12ac1cc
add config option to turn off zstandard compression
guzzijones Jul 6, 2023
fb1872d
black formatting
guzzijones Jul 6, 2023
20283bd
fix version on migration script
guzzijones Jul 6, 2023
f4c5b42
update sample config
guzzijones Jul 6, 2023
05970fa
initial header compression
guzzijones Jul 13, 2023
dd1c768
add new conf setting for zstandard
guzzijones Jul 13, 2023
b09b01e
add ability to change compression via config setting
guzzijones Jul 13, 2023
9a0e74f
black changes
guzzijones Jul 13, 2023
08f018c
change liveaction to liveaction_id; also move inquiry mask code into …
guzzijones Jul 14, 2023
97a53dd
remove liveaction from action execution api
guzzijones Jul 14, 2023
f12b829
Update st2common/bin/migrations/v3.9/st2-migrate-liveaction-executiondb
guzzijones Jul 14, 2023
6bca0bd
Update st2common/bin/migrations/v3.9/st2-migrate-liveaction-executiondb
guzzijones Jul 14, 2023
de4bad1
black fixes
guzzijones Jul 14, 2023
f08e2bf
untested migration script
guzzijones Jul 14, 2023
461e58e
flake fixes
guzzijones Jul 14, 2023
4b6f7b8
fix test actionchain liveaction
guzzijones Jul 14, 2023
babe63f
import fixes
guzzijones Jul 14, 2023
cd5fc5d
fix inquiry ttl liveaction_id
guzzijones Jul 14, 2023
3e3a9b7
zipp <=3.16 for python 3.6 compatibility
guzzijones Jul 14, 2023
98952aa
pin zipp < 3.16
guzzijones Jul 14, 2023
f948b96
liveaction return inside actionexecution api
guzzijones Jul 17, 2023
92b855f
fix unit tests to allow embedded liveaction
guzzijones Jul 17, 2023
ab33556
fix stream test
guzzijones Jul 17, 2023
2d0ca55
add back array params test for alias execution
guzzijones Jul 17, 2023
e34e3a5
migration script change
guzzijones Jul 17, 2023
eb2bbb5
compress and uncompress methods
guzzijones Jul 19, 2023
4e14d55
Update st2common/st2common/models/api/execution.py
guzzijones Jul 18, 2023
b7b3150
fix import error compress and uncompress
guzzijones Jul 19, 2023
af8bc86
add test for st2.inquiry.respond secret masking
guzzijones Jul 19, 2023
f189711
black formatting fix
guzzijones Jul 19, 2023
c81a131
lint fixes
guzzijones Jul 19, 2023
53b9e59
fix migration
guzzijones Jul 19, 2023
88590bc
add inheritance
guzzijones Jul 19, 2023
64bfefa
pymongo query for setting liveaction id
guzzijones Jul 19, 2023
e3a2df9
working migration script
guzzijones Jul 19, 2023
0d3b882
black fix
guzzijones Jul 19, 2023
ff00138
add required fields for liveaction in case where liveaction cannot be…
guzzijones Jul 19, 2023
84df1ab
remove breakpoint
guzzijones Jul 19, 2023
645acb5
migrate paused inquiries
guzzijones Jul 31, 2023
e640f1b
Update st2common/st2common/fields.py
guzzijones Jul 23, 2023
68a844b
black fix migration script
guzzijones Aug 2, 2023
92ae379
add pending actions to cover inquiries to migration
guzzijones Oct 10, 2023
798bd78
black and lint fixes
guzzijones Oct 10, 2023
19bcf81
make requirements
guzzijones Nov 19, 2024
2720022
fix test hanging due to monkey patch needed
guzzijones Nov 20, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
black reformat, fix log format string
  • Loading branch information
guzzijones committed Nov 19, 2024
commit d68660b0622b5e80d733731bb694c35574441720
5 changes: 4 additions & 1 deletion st2actions/st2actions/container/base.py
Original file line number Diff line number Diff line change
Expand Up @@ -145,7 +145,10 @@ def _do_run(self, runner):
# mark execution as failed.
status = action_constants.LIVEACTION_STATUS_FAILED
# include the error message and traceback to try and provide some hints.
LOG.exception("Failed to run action. traceback: %s".format("".join(traceback.format_tb(tb, 20))))
LOG.exception(
"Failed to run action. traceback: %s"
% "".join(traceback.format_tb(tb, 20))
)
result = {
"error": str(ex),
"traceback": "".join(traceback.format_tb(tb, 20)),
Expand Down
37 changes: 20 additions & 17 deletions st2actions/st2actions/policies/concurrency_by_attr.py
Original file line number Diff line number Diff line change
Expand Up @@ -44,35 +44,38 @@ def _apply_before(self, target):

scheduled_filters = {
"status": action_constants.LIVEACTION_STATUS_SCHEDULED,
"action": target.action
"action": target.action,
}
scheduled = [i for i in
action_access.LiveAction.query(**scheduled_filters)]
scheduled = [i for i in action_access.LiveAction.query(**scheduled_filters)]

running_filters = {
"status": action_constants.LIVEACTION_STATUS_RUNNING,
"action": target.action
"action": target.action,
}
running = [i for i in
action_access.LiveAction.query(**running_filters)]
running = [i for i in action_access.LiveAction.query(**running_filters)]
running.extend(scheduled)
count = 0
target_parameters = JSONDictEscapedFieldCompatibilityField(
).parse_field_value(target.parameters)
target_parameters = JSONDictEscapedFieldCompatibilityField().parse_field_value(
target.parameters
)
target_key_value_policy_attributes = {
k: v for k, v in
target_parameters.items() if k in self.attributes}
k: v for k, v in target_parameters.items() if k in self.attributes
}

for i in running:
running_event_parameters = \
JSONDictEscapedFieldCompatibilityField(
).parse_field_value(i.parameters)
running_event_parameters = (
JSONDictEscapedFieldCompatibilityField().parse_field_value(i.parameters)
)
# list of event parameter values that are also in policy
running_event_policy_item_key_value_attributes = {
k: v for k, v in
running_event_parameters.items() if k in self.attributes}
if running_event_policy_item_key_value_attributes == \
target_key_value_policy_attributes:
k: v
for k, v in running_event_parameters.items()
if k in self.attributes
}
if (
running_event_policy_item_key_value_attributes
== target_key_value_policy_attributes
):
count += 1

# Mark the execution as scheduled if threshold is not reached or delayed otherwise.
Expand Down