Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improve speed by removing IO process before insert_message() #858

Merged
merged 6 commits into from
Dec 29, 2022
Merged

Improve speed by removing IO process before insert_message() #858

merged 6 commits into from
Dec 29, 2022

Conversation

fukusuket
Copy link
Collaborator

@fukusuket fukusuket commented Dec 29, 2022 •
edited
Loading

What Changed

  • use static STORED_EKEY_ALIAS instead of loading the eventkey_alias.txt just before insert_message()
    • The above process is called on every detection(to store detection results), so this file loading process impacts speed.

Evidence

Environment

  • OS: Windows 10 Home edition
  • Hard: Memory 16GB , Core 8, SSD, laptop

Benchmark

I ran a benchmark using this procedure(6.1GB evtx) and the results were as follows.

Version Elapsed time Memory(peak) Events with hits / Total events Output file size(bytes)
1.9.0 00:13:35.166 5.4 GiB 1,593,660 / 4,817,181 574775133
2.0.0 00:17:41.226 5.3 GiB 1,593,660 / 4,817,181 574775133
This PR 00:13:31.811 5.4 GiB 1,593,660 / 4,817,181 574775133

Console output

1.9.0

PS C:\tmp\hayabusa-2.0-win-64-bit> .\hayabusa-1.9.0-win-x64.exe -d ..\hayabusa-big-evtx\ -o 1.csv --debug
...
Results Summary:

Events with hits / Total events: 1,593,660 / 4,817,181 (Data reduction: 3,223,521 events (66.92%))

Total | Unique detections: 1,626,081 | 145
Total | Unique critical detections: 0 (0.00%) | 0 (0.00%)
Total | Unique high detections: 12,057 (0.74%) | 20 (13.79%)
Total | Unique medium detections: 9,957 (0.61%) | 35 (24.14%)
Total | Unique low detections: 1,053,568 (64.79%) | 40 (27.59%)
Total | Unique informational detections: 550,499 (33.85%) | 50 (34.48%)

Elapsed time: 00:13:35.166
Saved file: 1.csv (574.8 MB)
Errors were generated. Please check ./logs/errorlog-20221229_155526.log for details.

Memory usage stats:
heap stats:    peak      total      freed    current       unit      count
  reserved:    5.4 GiB    5.4 GiB   56.0 MiB    5.3 GiB                        not all freed!
 committed:    4.7 GiB    6.3 GiB    1.7 GiB    4.6 GiB                        not all freed!

2.0.0

PS C:\tmp\hayabusa-2.0-win-64-bit> .\hayabusa-2.0.0-win-x64.exe csv-timeline -d ..\hayabusa-big-evtx\ -o 1.csv --debug
...
Results Summary:

Events with hits / Total events: 1,593,660 / 4,817,181 (Data reduction: 3,223,521 events (66.92%))

Total | Unique detections: 1,626,081 | 145
Total | Unique critical detections: 0 (0.00%) | 0 (0.00%)
Total | Unique high detections: 12,057 (0.74%) | 20 (13.79%)
Total | Unique medium detections: 9,957 (0.61%) | 35 (24.14%)
Total | Unique low detections: 1,053,568 (64.79%) | 40 (27.59%)
Total | Unique informational detections: 550,499 (33.85%) | 50 (34.48%)

Saved file: 1.csv (574.8 MB)
Elapsed time: 00:17:41.226
Errors were generated. Please check ./logs/errorlog-20221229_153818.log for details.

Rule Parse Processing Time: 00:00:25.802
Analysis Processing Time: 00:16:53.993
Output Processing Time: 00:00:21.428

Memory usage stats:
heap stats:    peak      total      freed    current       unit      count
  reserved:    5.3 GiB    5.3 GiB   56.0 MiB    5.3 GiB                        not all freed!
 committed:    4.7 GiB    7.1 GiB    2.4 GiB    4.6 GiB                        not all freed!

This PR

PS C:\tmp\hayabusa-2.0-win-64-bit> .\hayabusa-dev3.exe csv-timeline -d ..\hayabusa-big-evtx\ -o 1.csv --debug
...
Results Summary:

Events with hits / Total events: 1,593,660 / 4,817,181 (Data reduction: 3,223,521 events (66.92%))

Total | Unique detections: 1,626,081 | 145
Total | Unique critical detections: 0 (0.00%) | 0 (0.00%)
Total | Unique high detections: 12,057 (0.74%) | 20 (13.79%)
Total | Unique medium detections: 9,957 (0.61%) | 35 (24.14%)
Total | Unique low detections: 1,053,568 (64.79%) | 40 (27.59%)
Total | Unique informational detections: 550,499 (33.85%) | 50 (34.48%)
...
Saved file: 1.csv (574.8 MB)
Elapsed time: 00:13:31.811
Errors were generated. Please check ./logs/errorlog-20221229_173623.log for details.

Rule Parse Processing Time: 00:00:27.321
Analysis Processing Time: 00:12:42.335
Output Processing Time: 00:00:22.153

Memory usage stats:
heap stats:    peak      total      freed    current       unit      count
  reserved:    5.4 GiB    5.4 GiB   56.0 MiB    5.3 GiB                        not all freed!

I would appreciate it if you could review🙏

@fukusuket fukusuket changed the title Improve speed by removing io process beforeinsert_message() Improve speed by removing io process before insert_message()` Dec 29, 2022
@hitenkoku hitenkoku self-requested a review December 29, 2022 10:13
@hitenkoku hitenkoku added the enhancement New feature or request label Dec 29, 2022
@fukusuket fukusuket changed the title Improve speed by removing io process before insert_message()` Improve speed by removing io process before insert_message() Dec 29, 2022
hitenkoku
hitenkoku approved these changes Dec 29, 2022
View reviewed changes
Copy link
Collaborator

@hitenkoku hitenkoku left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your pull request!

LGTM

@hitenkoku
Copy link
Collaborator

I checked 6.1GB evtx and hayabusa-sample-evtx datas

  • hayabusa-sample-evtx main
Saved file: main.csv (16.4 MB)
Elapsed time: 00:00:57.649
Errors were generated. Please check ./logs/errorlog-20221229_200858.log for details.

Rule Parse Processing Time: 00:00:24.892
Analysis Processing Time: 00:00:31.715
Output Processing Time: 00:00:01.039

Memory usage stats:
heap stats:    peak      total      freed    current       unit      count
  reserved:    2.6 GiB    2.6 GiB      0        2.6 GiB                        not all freed!
 committed:    1.7 GiB    2.1 GiB  459.0 MiB    1.6 GiB                        not all freed!
     reset:      0          0          0          0                            ok
   touched:  128.5 KiB    3.1 MiB    3.6 GiB   -3.6 GiB                        ok
  segments:     19         25         17          8                            not all freed!
-abandoned:      0          0          0          0                            ok
   -cached:      0          0          0          0                            ok
     pages:      0          0       45.9 Ki   -45.9 Ki                         ok
-abandoned:      0          0          0          0                            ok
 -extended:      0
 -noretire:      0
     mmaps:      0
   commits:    515
   threads:     16         16          0         16                            not all freed!
  searches:     0.0 avg
numa nodes:       1
   elapsed:      57.701 s
   process: user: 102.437 s, system: 4.718 s, faults: 449341, rss: 1.7 GiB, commit: 1.8 GiB
  • hayabusa-sample-evtx this PR
Saved file: pr.csv (16.4 MB)
Elapsed time: 00:00:22.109
Errors were generated. Please check ./logs/errorlog-20221229_201015.log for details.

Rule Parse Processing Time: 00:00:02.889
Analysis Processing Time: 00:00:18.530
Output Processing Time: 00:00:00.688

Memory usage stats:
heap stats:    peak      total      freed    current       unit      count
  reserved:    2.6 GiB    2.6 GiB      0        2.6 GiB                        not all freed!
 committed:    1.8 GiB    2.0 GiB  355.3 MiB    1.6 GiB                        not all freed!
  • 6.1GB main
Saved file: all-main.csv (574.8 MB)
Elapsed time: 00:23:19.552
Errors were generated. Please check ./logs/errorlog-20221229_203553.log for details.

Rule Parse Processing Time: 00:00:03.211
Analysis Processing Time: 00:22:50.879
Output Processing Time: 00:00:25.460

Memory usage stats:
heap stats:    peak      total      freed    current       unit      count
  reserved:    5.3 GiB    5.3 GiB   56.0 MiB    5.3 GiB                        not all freed!
 committed:    4.6 GiB    7.6 GiB    3.0 GiB    4.5 GiB                        not all freed!
  • 6.1GB this PR
Saved file: all-pr.csv (574.8 MB)
Elapsed time: 00:18:52.842
Errors were generated. Please check ./logs/errorlog-20221229_205700.log for details.

Rule Parse Processing Time: 00:00:03.064
Analysis Processing Time: 00:18:25.808
Output Processing Time: 00:00:23.968

Memory usage stats:
heap stats:    peak      total      freed    current       unit      count
  reserved:    5.3 GiB    5.3 GiB   56.0 MiB    5.3 GiB                        not all freed!
 committed:    4.6 GiB    6.5 GiB    1.9 GiB    4.5 GiB                        not all freed!

@codecov
Copy link

codecov bot commented Dec 29, 2022 •
edited
Loading

Codecov Report

Base: 68.35% // Head: 68.43% // Increases project coverage by +0.07% 🎉

Coverage data is based on head (9c73f34) compared to base (063e4e2).
Patch coverage: 70.00% of modified lines in pull request are covered.

Additional details and impacted files 
@@            Coverage Diff             @@
##             main     #858      +/-   ##
==========================================
+ Coverage   68.35%   68.43%   +0.07%     
==========================================
  Files          23       23              
  Lines       13551    13537      -14     
==========================================
+ Hits         9263     9264       +1     
+ Misses       4288     4273      -15
Impacted Files Coverage Δ
src/detections/detection.rs 39.75% <0.00%> (+0.60%) ⬆️
src/afterfact.rs 32.55% <100.00%> (+0.04%) ⬆️
src/detections/message.rs 91.04% <100.00%> (ø)

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

@hitenkoku hitenkoku added this to the v2.1.0 milestone Dec 29, 2022
@hitenkoku hitenkoku merged commit f0e7c1d into main Dec 29, 2022
@hitenkoku hitenkoku deleted the improve-speed-by-removing-io-process branch December 29, 2022 13:46
@fukusuket
Copy link
Collaborator Author

Thank you so much for taking the benchmark and review :)

@fukusuket fukusuket changed the title Improve speed by removing io process before insert_message() Improve speed by removing IO process before insert_message() Jan 7, 2023
@fukusuket fukusuket mentioned this pull request Feb 18, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@hitenkoku hitenkoku hitenkoku approved these changes

@YamatoSecurity YamatoSecurity Awaiting requested review from YamatoSecurity

Assignees

@fukusuket fukusuket

Labels
enhancement New feature or request
Projects
None yet
Milestone
v2.1.0
Development

Successfully merging this pull request may close these issues.
2 participants
@fukusuket @hitenkoku