Check keepalive first, and throttle try_pop

[scallopboat]

Signed-off-by: Chris Meyer scallopboat@gmail.com

What type of PR is this?
/kind bug

Any specific area of the project related to this PR?
/area engine

What this PR does / why we need it:
Reduces the CPU usage of falco when the grpc server is enabled to ~15%.

Which issue(s) this PR fixes:
Fixes #1126

Special notes for your reviewer:
Submitted as a potential (short term?) fix. It'd likely be better to refactor such that a sleep isn't needed by using a local queue as suggested on 05-20-20 call, but wanted to submit this option regardless.

Does this PR introduce a user-facing change?:
NONE

fix: High CPU usage when grpc server is enabled [#1126]

[scallopboat]

What the profiler captured when falco::grpc::server_impl::subscribe is active.

[poiana]

Welcome @scallopboat! It looks like this is your first PR to falcosecurity/falco 🎉

[poiana]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
To complete the pull request process, please assign mstemm
You can assign the PR to them by writing /assign @mstemm in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[scallopboat]

/assign @mstemm

[fntlnz]

Thanks a lot for showing us this approach @scallopboat

@leodido and I are trying to change the server architecture a little bit to avoid the try_pop in a while loop situation at all, it that doesn't land the solution here is probably the best we can do short-term as you suggested.

Here's the motivation: #1240
Here's a possible implementation: #1241

We still need to plot some gperftools and gprof graphs.

Let's see how it goes!

[leodido]

Hey @scallopboat thanks a lot for submitting this, I really appreciated it! 🙏

Anyways PR #1241 is ready to be merged in and it shows serious improvements reducing the CPU consumption when gRPC Falco Outputs are on.

Thus we'll close this in favor of that PR.

[leodido]

Thanks a lot for trying out this approach!

We rewrote a part of the gRPC server to gain performances in PR #1241