Unexpected drop_pct value

[shane-lawrence]

Describe the bug

The expected value for drop_pct is the percentage of total events that were dropped, but that doesn't appear to be the value reported. While troubleshooting high numbers of dropped events on some systems, I found that the drop_pct sometimes exceeded 100%. It looks like Falco is not including the dropped events in the total when calculating drop_pct.

How to reproduce it

falco --stats-interval 60000

Note: this will reproduce the problem as long as it drops at least one event in one minute.

Expected behaviour

I expect the drop_pct to show dropped events as a percent of total events (e.g. if 1000 events are processed and 10 are dropped then drop_pct should be 1).

Screenshots

log: {"sample": 15, "cur": {"events": 7316740312, "drops": 13564074902, "preemptions": 0}, "delta": {"events": 488585741, "drops": 904962208, "preemptions": 0}, "drop_pct": 185.221},

Environment

• Falco version: 0.31.0

• System info:

• Cloud provider or hardware configuration: GKE
• OS: Debian GNU/Linux 10 (buster)

• Kernel: 5.10.109+

• Installation method:

Additional context

[shane-lawrence]

Thanks for looking into this with me @Andreagit97 and @jasondellaluce. Some of the surprising stats being reported in 0.31.1 included drop_pct > 200, so it couldn't have just been caused by individual events being counted as drops twice:

log: {"sample": 45, "cur": {"events": 2174324593, "drops": 4030518781, "preemptions": 0}, "delta": {"events": 36712191, "drops": 109353131, "preemptions": 0}, "drop_pct": 297.866},

I have now upgraded 20,000+ Falco instances to 0.32.1 so we should get some good data on the impact of previous changes. I also look forward to testing the impact of #2128.

[poiana]

Issues go stale after 90d of inactivity.

Mark the issue as fresh with /remove-lifecycle stale.

Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Provide feedback via /~https://github.com/falcosecurity/community.

/lifecycle stale

[jasondellaluce]

I think Falco 0.33.0 (released in few days) should have this solved! Looking forward to hear more once the release is available!

[poiana]

Stale issues rot after 30d of inactivity.

Mark the issue as fresh with /remove-lifecycle rotten.

Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Provide feedback via /~https://github.com/falcosecurity/community.

/lifecycle rotten

[Andreagit97]

Yeah, I think we could let poiana close it if we have no more feedback :)

[shane-lawrence]

Resolved by #2128.