Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add PSI support (threshold for swapin/swapout) #100

Closed
yangfl opened this issue Nov 10, 2018 · 43 comments
Closed

Add PSI support (threshold for swapin/swapout) #100

yangfl opened this issue Nov 10, 2018 · 43 comments
Labels
enhancement

Comments

@yangfl
Copy link
Contributor

yangfl commented Nov 10, 2018

earlyoom is only aware of the amount of free swap, but I've seen many times that with large swap, system stop responding due to heavy swapin/swapout, while earlyoom did nothing since there's still plenty of free swap. Setting a low free swap minimum is no better than completely disabling swap and letting kernel oom to handle low memory situation.

Ref #34
@hakavlad
Copy link
Contributor

hakavlad commented Nov 10, 2018 •
edited
Loading

while earlyoom did nothing

What was he supposed to do? Please describe in detail the algorithm you want to implement.

@hakavlad
Copy link
Contributor

The rate of filling of a swap partition should be a trigger?

@rfjakob
Copy link
Owner

rfjakob commented Nov 10, 2018 •
edited
Loading

Seems like the psi (pressure stall information) was merged recently: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=eb414681d5a07d28d2ff90dc05f69ec6b232ebd2
It should bei in linux 4.20. This would probably the best source of information when to start killing things, but it will not be available in older kernels.

@hakavlad
Copy link
Contributor

Do you plan to support PSI in earlyoom 1.3+?

@rfjakob
Copy link
Owner

rfjakob commented Nov 18, 2018

Once my distribution has it, yes! (Fedora)

@hakavlad
Copy link
Contributor

hakavlad commented Dec 24, 2018 •
edited
Loading

Congratulations!
Linux 4.20 released with PSI and is already in Kernel-vanilla-stable
https://fedoraproject.org/wiki/Kernel_Vanilla_Repositories
http://www.leemhuis.info/files/kernel-vanilla/repostatus.txt

@rfjakob
Copy link
Owner

rfjakob commented Dec 25, 2018

:)

Reading through https://lwn.net/Articles/759781/ , one problem with psi may be that you have to wait for the thrashing to happen before you can react. Maybe earlyoom will look at both psi and free mem (but much lower threshhold, maybe 1%)

@hakavlad
Copy link
Contributor

hakavlad commented Dec 25, 2018 •
edited
Loading

Next q.:
How to find a victim when using psi?

@rfjakob
Copy link
Owner

rfjakob commented Dec 25, 2018

I guess same as before, highest oom_score

@hakavlad
Copy link
Contributor

you have to wait

10 sec maximum

@hakavlad
Copy link
Contributor

hakavlad commented Dec 26, 2018 •
edited
Loading

it works

pic

pic

@hakavlad
Copy link
Contributor

hakavlad commented Dec 30, 2018 •
edited
Loading

PSI support implemented in nohang. It works pretty fast. The wait is not too long.
https://i.imgur.com/vEJbGMR.png

I just added 2 thresholds:
sigterm_psi = 60
sigkill_psi = 80

It uses some avg10 to quickly respond.

@hakavlad
Copy link
Contributor

hakavlad commented Jan 1, 2019

The next problem: PSI mem avg10 slowly falls after corrective action (may need more then 20 sec sleep to prevent multiple killings).

@rfjakob
Copy link
Owner

rfjakob commented Jan 1, 2019

Good catch. I guess this means we should use "total" instead of the averaged number.

@hakavlad
Copy link
Contributor

hakavlad commented Jan 1, 2019

Seems like total always grows. How we can use it?

@hakavlad
Copy link
Contributor

hakavlad commented Jan 1, 2019

https://youtu.be/YYJ9Af8Syyg
demo with PSI

@hakavlad
Copy link
Contributor

hakavlad commented Jan 1, 2019 •
edited
Loading

Eureka!
avg = (total2 - total1) / time
Now I need test it.

@rfjakob
Copy link
Owner

rfjakob commented Jan 1, 2019

Yes exactly, you check if it increases

@hakavlad
Copy link
Contributor

hakavlad commented Jan 1, 2019 •
edited
Loading

avg = (total1 - total0) / (time1 - time0) / 10000
It works.

Good news: it can works very quick and reacts if swapping starts.
http://okturing.com/src/5628/body
(in this output avg10 is avg found from formula above)

Bad news: it may give false positives. I just ran FF, and it was killed.
http://okturing.com/src/5630/body

Maybe we should use avg1 or avg2 as middle way.

@hakavlad
Copy link
Contributor

hakavlad commented Jan 1, 2019

I will try this settings:
$SIGTERM_AVG = 60
$AVG_TIME = 1
I want to give the user to choose any minimum amount of time for which AVG is calculated

@hakavlad
Copy link
Contributor

hakavlad commented Jan 2, 2019 •
edited
Loading

http://okturing.com/src/5636/body
avg1 monitor (or monitor for any time, change avg_min_time)

example of output with not very quick memhogging
http://okturing.com/src/5637/body

avg time: 1.0
PSI mem avg: 0.0
some avg10=2.67 avg60=5.62 avg300=9.39 total=229884775 

avg time: 1.0
PSI mem avg: 0.0
some avg10=2.67 avg60=5.62 avg300=9.39 total=229884775 

avg time: 1.0
PSI mem avg: 0.0
some avg10=2.19 avg60=5.43 avg300=9.32 total=229884775 


# START USING SWAP


avg time: 1.0
PSI mem avg: 16.92
some avg10=2.19 avg60=5.43 avg300=9.32 total=230056914 

avg time: 1.1
PSI mem avg: 84.1
some avg10=8.68 avg60=6.50 avg300=9.52 total=230972454 

avg time: 1.0
PSI mem avg: 87.9
some avg10=8.68 avg60=6.50 avg300=9.52 total=231888109 

avg time: 1.1
PSI mem avg: 99.97
some avg10=23.95 avg60=9.33 avg300=10.09 total=232967719 

avg time: 1.1
PSI mem avg: 99.99
some avg10=23.95 avg60=9.33 avg300=10.09 total=234037558 

avg time: 1.0
PSI mem avg: 94.99
some avg10=37.55 avg60=12.26 avg300=10.70 total=235007672 

avg time: 1.0
PSI mem avg: 98.21
some avg10=37.55 avg60=12.26 avg300=10.70 total=236033919 

avg time: 1.1
PSI mem avg: 99.01
some avg10=48.13 avg60=15.00 avg300=11.28 total=237084562 

avg time: 1.1
PSI mem avg: 71.53
some avg10=48.13 avg60=15.00 avg300=11.28 total=237881926 

avg time: 1.1
PSI mem avg: 74.66
some avg10=53.54 avg60=17.06 avg300=11.74 total=238669873 

avg time: 1.0
PSI mem avg: 91.74
some avg10=53.54 avg60=17.06 avg300=11.74 total=239630293 

avg time: 1.0
PSI mem avg: 95.02
some avg10=59.60 avg60=19.35 avg300=12.25 total=240594804 

avg time: 1.0
PSI mem avg: 99.95
some avg10=59.60 avg60=19.35 avg300=12.25 total=241643845 

avg time: 1.0
PSI mem avg: 98.67
some avg10=66.38 avg60=21.89 avg300=12.83 total=242666213 

avg time: 1.0
PSI mem avg: 49.99
some avg10=66.38 avg60=21.89 avg300=12.83 total=243167690 

avg time: 1.0
PSI mem avg: 84.66
some avg10=67.40 avg60=23.53 avg300=13.24 total=244038897 

avg time: 1.0
PSI mem avg: 70.76
some avg10=67.40 avg60=23.53 avg300=13.24 total=244771122 

avg time: 1.0
PSI mem avg: 92.33
some avg10=69.50 avg60=25.35 avg300=13.68 total=245703671 

avg time: 1.1
PSI mem avg: 99.31
some avg10=74.12 avg60=27.62 avg300=14.24 total=246781382 

avg time: 1.1
PSI mem avg: 99.97
some avg10=74.12 avg60=27.62 avg300=14.24 total=247864738 

avg time: 1.1
PSI mem avg: 99.97
some avg10=78.62 avg60=29.96 avg300=14.82 total=248954990 

avg time: 1.0
PSI mem avg: 99.44
some avg10=78.62 avg60=29.96 avg300=14.82 total=249980594 

avg time: 1.1
PSI mem avg: 99.06
some avg10=82.31 avg60=32.22 avg300=15.40 total=251050410 

avg time: 1.1
PSI mem avg: 95.75
some avg10=82.31 avg60=32.22 avg300=15.40 total=252087053 

avg time: 1.0
PSI mem avg: 97.02
some avg10=84.79 avg60=34.30 avg300=15.95 total=253064046 

avg time: 1.0
PSI mem avg: 99.41
some avg10=84.79 avg60=34.30 avg300=15.95 total=254100130 

avg time: 1.1
PSI mem avg: 98.17
some avg10=87.18 avg60=36.39 avg300=16.51 total=255139188 

avg time: 1.1
PSI mem avg: 84.71
some avg10=87.18 avg60=36.39 avg300=16.51 total=256069110 

avg time: 1.1
PSI mem avg: 83.95
some avg10=86.24 avg60=37.88 avg300=16.96 total=256952564 

avg time: 1.1
PSI mem avg: 98.74
some avg10=86.24 avg60=37.88 avg300=16.96 total=258030294 

avg time: 1.1
PSI mem avg: 99.98
some avg10=88.55 avg60=39.88 avg300=17.52 total=259080932 

avg time: 1.1
PSI mem avg: 80.52
some avg10=88.55 avg60=39.88 avg300=17.52 total=259943098 

avg time: 1.1
PSI mem avg: 60.02
some avg10=88.45 avg60=41.45 avg300=18.00 total=260590551 

avg time: 1.0
PSI mem avg: 84.79
some avg10=88.45 avg60=41.45 avg300=18.00 total=261455903 

avg time: 1.0
PSI mem avg: 77.87
some avg10=85.29 avg60=42.42 avg300=18.36 total=262255772 

avg time: 1.0
PSI mem avg: 86.76
some avg10=84.69 avg60=43.72 avg300=18.80 total=263132918 

avg time: 1.0
PSI mem avg: 89.88
some avg10=84.69 avg60=43.72 avg300=18.80 total=264042776 

avg time: 1.0
PSI mem avg: 69.67
some avg10=83.66 avg60=44.87 avg300=19.21 total=264758629 

avg time: 1.0
PSI mem avg: 70.96
some avg10=83.66 avg60=44.87 avg300=19.21 total=265486766 

avg time: 1.0
PSI mem avg: 88.18
some avg10=82.63 avg60=45.95 avg300=19.61 total=266382658 

avg time: 1.0
PSI mem avg: 82.89
some avg10=82.63 avg60=45.95 avg300=19.61 total=267224852 

avg time: 1.0
PSI mem avg: 83.4
some avg10=82.52 avg60=47.13 avg300=20.04 total=268071133 

avg time: 1.0
PSI mem avg: 84.65
some avg10=82.52 avg60=47.13 avg300=20.04 total=268926961 

avg time: 1.0
PSI mem avg: 86.77
some avg10=82.97 avg60=48.37 avg300=20.48 total=269808080 

avg time: 1.0
PSI mem avg: 87.66
some avg10=82.97 avg60=48.37 avg300=20.48 total=270705197 

avg time: 1.0
PSI mem avg: 83.86
some avg10=83.52 avg60=49.60 avg300=20.93 total=271558446 

avg time: 1.0
PSI mem avg: 91.06
some avg10=83.52 avg60=49.60 avg300=20.93 total=272485445 

avg time: 1.0
PSI mem avg: 89.83
some avg10=84.51 avg60=50.89 avg300=21.40 total=273399845 

avg time: 1.0
PSI mem avg: 99.46
some avg10=84.51 avg60=50.89 avg300=21.40 total=274409564 

avg time: 1.0
PSI mem avg: 95.79
some avg10=86.77 avg60=52.40 avg300=21.91 total=275390717 

avg time: 1.0
PSI mem avg: 84.82
some avg10=86.77 avg60=52.40 avg300=21.91 total=276246728 

avg time: 1.0
PSI mem avg: 99.99
some avg10=87.54 avg60=53.66 avg300=22.39 total=277262795 

avg time: 1.0
PSI mem avg: 98.98
some avg10=87.54 avg60=53.66 avg300=22.39 total=278271242 

avg time: 1.0
PSI mem avg: 92.34
some avg10=89.25 avg60=55.08 avg300=22.90 total=279209716 

avg time: 1.0
PSI mem avg: 90.39
some avg10=89.25 avg60=55.08 avg300=22.90 total=280125479 

avg time: 1.0
PSI mem avg: 96.1
some avg10=89.93 avg60=56.32 avg300=23.37 total=281097948 

avg time: 1.0
PSI mem avg: 90.81
some avg10=89.93 avg60=56.32 avg300=23.37 total=282033579 

avg time: 1.0
PSI mem avg: 94.96
some avg10=90.30 avg60=57.49 avg300=23.84 total=283005589 

avg time: 1.0
PSI mem avg: 99.84
some avg10=90.30 avg60=57.49 avg300=23.84 total=284014404 

avg time: 1.0
PSI mem avg: 95.37
some avg10=91.70 avg60=58.81 avg300=24.35 total=284980212 

avg time: 1.0
PSI mem avg: 95.8
some avg10=91.70 avg60=58.81 avg300=24.35 total=285948039 

avg time: 1.0
PSI mem avg: 99.98
some avg10=92.29 avg60=60.00 avg300=24.83 total=286968119 

avg time: 1.0
PSI mem avg: 99.94
some avg10=92.29 avg60=60.00 avg300=24.83 total=287974126 

avg time: 1.0
PSI mem avg: 97.2
some avg10=93.33 avg60=61.24 avg300=25.34 total=288957779 

avg time: 1.0
PSI mem avg: 93.15
some avg10=93.33 avg60=61.24 avg300=25.34 total=289909118 

avg time: 1.0
PSI mem avg: 85.54
some avg10=93.09 avg60=62.25 avg300=25.79 total=290779470 

avg time: 1.0
PSI mem avg: 92.57
some avg10=93.09 avg60=62.25 avg300=25.79 total=291724998 

avg time: 1.0
PSI mem avg: 99.11
some avg10=92.89 avg60=63.22 avg300=26.24 total=292728724 

avg time: 1.0
PSI mem avg: 99.97
some avg10=92.89 avg60=63.22 avg300=26.24 total=293748081 

avg time: 1.0
PSI mem avg: 99.99
some avg10=93.99 avg60=64.39 avg300=26.74 total=294776929 

avg time: 1.0
PSI mem avg: 99.98
some avg10=93.99 avg60=64.39 avg300=26.74 total=295785163 

avg time: 1.0
PSI mem avg: 99.95
some avg10=94.90 avg60=65.52 avg300=27.24 total=296787807 

avg time: 1.0
PSI mem avg: 99.98
some avg10=94.90 avg60=65.52 avg300=27.24 total=297821363 

avg time: 1.0
PSI mem avg: 98.74
some avg10=95.64 avg60=66.62 avg300=27.73 total=298820563 

# OOM!

@hakavlad
Copy link
Contributor

hakavlad commented Jan 2, 2019

nohang output with avg3
http://okturing.com/src/5638/body

@hakavlad
Copy link
Contributor

hakavlad commented Jan 2, 2019

sigterm_psi = 60
sigkill_psi = 95
avg_min_time = 4
psi_min_sleep_time_after_action = 16

output:

MemAvail: 1589 M, 79.8 % | SwapFree: 732 M, 100.0 %
MemAvail: 1391 M, 69.9 % | SwapFree: 732 M, 100.0 %
some avg10=0.00 avg60=0.00 avg300=0.00 total=0
PSI mem some avg: 0.0, PSI avg time: 4.1
PSI is OK  or  psi_min_sleep_time_after_action did not pass
MemAvail: 1050 M, 52.8 % | SwapFree: 732 M, 100.0 %
MemAvail:  755 M, 37.9 % | SwapFree: 732 M, 100.0 %
MemAvail:  496 M, 24.9 % | SwapFree: 732 M, 100.0 %
MemAvail:  260 M, 13.1 % | SwapFree: 732 M, 100.0 %
MemAvail:   54 M,  2.7 % | SwapFree: 727 M,  99.3 %
MemAvail:    0 M,  0.0 % | SwapFree: 718 M,  98.1 %
MemAvail:    0 M,  0.0 % | SwapFree: 718 M,  98.1 %
MemAvail:    0 M,  0.0 % | SwapFree: 715 M,  97.6 %
MemAvail:    0 M,  0.0 % | SwapFree: 704 M,  96.1 %
MemAvail:    0 M,  0.0 % | SwapFree: 701 M,  95.8 %
MemAvail:    0 M,  0.0 % | SwapFree: 694 M,  94.8 %
MemAvail:    0 M,  0.0 % | SwapFree: 688 M,  93.9 %
MemAvail:    0 M,  0.0 % | SwapFree: 685 M,  93.6 %
MemAvail:    0 M,  0.0 % | SwapFree: 680 M,  92.8 %
MemAvail:    0 M,  0.0 % | SwapFree: 676 M,  92.3 %
MemAvail:    0 M,  0.0 % | SwapFree: 667 M,  91.1 %
some avg10=9.41 avg60=1.70 avg300=0.35 total=1752059
PSI mem some avg: 41.83, PSI avg time: 4.2
PSI is OK  or  psi_min_sleep_time_after_action did not pass
MemAvail:    0 M,  0.0 % | SwapFree: 661 M,  90.2 %
MemAvail:    0 M,  0.0 % | SwapFree: 656 M,  89.6 %
MemAvail:    0 M,  0.0 % | SwapFree: 653 M,  89.2 %
MemAvail:    0 M,  0.0 % | SwapFree: 648 M,  88.5 %
MemAvail:    0 M,  0.0 % | SwapFree: 641 M,  87.6 %
MemAvail:    0 M,  0.0 % | SwapFree: 638 M,  87.1 %
MemAvail:    0 M,  0.0 % | SwapFree: 631 M,  86.1 %
MemAvail:    0 M,  0.0 % | SwapFree: 626 M,  85.5 %
MemAvail:    0 M,  0.0 % | SwapFree: 621 M,  84.8 %
MemAvail:    0 M,  0.0 % | SwapFree: 621 M,  84.8 %
MemAvail:    0 M,  0.0 % | SwapFree: 612 M,  83.6 %
MemAvail:    0 M,  0.0 % | SwapFree: 606 M,  82.7 %
MemAvail:    0 M,  0.0 % | SwapFree: 599 M,  81.8 %
MemAvail:    0 M,  0.0 % | SwapFree: 596 M,  81.4 %
MemAvail:    0 M,  0.0 % | SwapFree: 590 M,  80.5 %
MemAvail:    0 M,  0.0 % | SwapFree: 586 M,  80.1 %
MemAvail:    0 M,  0.0 % | SwapFree: 582 M,  79.5 %
MemAvail:    0 M,  0.0 % | SwapFree: 578 M,  79.0 %
MemAvail:    0 M,  0.0 % | SwapFree: 576 M,  78.7 %
MemAvail:    0 M,  0.0 % | SwapFree: 571 M,  77.9 %
some avg10=32.08 avg60=6.61 avg300=1.41 total=5262760
PSI mem some avg: 84.16, PSI avg time: 4.2
avg (84.16) > sigterm_psi (60)
Found the victim with highest badness:
  Name:    python3
  PID:     1077
  UID:     1000
  Badness: 663, oom_score: 663, oom_score_adj: 0
  VmSize:  1830 MiB
  VmRSS:   1768 MiB (Anon: 1762 MiB, File: 6 MiB, Shmem: 0 MiB)
  VmSwap:    35 MiB
  CmdLine: python3 /usr/bin/oom-trigger 
Implement corrective action:
  Sending SIGTERM to the victim; OK; response time: 16 ms

Duration of work: 45 sec; number of corrective actions:
  - Send SIGTERM to python3: 1
MemAvail: 1779 M, 89.4 % | SwapFree: 597 M,  81.5 %
MemAvail: 1791 M, 90.0 % | SwapFree: 585 M,  80.0 %
MemAvail: 1801 M, 90.5 % | SwapFree: 574 M,  78.4 %
MemAvail: 1811 M, 91.0 % | SwapFree: 564 M,  77.0 %
MemAvail: 1815 M, 91.2 % | SwapFree: 561 M,  76.6 %
MemAvail: 1814 M, 91.2 % | SwapFree: 561 M,  76.6 %
MemAvail: 1813 M, 91.1 % | SwapFree: 561 M,  76.6 %
some avg10=41.53 avg60=10.19 avg300=2.24 total=7650134
PSI mem some avg: 54.31, PSI avg time: 4.4
PSI is OK  or  psi_min_sleep_time_after_action did not pass
MemAvail: 1813 M, 91.1 % | SwapFree: 561 M,  76.6 %
MemAvail: 1811 M, 91.0 % | SwapFree: 562 M,  76.7 %
MemAvail: 1805 M, 90.7 % | SwapFree: 568 M,  77.5 %
MemAvail: 1798 M, 90.4 % | SwapFree: 574 M,  78.4 %
MemAvail: 1796 M, 90.2 % | SwapFree: 576 M,  78.7 %
MemAvail: 1796 M, 90.2 % | SwapFree: 576 M,  78.7 %
MemAvail: 1796 M, 90.2 % | SwapFree: 576 M,  78.7 %
MemAvail: 1796 M, 90.2 % | SwapFree: 576 M,  78.7 %
MemAvail: 1796 M, 90.2 % | SwapFree: 576 M,  78.7 %
some avg10=39.81 avg60=13.12 avg300=3.04 total=9217960
PSI mem some avg: 35.45, PSI avg time: 4.4
PSI is OK  or  psi_min_sleep_time_after_action did not pass
MemAvail: 1796 M, 90.2 % | SwapFree: 576 M,  78.7 %
MemAvail: 1796 M, 90.2 % | SwapFree: 576 M,  78.7 %

@hakavlad
Copy link
Contributor

hakavlad commented Jan 2, 2019

I saw false positives even if
avg_min_time = 4
and
psi_min_sleep_time_after_action = 20.
IMHO avg10 is now the safest for default using.
total need testing

@hakavlad
Copy link
Contributor

hakavlad commented Jan 2, 2019 

PSI mem some avg10:    0.0 | MemAvail:  235 M, 11.8 % | SwapFree: 731 M,  99.9 %
PSI mem some avg10:    0.0 | MemAvail:  165 M,  8.3 % | SwapFree: 731 M,  99.9 %
PSI mem some avg10:    0.0 | MemAvail:  101 M,  5.1 % | SwapFree: 731 M,  99.9 %
PSI mem some avg10:    0.0 | MemAvail:   49 M,  2.5 % | SwapFree: 731 M,  99.9 %
PSI mem some avg10:    0.0 | MemAvail:   17 M,  0.8 % | SwapFree: 731 M,  99.9 %
PSI mem some avg10:    0.0 | MemAvail:    1 M,  0.0 % | SwapFree: 727 M,  99.2 %
PSI mem some avg10:    0.0 | MemAvail:    0 M,  0.0 % | SwapFree: 717 M,  97.9 %
PSI mem some avg10:   1.99 | MemAvail:    0 M,  0.0 % | SwapFree: 713 M,  97.4 %
PSI mem some avg10:   1.99 | MemAvail:    0 M,  0.0 % | SwapFree: 708 M,  96.7 %
PSI mem some avg10:   1.99 | MemAvail:    0 M,  0.0 % | SwapFree: 703 M,  96.0 %
PSI mem some avg10:   1.99 | MemAvail:    7 M,  0.3 % | SwapFree: 695 M,  95.0 %
PSI mem some avg10:   1.99 | MemAvail:   13 M,  0.6 % | SwapFree: 688 M,  94.0 %
PSI mem some avg10:   1.99 | MemAvail:   19 M,  0.9 % | SwapFree: 681 M,  93.0 %
PSI mem some avg10:   1.99 | MemAvail:   27 M,  1.4 % | SwapFree: 673 M,  91.9 %
PSI mem some avg10:   1.99 | MemAvail:   32 M,  1.6 % | SwapFree: 668 M,  91.2 %
PSI mem some avg10:   1.99 | MemAvail:   34 M,  1.7 % | SwapFree: 664 M,  90.7 %
PSI mem some avg10:  10.32 | MemAvail:   39 M,  2.0 % | SwapFree: 659 M,  90.0 %
PSI mem some avg10:  10.32 | MemAvail:   45 M,  2.2 % | SwapFree: 653 M,  89.1 %
PSI mem some avg10:  10.32 | MemAvail:   49 M,  2.4 % | SwapFree: 648 M,  88.5 %
PSI mem some avg10:  10.32 | MemAvail:   55 M,  2.7 % | SwapFree: 641 M,  87.6 %
PSI mem some avg10:  10.32 | MemAvail:   61 M,  3.1 % | SwapFree: 634 M,  86.6 %
PSI mem some avg10:  10.32 | MemAvail:   64 M,  3.2 % | SwapFree: 632 M,  86.3 %
PSI mem some avg10:  10.32 | MemAvail:   51 M,  2.6 % | SwapFree: 630 M,  86.0 %
PSI mem some avg10:  10.32 | MemAvail:    4 M,  0.2 % | SwapFree: 630 M,  86.0 %
PSI mem some avg10:  10.32 | MemAvail:    0 M,  0.0 % | SwapFree: 621 M,  84.8 %
PSI mem some avg10:  20.05 | MemAvail:    0 M,  0.0 % | SwapFree: 612 M,  83.6 %
PSI mem some avg10:  20.05 | MemAvail:    0 M,  0.0 % | SwapFree: 611 M,  83.5 %
PSI mem some avg10:  20.05 | MemAvail:    0 M,  0.0 % | SwapFree: 607 M,  82.9 %
PSI mem some avg10:  20.05 | MemAvail:    0 M,  0.0 % | SwapFree: 600 M,  82.0 %
PSI mem some avg10:  20.05 | MemAvail:    0 M,  0.0 % | SwapFree: 600 M,  82.0 %
PSI mem some avg10:  20.05 | MemAvail:    1 M,  0.0 % | SwapFree: 591 M,  80.7 %
PSI mem some avg10:  20.05 | MemAvail:    0 M,  0.0 % | SwapFree: 583 M,  79.6 %
PSI mem some avg10:  20.05 | MemAvail:    0 M,  0.0 % | SwapFree: 583 M,  79.6 %
PSI mem some avg10:  20.05 | MemAvail:    0 M,  0.0 % | SwapFree: 582 M,  79.5 %
PSI mem some avg10:  20.05 | MemAvail:    0 M,  0.0 % | SwapFree: 578 M,  78.9 %
PSI mem some avg10:  31.09 | MemAvail:    0 M,  0.0 % | SwapFree: 572 M,  78.1 %
PSI mem some avg10:  31.09 | MemAvail:    0 M,  0.0 % | SwapFree: 567 M,  77.5 %
PSI mem some avg10:  31.09 | MemAvail:    0 M,  0.0 % | SwapFree: 563 M,  76.9 %
PSI mem some avg10:  31.09 | MemAvail:    0 M,  0.0 % | SwapFree: 557 M,  76.1 %
PSI mem some avg10:  31.09 | MemAvail:    0 M,  0.0 % | SwapFree: 556 M,  76.0 %
PSI mem some avg10:  31.09 | MemAvail:    0 M,  0.0 % | SwapFree: 555 M,  75.8 %
PSI mem some avg10:  31.09 | MemAvail:    0 M,  0.0 % | SwapFree: 550 M,  75.2 %
PSI mem some avg10:  31.09 | MemAvail:    0 M,  0.0 % | SwapFree: 548 M,  74.9 %
PSI mem some avg10:  31.09 | MemAvail:    0 M,  0.0 % | SwapFree: 541 M,  73.9 %
PSI mem some avg10:  31.09 | MemAvail:    0 M,  0.0 % | SwapFree: 535 M,  73.1 %
PSI mem some avg10:  31.09 | MemAvail:    0 M,  0.0 % | SwapFree: 532 M,  72.6 %
PSI mem some avg10:  40.85 | MemAvail:    0 M,  0.0 % | SwapFree: 527 M,  71.9 %
PSI mem some avg10:  40.85 | MemAvail:    0 M,  0.0 % | SwapFree: 527 M,  71.9 %
PSI mem some avg10:  40.85 | MemAvail:    0 M,  0.0 % | SwapFree: 527 M,  71.9 %
PSI mem some avg10:  40.85 | MemAvail:    0 M,  0.0 % | SwapFree: 518 M,  70.8 %
PSI mem some avg10:  40.85 | MemAvail:    0 M,  0.0 % | SwapFree: 514 M,  70.2 %
PSI mem some avg10:  40.85 | MemAvail:    0 M,  0.0 % | SwapFree: 507 M,  69.3 %
PSI mem some avg10:  40.85 | MemAvail:    0 M,  0.0 % | SwapFree: 506 M,  69.1 %
PSI mem some avg10:  40.85 | MemAvail:    0 M,  0.0 % | SwapFree: 506 M,  69.1 %
PSI mem some avg10:  40.85 | MemAvail:    0 M,  0.0 % | SwapFree: 506 M,  69.1 %
PSI mem some avg10:  40.85 | MemAvail:    0 M,  0.0 % | SwapFree: 501 M,  68.4 %
PSI mem some avg10:  40.85 | MemAvail:    0 M,  0.0 % | SwapFree: 498 M,  68.0 %
PSI mem some avg10:  50.66 | MemAvail:    0 M,  0.0 % | SwapFree: 490 M,  66.9 %
PSI mem some avg10:  50.66 | MemAvail:    0 M,  0.0 % | SwapFree: 485 M,  66.2 %
PSI mem some avg10:  50.66 | MemAvail:    0 M,  0.0 % | SwapFree: 481 M,  65.7 %
PSI mem some avg10:  50.66 | MemAvail:    0 M,  0.0 % | SwapFree: 472 M,  64.5 %
PSI mem some avg10:  50.66 | MemAvail:    0 M,  0.0 % | SwapFree: 466 M,  63.6 %
PSI mem some avg10:  50.66 | MemAvail:    0 M,  0.0 % | SwapFree: 460 M,  62.8 %
PSI mem some avg10:  50.66 | MemAvail:    0 M,  0.0 % | SwapFree: 456 M,  62.3 %
PSI mem some avg10:  50.66 | MemAvail:    0 M,  0.0 % | SwapFree: 449 M,  61.4 %
PSI mem some avg10:  50.66 | MemAvail:    0 M,  0.0 % | SwapFree: 444 M,  60.6 %
PSI mem some avg10:  50.66 | MemAvail:    0 M,  0.0 % | SwapFree: 438 M,  59.8 %
PSI mem some avg10:  50.66 | MemAvail:    0 M,  0.0 % | SwapFree: 437 M,  59.6 %
PSI mem some avg10:  50.66 | MemAvail:    0 M,  0.0 % | SwapFree: 433 M,  59.1 %
PSI mem some avg10:  50.66 | MemAvail:    0 M,  0.0 % | SwapFree: 424 M,  57.9 %
PSI mem some avg10:  55.25 | MemAvail:    0 M,  0.0 % | SwapFree: 424 M,  57.9 %
PSI mem some avg10:  55.25 | MemAvail:    0 M,  0.0 % | SwapFree: 424 M,  57.9 %
PSI mem some avg10:  55.25 | MemAvail:    0 M,  0.0 % | SwapFree: 424 M,  57.9 %
PSI mem some avg10:  55.25 | MemAvail:    0 M,  0.0 % | SwapFree: 421 M,  57.5 %
PSI mem some avg10:  55.25 | MemAvail:    0 M,  0.0 % | SwapFree: 418 M,  57.1 %
PSI mem some avg10:  55.25 | MemAvail:    0 M,  0.0 % | SwapFree: 412 M,  56.3 %
PSI mem some avg10:  55.25 | MemAvail:    0 M,  0.0 % | SwapFree: 409 M,  55.8 %
PSI mem some avg10:  55.25 | MemAvail:    0 M,  0.0 % | SwapFree: 404 M,  55.2 %
PSI mem some avg10:  55.25 | MemAvail:    0 M,  0.0 % | SwapFree: 399 M,  54.5 %
PSI mem some avg10:  55.25 | MemAvail:    0 M,  0.0 % | SwapFree: 398 M,  54.3 %
PSI mem some avg10:  55.25 | MemAvail:    0 M,  0.0 % | SwapFree: 398 M,  54.3 %
PSI mem some avg10:  55.25 | MemAvail:    0 M,  0.0 % | SwapFree: 393 M,  53.6 %
PSI mem some avg10:  55.25 | MemAvail:    0 M,  0.0 % | SwapFree: 391 M,  53.3 %
PSI mem some avg10:  55.25 | MemAvail:    0 M,  0.0 % | SwapFree: 386 M,  52.7 %
PSI mem some avg10:  55.25 | MemAvail:    0 M,  0.0 % | SwapFree: 384 M,  52.5 %
avg (62.99) > sigterm_psi (60.0)
Found the victim with highest badness:
  Name:    python3
  PID:     1486
  UID:     1000
  Badness: 621, oom_score: 621, oom_score_adj: 0
  VmSize:  1719 MiB
  VmRSS:   1624 MiB (Anon: 1618 MiB, File: 6 MiB, Shmem: 0 MiB)
  VmSwap:    66 MiB
  CmdLine: python3 /usr/bin/oom-trigger 
Implement corrective action:
  Sending SIGTERM to the victim; OK; response time: 70 ms

Duration of work: 1 min 0 sec; number of corrective actions:
  - Send SIGTERM to python3: 1
PSI mem some avg10:  62.99 | MemAvail: 1623 M, 81.5 % | SwapFree: 450 M,  61.4 %
PSI mem some avg10:  62.99 | MemAvail: 1663 M, 83.6 % | SwapFree: 446 M,  60.9 %
PSI mem some avg10:  62.99 | MemAvail: 1667 M, 83.7 % | SwapFree: 442 M,  60.4 %
PSI mem some avg10:  62.99 | MemAvail: 1667 M, 83.7 % | SwapFree: 442 M,  60.4 %
PSI mem some avg10:  68.43 | MemAvail: 1667 M, 83.7 % | SwapFree: 442 M,  60.4 %
PSI mem some avg10:  68.43 | MemAvail: 1666 M, 83.7 % | SwapFree: 442 M,  60.4 %
PSI mem some avg10:  68.43 | MemAvail: 1666 M, 83.7 % | SwapFree: 442 M,  60.4 %
PSI mem some avg10:  68.43 | MemAvail: 1665 M, 83.7 % | SwapFree: 443 M,  60.5 %
PSI mem some avg10:  68.43 | MemAvail: 1665 M, 83.6 % | SwapFree: 443 M,  60.5 %
PSI mem some avg10:  72.15 | MemAvail: 1664 M, 83.6 % | SwapFree: 443 M,  60.5 %
PSI mem some avg10:  72.15 | MemAvail: 1664 M, 83.6 % | SwapFree: 443 M,  60.5 %
PSI mem some avg10:  72.15 | MemAvail: 1663 M, 83.5 % | SwapFree: 443 M,  60.5 %
PSI mem some avg10:  72.15 | MemAvail: 1662 M, 83.5 % | SwapFree: 443 M,  60.5 %
PSI mem some avg10:  72.15 | MemAvail: 1661 M, 83.5 % | SwapFree: 443 M,  60.5 %
PSI mem some avg10:  71.94 | MemAvail: 1660 M, 83.4 % | SwapFree: 443 M,  60.5 %
PSI mem some avg10:  71.94 | MemAvail: 1660 M, 83.4 % | SwapFree: 443 M,  60.5 %
PSI mem some avg10:  71.94 | MemAvail: 1659 M, 83.3 % | SwapFree: 444 M,  60.6 %
PSI mem some avg10:  71.94 | MemAvail: 1658 M, 83.3 % | SwapFree: 444 M,  60.7 %
PSI mem some avg10:  71.94 | MemAvail: 1657 M, 83.2 % | SwapFree: 445 M,  60.8 %
PSI mem some avg10:  74.67 | MemAvail: 1655 M, 83.2 % | SwapFree: 446 M,  60.9 %
PSI mem some avg10:  74.67 | MemAvail: 1654 M, 83.1 % | SwapFree: 447 M,  61.0 %
PSI mem some avg10:  74.67 | MemAvail: 1653 M, 83.0 % | SwapFree: 447 M,  61.1 %
PSI mem some avg10:  74.67 | MemAvail: 1651 M, 82.9 % | SwapFree: 448 M,  61.2 %
PSI mem some avg10:  74.67 | MemAvail: 1649 M, 82.9 % | SwapFree: 450 M,  61.4 %
PSI mem some avg10:  78.35 | MemAvail: 1648 M, 82.8 % | SwapFree: 450 M,  61.5 %
PSI mem some avg10:  78.35 | MemAvail: 1648 M, 82.8 % | SwapFree: 450 M,  61.5 %
PSI mem some avg10:  78.35 | MemAvail: 1647 M, 82.7 % | SwapFree: 450 M,  61.5 %
PSI mem some avg10:  78.35 | MemAvail: 1647 M, 82.8 % | SwapFree: 450 M,  61.5 %
PSI mem some avg10:  79.56 | MemAvail: 1646 M, 82.7 % | SwapFree: 450 M,  61.5 %
PSI mem some avg10:  79.56 | MemAvail: 1646 M, 82.7 % | SwapFree: 451 M,  61.5 %
PSI mem some avg10:  79.56 | MemAvail: 1645 M, 82.7 % | SwapFree: 451 M,  61.6 %
PSI mem some avg10:  79.56 | MemAvail: 1644 M, 82.6 % | SwapFree: 452 M,  61.7 %
PSI mem some avg10:  79.56 | MemAvail: 1643 M, 82.5 % | SwapFree: 452 M,  61.8 %
PSI mem some avg10:   80.0 | MemAvail: 1642 M, 82.5 % | SwapFree: 453 M,  61.9 %
PSI mem some avg10:   80.0 | MemAvail: 1640 M, 82.4 % | SwapFree: 455 M,  62.1 %
PSI mem some avg10:   80.0 | MemAvail: 1640 M, 82.4 % | SwapFree: 455 M,  62.1 %
PSI mem some avg10:   80.0 | MemAvail: 1638 M, 82.3 % | SwapFree: 455 M,  62.1 %
PSI mem some avg10:   80.0 | MemAvail: 1637 M, 82.2 % | SwapFree: 455 M,  62.2 %
PSI mem some avg10:  79.09 | MemAvail: 1636 M, 82.2 % | SwapFree: 455 M,  62.2 %
PSI mem some avg10:  79.09 | MemAvail: 1637 M, 82.2 % | SwapFree: 455 M,  62.2 %
PSI mem some avg10:  79.09 | MemAvail: 1636 M, 82.2 % | SwapFree: 455 M,  62.2 %
PSI mem some avg10:  79.09 | MemAvail: 1635 M, 82.2 % | SwapFree: 455 M,  62.2 %
PSI mem some avg10:  79.09 | MemAvail: 1635 M, 82.1 % | SwapFree: 455 M,  62.2 %
PSI mem some avg10:  78.53 | MemAvail: 1635 M, 82.1 % | SwapFree: 455 M,  62.2 %
PSI mem some avg10:  78.53 | MemAvail: 1634 M, 82.1 % | SwapFree: 455 M,  62.2 %
PSI mem some avg10:  78.53 | MemAvail: 1634 M, 82.1 % | SwapFree: 455 M,  62.2 %
PSI mem some avg10:  78.53 | MemAvail: 1633 M, 82.1 % | SwapFree: 455 M,  62.2 %
PSI mem some avg10:  78.53 | MemAvail: 1633 M, 82.0 % | SwapFree: 455 M,  62.2 %
PSI mem some avg10:  77.71 | MemAvail: 1633 M, 82.0 % | SwapFree: 456 M,  62.2 %
PSI mem some avg10:  77.71 | MemAvail: 1632 M, 82.0 % | SwapFree: 456 M,  62.2 %
PSI mem some avg10:  77.71 | MemAvail: 1632 M, 82.0 % | SwapFree: 456 M,  62.2 %
PSI mem some avg10:  77.71 | MemAvail: 1632 M, 82.0 % | SwapFree: 456 M,  62.2 %
PSI mem some avg10:  78.67 | MemAvail: 1631 M, 82.0 % | SwapFree: 456 M,  62.2 %
PSI mem some avg10:  78.67 | MemAvail: 1631 M, 82.0 % | SwapFree: 456 M,  62.2 %
PSI mem some avg10:  78.67 | MemAvail: 1631 M, 81.9 % | SwapFree: 456 M,  62.2 %
PSI mem some avg10:  78.67 | MemAvail: 1631 M, 81.9 % | SwapFree: 456 M,  62.3 %
PSI mem some avg10:  78.67 | MemAvail: 1630 M, 81.9 % | SwapFree: 456 M,  62.3 %
PSI mem some avg10:  79.63 | MemAvail: 1630 M, 81.9 % | SwapFree: 456 M,  62.3 %
PSI mem some avg10:  79.63 | MemAvail: 1629 M, 81.9 % | SwapFree: 456 M,  62.3 %
PSI mem some avg10:  79.63 | MemAvail: 1629 M, 81.8 % | SwapFree: 456 M,  62.3 %
PSI mem some avg10:  79.63 | MemAvail: 1628 M, 81.8 % | SwapFree: 456 M,  62.3 %
PSI mem some avg10:  79.63 | MemAvail: 1628 M, 81.8 % | SwapFree: 456 M,  62.3 %
PSI mem some avg10:  81.15 | MemAvail: 1628 M, 81.8 % | SwapFree: 457 M,  62.4 %
PSI mem some avg10:  81.15 | MemAvail: 1627 M, 81.8 % | SwapFree: 457 M,  62.4 %
PSI mem some avg10:  81.15 | MemAvail: 1627 M, 81.7 % | SwapFree: 457 M,  62.4 %
PSI mem some avg10:  81.15 | MemAvail: 1626 M, 81.7 % | SwapFree: 457 M,  62.4 %
PSI mem some avg10:  81.15 | MemAvail: 1625 M, 81.7 % | SwapFree: 457 M,  62.4 %
PSI mem some avg10:  83.84 | MemAvail: 1625 M, 81.6 % | SwapFree: 457 M,  62.5 %
PSI mem some avg10:  83.84 | MemAvail: 1624 M, 81.6 % | SwapFree: 458 M,  62.5 %
PSI mem some avg10:  83.84 | MemAvail: 1624 M, 81.6 % | SwapFree: 458 M,  62.5 %
PSI mem some avg10:  83.84 | MemAvail: 1623 M, 81.5 % | SwapFree: 458 M,  62.5 %
PSI mem some avg10:  83.84 | MemAvail: 1622 M, 81.5 % | SwapFree: 458 M,  62.6 %
PSI mem some avg10:  84.77 | MemAvail: 1622 M, 81.5 % | SwapFree: 458 M,  62.6 %
PSI mem some avg10:  84.77 | MemAvail: 1621 M, 81.4 % | SwapFree: 458 M,  62.6 %
PSI mem some avg10:  84.77 | MemAvail: 1620 M, 81.4 % | SwapFree: 458 M,  62.6 %
PSI mem some avg10:  84.77 | MemAvail: 1620 M, 81.4 % | SwapFree: 459 M,  62.6 %
PSI mem some avg10:  86.26 | MemAvail: 1620 M, 81.4 % | SwapFree: 459 M,  62.6 %
PSI mem some avg10:  86.26 | MemAvail: 1619 M, 81.3 % | SwapFree: 459 M,  62.7 %
PSI mem some avg10:  86.26 | MemAvail: 1619 M, 81.3 % | SwapFree: 459 M,  62.7 %
PSI mem some avg10:  86.26 | MemAvail: 1618 M, 81.3 % | SwapFree: 459 M,  62.7 %
PSI mem some avg10:  86.26 | MemAvail: 1618 M, 81.3 % | SwapFree: 459 M,  62.7 %
PSI mem some avg10:  86.39 | MemAvail: 1618 M, 81.3 % | SwapFree: 459 M,  62.7 %
PSI mem some avg10:  86.39 | MemAvail: 1618 M, 81.3 % | SwapFree: 459 M,  62.7 %
PSI mem some avg10:  86.39 | MemAvail: 1617 M, 81.3 % | SwapFree: 459 M,  62.7 %
PSI mem some avg10:  86.39 | MemAvail: 1617 M, 81.2 % | SwapFree: 459 M,  62.7 %
PSI mem some avg10:  86.39 | MemAvail: 1617 M, 81.2 % | SwapFree: 459 M,  62.7 %
PSI mem some avg10:  86.68 | MemAvail: 1616 M, 81.2 % | SwapFree: 459 M,  62.7 %
PSI mem some avg10:  86.68 | MemAvail: 1616 M, 81.2 % | SwapFree: 459 M,  62.7 %
PSI mem some avg10:  86.68 | MemAvail: 1616 M, 81.2 % | SwapFree: 459 M,  62.7 %
PSI mem some avg10:  86.68 | MemAvail: 1615 M, 81.1 % | SwapFree: 460 M,  62.8 %
PSI mem some avg10:  86.68 | MemAvail: 1614 M, 81.1 % | SwapFree: 460 M,  62.8 %
PSI mem some avg10:  83.12 | MemAvail: 1614 M, 81.1 % | SwapFree: 460 M,  62.8 %
PSI mem some avg10:  83.12 | MemAvail: 1613 M, 81.0 % | SwapFree: 460 M,  62.8 %
PSI mem some avg10:  83.12 | MemAvail: 1613 M, 81.0 % | SwapFree: 460 M,  62.8 %
PSI mem some avg10:  83.12 | MemAvail: 1612 M, 81.0 % | SwapFree: 460 M,  62.8 %
PSI mem some avg10:  83.12 | MemAvail: 1611 M, 81.0 % | SwapFree: 460 M,  62.8 %
avg (79.47) > sigterm_psi (60.0)
Found the victim with highest badness:
  Name:    firefox
  PID:     1045
  UID:     1000
  Badness: 68, oom_score: 68, oom_score_adj: 0
  VmSize:  1843 MiB
  VmRSS:    115 MiB (Anon: 34 MiB, File: 22 MiB, Shmem: 58 MiB)
  VmSwap:    70 MiB
  CmdLine: /usr/lib/firefox/firefox 
Implement corrective action:
  Sending SIGTERM to the victim; OK; response time: 17 ms

Duration of work: 1 min 40 sec; number of corrective actions:
  - Send SIGTERM to python3: 1
  - Send SIGTERM to firefox: 1
PSI mem some avg10:  79.47 | MemAvail: 1611 M, 80.9 % | SwapFree: 460 M,  62.8 %
PSI mem some avg10:  79.47 | MemAvail: 1650 M, 82.9 % | SwapFree: 533 M,  72.7 %
PSI mem some avg10:  79.47 | MemAvail: 1649 M, 82.9 % | SwapFree: 533 M,  72.8 %
PSI mem some avg10:  79.47 | MemAvail: 1649 M, 82.8 % | SwapFree: 533 M,  72.8 %
PSI mem some avg10:  76.85 | MemAvail: 1649 M, 82.8 % | SwapFree: 533 M,  72.8 %
PSI mem some avg10:  76.85 | MemAvail: 1648 M, 82.8 % | SwapFree: 533 M,  72.8 %
PSI mem some avg10:  76.85 | MemAvail: 1647 M, 82.8 % | SwapFree: 534 M,  72.9 %
PSI mem some avg10:  76.85 | MemAvail: 1659 M, 83.4 % | SwapFree: 534 M,  72.9 %
PSI mem some avg10:  79.77 | MemAvail: 1659 M, 83.4 % | SwapFree: 534 M,  72.9 %
PSI mem some avg10:  79.77 | MemAvail: 1659 M, 83.4 % | SwapFree: 534 M,  73.0 %
PSI mem some avg10:  79.77 | MemAvail: 1658 M, 83.3 % | SwapFree: 534 M,  73.0 %
PSI mem some avg10:  79.77 | MemAvail: 1656 M, 83.2 % | SwapFree: 534 M,  73.0 %
PSI mem some avg10:  79.77 | MemAvail: 1656 M, 83.2 % | SwapFree: 534 M,  73.0 %
PSI mem some avg10:  75.65 | MemAvail: 1651 M, 82.9 % | SwapFree: 534 M,  73.0 %
PSI mem some avg10:  75.65 | MemAvail: 1650 M, 82.9 % | SwapFree: 534 M,  73.0 %
PSI mem some avg10:  75.65 | MemAvail: 1649 M, 82.9 % | SwapFree: 535 M,  73.0 %
PSI mem some avg10:  75.65 | MemAvail: 1646 M, 82.7 % | SwapFree: 535 M,  73.0 %
PSI mem some avg10:  68.28 | MemAvail: 1646 M, 82.7 % | SwapFree: 535 M,  73.1 %
PSI mem some avg10:  68.28 | MemAvail: 1645 M, 82.6 % | SwapFree: 535 M,  73.1 %
PSI mem some avg10:  68.28 | MemAvail: 1644 M, 82.6 % | SwapFree: 535 M,  73.1 %
PSI mem some avg10:  68.28 | MemAvail: 1642 M, 82.5 % | SwapFree: 536 M,  73.2 %
PSI mem some avg10:  68.28 | MemAvail: 1641 M, 82.4 % | SwapFree: 536 M,  73.2 %
PSI mem some avg10:  66.24 | MemAvail: 1637 M, 82.3 % | SwapFree: 536 M,  73.2 %
PSI mem some avg10:  66.24 | MemAvail: 1637 M, 82.3 % | SwapFree: 536 M,  73.2 %
PSI mem some avg10:  66.24 | MemAvail: 1633 M, 82.1 % | SwapFree: 536 M,  73.2 %
PSI mem some avg10:  66.24 | MemAvail: 1631 M, 82.0 % | SwapFree: 536 M,  73.2 %
PSI mem some avg10:  63.11 | MemAvail: 1630 M, 81.9 % | SwapFree: 536 M,  73.2 %
PSI mem some avg10:  63.11 | MemAvail: 1628 M, 81.8 % | SwapFree: 536 M,  73.2 %
PSI mem some avg10:  63.11 | MemAvail: 1625 M, 81.6 % | SwapFree: 536 M,  73.2 %
PSI mem some avg10:  63.11 | MemAvail: 1625 M, 81.6 % | SwapFree: 536 M,  73.2 %
PSI mem some avg10:  63.11 | MemAvail: 1623 M, 81.5 % | SwapFree: 536 M,  73.3 %
PSI mem some avg10:   62.0 | MemAvail: 1619 M, 81.4 % | SwapFree: 536 M,  73.3 %
PSI mem some avg10:   62.0 | MemAvail: 1615 M, 81.1 % | SwapFree: 536 M,  73.3 %
PSI mem some avg10:   62.0 | MemAvail: 1612 M, 81.0 % | SwapFree: 536 M,  73.3 %
PSI mem some avg10:   62.0 | MemAvail: 1611 M, 80.9 % | SwapFree: 536 M,  73.3 %
PSI mem some avg10:  56.39 | MemAvail: 1610 M, 80.9 % | SwapFree: 536 M,  73.3 %
PSI mem some avg10:  56.39 | MemAvail: 1605 M, 80.7 % | SwapFree: 536 M,  73.3 %
PSI mem some avg10:  56.39 | MemAvail: 1732 M, 87.0 % | SwapFree: 583 M,  79.6 %
PSI mem some avg10:  56.39 | MemAvail: 1732 M, 87.0 % | SwapFree: 583 M,  79.6 %
PSI mem some avg10:  56.39 | MemAvail: 1732 M, 87.0 % | SwapFree: 583 M,  79.6 %
PSI mem some avg10:  47.98 | MemAvail: 1732 M, 87.0 % | SwapFree: 583 M,  79.6 %
PSI mem some avg10:  47.98 | MemAvail: 1732 M, 87.0 % | SwapFree: 583 M,  79.7 %
PSI mem some avg10:  47.98 | MemAvail: 1732 M, 87.0 % | SwapFree: 583 M,  79.7 %
PSI mem some avg10:  47.98 | MemAvail: 1732 M, 87.0 % | SwapFree: 583 M,  79.7 %
PSI mem some avg10:  41.28 | MemAvail: 1732 M, 87.0 % | SwapFree: 583 M,  79.7 %
PSI mem some avg10:  41.28 | MemAvail: 1732 M, 87.0 % | SwapFree: 583 M,  79.7 %
PSI mem some avg10:  41.28 | MemAvail: 1732 M, 87.0 % | SwapFree: 583 M,  79.7 %
PSI mem some avg10:  41.28 | MemAvail: 1732 M, 87.0 % | SwapFree: 584 M,  79.7 %
PSI mem some avg10:  33.98 | MemAvail: 1732 M, 87.0 % | SwapFree: 584 M,  79.7 %
PSI mem some avg10:  33.98 | MemAvail: 1732 M, 87.0 % | SwapFree: 584 M,  79.8 %
PSI mem some avg10:  33.98 | MemAvail: 1732 M, 87.0 % | SwapFree: 584 M,  79.8 %
PSI mem some avg10:  33.98 | MemAvail: 1732 M, 87.0 % | SwapFree: 584 M,  79.8 %
PSI mem some avg10:  28.19 | MemAvail: 1732 M, 87.0 % | SwapFree: 584 M,  79.8 %
PSI mem some avg10:  28.19 | MemAvail: 1732 M, 87.0 % | SwapFree: 584 M,  79.8 %
PSI mem some avg10:  28.19 | MemAvail: 1732 M, 87.0 % | SwapFree: 584 M,  79.8 %
PSI mem some avg10:  28.19 | MemAvail: 1732 M, 87.0 % | SwapFree: 584 M,  79.8 %
PSI mem some avg10:  23.26 | MemAvail: 1732 M, 87.0 % | SwapFree: 584 M,  79.8 %
PSI mem some avg10:  23.26 | MemAvail: 1732 M, 87.0 % | SwapFree: 584 M,  79.8 %
PSI mem some avg10:  23.26 | MemAvail: 1732 M, 87.0 % | SwapFree: 584 M,  79.8 %
PSI mem some avg10:  23.26 | MemAvail: 1731 M, 87.0 % | SwapFree: 586 M,  80.1 %
PSI mem some avg10:  28.29 | MemAvail: 1730 M, 86.9 % | SwapFree: 587 M,  80.2 %
PSI mem some avg10:  28.29 | MemAvail: 1730 M, 86.9 % | SwapFree: 587 M,  80.2 %
PSI mem some avg10:  28.29 | MemAvail: 1730 M, 86.9 % | SwapFree: 587 M,  80.2 %
PSI mem some avg10:  28.29 | MemAvail: 1730 M, 86.9 % | SwapFree: 587 M,  80.2 %
PSI mem some avg10:  28.29 | MemAvail: 1730 M, 86.9 % | SwapFree: 587 M,  80.2 %
PSI mem some avg10:  23.16 | MemAvail: 1730 M, 86.9 % | SwapFree: 587 M,  80.2 %
PSI mem some avg10:  23.16 | MemAvail: 1730 M, 86.9 % | SwapFree: 587 M,  80.2 %
PSI mem some avg10:  23.16 | MemAvail: 1730 M, 86.9 % | SwapFree: 587 M,  80.2 %
PSI mem some avg10:  23.16 | MemAvail: 1730 M, 86.9 % | SwapFree: 587 M,  80.2 %
PSI mem some avg10:  18.96 | MemAvail: 1730 M, 86.9 % | SwapFree: 587 M,  80.2 %
PSI mem some avg10:  18.96 | MemAvail: 1730 M, 86.9 % | SwapFree: 587 M,  80.2 %
PSI mem some avg10:  18.96 | MemAvail: 1730 M, 86.9 % | SwapFree: 587 M,  80.2 %
PSI mem some avg10:  18.96 | MemAvail: 1730 M, 86.9 % | SwapFree: 587 M,  80.2 %
PSI mem some avg10:  15.71 | MemAvail: 1730 M, 86.9 % | SwapFree: 587 M,  80.2 %
PSI mem some avg10:  15.71 | MemAvail: 1730 M, 86.9 % | SwapFree: 587 M,  80.2 %

PSI is very slow to bounce back.
FF is innocent victim in this output (I used psi_min_sleep_time_after_action = 40).

@hakavlad hakavlad mentioned this issue Mar 2, 2019
Closed
@rfjakob rfjakob changed the title Add threshold for swapin/swapout Add PSI support (threshold for swapin/swapout) May 4, 2019
@hakavlad
Copy link
Contributor

hakavlad commented Jun 4, 2019 •
edited
Loading

Info:
Switching off a swapspace (on HDD if it was not empty) can cause a significant increasing PSI value (I've seen some avg10 = 70 when ran swapoff -a).

@rfjakob
Copy link
Owner

rfjakob commented Jul 4, 2019

I have tested a few things with PSI, and it seems pretty difficult to do "the right thing".

The difficult example is this: /~https://github.com/rfjakob/earlyoom/blob/master/contrib/oomstat/loadshift.txt
You have an app that uses 70% of your RAM but is idle, and you have enough swap free. Then another app wants to use 70% of your RAM. PSI goes crazy, but the situation calms down as the idle app is swapped out. The right thing here is not killing anything.

@hakavlad
Copy link
Contributor

hakavlad commented Jul 14, 2019 •
edited
Loading

The maximum PSI values depend on the type of swapspace.
99 for slow HDD, 60-70 for zram and SSD.

Not very long-term exceedances are normal.

Only long-term threshold exceedances are a clear pathology and require a reaction.

The optimal settings are very individual.

The right things are:

  • disable PSI monitoring by default
  • respond to only very long (>60 sec) thresholds exceeding.

Offer the following default settings:

psi_checking_enabled = False
psi_metrics = some_avg10
sigterm_psi_threshold = 60
sigkill_psi_threshold = 90
psi_excess_duration = 60 sec
psi_post_action_delay = 60 sec

@hakavlad hakavlad mentioned this issue Aug 14, 2019
Closed
@hakavlad
Copy link
Contributor

hakavlad commented Aug 16, 2019 •
edited
Loading

@rfjakob see
https://www.kernel.org/doc/html/latest/accounting/psi.html#monitoring-for-pressure-thresholds
and
https://www.kernel.org/doc/html/latest/accounting/psi.html#userspace-monitor-usage-example (using poll() since Linux 5.2)

Users can register triggers and use poll() to be woken up when resource pressure exceeds certain thresholds

@rfjakob
Copy link
Owner

rfjakob commented Aug 18, 2019

I have read through hakavlad/nohang#25, oh boy, PSI is such a headache. Although in this case it seems that muqss is interacting badly with PSI.

@hakavlad
Copy link
Contributor

PSI is such a headache

Only on desktops. But don't worry, rfjakob-sensei: PSI should be very useful for servers. Disable PSI by default, add warnings to the documentation, and PSI will no longer be a headache.

@hakavlad
Copy link
Contributor

hakavlad commented Sep 1, 2019

muqss is interacting badly with PSI

No.

Do you use also btrfs? See https://bugzilla.kernel.org/show_bug.cgi?id=196729

hakavlad/nohang#25 (comment)

Yes, I do.

Seems like this problem was with swap on btrfs.

@polarathene
Copy link

Please note that MuQSS is a valid possibility still, the linked issue didn't verify that it wasn't. Nor does the linked bugreport by @hakavlad to claim BTRFS at fault have anything to do with MuQSS or PSI. MuQSS only has partial support for cgroups afaik, and PSI relies on cgroups support, so it's more likely due to MuQSS lacking proper support for cgroups that PSI requires.

While the user also uses BTRFS, there is no verification from that user by using a different filesystem to confirm that it was caused by BTRFS alone. The linked bug report also has users with other filesystems other than BTRFS claiming they are experiencing the same/similar problems.

@hakavlad
Copy link
Contributor

hakavlad commented Jan 4, 2020

users with other filesystems other than BTRFS claiming they are experiencing the same/similar problems.

Could you provide links, please?

@hakavlad
Copy link
Contributor

hakavlad commented Jan 4, 2020

PSI relies on cgroups support

PSI is not directly dependent on сgroups. PSI provides metrics for each group with cgroup_v2. Without cgroup_v2 PSI works well and provides files only in /proc/pressure.

@polarathene
Copy link

The problem happen much more frequently when I used BtrFS. After switching to XFS, this happen less frequently (weekly instead of daily). - Comment 17

So, something was fixed in upstream, backported to LTS kernel 4.19 and to Ubuntu kernel. I don't know what. And that issue is 100% not in BTRFS but is another problem or another aspect of the problem. - Comment 54

I don't use btrfs, but only ext4 on an SSD. Since updating my system from kernel 5.1.7 to kernel 5.3.12 in Tumbleweed, I get regular ~1-2 second freezes - Comment 62

Probably the last one is unrelated though. Another comment(55 I think), pointed out their issue was related to CPU activity, which reminds me of another issue on github about BTRFS maintenance scripts which openSUSE at least shipped by default. As it's a non-traditional filesystem, and can require treating it differently along with maintenance, I could understand how it adds additional complications/complexities that can make issues more pronounced.

If you just want to find issues/reports of users with bad performance related to swap, there are plenty of those without being specific to BTRFS. This bug report has nothing to do with PSI or earlyoom/nohang? So it's relevance to identifying the problem a user experienced with PSI not working as expected is low.

The user would need to verify with a different filesystem that the symptoms were resolved, or better yet and easier verify that MuQSS kernel is not to blame by trying a kernel without MuQSS, just plain CFS and vanilla kernel that enables PSI.

Without cgroup_v2 PSI works well and provides files only in /proc/pressure.

Ah.. my mistake. But the log the user shared does seem to indicate that their system might indicate cgroup v2 support, but as MuQSS afaik only provides a partial implementation, what happens if PSI acknowledges cgroup v2 support and tries to use unimplemented features? It'd presumably break or behave incorrectly as the user indicates?

2019-08-14T21:15:47+0300 gentoo nohang[19609]:   CGroup_v2: /system.slice/system-beesd.slice/beesd@-----------------------------------------

That suggests that might be the case with that BTRFS daemon utility?

@hakavlad
Copy link
Contributor

hakavlad commented Mar 7, 2020

It's time to use PSI metrics to improve interactivity.

In this post I want to tell you about psi2log and psi-top scripts from nohang package. These simple tools can help you to conveniently measure and log various PSI metrics. See https://pagure.io/fedora-workstation/issue/98#comment-631355.

And PSI is not goes crazy, PSI metrics just provide stall information.

I have read through hakavlad/nohang#25, oh boy, PSI is such a headache.

PSI works incorrect with out-of-tree shedulers like MuQSS. But it is not a problem.

@hakavlad
Copy link
Contributor

hakavlad commented Mar 7, 2020 •
edited
Loading

I offer you to add optional PSI support.

For example, earlyoom --psi some,40.
What this should mean? If -m threshold exceeded, we can shoot if memory some avg10 > 40. I offer to hardcode 15 sec reclaim time.

So, I offer hardcode avg10 metrics and 15 sec reclaim time. User can choose some/full and threshold. Checking -m should prevent false-positives with a lot of available memory. And of course, this shoulds be optional.

@hakavlad
Copy link
Contributor

hakavlad commented Mar 7, 2020

nohang with PSI: no problem: https://www.youtube.com/watch?v=Y6GJqFE_ke4

@hakavlad
Copy link
Contributor

See also /~https://github.com/cdown/psi-notify

@rfjakob
Copy link
Owner

rfjakob commented Apr 10, 2024

I think now that nohang has psi support, there is no need for earlyoom to duplicate the functionality.

@rfjakob rfjakob closed this as completed Apr 10, 2024
@saurik
Copy link

saurik commented Apr 11, 2024

FWIW, the reason I like earlyoom is because it is written in C, and I thereby feel more confident that its own swap usage behavior can be more obviously controlled. I was thereby interested in PSI support one day being added to earlyoom, but am not terribly interested in switching to nohang. (Maybe my thought there is daft, and nohang can be fully trusted... but, if that is the case, why does this project still exist at all? Wouldn't it be better to just redirect the user base and developer interest by formally anointing a successor?)

@rfjakob
Copy link
Owner

rfjakob commented Apr 11, 2024

Let's take a step back for a second: Where do you want to use psi support? On a server? And what would you expect to be better than checking available mem/swap?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@saurik @rfjakob @polarathene @yangfl @hakavlad