falkon: insufficient kernel memory quota on Genode@NOVA

[cproc]

I'm currently testing the Falkon web browser on current staging Sculpt and often get errors like the following:

[ 0] sys_pd_ctrl: PD 0xffffffff810306a0 has insufficient kernel memory quota                                                                                 
Warning: kernel memory quota upgrade failed - trigger memory free up for causing 'init -> runtime -> falkon-jemalloc -> falkon':'pthread.12' - donator is 'core':'unknown', policy=2

I'm not sure how to fix this.

[alex-ab]

Actually it should be just a warning, does Falkon afterwards still work or is it done ? What does 162cdc4 show for numbers regarding the kernel resource usage of Falkon.

[cproc]

Falkon does not respond to mouse input anymore when it happens and the log messages keep repeating every 2 seconds or so:

[ 1] sys_pd_ctrl: PD 0xffffffff81030660 has insufficient kernel memory quota                                                                                 
Warning: kernel memory quota upgrade failed - trigger memory free up for causing 'init -> runtime -> falkon-jemalloc -> falkon':'pthread.13' - donator is 'co
re':'unknown', policy=2
Warning: usage before=34989 after=34988 - limit before=34992 after=34992
[ 1] sys_pd_ctrl: PD 0xffffffff81030660 has insufficient kernel memory quota
Warning: kernel memory quota upgrade failed - trigger memory free up for causing 'init -> runtime -> falkon-jemalloc -> falkon':'pthread.13' - donator is 'co
re':'unknown', policy=2
Warning: usage before=34989 after=34989 - limit before=34992 after=34992
[ 1] sys_pd_ctrl: PD 0xffffffff81030660 has insufficient kernel memory quota
Warning: kernel memory quota upgrade failed - trigger memory free up for causing 'init -> runtime -> falkon-jemalloc -> falkon':'pthread.13' - donator is 'co
re':'unknown', policy=2
Warning: usage before=34989 after=34989 - limit before=34992 after=34992
[ 0] sys_pd_ctrl: PD 0xffffffff81030660 has insufficient kernel memory quota
Warning: kernel memory quota upgrade failed - trigger memory free up for causing 'init -> runtime -> falkon-jemalloc -> falkon':'signal handler' - donator is
 'core':'unknown', policy=2
Warning: usage before=34989 after=34989 - limit before=34992 after=34992
[ 1] sys_pd_ctrl: PD 0xffffffff81030660 has insufficient kernel memory quota
Warning: kernel memory quota upgrade failed - trigger memory free up for causing 'init -> runtime -> falkon-jemalloc -> falkon':'pthread.13' - donator is 'co
re':'unknown', policy=2
Warning: usage before=34989 after=34989 - limit before=34992 after=34992
[ 0] sys_pd_ctrl: PD 0xffffffff81030660 has insufficient kernel memory quota
Warning: kernel memory quota upgrade failed - trigger memory free up for causing 'init -> runtime -> falkon-jemalloc -> falkon':'signal handler' - donator is
 'core':'unknown', policy=2
Warning: usage before=34989 after=34989 - limit before=34992 after=34992

[cproc]

When I clicked "Restart" in the Leitzentrale after the warnings appeared, I got the following log output and the browser did not start:

[init -> runtime] child "falkon-jemalloc"                                                                                                                    
[init -> runtime]   RAM quota:  2096904K                                                                                                                     
[init -> runtime]   cap quota:  9966                                                                                                                         
[init -> runtime]   ELF binary: init                                                                                                                         
[init -> runtime]   priority:   2                                                                                                                            
[ 0] Insufficient kernel memory for creating new PD                                                                                                          
Error: create_pd returned 5                                                                                                                                  
[ 0] sys_pd_ctrl: PD 0xffffffff81030660 has insufficient kernel memory quota                                                                                 
Warning: kernel memory quota upgrade failed - trigger memory free up for causing 'init -> report_fs_rom':'ep' - donator is 'core':'unknown', policy=2        
Warning: usage before=55 after=55 - limit before=58 after=58                                                                                                 
[ 0] sys_create_ec: Non-PD CAP (0x7130)                                                                                                                      
Error: create_ec returned 4

When I clicked "Restart" again, the browser started successfully.

[alex-ab]

Well, for some reasons Falkon consumes a lot of kernel resources. When killing Falkon it will take a bit of time to free up all the resources (until all CPUs ack that the dead kernel objects in the RCU list can be freed). Can you please add commit 68af359, which will give some more details about which class of objects consumes what amount (first part will be core PD, second part will be Falkon PD)

[cproc]

Yeah, I'm running the JetStream2 benchmark to trigger the problem, which probably consumes more resources than usual.

Here's the output:

[ 3] sys_pd_ctrl: PD 0xffffffff81030860 has insufficient kernel memory quota
[ 3] 0xffffffff81030860 pt : slabs=758 elements=15533/15918, suspicious=0
[ 3] 0xffffffff81030860 mdb: slabs=116 elements=3090/3248, suspicious=0
[ 3] 0xffffffff81030860 sm : slabs=102 elements=1793/1836, suspicious=0
[ 3] 0xffffffff81030860 sc : slabs=2 elements=6/36, suspicious=0
[ 3] 0xffffffff81030860 ec : slabs=4 elements=18/24, suspicious=0
[ 3] 0xffffffff81030860 fpu: slabs=9 elements=9/9, suspicious=0
[ 3] 0xffffffff8a1c5540 pt : slabs=1 elements=3/21, suspicious=0
[ 3] 0xffffffff8a1c5540 mdb: slabs=1098 elements=29827/30744, suspicious=0
[ 3] 0xffffffff8a1c5540 sm : slabs=16 elements=257/288, suspicious=0
[ 3] 0xffffffff8a1c5540 sc : slabs=4 elements=53/72, suspicious=0
[ 3] 0xffffffff8a1c5540 ec : slabs=12 elements=54/72, suspicious=0
[ 3] 0xffffffff8a1c5540 fpu: slabs=72 elements=53/72, suspicious=0
Warning: kernel memory quota upgrade failed - trigger memory free up for causing 'init -> runtime -> falkon-jemalloc -> falkon':'pthread.11' - donator is 'co
re':'unknown', policy=2
[ 0] sys_pd_ctrl: PD 0xffffffff81030860 has insufficient kernel memory quota
[ 0] 0xffffffff81030860 pt : slabs=758 elements=15495/15918, suspicious=0
[ 0] 0xffffffff81030860 mdb: slabs=116 elements=2983/3248, suspicious=0
[ 0] 0xffffffff81030860 sm : slabs=102 elements=1793/1836, suspicious=0
[ 0] 0xffffffff81030860 sc : slabs=2 elements=6/36, suspicious=0
[ 0] 0xffffffff81030860 ec : slabs=4 elements=18/24, suspicious=0
[ 0] 0xffffffff81030860 fpu: slabs=9 elements=9/9, suspicious=0
[ 0] 0xffffffff8a1c5540 pt : slabs=1 elements=3/21, suspicious=0
[ 0] 0xffffffff8a1c5540 mdb: slabs=1098 elements=29634/30744, suspicious=0
[ 0] 0xffffffff8a1c5540 sm : slabs=16 elements=144/288, suspicious=0
[ 0] 0xffffffff8a1c5540 sc : slabs=4 elements=53/72, suspicious=0
[ 0] 0xffffffff8a1c5540 ec : slabs=12 elements=54/72, suspicious=0
[ 0] 0xffffffff8a1c5540 fpu: slabs=72 elements=53/72, suspicious=0
Warning: usage before=35154 after=35154 - limit before=35156 after=35156
Warning: kernel memory quota upgrade failed - trigger memory free up for causing 'init -> runtime -> falkon-jemalloc -> falkon':'signal handler' - donator is 'core':'unknown', policy=2
Warning: usage before=35154 after=35154 - limit before=35156 after=35156

[alex-ab]

@cproc: Thanks, may you please replace the debug commit with 544dc59 and please re-run ?

[cproc]

This is the output with 544dc59:

[ 0] sys_pd_ctrl: PD 0xffffffff81030a20 has insufficient kernel memory quota                                                                                 
[ 0] 0xffffffff81030a20 usage/limit 1184/1441 o=0                                                                                                            
[ 0] 0xffffffff81030a20 pt : slabs=751 elements=15531/15771, suspicious=0                                                                                    
[ 0] 0xffffffff81030a20 mdb: slabs=122 elements=3390/3416, suspicious=0                                                                                      
[ 0] 0xffffffff81030a20 sm : slabs=100 elements=1763/1800, suspicious=0                                                                                      
[ 0] 0xffffffff81030a20 sc : slabs=2 elements=6/36, suspicious=0                                                                                             
[ 0] 0xffffffff81030a20 ec : slabs=4 elements=18/24, suspicious=0                                                                                            
[ 0] 0xffffffff81030a20 fpu: slabs=9 elements=9/9, suspicious=0
[ 0] 0xffffffff81bca540 usage/limit 35159/35161 o=0
[ 0] 0xffffffff81bca540 pt : slabs=1 elements=3/21, suspicious=0
[ 0] 0xffffffff81bca540 mdb: slabs=1084 elements=30326/30352, suspicious=0
[ 0] 0xffffffff81bca540 sm : slabs=2 elements=12/36, suspicious=0
[ 0] 0xffffffff81bca540 sc : slabs=4 elements=53/72, suspicious=0
[ 0] 0xffffffff81bca540 ec : slabs=12 elements=54/72, suspicious=0
[ 0] 0xffffffff81bca540 fpu: slabs=68 elements=53/68, suspicious=0
Warning: kernel memory quota upgrade failed - trigger memory free up for causing 'init -> runtime -> falkon-jemalloc -> falkon':'pthread.12' - donator is 'core':'unknown', policy=2
Warning: usage before=35159 after=35159 - limit before=35161 after=35161
[ 0] sys_pd_ctrl: PD 0xffffffff81030a20 has insufficient kernel memory quota
[ 0] 0xffffffff81030a20 usage/limit 1184/1441 o=0
[ 0] 0xffffffff81030a20 pt : slabs=751 elements=15435/15771, suspicious=0
[ 0] 0xffffffff81030a20 mdb: slabs=122 elements=3041/3416, suspicious=0
[ 0] 0xffffffff81030a20 sm : slabs=100 elements=1763/1800, suspicious=0
[ 0] 0xffffffff81030a20 sc : slabs=2 elements=6/36, suspicious=0
[ 0] 0xffffffff81030a20 ec : slabs=4 elements=18/24, suspicious=0
[ 0] 0xffffffff81030a20 fpu: slabs=9 elements=9/9, suspicious=0
[ 0] 0xffffffff81bca540 usage/limit 35159/35161 o=0
[ 0] 0xffffffff81bca540 pt : slabs=1 elements=3/21, suspicious=0
[ 0] 0xffffffff81bca540 mdb: slabs=1084 elements=30001/30352, suspicious=0
[ 0] 0xffffffff81bca540 sm : slabs=2 elements=3/36, suspicious=0
[ 0] 0xffffffff81bca540 sc : slabs=4 elements=53/72, suspicious=0
[ 0] 0xffffffff81bca540 ec : slabs=12 elements=54/72, suspicious=0
[ 0] 0xffffffff81bca540 fpu: slabs=68 elements=53/68, suspicious=0
Warning: kernel memory quota upgrade failed - trigger memory free up for causing 'init -> runtime -> falkon-jemalloc -> falkon':'signal handler' - donator is
 'core':'unknown', policy=2
Warning: usage before=35159 after=35159 - limit before=35161 after=35161

[alex-ab]

@cproc: can you please try to adjust in repos/base-nova/src/kernel/nova/target.mk the value of CONFIG_MEMORY_DYN_PER_MILL to some higher value (default is 1% of system memory is for the kernel), just to see whether the kernel memory ever will be sufficient. core will report the effective kernel memory during boot.