RSS growing into several hundreds of megabytes and never going down in size in any significant way when no activity is being served.

[reportingstuffandthings]

Version

18.20.4

Platform

FreeBSD 14.1

Subsystem

No response

What steps will reproduce the bug?

I would like to ask if following behaviour of RSS part of memory of node process
is within a norm. Or is this an abnormal behaviour caused by memory leaks in some ibraries.
It's hard to judge based on valgrind, as it producess several dozens of thousands of lines of log
covering several libraries after termination of server.js process by Ctrl-C.

mkdir certs
openssl req -x509 -newkey rsa:2048 -nodes -sha256 -subj '/CN=localhost' -keyout certs/cert.key -out certs/cert.pem
node server.js

run several instances of at the same time
node client

RSS starts to grow into region of several hundred of megabytes when multiple instances of cliet.js are connecting to it.
No substantial decrease of size of RSS reported by server.js been observed
in time period of up to one hour. It decreased only by few dozens of megabytes
shortly after connections stopped but remained elevated into many hundreds
of megabytes region.

It's unclear if that memory would ever be available to the OS in case on need for memory by other processes. There are many environments with limited resources where node holding right to big areas of memory without actual need to hold it for prolonged periods of time could cause problems and starve other processes of needed memory. This is why some people could find it concerning.

After terminating (Ctrl+C) server.js running as result of

valgrind --leak-check=full --show-leak-kinds=all node server.js

Few dozens of thousands of lines is being generated covering many libraries.

Very last lines of them are following

==79512==
==79512== LEAK SUMMARY:
==79512== definitely lost: 0 bytes in 0 blocks
==79512== indirectly lost: 0 bytes in 0 blocks
==79512== possibly lost: 0 bytes in 0 blocks
==79512== still reachable: 5,348,193 bytes in 23,822 blocks
==79512== suppressed: 12,556 bytes in 37 blocks
==79512==
==79512== Use --track-origins=yes to see where uninitialised values come from
==79512== For lists of detected and suppressed errors, rerun with: -s
==79512== ERROR SUMMARY: 24 errors from 12 contexts (suppressed: 0 from 0)

At this point I'm not sure if adding full log would be of any help as it's szie
reached over 3 megabytes.

Files below:

server.js

const process = require('node:process');
const https = require('https');
const fs = require('fs');
//const heapdump = require('heapdump');

const port = ; //TODO set port number 

process.on('uncaughtException', (err, origin) => {
  console.log(err);
  fs.writeSync(
    process.stderr.fd,
    `Caught exception: ${err}\n` +
    `Exception origin: ${origin}\n`,
  );
});


const privateKey = fs.readFileSync(__dirname + '/certs/cert.key');
const certificate = fs.readFileSync(__dirname + '/certs/cert.pem');

const options = { key: privateKey, 
                  cert: certificate, enableTrace: false };

const server = https.createServer(options, (req, res) => {
    res.write("Hello there!");
   // res.writeHead(200); 
    res.end();
});
  
server.on('error', (e) => {
  console.error(e);
});
  
server.on('connection', (socket) => {
  //console.log('New connection established.');

  socket.on('error', (err) => {
    console.error('Socket error:', err);
    socket.destroy(); // Close in case of error
  });

  socket.on('close', () => {
    //console.log('Socket closed.');
  });
});

  
server.listen(port, () => {
    console.log(`Server listen on https://:${port}`);
});


setInterval(function(){
  
    const formatMemoryUsage = (data) => `${Math.round(data / 1024 / 1024 * 100) / 100} MB`;
    const memoryData = process.memoryUsage();

        const memoryUsage = {
        rss: `${formatMemoryUsage(memoryData.rss)} -> Resident Set Size - total memory allocated for the process execution`,
        heapTotal: `${formatMemoryUsage(memoryData.heapTotal)} -> total size of the allocated heap`,
        heapUsed: `${formatMemoryUsage(memoryData.heapUsed)} -> actual memory used during the execution`,
        external: `${formatMemoryUsage(memoryData.external)} -> V8 external memory`,
        };

        console.log(memoryUsage);

}, 5*1000);


setInterval(function(){
     server.getConnections((err, count) => console.log('Active connections: ' + count));

    if(global.gc){
        global.gc();
       console.log("garbage free");
    }
}, 5*1000);



  /**
setTimeout(function () {
    const filename = Date.now() + '.heapsnapshot';
    heapdump.writeSnapshot(function(err, filename) {
 //   console.log('dump written to', filename);
    });    
}, 10*1000);

setTimeout(function () {
     const filename = Date.now() + '.heapsnapshot';
    heapdump.writeSnapshot(function(err, filename) {
 //   console.log('dump2 written to', filename);
    });    
}, 120*1000);
**/  

---

client.js

const https = require('https');
const fs = require('fs');

const port = ; //TODO set port number here
const host = 'host address here'; //TODO set host name here
const requestCount = 3000;


const options = {
  hostname: host,
  port: port,
  path: '/',
  method: 'GET',
  rejectUnauthorized: false
};

function makeRequest(id) {
  const req = https.request(options, (res) => {
    let data = '';

    res.on('data', (chunk) => {
      data += chunk;
    });

    res.on('end', () => {
      console.log(`Request ${id} completed. Response: ${data}`);
    });
  });

  req.on('error', (e) => {
    console.error(`Request ${id} encountered an error: ${e.message}`);
    console.error(e);
  });

  req.end();
}


for (let i = 1; i <= requestCount; i++) {
  makeRequest(i);
}

---

How often does it reproduce? Is there a required condition?

Always.

What is the expected behavior? Why is that the expected behavior?

Expected behavior: Passing control over unused memory back to the OS.

Rationale: There are many environments with limited resources where node holding right to big areas of memory without actual need to hold it for prolonged periods of time could cause problems and starve other processes of needed memory. This is why some people could find it concerning.

What do you see instead?

RSS growing into hundreds of megabytes and never decreasing significantly even after very long time.

Additional information

No response

[joyeecheung]

What memory allocator are you using? Have you tried using a memory allocator that prioritizes low RSS, like jemalloc? (Not sure if LD_PRELOAD works on FreeBSD, though, so you might need to find another way to customize the memory allocator used). It's a known issue on Linux at least that if you use glibc's memory allocator, it can retain a huge amount of memory without giving it back to the OS even though it's freeable, and it could be a 1.5GB v.s. 70MB difference when one switches to jemalloc via LD_PRELOAD on Linux IIRC. In that case there's not much Node.js can do since it depends on the memory management library you use.

[reportingstuffandthings]

This is running on FreeBSD (amd64 compatible CPU).
It's libc's malloc() implementation is jemalloc (for many years now).
List of libraries generated by ldd executed on node binary

        libz.so.6 => /lib/libz.so.6 
	libuv.so.1 => /usr/local/lib/libuv.so.1
	libbrotlidec.so.1 => /usr/local/lib/libbrotlidec.so.1
	libbrotlienc.so.1 => /usr/local/lib/libbrotlienc.so.1
	libcares.so.2 => /usr/local/lib/libcares.so.2
	libnghttp2.so.14 => /usr/local/lib/libnghttp2.so.14
	libcrypto.so.30 => /lib/libcrypto.so.30
	libssl.so.30 => /usr/lib/libssl.so.30
	libicui18n.so.74 => /usr/local/lib/libicui18n.so.74
	libicuuc.so.74 => /usr/local/lib/libicuuc.so.74
	libicudata.so.74 => /usr/local/lib/libicudata.so.74
	libutil.so.9 => /lib/libutil.so.9
	libkvm.so.7 => /lib/libkvm.so.7
	libexecinfo.so.1 => /usr/lib/libexecinfo.so.1
	libc++.so.1 => /lib/libc++.so.1
	libcxxrt.so.1 => /lib/libcxxrt.so.1
	libm.so.5 => /lib/libm.so.5
	libthr.so.3 => /lib/libthr.so.3
	libc.so.7 => /lib/libc.so.7
	libbrotlicommon.so.1 => /usr/local/lib/libbrotlicommon.so.1
	libgcc_s.so.1 => /lib/libgcc_s.so.1
	libelf.so.2 => /lib/libelf.so.2