pm2 god daemon crash. malloc(): unaligned tcache chunk detected

[amit777]

What's going wrong?

pm2 got daemon crashes on a couple of our Ubuntu 22.04 servers

How could we reproduce this issue?

I can't reproduce at will it will happen once every day or 2.

Supporting information

$ pm2 report
PM2        | 2023-04-01T17:02:21: PM2 log: [PM2][WORKER] Reset the restart delay, as app notifications has been up for more than 30000ms
PM2        | malloc(): unaligned tcache chunk detected
PM2        | 2023-04-02T14:41:41: PM2 log: ===============================================================================
PM2        | 2023-04-02T14:41:41: PM2 log: --- New PM2 Daemon started ----------------------------------------------------
PM2        | 2023-04-02T14:41:41: PM2 log: Time                 : Sun Apr 02 2023 14:41:41 GMT-0400 (Eastern Daylight Time)
PM2        | 2023-04-02T14:41:41: PM2 log: PM2 version          : 5.3.0
PM2        | 2023-04-02T14:41:41: PM2 log: Node.js version      : 18.15.0
PM2        | 2023-04-02T14:41:41: PM2 log: Current arch         : x64
PM2        | 2023-04-02T14:41:41: PM2 log: PM2 home             : /root/.pm2
PM2        | 2023-04-02T14:41:41: PM2 log: PM2 PID file         : /root/.pm2/pm2.pid
PM2        | 2023-04-02T14:41:41: PM2 log: RPC socket file      : /root/.pm2/rpc.sock
PM2        | 2023-04-02T14:41:41: PM2 log: BUS socket file      : /root/.pm2/pub.sock
PM2        | 2023-04-02T14:41:41: PM2 log: Application log path : /root/.pm2/logs
PM2        | 2023-04-02T14:41:41: PM2 log: Worker Interval      : 30000
PM2        | 2023-04-02T14:41:41: PM2 log: Process dump file    : /root/.pm2/dump.pm2
PM2        | 2023-04-02T14:41:41: PM2 log: Concurrent actions   : 2
PM2        | 2023-04-02T14:41:41: PM2 log: SIGTERM timeout      : 1600
PM2        | 2023-04-02T14:41:41: PM2 log: ===============================================================================
PM2        | 2023-04-02T14:41:41: PM2 log: App [pm2-logrotate:0] starting in -fork mode-
PM2        | 2023-04-02T14:41:41: PM2 log: App [pm2-logrotate:0] online

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /root/.nvm/versions/node/v18.15.0/bin/node...
[New LWP 8158]
[New LWP 8164]
[New LWP 8159]
[New LWP 8160]
[New LWP 8167]
[New LWP 8163]
[New LWP 8161]
[New LWP 8162]
[New LWP 8168]
[New LWP 8166]
[New LWP 8169]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `PM2 v5.3.0: God Daemon (/root/.pm2)                                      '.
Program terminated with signal SIGABRT, Aborted.
#0  __pthread_kill_implementation (no_tid=0, signo=6, threadid=140525053355968) at ./nptl/pthread_kill.c:44
44	./nptl/pthread_kill.c: No such file or directory.
[Current thread is 1 (Thread 0x7fce89e327c0 (LWP 8158))]
(gdb) bt
#0  __pthread_kill_implementation (no_tid=0, signo=6, threadid=140525053355968) at ./nptl/pthread_kill.c:44
#1  __pthread_kill_internal (signo=6, threadid=140525053355968) at ./nptl/pthread_kill.c:78
#2  __GI___pthread_kill (threadid=140525053355968, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
#3  0x00007fce89842476 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#4  0x00007fce898287f3 in __GI_abort () at ./stdlib/abort.c:79
#5  0x00007fce898896f6 in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x7fce899dbb8c "%s\n") at ../sysdeps/posix/libc_fatal.c:155
#6  0x00007fce898a0d7c in malloc_printerr (str=str@entry=0x7fce899ded40 "malloc(): unaligned tcache chunk detected") at ./malloc/malloc.c:5664
#7  0x00007fce898a545c in tcache_get (tc_idx=<optimized out>) at ./malloc/malloc.c:3195
#8  __GI___libc_malloc (bytes=64) at ./malloc/malloc.c:3313
#9  0x00007fce89cae9cc in operator new(unsigned long) () from /lib/x86_64-linux-gnu/libstdc++.so.6
#10 0x00000000010805c0 in v8::internal::BackingStore::Allocate(v8::internal::Isolate*, unsigned long, v8::internal::SharedFlag, v8::internal::InitializedFlag)
    ()
#11 0x0000000000d7a1a4 in v8::ArrayBuffer::NewBackingStore(v8::Isolate*, unsigned long) ()
#12 0x0000000000b11fa3 in node::Environment::allocate_managed_buffer(unsigned long) ()
#13 0x0000000000c6f528 in node::LibuvStreamWrap::ReadStart()::{lambda(uv_handle_s*, unsigned long, uv_buf_t*)#1}::_FUN(uv_handle_s*, unsigned long, uv_buf_t*)
    ()
#14 0x0000000001676dba in uv__read (stream=stream@entry=0x7fce88566390) at ../deps/uv/src/unix/stream.c:1105
#15 0x0000000001677790 in uv__stream_io (loop=<optimized out>, w=0x7fce88566418, events=1) at ../deps/uv/src/unix/stream.c:1270
#16 0x000000000167d534 in uv__io_poll (loop=loop@entry=0x526dfc0 <default_loop_struct>, timeout=<optimized out>) at ../deps/uv/src/unix/epoll.c:374
#17 0x000000000166b95e in uv_run (loop=0x526dfc0 <default_loop_struct>, mode=UV_RUN_DEFAULT) at ../deps/uv/src/unix/core.c:406
#18 0x0000000000abda2d in node::SpinEventLoop(node::Environment*) ()
#19 0x0000000000bc1874 in node::NodeMainInstance::Run() ()
#20 0x0000000000b36434 in node::LoadSnapshotDataAndRun(node::SnapshotData const**, node::InitializationResult const*) ()
#21 0x0000000000b3a02f in node::Start(int, char**) ()
#22 0x00007fce89829d90 in __libc_start_call_main (main=main@entry=0xab1f70 <main>, argc=argc@entry=2, argv=argv@entry=0x7fff3513b198)
    at ../sysdeps/nptl/libc_start_call_main.h:58
#23 0x00007fce89829e40 in __libc_start_main_impl (main=0xab1f70 <main>, argc=2, argv=0x7fff3513b198, init=<optimized out>, fini=<optimized out>,
    rtld_fini=<optimized out>, stack_end=0x7fff3513b188) at ../csu/libc-start.c:392
#24 0x0000000000aba37e in _start ()

[Anirudhx234]

Ever find out what the issue was? Getting the same error

[amit777]

Try deleted node_modules and reinstalling. Also try reinstalling global pm2 daemon. If you have multiple versions of node installed via nvm, make sure you don't have any weird inconsistencies in running procs.

[Anirudhx234]

Seems to be working so far. Thanks!