epmd monitor consumes one full CPU core on Windows

[xjlixd]

RabbitMQ 3.7.18,Erlang 21.3.8.6.

Firstly,epmd-starter process 3701 is startting,maybe the start-time is 15:11,the command result as follow:
3701 41924 Thu Jun 24 15:11:13 2021 01:56:48 100 52376 /usr/lib64/erlang/erts-10.3.5.4/bin/beam.smp -- -root /usr/lib64/erlang -progname erl -- -home /var/lib/rabbitmq -- -sname epmd-starter-167737917 -noshell -eval halt().

and the command is :
# ps -eo pid,ppid,lstart,etime,pcpu,rss,cmd | grep beam.smp | grep -Ev "grep|rabbit@hcapp-test"

Now , the RabbitMQ's process Id is 40203, its CPU usage is 101.0% .
When I kill the process 3701 ,the CPU usage of RabbitMQ will be work normally.

and meanwhile the GC and context switch of Rabbitmq is very high, and the most CPU intensive thread is rabbit_epmd_monitor, the detail info as following image.

So，I want to how to solve this problem, and the root cause of this phenomenon.

[michaelklishin]

I will convert this issue to a GitHub discussion. Currently GitHub will automatically close and lock the issue even though your question will be transferred and responded to elsewhere. This is to let you know that we do not intend to ignore this but this is how the current GitHub conversion mechanism makes it seem for the users :(

[michaelklishin]

RabbitMQ 3.7 has beenout of support for more than six months now. Erlang 21 is three major versions behind.

Unless you can provide a set of steps to reasonably reliably reproduce this behavior on 3.8.18 and Erlang 23+, I'm afraid it's on you to investigate.

The epmd monitor module is small, so narrowing things down should be easier. strace(1) can help understand what OS process maybe doing. I don't remember if 3.7.18 has runtime thread metrics.

My best hypothesis is that the module cannot connect to epmd or fails to run a port because of e.g. a security policy, so it retries in what becomes an infinite loop.

[xjlixd]

today,the empd-stater is running well, for example:
16804 41924 Wed Jun 30 19:19:23 2021 00:00 0.0 36872 /usr/lib64/erlang/erts-10.3.5.4/bin/beam.smp -- -root /usr/lib64/erlang -progname erl -- -home /var/lib/rabbitmq -- -sname epmd-starter-407557515 -noshell -eval halt().
......... about 10 rows。
25729 41924 Wed Jun 30 19:35:29 2021 00:00 0.0 29304 /usr/lib64/erlang/erts-10.3.5.4/bin/beam.smp -- -root /usr/lib64/erlang -progname erl -- -home /var/lib/rabbitmq -- -sname epmd-starter-889460534 -noshell -eval halt().

[michaelklishin]

Here are some relevant functions. They start a subprocess VM that would start epmd. This is necessary because on some operating systems, epmd can be terminated in certain cases without RabbitMQ being notified this in any way. IIRC
on Windows, a logout of the effective user would lead to this.

So here's how epmd is kept running periodically:

ensure_epmd() ->
    Exe = rabbit_runtime:get_erl_path(),
    ID = rabbit_misc:random(1000000000),
    Port = open_port(
             {spawn_executable, Exe},
             [{args, ["-boot", "no_dot_erlang",
                      "-sname", rabbit_misc:format("epmd-starter-~b", [ID]),
                      "-noinput", "-s", "erlang", "halt"]},
              exit_status, stderr_to_stdout, use_stdio]),
    port_shutdown_loop(Port).

port_shutdown_loop(Port) ->
    receive
        {Port, {exit_status, _Rc}} -> ok;
        {Port, _}                  -> port_shutdown_loop(Port)
    end.

Note that second port_shutdown_loop/1 function. It accepts a port (a subprocess if you will, nothing to do with TCP or UDP ports)
and waits for it to exit. If it does not exit with a clearly communicated status, the loop immediately recurses.

While I cannot immediately tell why the port exit is never received, we can introduce a timeout that would at least avoid the loop.
This check is performed every so often so if it fails once, it's not going to matter much: the "check" is best effort anyway.

[michaelklishin]

erlang:open_port/2 is called on the RabbitMQ node. You are correct that the subprocess is started with a "trampoline" erl_child_setup process.

Note that the name of the subprocess Erlang VM (--sname …) is identical in all those calls. So rabbit_nodes:ensure_epmd/0 is called just once. This looks like a behavior of erlang:open_port/2 then. Anyhow,
we should have a timeout in the waiting part.

[michaelklishin]

I could not find anything in runtime release notes for erlang:open_port/2 entering a loop but there are definitely bugs that specifically mention it that were addressed in Erlang 22.x (ERTS 10.x).

[xjlixd]

Yestoday evening this problem occurs again.The epmd-started process 55050 start at Wed Jun 30 22:21:25 2021,and it ran for 11 hours, whose parent process is 41924,the erl_child_setup.
55050 41924 Wed Jun 30 22:21:25 2021 10:47:19 100 51252 /usr/lib64/erlang/erts-10.3.5.4/bin/beam.smp -- -root /usr/lib64/erlang -progname erl -- -home /var/lib/rabbitmq -- -sname epmd-starter-734552564 -noshell -eval halt().

• And epmd-started and RabbitMQ all cosume more and more system CPU usage.

• 2 strace for epmd-started
  Only select in system call.
  [root@hcapp-test ~]# strace -p 55050
  strace: Process 55050 attached
  select(0, NULL, NULL, NULL, NULL^Cstrace: Process 55050 detached
  <detached ...>

• 3 perf report for RabbitMQ process 40203
  

• 4 perf report for epmd-starter 55050

• 5 perf report for RabbitMQ 40203 after kill the epmd-starter，a liitle time for system call
  
  
  

FYI, maybe these system call have bo relation to open_port , and the latter maybe call tcp satck .

[xjlixd]

maybe the spink lock is the root cause.

[michaelklishin]

This increasingly looks like a problem with the runtime implementation of erlang:open_port/2. Please upgrade to the latest release series you can.