Unable to see logstash under Cluster overview and encountering error [ERROR][logstash.licensechecker.licensereader]

[btmndkh]

I am experiencing an issue where I am unable to see logstash under Cluster overview in my setup. Additionally, I am encountering the following error messages repeatedly:

elastic-logstash-1 | [2023-06-07T06:53:13,041][ERROR][logstash.licensechecker.licensereader] Unable to retrieve license information from license server {:message=>"No Available connections"}
elastic-logstash-1 | [2023-06-07T06:53:13,706][INFO ][logstash.licensechecker.licensereader] Failed to perform request {:message=>"PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target", :exception=>Manticore::ClientProtocolException, :cause=>#<Java::JavaxNetSsl::SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target>}
elastic-logstash-1 | [2023-06-07T06:53:13,709][WARN ][logstash.licensechecker.licensereader] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"https://elastic:xxxxxx@elasticsearch:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [https://elasticsearch:9200/][Manticore::ClientProtocolException] PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target"}
elastic-logstash-1 | [2023-06-07T06:53:43,041][ERROR][logstash.licensechecker.licensereader] Unable to retrieve license information from license server {:message=>"No Available connections"}
elastic-logstash-1 | [2023-06-07T06:53:43,738][INFO ][logstash.licensechecker.licensereader] Failed to perform request {:message=>"PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target", :exception=>Manticore::ClientProtocolException, :cause=>#<Java::JavaxNetSsl::SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target>}
elastic-logstash-1 | [2023-06-07T06:53:43,744][WARN ][logstash.licensechecker.licensereader] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"https://elastic:xxxxxx@elasticsearch:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [https://elasticsearch:9200/][Manticore::ClientProtocolException] PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target"}
elastic-logstash-1 | [2023-06-07T06:54:13,040][ERROR][logstash.licensechecker.licensereader] Unable to retrieve license information from license server {:message=>"No Available connections"}
elastic-logstash-1 | [2023-06-07T06:54:13,771][INFO ][logstash.licensechecker.licensereader] Failed to perform request {:message=>"PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target", :exception=>Manticore::ClientProtocolException, :cause=>#<Java::JavaxNetSsl::SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target>}
elastic-logstash-1 | [2023-06-07T06:54:13,772][WARN ][logstash.licensechecker.licensereader] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"https://elastic:xxxxxx@elasticsearch:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [https://elasticsearch:9200/][Manticore::ClientProtocolException] PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target"}

My configuration settings are set to the default values as follows:
*ELK 8.8.0

git clone /~https://github.com/sherifabdlnaby/elastdocker.git
make setup
make elk

I have checked the status of the logstash containers and they appear to be healthy:

[root@elk1 elastdocker]# docker ps -a
CONTAINER ID   IMAGE                             COMMAND                  CREATED         STATUS                   PORTS                                                                                  NAMES
fb3e1e604b3b   elastdocker/logstash:8.8.0        "/usr/local/bin/dock…"   6 minutes ago   Up 6 minutes (healthy)   0.0.0.0:5044->5044/tcp, :::5044->5044/tcp, 0.0.0.0:9600->9600/tcp, :::9600->9600/tcp   elastic-logstash-1
1aa62265a433   elastdocker/kibana:8.8.0          "/bin/tini -- /usr/l…"   6 minutes ago   Up 6 minutes             0.0.0.0:5601->5601/tcp, :::5601->5601/tcp                                              elastic-kibana-1
ea7e23818a0a   elastdocker/apm-server:8.8.0      "/usr/bin/tini -- /u…"   6 minutes ago   Up 6 minutes             0.0.0.0:8200->8200/tcp, :::8200->8200/tcp                                              elastic-apm-server-1
814c7980cd7b   elastdocker/elasticsearch:8.8.0   "/bin/tini -- /usr/l…"   6 minutes ago   Up 6 minutes (healthy)   0.0.0.0:9200->9200/tcp, :::9200->9200/tcp, 0.0.0.0:9300->9300/tcp, :::9300->9300/tcp   elastic-elasticsearch-1

Anyone else have a similar issue too?

[btmndkh]

I tried both old and latest versions of Docker, but I am still encountering some issues, FYI.

I tested it with version 8.7.1, and there were no issues at all.

Here is the output of the "docker version":

Client: Docker Engine - Community
 Version: 24.0.2
 API version: 1.43
 Go version: go1.20.4
 Git commit: cb74dfc
 Built: Thu May 25 21:53:10 2023
 OS/Arch: linux/amd64
 Context: default

Server: Docker Engine - Community
 Engine:
  Version: 24.0.2
  API version: 1.43 (minimum version 1.12)
  Go version: go1.20.4
  Git commit: 659604f
  Built: Thu May 25 21:52:10 2023
  OS/Arch: linux/amd64
  Experimental: false
 containerd:
  Version: 1.6.21
  GitCommit: 3dce8eb055cbb6872793272b4f20ed16117344f8
 runc:
  Version: 1.1.7
  GitCommit: v1.1.7-0-g860f061
 docker-init:
  Version: 0.19.0
  GitCommit: de40ad0

And the output of the "docker-compose version" :

Docker Compose version v2.18.1

[db0109]

I received the same error.

[btmndkh]

I tried using v8.8.1, but I am still see the same issue. I am unable to view Logstash on Kibana, and I keep receiving the same error from elastic-logstash-1.

Any ideas?

[btmndkh]

It appears that the ssl_certificate_verification (main.conf-output) option is deprecated.

ssl_certificate_verification

[db0109]

Awesome what's the fix for that?

…

On Sun, Jun 11, 2023, 8:01 PM bbatmandakh ***@***.***> wrote: It appears that the ssl_certificate_verification (main.conf-output) option is deprecated. ssl_certificate_verification <logstash-plugins/logstash-output-elasticsearch#1138> — Reply to this email directly, view it on GitHub <#102 (comment)>, or unsubscribe </~https://github.com/notifications/unsubscribe-auth/APD7WYOJ52AVQKRC6KOQS23XKZ2BJANCNFSM6AAAAAAY5NJLDI> . You are receiving this because you commented.Message ID: ***@***.***>

[btmndkh]

I tried with the following option, but still no luck.
ssl_verification_mode => none.

It looks like I'll stuck with version 8.7.1

[btmndkh]

found related issue

elastic/logstash#15068