Immediately get "Disconnected. Will refresh when connected." after picking a team, or /reconnect ing?

[CaerphillyMediaLtd]

I can't figure out this now either. When I sign in on the clieent I get to the pick a team screen, then I pick one and immediately I get "Disconnected. Will refresh when connected.". If I type /reconnect then it immediately comes back with "Disconnected. Will refresh when connected." instantly.

I've tried setting:

showTypingIndicator = False

But it still seems to happen?

I'm on version:

matterhorn 50200.6.0 (master@a2e55f7)
     using mattermost-api 50200.3.0 (master@a2e55f7)

Our MatterMost is setup through an nginx proxy, which handles the SSL, then connects via HTTP to MM locally.

[jtdaugherty]

A couple of questions:

• Have you tried Matterhorn 50200.7.0?
• Could you enable logging with matterhorn -l logfile.txt, run Matterhorn long enough to see some of these disconnections, and then provide the log here?

[jtdaugherty]

Another question:

Our MatterMost is setup through an nginx proxy, which handles the SSL, then connects via HTTP to MM locally.

Just to confirm: is it the case that the web client works without issues in that setup?

[CaerphillyMediaLtd]

Could you enable logging with matterhorn -l logfile.txt, run Matterhorn long enough to see some of these disconnections, and then provide the log here?

I solved it myself by looking at them. I'll go through it with you so maybe this fix doesn't have to be added?

The log from Matterhorn was (I can supply full logs if you like, but this is the problematic part):

[2020-03-17 18:47:46.969347955 UTC] [LogWebsocket] [*] Websocket error: MalformedResponse (ResponseHead {responseCode = 400, responseMessage = "Bad Request", responseHeaders = [("Server","nginx/1.14.0 (Ubuntu)"),("Date","Tue, 17 Mar 2020 18:47:46 GMT"),("Content-Type","text/plain; charset=utf-8"),("Content-Length","192"),("Connection","keep-alive"),("Expires","0"),("Sec-Websocket-Version","13"),("Vary","Accept-Encoding"),("X-Content-Type-Options","nosniff"),("X-Request-Id","6crwcd8jrfy7fqyz4gjs7er38h"),("X-Version-Id","5.20.0.5.20.2.e17ee90598054f200aeb55a3646bd7ef.false")]}) "Wrong response status or message."
[2020-03-17 18:47:51.503926111 UTC] [LogWebsocket] [*] Terminating previous websocket thread
[2020-03-17 18:47:51.503931689 UTC] [LogWebsocket] [*] Starting new websocket thread
[2020-03-17 18:47:51.657441207 UTC] [LogWebsocket] [*] Websocket error: MalformedResponse (ResponseHead {responseCode = 400, responseMessage = "Bad Request", responseHeaders = [("Server","nginx/1.14.0 (Ubuntu)"),("Date","Tue, 17 Mar 2020 18:47:51 GMT"),("Content-Type","text/plain; charset=utf-8"),("Content-Length","192"),("Connection","keep-alive"),("Expires","0"),("Sec-Websocket-Version","13"),("Vary","Accept-Encoding"),("X-Content-Type-Options","nosniff"),("X-Request-Id","p89k8ur13irttm7stgps5hz1nh"),("X-Version-Id","5.20.0.5.20.2.e17ee90598054f200aeb55a3646bd7ef.false")]}) "Wrong response status or message."
[2020-03-17 18:47:54.665978028 UTC] <<< Logging end >>>

I checked the nginx proxy logs, and it wasn't creating any issues. So I checked the MatterMost logs, and found the following corresponding to a request:

{"level":"error","ts":1584472860.4649854,"caller":"api4/websocket.go:28","msg":"websocket connect err.","error":"websocket: the client is not using the websocket protocol: 'upgrade' token not found in 'Connection' header"}
{"level":"error","ts":1584472860.4650764,"caller":"mlog/log.go:175","msg":"Failed to upgrade websocket connection","path":"/api/v4/websocket","request_id":"x6bszgdhpfyzfcehhtsit94o6r","ip_addr":"89.242.208.192","user_id":"9om63om8rjnxzxj1za84435f5c","method":"GET","err_where":"connect","http_code":500,"err_details":""}
{"level":"error","ts":1584472860.4651434,"caller":"http/server.go:3053","msg":"http: superfluous response.WriteHeader call from github.com/mattermost/mattermost-server/v5/web.Handler.ServeHTTP (handlers.go:200)","source":"httpserver"}

So it looks like it's an issue with missing Upgrade headers in the http header? So I added:

proxy_set_header        Upgrade $http_upgrade;
proxy_set_header        Connection "upgrade";

It works fine now, I'm not sure if this is something I should have had in there to begin with? Although I'm also reading that this has been removed in HTTP2. Maybe a patch could be added to re-establish a new connection if the upgrade fails (if I'm understanding the problem correctly)?

[kquick]

That's great news, @CaerphillyMediaLtd ! Would you be willing to share the (anonymized) set of nginx configuration elements you are running with that pertain to matterhorn/mattermost? It might be worth adding that to our documentation to help other folks setting things up behind a proxy like you have.

[CaerphillyMediaLtd]

@kquick Sure, here's the config we use. We don't include:

proxy_set_header        Upgrade $http_upgrade;
proxy_set_header        Connection "upgrade";

On other configs, for e.g. gitlab, nextcloud, etc.

[jtdaugherty]

An FAQ entry about this has been added to the README. Thanks!