Ingress overwrites instead of adds X-Forwarded-* headers

[ractive]

The ingress proxy sets the X-Forwarded-* headers and hereby overwrites previous values set by other proxies. So the application only sees the X-Forwarded-* header values set by the ingress, but not by the first proxy and can therefore not create URLs pointing to itself. The ingress proxy should either add these headers (using add_header) or add its value in a comma separated list to an existing header value. Unfortunately there's no standard describing these X-Forwarded headers, but it seems as if both variants (multiple headers and comma separated values) are seen in the wild.
The changes would be probably needed to be done here: /~https://github.com/kubernetes/contrib/blob/master/ingress/controllers/nginx/nginx.tmpl#L276

We concretely face this issue because we terminate all https connections on an external loadbalancer (e.g. https://api.example.com) that then routes the traffic via http to an ingress endpoint (e.g. http://internal.example.com). The redirect URI that is created by the internal app is now https://api.example.com:80 - which is wrong. This is because the ingress sets the X-Forwarded-Port header to 80. The X-Forwarded-Port value 443 which is set by the loadbalancer (where ssl is terminated) should be preserved.

This issue shows up in a Spring Boot application with OAuth2 protected resources, where the app creates the redirect_uri pointing to itself sent to the OAuth provider using a UriComponentsBuilder [1] in the OAuth2ClientContextFilter [2]. The UriComponentsBuilder builds the URL using the X-Forwarded-* headers, picking the first header and the first value (if comma spearated).

[1] /~https://github.com/spring-projects/spring-framework/blob/master/spring-web/src/main/java/org/springframework/web/util/UriComponentsBuilder.java#L705

[2] /~https://github.com/spring-projects/spring-security-oauth/blob/master/spring-security-oauth2/src/main/java/org/springframework/security/oauth2/client/filter/OAuth2ClientContextFilter.java#L122

[aledbf]

@ractive thanks for the report. Please check this /~https://github.com/kubernetes/contrib/blob/master/ingress/controllers/nginx/nginx.tmpl#L110
I will add the same logic (pass the value if the header exists or add a default) in:

• X-Forwarded-Host
• X-Forwarded-Port
• X-Forwarded-Proto

[aledbf]

@ractive sorry but this is already fixed in the new version. Please check /~https://github.com/kubernetes/ingress/blob/master/controllers/nginx/rootfs/etc/nginx/template/nginx.tmpl#L110

[ractive]

Ah ok. So I looked in the wrong nginx.tmpl file. Is this already part of a released version?

[aledbf]

@ractive not yet, next week.

[ractive]

Great! We're looking forward to this release!
Thanks.

[ractive]

We still have this issue. We're using this server version:

Server Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.4+coreos.0", GitCommit:"97c11b097b1a2b194f1eddca8ce5468fcc83331c", GitTreeState:"clean", BuildDate:"2017-03-08T23:54:21Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"}

Do we need to somehow update the current ingress instances to use the updated template?

[alikhan-io]

+1 on this. Has this been resolved to anyones knowledge?

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle rotten
/remove-lifecycle stale

[yveszoundi]

-After couple of rather long investigations, I found this page. I have exactly the same problem for an application using Java libraries (spring boot + spring oauth2) deployed into Kubernetes (ingress-nginx).

@aledbf, it seems to me that there's something off about the "rewrite-target' annotation.

Scenario
Let's look at the following scenario where I deploy an app "/app" via an ingress at "/secureapp/app".
This is a java oauth2 application (Spring Boot + Tomcat application server) deployed into Kubernetes on either Azure AKS or Minikube for local integration testing.

annotations:
  nginx.ingress.kubernetes.io/rewrite-target: /
  other_annotations: other_annotations_values
path: /secureapp # The regular path is /app so it becomes /secureapp/app

Expectations

• I would expect the rewrite rules to automatically set the "x-forwarded-prefix" maybe via another annotation? Tomcat would then automatically use "/secureapp/app" instead of just "/app". Do I misunderstand how "rewrite-target" behaves?
• If possible, the cookie path would be automatically adjusted at the ingress level for tomcat "JSESSIONID" cookie (as well as other cookies).

Underlying problems

• At the application level, some "bad redirects" are happening because the application server proxied via nginx doesn't know how to use the ingress path as prefix -> The first attempt to deal with this is to correct URLs manually in the browser.
• During the oauth2 authentication/authorization process, potential CSRF errors are thrown because session cookies get lost due to wrong path settings

[yveszoundi]

@ractive, I found a rather intrusive workaround per code/configuration below, it was a rather painful exercise, if it solves your problems too, you owe me beer :-) !
@aledbf , is there something that I'm missing that could reduce complexity that is obvious to you?

A) Ingress side work

    nginx.ingress.kubernetes.io/session-cookie-hash: "sha1"
    nginx.ingress.kubernetes.io/enable-cors: "true"
    nginx.ingress.kubernetes.io/cors-allow-headers: "X-Forwarded-For, X-Forwarded-Proto, X-Forwarded-Port, X-Forwarded-Prefix"    
    nginx.ingress.kubernetes.io/configuration-snippet: |
      more_set_headers "Request-Id: $request_id";
      more_set_headers "X-Forwarded-Host: $host";
      more_set_headers "X-Forwarded-Proto: $scheme";
      more_set_headers "X-Forwarded-For: $proxy_add_x_forwarded_for";
      proxy_set_header X-Forwarded-Prefix "/secureapp/app";
      #doesnotwork proxy_cookie_path /app /secureapp/app;

  path: /secureapp

B) Java side work

a) Use the X-forwarded- headers in Tomcat (Spring oauth2 application)

We want to force Java to pick up x-forwarded-host, x-forwarded-proto and x-forwarded-prefix.

@Bean
public FilterRegistrationBean forwardHeadersFilterBean() {
   FilterRegistrationBean bean = new FilterRegistrationBean(new ForwardedHeaderFilter());
   bean.setOrder(Ordered.HIGHEST_PRECEDENCE);
   return bean;
 }

b) Prepend the ingress location to the application cookie path

By default to the application cookie path is "/app", we want it to be "/secureapp/app" inside Kubernetes.

// At the ingress level, proxy_cookie_path doesn't seem to work for me
@Bean
public ServletContextInitializer servletContextInitializer() {
  return new ServletContextInitializer() {

     @Override
     public void onStartup(ServletContext servletContext) throws ServletException {
       SessionCookieConfig sessionCookieConfig = servletContext.getSessionCookieConfig();
       sessionCookieConfig.setPath("/secureapp/app");
     }
  };
}

[ractive]

To do a quick fix we called the internal service on kubernetes also via https routed through our loadbalancer that terminates SSL. So honestly I did not follow this any further. Isn't this fixed as stated in #2312 (comment) ?

[SEJeff]

@ractive does your external load balancer support the PROXY protocol? haproxy, nginx, and elb all support it. It will mitigate this problem for you entirely (and is generally a better / protocol agnostic way of preserving source ip).

See this document for how to set it up.

[ractive]

Unfortunately not, no. :-/

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[ryandawsonuk]

Another way to set x-forwarded-prefix is kubernetes/ingress-nginx#1805 (comment)