Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Setting for local_port_range overlaps with very common listen ports #1386

Closed
fasaxc opened this issue Mar 12, 2021 · 3 comments · Fixed by #1437 or #1560
Closed

Setting for local_port_range overlaps with very common listen ports #1386

fasaxc opened this issue Mar 12, 2021 · 3 comments · Fixed by #1437 or #1560
Assignees
Labels
area/kubernetes K8s including EKS, EKS-A, and including VMW type/bug Something isn't working
Milestone

Comments

@fasaxc
Copy link

fasaxc commented Mar 12, 2021

Image I'm using:

Bottlerocket appears to set the local port range to

net.ipv4.ip_local_port_range = 1025 65000

In this file:
/~https://github.com/bottlerocket-os/bottlerocket/blob/v1.0.5/packages/release/release-sysctl.conf#L23

What I expected to happen:

Ports in the 8000-10000 range should be available for services to listen on (e.g. HTTP servers on 8081, common healthcheck ports etc).

What actually happened:

Outbound sockets can claim those ports resulting in sporadic failures to bind. I work on a project that opens port 9090; one of our users reported sporadic failures on bottlerocket because port 9090 gets used at random for outbound connections.

How to reproduce the problem:

Make lots of outbound connections, try to bind to say port 8080; it may fail at random if an outbound connection happened to use that port.

@bcressey bcressey added the status/needs-triage Pending triage or re-evaluation label Mar 15, 2021
@arnaldo2792
Copy link
Contributor

Hello @fasaxc , thanks for the issue!

As you mentioned, Bottlerocket uses a wide port range for net.ipv4.ip_local_port_range, the reason being that applications and services running in the host could create many outbound connections.

You can override this value through the API as a workaround for now, either using user data or API calls from the control/admin containers:

# With user data
#...
[settings.kernel.sysctl]
"net.ipv4.ip_local_port_range" = "<range>"
# With API calls
apiclient set --json '{"kernel": {"sysctl": {"net.ipv4.ip_local_port_range": "<range>"}}}'

We will evaluate reducing the range for the next release. This raises some compatibility concerns since users may be relying on the existing value, but there are compatibility concerns with the current default as well.

@fasaxc
Copy link
Author

fasaxc commented Mar 17, 2021

Thanks, I'll let the user who raised it know how to configure that.

Clearly, there's a trade-off here but we certainly found it surprising that there were no guaranteed-available server ports > 1024.

@gregdek gregdek added type/bug Something isn't working priority/p1 and removed status/needs-triage Pending triage or re-evaluation labels Mar 19, 2021
@bcressey
Copy link
Contributor

bcressey commented Mar 19, 2021

Current thinking is that we'll align this with the ECS values in the variants for Kubernetes 1.20 onward.

@bcressey bcressey added the area/kubernetes K8s including EKS, EKS-A, and including VMW label Mar 22, 2021
@jhaynes jhaynes added this to the next milestone Mar 26, 2021
@jhaynes jhaynes assigned etungsten and unassigned jhaynes Mar 31, 2021
@etungsten etungsten modified the milestones: next, next+1 Apr 1, 2021
@etungsten etungsten linked a pull request Apr 5, 2021 that will close this issue
5 tasks
@bcressey bcressey modified the milestones: next+1, next Apr 5, 2021
@jhaynes jhaynes added the status/in-progress This issue is currently being worked on label Apr 5, 2021
@bcressey bcressey removed the status/in-progress This issue is currently being worked on label Nov 11, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/kubernetes K8s including EKS, EKS-A, and including VMW type/bug Something isn't working
Projects
None yet
6 participants