Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kernel-5.10: enable mellanox modules #2241

Merged
merged 1 commit into from
Jun 28, 2022

Conversation

etungsten
Copy link
Contributor

@etungsten etungsten commented Jun 24, 2022

Issue number:
N/A

Description of changes:

    kernel-5.10: enable mellanox modules
    
    This adds the kernel configs necessary for Mellanox NICs.

Testing done:
Built metal-k8s-1.23 without problem. Provisioned onto a HP DL160 with Mellanox Connect X-5 ethernet cards and the host booted up with network interfaces correctly configured.

Kernel configuration changes:
x86_64:

diff --git a/x86_64 without b/x86_64 with
index a0cfca8..adc3268 100644
--- a/x86_64 without    
+++ b/x86_64 with       
@@ -1677,7 +1677,7 @@
  CONFIG_MPLS_IPTUNNEL=m
  CONFIG_NET_NSH=m
  CONFIG_HSR=m
- # CONFIG_NET_SWITCHDEV is not set
+ CONFIG_NET_SWITCHDEV=y
  CONFIG_NET_L3_MASTER_DEV=y
  # CONFIG_QRTR is not set
  # CONFIG_NET_NCSI is not set
@@ -2411,7 +2411,22 @@
  # CONFIG_IGC is not set
  # CONFIG_JME is not set
  # CONFIG_NET_VENDOR_MARVELL is not set
- # CONFIG_NET_VENDOR_MELLANOX is not set
+ CONFIG_NET_VENDOR_MELLANOX=y
+ # CONFIG_MLX4_EN is not set
+ CONFIG_MLX5_CORE=m
+ # CONFIG_MLX5_FPGA is not set
+ CONFIG_MLX5_CORE_EN=y
+ CONFIG_MLX5_EN_ARFS=y
+ CONFIG_MLX5_EN_RXNFC=y
+ CONFIG_MLX5_MPFS=y
+ CONFIG_MLX5_ESWITCH=y
+ CONFIG_MLX5_CLS_ACT=y
+ CONFIG_MLX5_CORE_EN_DCB=y
+ # CONFIG_MLX5_CORE_IPOIB is not set
+ # CONFIG_MLX5_IPSEC is not set
+ CONFIG_MLX5_SW_STEERING=y
+ # CONFIG_MLXSW_CORE is not set
+ CONFIG_MLXFW=m
  # CONFIG_NET_VENDOR_MICREL is not set
  # CONFIG_NET_VENDOR_MICROCHIP is not set
  # CONFIG_NET_VENDOR_MICROSEMI is not set
@@ -4026,6 +4041,7 @@
  # CONFIG_INFINIBAND_MTHCA is not set
  # CONFIG_INFINIBAND_EFA is not set
  # CONFIG_MLX4_INFINIBAND is not set
+ CONFIG_MLX5_INFINIBAND=m
  # CONFIG_INFINIBAND_OCRDMA is not set
  # CONFIG_INFINIBAND_VMWARE_PVRDMA is not set
  # CONFIG_INFINIBAND_USNIC is not set

aarch64:
None, presumably it's already enabled from upstream kernel package for aarch64.

Terms of contribution:

By submitting this pull request, I agree that this contribution is dual-licensed under the terms of both the Apache License, version 2.0, and the MIT license.

@etungsten etungsten marked this pull request as ready for review June 24, 2022 20:34
packages/kernel-5.10/config-bottlerocket Outdated Show resolved Hide resolved
packages/kernel-5.10/config-bottlerocket Outdated Show resolved Hide resolved
packages/kernel-5.10/config-bottlerocket Outdated Show resolved Hide resolved
packages/kernel-5.10/config-bottlerocket Show resolved Hide resolved
@etungsten etungsten force-pushed the mellanox branch 4 times, most recently from 70b4f79 to 9a07ec7 Compare June 24, 2022 21:47
@etungsten etungsten changed the title kernel-5.10: enable infiniband, mellanox modules kernel-5.10: enable mellanox modules Jun 24, 2022
Copy link
Contributor

@zmrow zmrow left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good pending a test!

🧙

CONFIG_NET_VENDOR_MELLANOX=y
CONFIG_MLX5_CORE_EN=y
CONFIG_NET_SWITCHDEV=y
CONFIG_MLX5_BRIDGE=n
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This option would not be interesting for 5.10, as it was introduced first in 5.14.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does that mean it's safe to assume it defaults to =n and we wouldn't need to specify it here?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In a way, yes.

The option does not exist in kernels prior to 5.14. It will be ignored as there is no code that switches anything off or on, no matter what we choose here. It will probably produce something like an unknown option log line somewhere, that is all.

No occurrence of MLX5_BRIDGE in the kernel for 5.4.124:

HEAD is now at 4f3fee72a74c Linux 5.10.124
[foersleo@devbox linux]$ grep -nr MLX5_BRIDGE *
[foersleo@devbox linux]$ 

On kernels after 5.14 however, it will default to =y:

config MLX5_BRIDGE
	bool
	depends on MLX5_ESWITCH && BRIDGE
	default y
	help
	  mlx5 ConnectX offloads support for Ethernet Bridging (BRIDGE).
	  Enable adding representors of mlx5 uplink and VF ports to Bridge and
	  offloading rules for traffic between such ports. Supports VLANs (trunk and
	  access modes).

There is some handy webpages, that can show some info on which kernel versions have certain options, without having to go through the versions oneself. So far I have not seen any problems with regards to reliability of the data (probably automagically created). For example https://cateee.net/lkddb/web-lkddb/MLX5_BRIDGE.html

# Mellanox network support
CONFIG_MLXFW=m
CONFIG_MLX5_CORE=m
CONFIG_MLX5_INFINIBAND=m
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure if Infiniband is actually needed. If I read the Connect X-4 page at nvidia [1] correctly the Cards mentioned in the Overview do not support Infiniband. Connect X-[5,6,7] seem to be the infiniband offerings by Mellanox [2].

[1] https://www.nvidia.com/en-us/networking/ethernet/connectx-4-lx/
[2] https://www.nvidia.com/en-us/networking/infiniband-adapters/

Copy link
Contributor Author

@etungsten etungsten Jun 27, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I was mistaken in assuming it was Connect X-4. I was just basing it off of a HP DL160 spec file I found but apparently DL160 has the option for Connect X-5 which is what the machine needing the driver has (see here). When the machine booted with ubuntu, we saw MLX5 drivers get loaded.

(I fixed the PR description)

@etungsten etungsten requested a review from foersleo June 27, 2022 18:24
This adds the kernel configs necessary for Mellanox NICs.
@etungsten
Copy link
Contributor Author

etungsten commented Jun 28, 2022

Push above removes CONFIG_MLX5_BRIDGE=n as it's not an existing option for the 5.10 kernel.

@etungsten etungsten merged commit 6fa9903 into bottlerocket-os:develop Jun 28, 2022
@etungsten etungsten deleted the mellanox branch June 28, 2022 19:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants