Skip to content

Commit

Permalink
Merge pull request #2596 from yeazelm/bonding_vlan
Browse files Browse the repository at this point in the history
Add bonding and vlan tagging support to Bottlerocket
  • Loading branch information
yeazelm authored Dec 13, 2022
2 parents f915de1 + 9e43a1a commit f3dfa53
Show file tree
Hide file tree
Showing 43 changed files with 1,324 additions and 86 deletions.
101 changes: 97 additions & 4 deletions PROVISIONING-METAL.md
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,7 @@ When these services fail, your machine will not connect to any cluster and will
#### `net.toml` structure

The configuration file must be valid TOML and have the filename `net.toml`.
The first and required top level key in the file is `version`; the latest is version `2`.
The first and required top level key in the file is `version`; the latest is version `3`.
The rest of the file is a map of interface name to supported settings.
Interface names are expected to be correct as per `udevd` naming, no interface naming or matching is supported.
(See the note below regarding `udevd` interface naming.)
Expand All @@ -95,6 +95,7 @@ Interface names are expected to be correct as per `udevd` naming, no interface n

As of version `2` static addressing with simple routes is supported via the below settings.
Please keep in mind that when using static addresses, DNS information must be supplied to the system via user data: [`settings.dns`](/~https://github.com/bottlerocket-os/bottlerocket#network-settings).

* `static4` (map): IPv4 static address settings.
* `addresses` (list of quoted IPv4 address including prefix): The desired IPv4 IP addresses, including prefix i.e. `["192.168.14.2/24"]`. The first IP in the list will be used as the primary IP which `kubelet` will use when joining the cluster. If IPv4 and IPv6 static addresses exist, the first IPv4 address is used.
* `static6` (map): IPv6 static address settings.
Expand All @@ -106,10 +107,50 @@ Please keep in mind that when using static addresses, DNS information must be su
* `via` (IP address): Gateway IP address. If no gateway is provided, a scope of `link` is assumed.
* `route-metric` (integer): Relative route priority.

Example `net.toml` with comments:
Version `3` adds in support for bonding and vlan tagging.
The support is limited to mode `1` (`active-backup`) for [bonding](https://www.kernel.org/doc/Documentation/networking/bonding.txt).
Future support may include other bonding options - pull requests are welcome!
Version `3` adds the concept of virtual network devices in addition to interfaces.
The default type of device is an interface and the syntax is the same as previous versions.
The name of an interface must match an existing interface on the system such as `eno1` or `enp0s16`.
For virtual network devices, a `kind` is required.
If no `kind` is specified, it is assumed to be an interface.
Currently, `bond` and `vlan` are the two supported `kind`s.
Virtual network devices are created, and therefore a name has to be chosen.

Names for virtual network devices must conform to kernel naming restrictions:
* Names must not have line terminators in them
* Names must be between 1-15 characters
* Names must not contain `.`, `/` or whitespace

Bonding configuration creates a virtual network device across several other devices:

* Bonding configuration (map):
* `kind = "bond"`: This setting is required to specify a bond device. Required.
* `interfaces` (list of quoted strings of interfaces): Which interfaces should be added to the bond (i.e. `["eno1"]`). The first in the list is considered the default `primary`. These interfaces are "consumed" so no other configuration can refer to them. Required.
* `mode` (string): Currently `active-backup` is the only supported option. Required.
* `min-links` (integer): Number of links required to bring up the device
* `monitoring` (map): Values m ust all be of `miimon` or `arpmon` type.
The user must choose one type of monitoring and configure it fully in order for the bond to properly function.
See [section 7](https://www.kernel.org/doc/Documentation/networking/bonding.txt) for more background on what to choose.
* `miimon-frequency-ms` (integer): MII Monitoring frequency in milliseconds
* `miimon-updelay-ms` (integer): MII Monitoring delay before the link is enabled after link is detected in milliseconds
* `miimon-downdelay-ms` (integer): MII Monitoring delay before the link is disabled after link is no longer detected in milliseconds
* `arpmon-interval-ms` (integer): Number of milliseconds between intervals to determine link status, must be greater than 0
* `arpmon-validate` (one of `all`, `none`, `active`, or `backup`): What packets should be used to validate link
* `arpmon-targets` (list of quoted IPv4 address including prefix): List of targets to use for validating ARP. Min = 1, Max = 16

Vlan tagging is configured as a new virtual network device stacked on another device:

* Vlan configuration (map):
* `kind = "vlan"`: This setting is required to specify a vlan device.
* `device` (string for device): Defines the device the vlan should be configured on. If VLAN tagging is required, this device should recieve all IP address configuration instead of the underlying device.
* `id` (integer): Number between 0 and 4096 specifying the vlan tag on the device

Example `net.toml` version `3` with comments:

```toml
version = 2
version = 3

# "eno1" is the interface name
[eno1]
Expand Down Expand Up @@ -152,13 +193,65 @@ addresses = ["192.168.14.5/24"]
to = "10.10.10.0/24"
from = "192.168.14.5"
via = "192.168.14.25"

# A bond is a network device that is of `kind` `bond`
[bond0]
kind = "bond"
# Currently `active-backup` is the only supported option
mode = "active-backup"
# In this case, the vlan will have addressing, the bond is simply there for use in the vlan
dhcp4 = false
dhcp6 = false
# The first interface in the array is considered `primary` by default
interfaces = ["eno11", "eno12"]

[bond0.monitoring]
miimon-frequency-ms = 100 # 100 milliseconds
miimon-updelay-ms = 200 # 200 milliseconds
miimon-downdelay-ms = 200 # 200 milliseconds

[bond1]
kind = "bond"
mode = "active-backup"
interfaces = ["eno51" , "eno52", "eno53"]
min-links = 2 # Optional min-links
dhcp4 = true

[bond1.monitoring]
arpmon-interval-ms = 200 # 200 milliseconds
arpmon-validate = "all"
arpmon-targets = ["192.168.1.1", "10.0.0.2"]

# A vlan is a network device that is of `kind` `vlan`
# VLAN42 is the name of the device, can be anything that is a valid network interface name
[VLAN42]
kind = "vlan"
device = "bond0"
id = 42
dhcp4 = true

[internal_vlan]
kind = "vlan"
device = "eno2"
id = 1234
dhcp6 = true
```

**An additional note on network device names**
#### **An additional note on network device names**

Interface name policies are [specified in this file](/~https://github.com/bottlerocket-os/bottlerocket/blob/develop/packages/release/80-release.link#L6); with name precedence in the following order: onboard, slot, path.
Typically on-board devices are named `eno*`, hot-plug devices are named `ens*`, and if neither of those names are able to be generated, the “path” name is given, i.e `enp*s*f*`.

#### Networking configuration versions and Releases

Older networking configuration versions (such as `1` or `2`) are supported in newer releases. In order to use a newer version, the following table provides guidance on what release first enabled the version.

| Network Configuration Version | First Release |
|-------------------------------|---------------------------------------------------------------------------------|
| Version 1 | [v1.9.0](/~https://github.com/bottlerocket-os/bottlerocket/releases/tag/v1.9.0) |
| Version 2 | [v1.10.0](/~https://github.com/bottlerocket-os/bottlerocket/releases/tag/v1.10.0) |
| Version 3 | Unreleased |

### Boot Configuration

Bottlerocket for bare metal uses a feature of the Linux kernel called [Boot Configuration](https://www.kernel.org/doc/html/latest/admin-guide/bootconfig.html), which allows a user to pass additional arguments to the kernel command line at runtime.
Expand Down
163 changes: 163 additions & 0 deletions sources/api/netdog/src/net_config/devices/bonding.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,163 @@
use super::validate_addressing;
use super::{error, Dhcp4ConfigV1, Dhcp6ConfigV1, Result, RouteV1, StaticConfigV1, Validate};
use crate::interface_name::InterfaceName;
use crate::net_config::devices::generate_addressing_validation;
use serde::de::Error;
use serde::{Deserialize, Deserializer};
use snafu::ensure;
use std::net::IpAddr;

#[derive(Debug, Deserialize)]
#[serde(deny_unknown_fields)]
#[serde(remote = "Self")]
pub(crate) struct NetBondV1 {
pub(crate) primary: Option<bool>,
pub(crate) dhcp4: Option<Dhcp4ConfigV1>,
pub(crate) dhcp6: Option<Dhcp6ConfigV1>,
pub(crate) static4: Option<StaticConfigV1>,
pub(crate) static6: Option<StaticConfigV1>,
#[serde(rename = "route")]
pub(crate) routes: Option<Vec<RouteV1>>,
kind: String,
pub(crate) mode: BondMode,
#[serde(rename = "min-links")]
pub(crate) min_links: Option<usize>,
#[serde(rename = "monitoring")]
pub(crate) monitoring_config: BondMonitoringConfig,
pub(crate) interfaces: Vec<InterfaceName>,
}

impl<'de> Deserialize<'de> for NetBondV1 {
fn deserialize<D>(deserializer: D) -> std::result::Result<Self, D::Error>
where
D: Deserializer<'de>,
{
let this = Self::deserialize(deserializer)?;
if this.kind.to_lowercase().as_str() != "bond" {
return Err(D::Error::custom(format!(
"kind of '{}' does not match 'bond'",
this.kind.as_str()
)));
}

Ok(this)
}
}

generate_addressing_validation!(&NetBondV1);

impl Validate for NetBondV1 {
fn validate(&self) -> Result<()> {
validate_addressing(self)?;

// TODO: We should move this and other validation logic into Deserialize when messaging
// is better for enum failures /~https://github.com/serde-rs/serde/issues/2157
let interfaces_count = self.interfaces.len();
ensure!(
interfaces_count > 0,
error::InvalidNetConfigSnafu {
reason: "bonds must have 1 or more interfaces specified"
}
);
if let Some(min_links) = self.min_links {
ensure!(
min_links <= interfaces_count,
error::InvalidNetConfigSnafu {
reason: "min-links is greater than number of interfaces configured"
}
)
}
// Validate monitoring configuration
match &self.monitoring_config {
BondMonitoringConfig::MiiMon(config) => config.validate()?,
BondMonitoringConfig::ArpMon(config) => config.validate()?,
}

Ok(())
}
}

// Currently only mode 1 (active-backup) is supported but eventually 0-6 could be added
#[derive(Clone, Debug, Deserialize)]
#[serde(rename_all = "kebab-case")]
pub(crate) enum BondMode {
ActiveBackup,
}

#[derive(Clone, Debug, Deserialize)]
#[serde(untagged)]
pub(crate) enum BondMonitoringConfig {
MiiMon(MiiMonitoringConfig),
ArpMon(ArpMonitoringConfig),
}

#[derive(Clone, Debug, Deserialize)]
#[serde(deny_unknown_fields)]
pub(crate) struct MiiMonitoringConfig {
#[serde(rename = "miimon-frequency-ms")]
pub(crate) frequency: u32,
#[serde(rename = "miimon-updelay-ms")]
pub(crate) updelay: u32,
#[serde(rename = "miimon-downdelay-ms")]
pub(crate) downdelay: u32,
}

impl Validate for MiiMonitoringConfig {
fn validate(&self) -> Result<()> {
ensure!(
self.frequency > 0,
error::InvalidNetConfigSnafu {
reason: "miimon-frequency-ms of 0 disables Mii Monitoring, either set a value or configure Arp Monitoring"
}
);
// updelay and downdelay should be a multiple of frequency, but will be rounded down
// by the kernel, this ensures they are at least the size of frequency (non-zero)
ensure!(
self.frequency <= self.updelay && self.frequency <= self.downdelay,
error::InvalidNetConfigSnafu {
reason: "miimon-updelay-ms and miimon-downdelay-ms must be equal to or larger than miimon-frequency-ms"
}
);
Ok(())
}
}

#[derive(Clone, Debug, Deserialize)]
#[serde(deny_unknown_fields)]
pub(crate) struct ArpMonitoringConfig {
#[serde(rename = "arpmon-interval-ms")]
pub(crate) interval: u32,
#[serde(rename = "arpmon-validate")]
pub(crate) validate: ArpValidate,
#[serde(rename = "arpmon-targets")]
pub(crate) targets: Vec<IpAddr>,
}

impl Validate for ArpMonitoringConfig {
fn validate(&self) -> Result<()> {
ensure!(
self.interval > 0,
error::InvalidNetConfigSnafu {
reason: "arpmon-interval-ms of 0 disables Arp Monitoring, either set a value or configure Mii Monitoring"
}
);
// If using Arp Monitoring, 1-16 targets must be specified
let targets_length: u32 = self.targets.len() as u32;
ensure!(
targets_length > 0 && targets_length <= 16,
error::InvalidNetConfigSnafu {
reason: "arpmon-targets must include between 1 and 16 targets"
}
);
Ok(())
}
}

#[derive(Clone, Debug, Deserialize)]
#[serde(rename_all = "kebab-case")]
pub(crate) enum ArpValidate {
Active,
All,
Backup,
None,
}
27 changes: 27 additions & 0 deletions sources/api/netdog/src/net_config/devices/interface.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
use super::validate_addressing;
use super::{Dhcp4ConfigV1, Dhcp6ConfigV1, Result, Validate};
use crate::net_config::devices::generate_addressing_validation;
use crate::net_config::{RouteV1, StaticConfigV1};
use serde::Deserialize;

#[derive(Debug, Deserialize)]
#[serde(deny_unknown_fields)]
pub(crate) struct NetInterfaceV2 {
// Use this interface as the primary interface for the system
pub(crate) primary: Option<bool>,
pub(crate) dhcp4: Option<Dhcp4ConfigV1>,
pub(crate) dhcp6: Option<Dhcp6ConfigV1>,
pub(crate) static4: Option<StaticConfigV1>,
pub(crate) static6: Option<StaticConfigV1>,
#[serde(rename = "route")]
pub(crate) routes: Option<Vec<RouteV1>>,
}

impl Validate for NetInterfaceV2 {
fn validate(&self) -> Result<()> {
validate_addressing(self)
}
}

// Generate the traits for IP Address validation
generate_addressing_validation!(&NetInterfaceV2);
Loading

0 comments on commit f3dfa53

Please sign in to comment.