core kit migration 🚧 tracking issue

[bcressey]

What's happening?

As the culmination of work done over the past 18 months, this repo - aka "the Bottlerocket main repo" - will be migrating over to the out-of-tree builds infrastructure.

This will mark a major change in the development workflow:

1. Most packages and first-party agents (like updog and the API server) will move to bottlerocket-os/bottlerocket-core-kit.
2. Reusable settings definitions will move to bottlerocket-os/bottlerocket-settings-sdk.
3. Variant definitions, and their associated settings and defaults, will stay in bottlerocket-os/bottlerocket.

There will be no impact to past or future Bottlerocket releases. The only change is to how Bottlerocket is built, not to how it is published or consumed.

Why is this happening?

One of the major lessons of the out-of-tree builds project has been that without the forcing function of a repository boundary, it's easy to miss adding useful abstractions and end up with software that's nominally reusable but is actually very tightly coupled to specific use cases. While that's helpful for the velocity of feature development in the moment, it's harmful to the larger goal of fostering a community of out-of-tree builders.

Bottlerocket has had two similar migrations in its history - first when the SDK moved out-of-tree in #525, and again when the build and publish tools moved out-of-tree in #3429. Both were successful from the perspective of streamlining development and reducing compile times. This will be the largest migration yet, and I anticipate similar benefits.

When it will happen?

The bottlerocket-os/bottlerocket-core-kit repo will be created and publish its first release on June 13, 2024. That repo will start with a copy of the main repo's history, and the non-core sources will be removed in a subsequent commit.

Once the core kit is available, bottlerocket-os/bottlerocket will then remove the core sources in a subsequent commit, retaining only the variant-specific sources. This is expected to happen on June 14, 2024.

What about my existing private fork?

The 1.20.x branch will not be updated with these changes, so that can be used (temporarily) as the basis for ongoing development. The shared Git history should make it straightforward to cherry-pick fixes by adding bottlerocket-core-kit as an addition remote.

Long-term, I invite you to collaborate to get useful changes upstream, such as new packages into the core kit and new modeled types into the settings SDK.

[bcressey]

Pull requests that need to be merged before the migration can happen.

bottlerocket

• migrate to settings plugins and eliminate variant-based conditional compilation #4038

bottlerocket-settings-sdk

• feat: add defaults helper bottlerocket-settings-sdk#37
• feat: add crates for settings plugins bottlerocket-settings-sdk#38

bottlerocket-sdk

• release v0.42.0 with the grub-bios-setup binary available

twoliter

• twoliter: use grub-bios-setup from SDK twoliter#242
• application-inventory: use RPM's Version and Release, set Epoch twoliter#263
• twoliter: fetch command for downloading and extracting external kits twoliter#270
• buildsys: generate kit metadata and create oci image twoliter#271
• twoliter: external kit test project and include rpm repositories twoliter#272
• external kit support: publish-kit functionality twoliter#230

[bcressey]

The migration is now complete, though there's work left to bring documentation up to date. We're also iterating on some quality-of-life improvements over in twoliter.