Add support for NVIDIA MIG #4418

piyush-jena · 2025-02-27T01:36:42Z

Issue number:
Fixes: #4406, #4252

Description of changes:

Testing done:

Migration Testing

v1.32.0

bash-5.1# apiclient get os
{
  "os": {
    "arch": "x86_64",
    "build_id": "cacc4ce9",
    "pretty_name": "Bottlerocket OS 1.32.0 (aws-k8s-1.29-nvidia)",
    "variant_id": "aws-k8s-1.29-nvidia",
    "version_id": "1.32.0"
  }
}
bash-5.1# apiclient set settings.kubelet-device-plugins.nvidia.device-partitioning-strategy="mig"
Failed to change settings: Failed PATCH request to '/settings/keypair?tx=apiclient-set-vELpjamiDtvsQslp': Status 400 when PATCHing /settings/keypair?tx=apiclient-set-vELpjamiDtvsQslp: Unable to match your input to the data model.  We may not have enough type information.  Please try the --json input form.  Cause: Error during deserialization: unknown field `device-partitioning-strategy`, expected one of `pass-device-specs`, `device-id-strategy`, `device-list-strategy`, `device-sharing-strategy`, `time-slicing` at line 1 column 67
bash-5.1# apiclient apply <<EOF
> [settings.kubelet-device-plugins.nvidia.mig.profile]
> "a100.40gb"="7"
> "h100.80gb"="4"
> "g100.80gb"="1g.5gb"
> EOF
Failed to apply settings: Failed to PATCH settings from '-' to '/settings?tx=apiclient-apply-jnaXTAbi9QIbJm0M': Status 400 when PATCHing /settings?tx=apiclient-apply-jnaXTAbi9QIbJm0M: Json deserialize error: unknown field `mig`, expected one of `pass-device-specs`, `device-id-strategy`, `device-list-strategy`, `device-sharing-strategy`, `time-slicing` at line 1 column 42
bash-5.1# updog check-update -a --json
[
  {
    "variant": "aws-k8s-1.29-nvidia",
    "arch": "x86_64",
    "version": "1.34.0",
    "max_version": "1.34.0",
    "waves": {
      "0": "2025-02-27T03:28:57.496827355Z",
      "20": "2025-02-27T06:28:57.496827355Z",
      "102": "2025-02-28T02:28:57.496827355Z",
      "307": "2025-03-01T02:28:57.496827355Z",
      "819": "2025-03-03T02:28:57.496827355Z",
      "1228": "2025-03-04T02:28:57.496827355Z",
      "1843": "2025-03-05T02:28:57.496827355Z"
    },
    "images": {
      "boot": "bottlerocket-aws-k8s-1.29-nvidia-x86_64-1.34.0-2ac59cde-boot.ext4.lz4",
      "root": "bottlerocket-aws-k8s-1.29-nvidia-x86_64-1.34.0-2ac59cde-root.ext4.lz4",
      "hash": "bottlerocket-aws-k8s-1.29-nvidia-x86_64-1.34.0-2ac59cde-root.verity.lz4"
    }
  }
]
bash-5.1# updog update -i 1.34.0 -r -n
Starting update to 1.34.0
Reboot scheduled for Thu 2025-02-27 02:37:43 UTC, use 'shutdown -c' to cancel.
Update applied: aws-k8s-1.29-nvidia 1.34.0

Upgrade to v1.34.0

bash-5.1# apiclient get os
{
  "os": {
    "arch": "x86_64",
    "build_id": "2ac59cde",
    "pretty_name": "Bottlerocket OS 1.34.0 (aws-k8s-1.29-nvidia)",
    "variant_id": "aws-k8s-1.29-nvidia",
    "version_id": "1.34.0"
  }
}
bash-5.1# apiclient set settings.kubelet-device-plugins.nvidia.device-partitioning-strategy="mig"
bash-5.1# apiclient apply <<EOF
> [settings.kubelet-device-plugins.nvidia.mig.profile]
> "a100.40gb"="7"
> "h100.80gb"="4"
> "g100.80gb"="1g.5gb"
> EOF
bash-5.1# signpost rollback-to-inactive

Downgrade to v1.32.0

[ssm-user@control]$ apiclient get os
{
  "os": {
    "arch": "x86_64",
    "build_id": "cacc4ce9",
    "pretty_name": "Bottlerocket OS 1.32.0 (aws-k8s-1.29-nvidia)",
    "variant_id": "aws-k8s-1.29-nvidia",
    "version_id": "1.32.0"
  }
}
[ssm-user@control]$ apiclient set settings.kubelet-device-plugins.nvidia.device-partitioning-strategy="mig"
Failed to change settings: Failed PATCH request to '/settings/keypair?tx=apiclient-set-3ypsF3ruTrXjCzNq': Status 400 when PATCHing /settings/keypair?tx=apiclient-set-3ypsF3ruTrXjCzNq: Unable to match your input to the data model.  We may not have enough type information.  Please try the --json input form.  Cause: Error during deserialization: unknown field `device-partitioning-strategy`, expected one of `pass-device-specs`, `device-id-strategy`, `device-list-strategy`, `device-sharing-strategy`, `time-slicing` at line 1 column 67
[ssm-user@control]$ apiclient apply <<EOF
> [settings.kubelet-device-plugins.nvidia.mig.profile]
> "a100.40gb"="7"
> "h100.80gb"="4"
> "g100.80gb"="1g.5gb"
> EOF
Failed to apply settings: Failed to PATCH settings from '-' to '/settings?tx=apiclient-apply-yGwq7oN11jDcLKqn': Status 400 when PATCHing /settings?tx=apiclient-apply-yGwq7oN11jDcLKqn: Json deserialize error: unknown field `mig`, expected one of `pass-device-specs`, `device-id-strategy`, `device-list-strategy`, `device-sharing-strategy`, `time-slicing` at line 1 column 42

Feature Testing

Settings

bash-5.1# apiclient set settings.kubelet-device-plugins.nvidia.device-partitioning-strategy="mig"
bash-5.1# apiclient apply <<EOF
> [settings.kubelet-device-plugins.nvidia.mig.profile]
> "a100.40gb"="7"
> "h100.80gb"="4"
> "g100.80gb"="1g.5gb"
> EOF

`kubectl describe node` output

Capacity:
  cpu:                96
  ephemeral-storage:  418812624Ki
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             1176277888Ki
  nvidia.com/gpu:     56
  pods:               737
Allocatable:
  cpu:                95690m
  ephemeral-storage:  384903971816
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             1167612800Ki
  nvidia.com/gpu:     56
  pods:               737

`nvidia-k8s-device-plugin` status

ash-5.1# systemctl status nvidia-k8s-device-plugin
● nvidia-k8s-device-plugin.service - Start NVIDIA kubernetes device plugin
     Loaded: loaded (/x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/systemd/system/nvidia-k8s-device-plugin.service; enabled; preset: enabled)
    Drop-In: /x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/systemd/system/service.d
             └─00-aws-config.conf
             /etc/systemd/system/nvidia-k8s-device-plugin.service.d
             └─exec-start.conf
     Active: active (running) since Thu 2025-02-27 02:34:58 UTC; 15min ago
   Main PID: 14870 (nvidia-device-p)
      Tasks: 21 (limit: 629145)
     Memory: 61.7M
        CPU: 16.438s
     CGroup: /system.slice/nvidia-k8s-device-plugin.service
             └─14870 /usr/bin/nvidia-device-plugin --config-file=/etc/nvidia-k8s-device-plugin/settings.yaml

Feb 27 02:34:59 ip-192-168-155-1.us-west-2.compute.internal nvidia-device-plugin[14870]:   },
Feb 27 02:34:59 ip-192-168-155-1.us-west-2.compute.internal nvidia-device-plugin[14870]:   "sharing": {
Feb 27 02:34:59 ip-192-168-155-1.us-west-2.compute.internal nvidia-device-plugin[14870]:     "timeSlicing": {}
Feb 27 02:34:59 ip-192-168-155-1.us-west-2.compute.internal nvidia-device-plugin[14870]:   },
Feb 27 02:34:59 ip-192-168-155-1.us-west-2.compute.internal nvidia-device-plugin[14870]:   "imex": {}
Feb 27 02:34:59 ip-192-168-155-1.us-west-2.compute.internal nvidia-device-plugin[14870]: }
Feb 27 02:34:59 ip-192-168-155-1.us-west-2.compute.internal nvidia-device-plugin[14870]: I0227 02:34:58.722298   14870 main.go:356] Retrieving plugins.
Feb 27 02:35:05 ip-192-168-155-1.us-west-2.compute.internal nvidia-device-plugin[14870]: I0227 02:35:05.440287   14870 server.go:195] Starting GRPC server fo…om/gpu'
Feb 27 02:35:05 ip-192-168-155-1.us-west-2.compute.internal nvidia-device-plugin[14870]: I0227 02:35:05.446465   14870 server.go:139] Starting to serve 'nvid…pu.sock
Feb 27 02:35:05 ip-192-168-155-1.us-west-2.compute.internal nvidia-device-plugin[14870]: I0227 02:35:05.451689   14870 server.go:146] Registered device plugi…Kubelet
Hint: Some lines were ellipsized, use -l to show in full.

Workload test

[fedora@ip-172-31-48-208 kubernetes]$ cat << EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nvidia-cuda-workload-deployment-1
spec:
  replicas: 20
  selector:
    matchLabels:
      app: vector-add
  template:
    metadata:
      labels:
        app: vector-add
    spec:
      containers:
      - name: vector-add
        image: nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda10.2
        resources:
          limits:
              nvidia.com/gpu: 1
EOF
deployment.apps/nvidia-cuda-workload-deployment-1 created
[fedora@ip-172-31-48-208 kubernetes]$ kubectl get pods
NAME                                                READY   STATUS              RESTARTS   AGE
nvidia-cuda-workload-deployment-1-c9548dfc4-2wxw2   0/1     ContainerCreating   0          6s
nvidia-cuda-workload-deployment-1-c9548dfc4-44wmq   0/1     ContainerCreating   0          6s
nvidia-cuda-workload-deployment-1-c9548dfc4-5j8xd   0/1     ContainerCreating   0          6s
nvidia-cuda-workload-deployment-1-c9548dfc4-5zldn   0/1     ContainerCreating   0          6s
nvidia-cuda-workload-deployment-1-c9548dfc4-69gfp   0/1     ContainerCreating   0          6s
nvidia-cuda-workload-deployment-1-c9548dfc4-9nnqf   0/1     ContainerCreating   0          6s
nvidia-cuda-workload-deployment-1-c9548dfc4-b74mp   0/1     ContainerCreating   0          6s
nvidia-cuda-workload-deployment-1-c9548dfc4-bdbwf   0/1     ContainerCreating   0          6s
nvidia-cuda-workload-deployment-1-c9548dfc4-brvrt   0/1     ContainerCreating   0          6s
nvidia-cuda-workload-deployment-1-c9548dfc4-dfsnd   0/1     ContainerCreating   0          6s
nvidia-cuda-workload-deployment-1-c9548dfc4-dhddn   0/1     ContainerCreating   0          6s
nvidia-cuda-workload-deployment-1-c9548dfc4-dtxnz   0/1     ContainerCreating   0          6s
nvidia-cuda-workload-deployment-1-c9548dfc4-fwcmn   0/1     ContainerCreating   0          6s
nvidia-cuda-workload-deployment-1-c9548dfc4-gp86x   0/1     ContainerCreating   0          6s
nvidia-cuda-workload-deployment-1-c9548dfc4-k4sjd   0/1     ContainerCreating   0          6s
nvidia-cuda-workload-deployment-1-c9548dfc4-mqwkt   0/1     ContainerCreating   0          6s
nvidia-cuda-workload-deployment-1-c9548dfc4-p5bt7   0/1     ContainerCreating   0          6s
nvidia-cuda-workload-deployment-1-c9548dfc4-wl5z8   0/1     ContainerCreating   0          6s
nvidia-cuda-workload-deployment-1-c9548dfc4-z9vxr   0/1     ContainerCreating   0          6s
nvidia-cuda-workload-deployment-1-c9548dfc4-zhls8   0/1     ContainerCreating   0          6s
[fedora@ip-172-31-48-208 kubernetes]$ kubectl get pods
NAME                                                READY   STATUS      RESTARTS   AGE
nvidia-cuda-workload-deployment-1-c9548dfc4-2wxw2   1/1     Running     0          29s
nvidia-cuda-workload-deployment-1-c9548dfc4-44wmq   1/1     Running     0          29s
nvidia-cuda-workload-deployment-1-c9548dfc4-5j8xd   1/1     Running     0          29s
nvidia-cuda-workload-deployment-1-c9548dfc4-5zldn   0/1     Completed   0          29s
nvidia-cuda-workload-deployment-1-c9548dfc4-69gfp   1/1     Running     0          29s
nvidia-cuda-workload-deployment-1-c9548dfc4-9nnqf   1/1     Running     0          29s
nvidia-cuda-workload-deployment-1-c9548dfc4-b74mp   0/1     Completed   0          29s
nvidia-cuda-workload-deployment-1-c9548dfc4-bdbwf   1/1     Running     0          29s
nvidia-cuda-workload-deployment-1-c9548dfc4-brvrt   1/1     Running     0          29s
nvidia-cuda-workload-deployment-1-c9548dfc4-dfsnd   1/1     Running     0          29s
nvidia-cuda-workload-deployment-1-c9548dfc4-dhddn   1/1     Running     0          29s
nvidia-cuda-workload-deployment-1-c9548dfc4-dtxnz   0/1     Completed   0          29s
nvidia-cuda-workload-deployment-1-c9548dfc4-fwcmn   0/1     Completed   0          29s
nvidia-cuda-workload-deployment-1-c9548dfc4-gp86x   1/1     Running     0          29s
nvidia-cuda-workload-deployment-1-c9548dfc4-k4sjd   1/1     Running     0          29s
nvidia-cuda-workload-deployment-1-c9548dfc4-mqwkt   1/1     Running     0          29s
nvidia-cuda-workload-deployment-1-c9548dfc4-p5bt7   0/1     Completed   0          29s
nvidia-cuda-workload-deployment-1-c9548dfc4-wl5z8   1/1     Running     0          29s
nvidia-cuda-workload-deployment-1-c9548dfc4-z9vxr   1/1     Running     0          29s
nvidia-cuda-workload-deployment-1-c9548dfc4-zhls8   1/1     Running     0          29s
[fedora@ip-172-31-48-208 kubernetes]$ kubectl logs nvidia-cuda-workload-deployment-1-c9548dfc4-5j8xd
[Vector addition of 50000 elements]
Copy input data from the host memory to the CUDA device
CUDA kernel launch with 196 blocks of 256 threads
Copy output data from the CUDA device to the host memory
Test PASSED
Done

MIG + Time-slicing `kubectl describe node` output (gpus = 8, partitions = 4, replicas = 4)

bash-5.1# apiclient get settings.kubelet-device-plugins.nvidia
{
  "settings": {
    "kubelet-device-plugins": {
      "nvidia": {
        "device-id-strategy": "index",
        "device-list-strategy": "volume-mounts",
        "device-partitioning-strategy": "mig",
        "device-sharing-strategy": "time-slicing",
        "mig": {
          "profile": {
            "a100.40gb": "7",
            "g100.80gb": "1g.5gb",
            "h100.80gb": "4"
          }
        },
        "pass-device-specs": true,
        "time-slicing": {
          "replicas": 4
        }
      }
    }
  }
}

[fedora@ip-172-31-48-208 kubernetes]$ kubectl describe node
Capacity:
  cpu:                    192
  ephemeral-storage:      418124376Ki
  hugepages-1Gi:          0
  hugepages-2Mi:          0
  memory:                 2097096404Ki
  nvidia.com/gpu:         0
  nvidia.com/gpu.shared:  128
  pods:                   100
Allocatable:
  cpu:                    191450m
  ephemeral-storage:      384269682460
  hugepages-1Gi:          0
  hugepages-2Mi:          0
  memory:                 2095606484Ki
  nvidia.com/gpu:         0
  nvidia.com/gpu.shared:  128
  pods:                   100

P5 Testing

NVLink status when MIG is enabled

bash-5.1# ./x86_64-bottlerocket-linux-gnu/sys-root/usr/libexec/nvidia/tesla/bin/nvidia-smi nvlink --status

NVLink status when MIG is disabled

bash-5.1# ./x86_64-bottlerocket-linux-gnu/sys-root/usr/libexec/nvidia/tesla/bin/nvidia-smi nvlink --status
GPU 0: NVIDIA H100 80GB HBM3 (UUID: GPU-4039dfe4-0c97-4b1e-683e-eb7d42699053)
         Link 0: 26.562 GB/s
         Link 1: 26.562 GB/s
         Link 2: 26.562 GB/s
         Link 3: 26.562 GB/s
         Link 4: 26.562 GB/s
         Link 5: 26.562 GB/s
         Link 6: 26.562 GB/s
         Link 7: 26.562 GB/s
         Link 8: 26.562 GB/s
         Link 9: 26.562 GB/s
         Link 10: 26.562 GB/s
         Link 11: 26.562 GB/s
         Link 12: 26.562 GB/s
         Link 13: 26.562 GB/s
         Link 14: 26.562 GB/s
         Link 15: 26.562 GB/s
         Link 16: 26.562 GB/s
         Link 17: 26.562 GB/s
GPU 1: NVIDIA H100 80GB HBM3 (UUID: GPU-dd3b269a-8f3a-d2d5-1518-2713ba03fcf7)
         Link 0: 26.562 GB/s
         Link 1: 26.562 GB/s
         Link 2: 26.562 GB/s
         Link 3: 26.562 GB/s
         Link 4: 26.562 GB/s
         Link 5: 26.562 GB/s
         Link 6: 26.562 GB/s
         Link 7: 26.562 GB/s
         Link 8: 26.562 GB/s
         Link 9: 26.562 GB/s
         Link 10: 26.562 GB/s
         Link 11: 26.562 GB/s
         Link 12: 26.562 GB/s
         Link 13: 26.562 GB/s
         Link 14: 26.562 GB/s
         Link 15: 26.562 GB/s
         Link 16: 26.562 GB/s
         Link 17: 26.562 GB/s
GPU 2: NVIDIA H100 80GB HBM3 (UUID: GPU-b6c58f85-1f52-9905-5a8e-06ac27805eb9)
         Link 0: 26.562 GB/s
         Link 1: 26.562 GB/s
         Link 2: 26.562 GB/s
         Link 3: 26.562 GB/s
         Link 4: 26.562 GB/s
         Link 5: 26.562 GB/s
         Link 6: 26.562 GB/s
         Link 7: 26.562 GB/s
         Link 8: 26.562 GB/s
         Link 9: 26.562 GB/s
         Link 10: 26.562 GB/s
         Link 11: 26.562 GB/s
         Link 12: 26.562 GB/s
         Link 13: 26.562 GB/s
         Link 14: 26.562 GB/s
         Link 15: 26.562 GB/s
         Link 16: 26.562 GB/s
         Link 17: 26.562 GB/s
GPU 3: NVIDIA H100 80GB HBM3 (UUID: GPU-0841636d-8d48-9195-ea7f-9d4a9ad9f846)
         Link 0: 26.562 GB/s
         Link 1: 26.562 GB/s
         Link 2: 26.562 GB/s
         Link 3: 26.562 GB/s
         Link 4: 26.562 GB/s
         Link 5: 26.562 GB/s
         Link 6: 26.562 GB/s
         Link 7: 26.562 GB/s
         Link 8: 26.562 GB/s
         Link 9: 26.562 GB/s
         Link 10: 26.562 GB/s
         Link 11: 26.562 GB/s
         Link 12: 26.562 GB/s
         Link 13: 26.562 GB/s
         Link 14: 26.562 GB/s
         Link 15: 26.562 GB/s
         Link 16: 26.562 GB/s
         Link 17: 26.562 GB/s
GPU 4: NVIDIA H100 80GB HBM3 (UUID: GPU-fd607481-793f-5e10-278c-109552132250)
         Link 0: 26.562 GB/s
         Link 1: 26.562 GB/s
         Link 2: 26.562 GB/s
         Link 3: 26.562 GB/s
         Link 4: 26.562 GB/s
         Link 5: 26.562 GB/s
         Link 6: 26.562 GB/s
         Link 7: 26.562 GB/s
         Link 8: 26.562 GB/s
         Link 9: 26.562 GB/s
         Link 10: 26.562 GB/s
         Link 11: 26.562 GB/s
         Link 12: 26.562 GB/s
         Link 13: 26.562 GB/s
         Link 14: 26.562 GB/s
         Link 15: 26.562 GB/s
         Link 16: 26.562 GB/s
         Link 17: 26.562 GB/s
GPU 5: NVIDIA H100 80GB HBM3 (UUID: GPU-7da3770c-f6be-cc04-c1c8-a6e0eff2c89c)
         Link 0: 26.562 GB/s
         Link 1: 26.562 GB/s
         Link 2: 26.562 GB/s
         Link 3: 26.562 GB/s
         Link 4: 26.562 GB/s
         Link 5: 26.562 GB/s
         Link 6: 26.562 GB/s
         Link 7: 26.562 GB/s
         Link 8: 26.562 GB/s
         Link 9: 26.562 GB/s
         Link 10: 26.562 GB/s
         Link 11: 26.562 GB/s
         Link 12: 26.562 GB/s
         Link 13: 26.562 GB/s
         Link 14: 26.562 GB/s
         Link 15: 26.562 GB/s
         Link 16: 26.562 GB/s
         Link 17: 26.562 GB/s
GPU 6: NVIDIA H100 80GB HBM3 (UUID: GPU-a0620bb0-8ae2-7101-4b52-2fa2ac096343)
         Link 0: 26.562 GB/s
         Link 1: 26.562 GB/s
         Link 2: 26.562 GB/s
         Link 3: 26.562 GB/s
         Link 4: 26.562 GB/s
         Link 5: 26.562 GB/s
         Link 6: 26.562 GB/s
         Link 7: 26.562 GB/s
         Link 8: 26.562 GB/s
         Link 9: 26.562 GB/s
         Link 10: 26.562 GB/s
         Link 11: 26.562 GB/s
         Link 12: 26.562 GB/s
         Link 13: 26.562 GB/s
         Link 14: 26.562 GB/s
         Link 15: 26.562 GB/s
         Link 16: 26.562 GB/s
         Link 17: 26.562 GB/s
GPU 7: NVIDIA H100 80GB HBM3 (UUID: GPU-3b80af25-3116-b375-0333-fcb4c42b1627)
         Link 0: 26.562 GB/s
         Link 1: 26.562 GB/s
         Link 2: 26.562 GB/s
         Link 3: 26.562 GB/s
         Link 4: 26.562 GB/s
         Link 5: 26.562 GB/s
         Link 6: 26.562 GB/s
         Link 7: 26.562 GB/s
         Link 8: 26.562 GB/s
         Link 9: 26.562 GB/s
         Link 10: 26.562 GB/s
         Link 11: 26.562 GB/s
         Link 12: 26.562 GB/s
         Link 13: 26.562 GB/s
         Link 14: 26.562 GB/s
         Link 15: 26.562 GB/s
         Link 16: 26.562 GB/s
         Link 17: 26.562 GB/s

Fabric Statuses

MIG enabled - P5

bash-5.1# ./x86_64-bottlerocket-linux-gnu/sys-root/usr/libexec/nvidia/tesla/bin/nvidia-smi  -q -i 0 | grep -i -A 2 Fabric
    Fabric
        State                             : In Progress
        Status                            : N/A

MIG disabled - P5

bash-5.1# ./x86_64-bottlerocket-linux-gnu/sys-root/usr/libexec/nvidia/tesla/bin/nvidia-smi  -q -i 0 | grep -i -A 2 Fabric
    Fabric
        State                             : Completed
        Status                            : Success

MIG disabled - P4

bash-5.1# ./x86_64-bottlerocket-linux-gnu/sys-root/usr/libexec/nvidia/tesla/bin/nvidia-smi  -q -i 0 | grep -i -A 2 Fabric
    Fabric
        State                             : N/A
        Status                            : N/A

MIG enabled - P4

bash-5.1# ./x86_64-bottlerocket-linux-gnu/sys-root/usr/libexec/nvidia/tesla/bin/nvidia-smi  -q -i 0 | grep -i -A 2 Fabric
    Fabric
        State                             : N/A
        Status                            : N/A

Terms of contribution:

By submitting this pull request, I agree that this contribution is dual-licensed under the terms of both the Apache License, version 2.0, and the MIT license.

sources/Cargo.lock

sources/Cargo.toml

Signed-off-by: Piyush Jena <piyushjena1996@gmail.com>

bcressey

LGTM if testing comes back green.

arnaldo2792 · 2025-02-27T05:48:42Z

Is there a test to make sure that once timeSlicing is enabled, the Device Plugin advertises the correct device name based of the time slice?

piyush-jena · 2025-02-27T17:19:00Z

@arnaldo2792 I have added the test towards the end.

yeazelm · 2025-02-27T18:29:35Z

CHANGELOG.md

+* Add support for NVIDIA Multi-Instance GPU (MIG) ([#4418])
+
+## Build Changes
+* Update bottlerocket-core-kit to 6.0.2 ([#4416])


I'd like to have links to the changelogs to help one confirm what the actual delta is here. For example, I have to check the previous entry to see we went from core kit 6.0.1 -> 6.0.2 and it would be great if there was a link on this line giving me the diff for that repo.

piyush-jena force-pushed the roll-fwd branch from 86ad6e0 to 9a70605 Compare February 27, 2025 01:43

piyush-jena changed the title ~~migration: add migrations for NVIDIA MIG~~ Add support for NVIDIA MIG Feb 27, 2025

bcressey reviewed Feb 27, 2025

View reviewed changes

sources/Cargo.lock Outdated Show resolved Hide resolved

sources/Cargo.lock Outdated Show resolved Hide resolved

sources/Cargo.toml Outdated Show resolved Hide resolved

piyush-jena force-pushed the roll-fwd branch from 08cee62 to 0d8e9ec Compare February 27, 2025 01:57

migration: add migrations for NVIDIA MIG

c3ea569

Signed-off-by: Piyush Jena <piyushjena1996@gmail.com>

piyush-jena force-pushed the roll-fwd branch from 0d8e9ec to 2ac59cd Compare February 27, 2025 02:02

bcressey reviewed Feb 27, 2025

View reviewed changes

piyush-jena marked this pull request as ready for review February 27, 2025 06:42

KCSesh approved these changes Feb 27, 2025

View reviewed changes

yeazelm reviewed Feb 27, 2025

View reviewed changes

yeazelm approved these changes Feb 27, 2025

View reviewed changes

piyush-jena force-pushed the roll-fwd branch 3 times, most recently from 07dc803 to e581904 Compare February 27, 2025 18:52

changelog: Add 1.34.0 release notes

e7f43c3

piyush-jena force-pushed the roll-fwd branch from e581904 to e7f43c3 Compare February 27, 2025 18:53

yeazelm approved these changes Feb 27, 2025

View reviewed changes

KCSesh approved these changes Feb 27, 2025

View reviewed changes

piyush-jena merged commit 18d04e5 into bottlerocket-os:develop Feb 27, 2025
2 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for NVIDIA MIG #4418

Add support for NVIDIA MIG #4418

piyush-jena commented Feb 27, 2025 •

edited

Loading

bcressey left a comment

arnaldo2792 commented Feb 27, 2025

piyush-jena commented Feb 27, 2025

yeazelm Feb 27, 2025

Add support for NVIDIA MIG #4418

Add support for NVIDIA MIG #4418

Conversation

piyush-jena commented Feb 27, 2025 • edited Loading

Migration Testing

v1.32.0

Upgrade to v1.34.0

Downgrade to v1.32.0

Feature Testing

Settings

kubectl describe node output

nvidia-k8s-device-plugin status

Workload test

MIG + Time-slicing kubectl describe node output (gpus = 8, partitions = 4, replicas = 4)

P5 Testing

NVLink status when MIG is enabled

NVLink status when MIG is disabled

Fabric Statuses

MIG enabled - P5

MIG disabled - P5

MIG disabled - P4

MIG enabled - P4

bcressey left a comment

Choose a reason for hiding this comment

arnaldo2792 commented Feb 27, 2025

piyush-jena commented Feb 27, 2025

yeazelm Feb 27, 2025

Choose a reason for hiding this comment

piyush-jena commented Feb 27, 2025 •

edited

Loading

`kubectl describe node` output

`nvidia-k8s-device-plugin` status

MIG + Time-slicing `kubectl describe node` output (gpus = 8, partitions = 4, replicas = 4)