Skip to content

Add aws-mantle-1 variants - #4908

Open
piyush-jena wants to merge 3 commits into
bottlerocket-os:developfrom
piyush-jena:ukis/bottlerocket
Open

Add aws-mantle-1 variants#4908
piyush-jena wants to merge 3 commits into
bottlerocket-os:developfrom
piyush-jena:ukis/bottlerocket

Conversation

@piyush-jena

@piyush-jena piyush-jena commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

Description of changes:

  • add a new minimal variant
  • This variant has Unified Kernel Image (UKI), ephemeral-encryption-keys
  • The following settings are absent - updates, boot, ecs, metrics, autoscaling
  • This is an experimental variant

Testing done:

Test area mantle-1 mantle-1-fips mantle-1-nvidia mantle-1-nvidia-fips
Build + AMI registered (uefi / v2.0 / x86_64 / available)
Boot health (running, SSM Online, no failed units, journalctl clean)
UKI /boot mounted (vfat on BOTTLEROCKET-BOOT-A, boot-integrity HMAC present)
No kubelet / docker units
dotnet-test container runs via ctr (CPU)
nvidia-smoke-test container runs via ctr (GPU detected)
FIPS enabled (crypto/fips_enabled=1, kernel fips=1, OpenSSL FIPS)
ephemeral-encryption-keys rotation + /local wipe across reboot
Warm-reboot endurance test (0 bricks / 2400 reboots)
Plain dm-crypt used in BOTTLEROCKET-DATA and BOTTLEROCKET-PRIVATE
$ apiclient get os
{
  "os": {
    "arch": "x86_64",
    "build_id": "1ed7654e-dirty",
    "pretty_name": "Bottlerocket OS 1.64.0 (aws-mantle-1)",
    "variant_id": "aws-mantle-1",
    "version_id": "1.64.0"
  }
}

PASS — identity of the host under test (variant / variant-flavor / arch / version)

$ apiclient exec admin sheltie cryptsetup status BOTTLEROCKET-DATA
/dev/mapper/BOTTLEROCKET-DATA is active and is in use.
  type:    PLAIN
  cipher:  aes-xts-plain64
  keysize: 512 [bits]
  key location: dm-crypt
  device:  /dev/nvme1n1p1
  sector size:  512 [bytes]
  offset:  0 [512-byte units] (0 [bytes])
  size:    4194299904 [512-byte units] (2147481550848 [bytes])
  mode:    read/write

PASS — BOTTLEROCKET-DATA is headerless plain-mode dm-crypt (type: PLAIN, key location: dm-crypt)

$ apiclient exec admin sheltie cryptsetup status BOTTLEROCKET-PRIVATE
/dev/mapper/BOTTLEROCKET-PRIVATE is active and is in use.
  type:    PLAIN
  cipher:  aes-xts-plain64
  keysize: 512 [bits]
  key location: dm-crypt
  device:  /dev/nvme0n1p7
  sector size:  512 [bytes]
  offset:  0 [512-byte units] (0 [bytes])
  size:    83968 [512-byte units] (42991616 [bytes])
  mode:    read/write

PASS — BOTTLEROCKET-PRIVATE is headerless plain-mode dm-crypt (type: PLAIN, key location: dm-crypt)

$ ./scripts/dotnet-ctr.sh i-0fee4e9804da1a47d us-west-2 public.ecr.aws/s2v1a1q8/hello-dotnet:latest
### target: i-0fee4e9804da1a47d  region=us-west-2  image=public.ecr.aws/s2v1a1q8/hello-dotnet:latest  runid=1788505659-1018273-0fee4e9804da1a47d
### [1/8] pull image (up to ~15m)
### [2/8] materialize writable rootfs -> /local/dotnet-rootfs-1788505659-1018273-0fee4e9804da1a47d
### [3/8] relabel rootfs -> data_t
### [4/8] capture image base OCI spec via a helper container
### [5/8] patch SELinux labels + writable rootfs into the spec (local; no CDI merge)
    spec bytes: 3383
### [6/8] deliver spec -> /local/dotnet-oci-spec-1788505659-1018273-0fee4e9804da1a47d.json
### DELIVER OK: /tmp/tmp.hSzGBKjuI0/dotnet-oci-spec.json -> /local/dotnet-oci-spec-1788505659-1018273-0fee4e9804da1a47d.json (1 chunks of 6000)
### [7/8] run dotnet-test via ctr — output below
-----------------------------------------------------------------

Welcome to .NET 9.0!
---------------------
SDK Version: 9.0.107

----------------
Installed an ASP.NET Core HTTPS development certificate.
To trust the certificate, run 'dotnet dev-certs https --trust'
Learn about HTTPS: https://aka.ms/dotnet-https

----------------
Write your first app: https://aka.ms/dotnet-hello-world
Find out what's new: https://aka.ms/dotnet-whats-new
Explore documentation: https://aka.ms/dotnet-docs
Report issues and find source on GitHub: https://github.com/dotnet/core
Use 'dotnet --help' to see available commands or visit: https://aka.ms/dotnet-cli
--------------------------------------------------------------------------------------
The template "Console App" was created successfully.

Processing post-creation actions...
Restoring /Hello/Hello.csproj:
  Determining projects to restore...
  Restored /Hello/Hello.csproj (in 105 ms).
Restore succeeded.


hello, dotnet-----------------------------------------------------------------
### [8/8] cleanup
### DONE (run rc=0) — expect the hello-dotnet greeting and rc=0

PASS — hello-dotnet ran via ctr (writable rootfs + container_t/data_t OCI spec) and exited 0.

The same greeting + rc=0 pattern appears in logs/F2-workloads-i-*.txt for all 12
hosts, including the 4 NVIDIA hosts (logs/F2-workloads-i-071a98c037d59bb28.txt,
i-0b6f740823ea7fde6.txt, i-033b955da8842ad6b.txt, i-09b2cdc80ddbfd740.txt).

nvidia-smoke-test — the 4 GPU hosts

Sampled host: i-071a98c037d59bb28 (aws-mantle-1-nvidia, x86_64, g6.xlarge). Identical
pattern on the other 3 x86_64 NVIDIA hosts.

$ ./scripts/nvidia-smoke-ctr.sh i-071a98c037d59bb28 us-west-2 public.ecr.aws/s2v1a1q8/nvidia-smoke-test:latest
### target: i-071a98c037d59bb28  region=us-west-2  image=public.ecr.aws/s2v1a1q8/nvidia-smoke-test:latest  runid=1788505896-1025654-071a98c037d59bb28
### [1/9] pull image (up to ~15m)
### [2/9] materialize writable rootfs -> /local/nvidia-rootfs-1788505896-1025654-071a98c037d59bb28
### [3/9] relabel rootfs -> data_t
### [4/9] capture image base OCI spec via a helper container
### [5/9] fetch boot-generated CDI spec (/etc/cdi/nvidia.json)
### [6/9] merge CDI containerEdits + patch SELinux/rootfs (local)
    spec bytes: 25488
### [7/9] deliver spec -> /local/nvidia-oci-spec-1788505896-1025654-071a98c037d59bb28.json
### DELIVER OK: /tmp/tmp.8aIC2juAIk/spec.json -> /local/nvidia-oci-spec-1788505896-1025654-071a98c037d59bb28.json (6 chunks of 6000)
### [8/9] run nvidia-smoke-test via ctr (GPU) — output below
-----------------------------------------------------------------
[... UnifiedMemoryPerf, globalToShmemAsyncCopy, immaTensorCoreGemm, reductionMultiBlockCG, shfl_scan, simpleAWBarrier, simpleAtomicIntrinsics, simpleVoteIntrinsics samples all ran and reported PASSED/OK; full raw text in logs/F2-workloads-i-071a98c037d59bb28.txt ...]

=========================================
  Running sample deviceQuery
=========================================

./deviceQuery Starting...

 CUDA Device Query (Runtime API) version (CUDART static linking)

Detected 1 CUDA Capable device(s)

Device 0: "NVIDIA L4"
  CUDA Driver Version / Runtime Version          13.0 / 12.9
  CUDA Capability Major/Minor version number:    8.9
  Total amount of global memory:                 22563 MBytes (23659151360 bytes)
  (058) Multiprocessors, (128) CUDA Cores/MP:    7424 CUDA Cores
  ...
deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 13.0, CUDA Runtime Version = 12.9, NumDevs = 1
Result = PASS

=========================================
  Running sample vectorAdd
=========================================

[Vector addition of 50000 elements]
Copy input data from the host memory to the CUDA device
CUDA kernel launch with 196 blocks of 256 threads
Copy output data from the CUDA device to the host memory
Test PASSED
Done
-----------------------------------------------------------------
### [9/9] cleanup
### DONE (run rc=0) — expect deviceQuery 'Device 0: "NVIDIA <gpu>"', NumDevs = 1, Result = PASS, vectorAdd Test PASSED

PASS — nvidia-smoke-test saw the GPU via the boot-generated CDI merge; deviceQuery Result = PASS.

$ apiclient get os
{
  "os": {
    "arch": "x86_64",
    "build_id": "1ed7654e-dirty",
    "pretty_name": "Bottlerocket OS 1.64.0 (aws-mantle-1-fips)",
    "variant_id": "aws-mantle-1-fips",
    "version_id": "1.64.0"
  }
}

PASS — identity of the host under test (variant / variant-flavor / arch / version)

$ apiclient report fips
Benchmark name:  FIPS Security Policy
Version:         v1.0.0
Reference:       https://csrc.nist.gov/
Benchmark level: 1
Start time:      2026-09-04T07:09:46.002757644Z

[PASS] 1.0       FIPS mode is enabled. (Automatic)
[PASS] 1.1       FIPS module is Amazon Linux 2023 Kernel Cryptographic API. (Automatic)
[PASS] 1.2       FIPS self-tests passed. (Automatic)

Passed:          3
Failed:          0
Skipped:         0
Total checks:    3

Compliance check result: PASS

PASS — apiclient reports FIPS compliance on this -fips variant

Terms of contribution:

By submitting this pull request, I agree that this contribution is dual-licensed under the terms of both the Apache License, version 2.0, and the MIT license.

@piyush-jena
piyush-jena force-pushed the ukis/bottlerocket branch 2 times, most recently from dfd998f to 3848d53 Compare August 13, 2026 01:51
Comment thread sources/settings-plugins/aws-ecs-5/src/lib.rs Outdated
@piyush-jena
piyush-jena force-pushed the ukis/bottlerocket branch 4 times, most recently from 10b3860 to 49c7db5 Compare August 19, 2026 02:10
@piyush-jena piyush-jena changed the title Add ecs-5 variants Add mantle-1 variants Aug 19, 2026
@piyush-jena
piyush-jena force-pushed the ukis/bottlerocket branch 2 times, most recently from 413004b to 5205ece Compare August 19, 2026 03:18
@piyush-jena piyush-jena changed the title Add mantle-1 variants Add new variants Aug 19, 2026
@piyush-jena
piyush-jena force-pushed the ukis/bottlerocket branch 2 times, most recently from 12c0b8e to 841ea62 Compare August 19, 2026 23:03
@piyush-jena

Copy link
Copy Markdown
Contributor Author

This breaks aws-dev variant because of the earlier metric-dog changes

@jpculp jpculp left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall this looks great! Can we rename new-variant.toml to something more specific? Long term we might have a newer new variant with different defaults.

@piyush-jena
piyush-jena force-pushed the ukis/bottlerocket branch 2 times, most recently from 1ed7654 to fcb1919 Compare September 4, 2026 10:47
Comment thread sources/settings-defaults/aws-dev/defaults.d/30-metrics.toml
Signed-off-by: Piyush Jena <jepiyush@amazon.com>
@piyush-jena piyush-jena changed the title Add new variants Add aws-mantle-1 variants Sep 4, 2026
@piyush-jena
piyush-jena marked this pull request as ready for review September 4, 2026 21:23
@piyush-jena

Copy link
Copy Markdown
Contributor Author

Force push gets rid of oci-defaults settings, rebases to get latest kits and sdk.

Many setting defaults are inter-connected to trigger the right services
upon setting changes. As we prepare for a minimal variant, we introduce
these shared-defaults to be able to boot with the minimal set of
settings.

Signed-off-by: Piyush Jena <jepiyush@amazon.com>
Signed-off-by: Piyush Jena <jepiyush@amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants