Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 22 additions & 24 deletions .github/workflows/build-image.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -31,22 +31,20 @@ jobs:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v4
uses: actions/checkout@v7

# GCP credentials
# GCP credentials - only when a GCP builder is in scope
- name: Packer GCP Service Account
uses: google-github-actions/auth@v2
if: contains(github.event.inputs.only_builders, 'googlecompute')
uses: google-github-actions/auth@7c6bc770dae815cd3e89ee6cdf493a5fab2cc093 # v3.0.0
with:
workload_identity_provider: 'projects/17033121890/locations/global/workloadIdentityPools/duplo-githubactions/providers/duplo-githubactions'
service_account: 'packer@msp-duplocloud-01.iam.gserviceaccount.com'

# AWS credentials
- name: Packer AWS Role
uses: aws-actions/configure-aws-credentials@v4
uses: aws-actions/configure-aws-credentials@e6de054238d6b7531b4efff3b6587d9aade6a06c # v6.2.3
with:
aws-access-key-id: ${{ env.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ env.AWS_SECRET_ACCESS_KEY }}
aws-session-token: ${{ env.AWS_SESSION_TOKEN }}
aws-region: us-west-2
role-to-assume: arn:aws:iam::227120241369:role/packer-builder
role-session-name: github-duplocloud-linuxagent
Expand All @@ -71,10 +69,10 @@ jobs:
# Validate the template.
packer validate -syntax-only ./packer

# Parse build options.
if [ "$ONLY_BUILDERS" = "all" ]; then
# Parse build options. An empty input would otherwise reach packer unfiltered and build the GCP sources too.
if [ -z "$ONLY_BUILDERS" ] || [ "$ONLY_BUILDERS" = "all" ]; then
ONLY_BUILDERS="-except=googlecompute.ubuntu-20,googlecompute.ubuntu-22"
elif [ -n "$ONLY_BUILDERS" ]; then
else
ONLY_BUILDERS="-only=$ONLY_BUILDERS"
fi

Expand Down Expand Up @@ -105,7 +103,7 @@ jobs:

# Upload the image manifest
- name: Attach Manifest
uses: actions/upload-artifact@v4
uses: actions/upload-artifact@v7
with:
name: packer-manifest.json
path: packer-manifest.json
Expand All @@ -117,17 +115,17 @@ jobs:
duplo_token: "${{ secrets.GOVCLOUD_DUPLO_TOKEN }}"
steps:
- name: Checkout
uses: actions/checkout@v4
uses: actions/checkout@v7

# AWS credentials
- name: Tenant AWS JIT
uses: duplocloud/ghactions-aws-jit@master
uses: duplocloud/ghactions-aws-jit@8add5087285847c4f96923be523cee313aa4009d # master, newer than the v0.3.3 tag
with:
tenant: github

# AWS credentials
- name: Packer AWS Role
uses: aws-actions/configure-aws-credentials@v4
uses: aws-actions/configure-aws-credentials@e6de054238d6b7531b4efff3b6587d9aade6a06c # v6.2.3
with:
aws-access-key-id: ${{ env.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ env.AWS_SECRET_ACCESS_KEY }}
Expand Down Expand Up @@ -157,10 +155,10 @@ jobs:
# Validate the template.
packer validate -syntax-only ./packer

# Parse build options.
if [ "$ONLY_BUILDERS" = "all" ]; then
# Parse build options. An empty input would otherwise reach packer unfiltered and build the GCP sources too.
if [ -z "$ONLY_BUILDERS" ] || [ "$ONLY_BUILDERS" = "all" ]; then
ONLY_BUILDERS="-except=googlecompute.ubuntu-20,googlecompute.ubuntu-22"
elif [ -n "$ONLY_BUILDERS" ]; then
else
ONLY_BUILDERS="-only=$ONLY_BUILDERS"
fi

Expand All @@ -179,7 +177,7 @@ jobs:
done

# Build the images.
rm -f *packer-manifest.json # always be clean
rm -f ./*packer-manifest.json # always be clean
packer build $ONLY_BUILDERS \
-color=false -on-error=cleanup -parallel-builds=10 -timestamp-ui \
-var-file=packer/duplo-gov.json \
Expand All @@ -193,7 +191,7 @@ jobs:

# Upload the image manifest
- name: Attach Manifest
uses: actions/upload-artifact@v4
uses: actions/upload-artifact@v7
with:
name: govcloud-packer-manifest.json
path: govcloud-packer-manifest.json
Expand All @@ -208,9 +206,9 @@ jobs:
steps:
# Get the code for the image JSON generation, and the code for Duplo master.
- name: Checkout duplo-infra
uses: actions/checkout@v4
uses: actions/checkout@v7
- name: Checkout duplo (backend)
uses: actions/checkout@v4
uses: actions/checkout@v7
with:
repository: duplocloud-internal/duplo
ref: master # always start from master
Expand All @@ -219,12 +217,12 @@ jobs:

# Download the image manifest
- name: Download Manifest (Commercial)
uses: actions/download-artifact@v4
uses: actions/download-artifact@v8
with:
name: packer-manifest.json
path: packer
- name: Download Manifest (Govcloud)
uses: actions/download-artifact@v4
uses: actions/download-artifact@v8
with:
name: govcloud-packer-manifest.json
path: packer
Expand All @@ -238,7 +236,7 @@ jobs:

# Create a PR
- name: Create Pull Request
uses: peter-evans/create-pull-request@v4
uses: peter-evans/create-pull-request@5f6978faf089d4d20b00c7766989d076bb2fc7f1 # v8.1.1
with:
title: '[duplo-bot] Update Duplo Docker AMI(s)'
branch: auto-update/duplo-docker-amis
Expand Down
5 changes: 3 additions & 2 deletions .github/workflows/lint.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -6,11 +6,12 @@ jobs:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v3
uses: actions/checkout@v7

# Validate templates
- name: Validate Template
uses: hashicorp/packer-github-actions@master
# Last commit of the command-based action. v2+ is a different, setup-only action with no command/target inputs.
uses: hashicorp/packer-github-actions@8c999b2ff7c48cce2adc5fe43ad7fd8a80e6a8b5 # master
with:
command: validate
arguments: -syntax-only
Expand Down
46 changes: 46 additions & 0 deletions AgentAmazonLinux2023/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
### Amazon Linux 2023 base image

Use the plain AL2023 base image, published by `amazon` and named `al2023-ami-2023.*-kernel-6.1-x86_64` (or `-arm64`). The ECS-optimized and minimal variants also match the looser `al2023-ami-*` pattern and are not supported here.

The agent script and the systemd unit are downloaded from `AgentAmazonLinux2/`. Only the install steps differ on AL2023, so there is no copy of `NetworkAgentV2.py` or `NetworkAgent.service` in this directory.

### Amazon Linux 2023 user data

```
#!/bin/bash

curl -H "Accept: application/vnd.github.v3.raw" -O -L https://api.github.com/repos/duplocloud/linuxagent/contents/AgentAmazonLinux2023/Setup.sh
chmod +x ./Setup.sh
sudo bash ./Setup.sh
#

```

### Amazon Linux 2023 user data, base64

```
IyEvYmluL2Jhc2gKCmN1cmwgLUggIkFjY2VwdDogYXBwbGljYXRpb24vdm5kLmdpdGh1Yi52My5yYXciIC1PIC1MIGh0dHBzOi8vYXBpLmdpdGh1Yi5jb20vcmVwb3MvZHVwbG9jbG91ZC9saW51eGFnZW50L2NvbnRlbnRzL0FnZW50QW1hem9uTGludXgyMDIzL1NldHVwLnNoCmNobW9kICt4IC4vU2V0dXAuc2gKc3VkbyBiYXNoIC4vU2V0dXAuc2gKIwo=
```

### Amazon Linux 2023 manual

```
curl -H "Accept: application/vnd.github.v3.raw" -O -L https://api.github.com/repos/duplocloud/linuxagent/contents/AgentAmazonLinux2023/Setup.sh
chmod +x ./Setup.sh
sudo bash ./Setup.sh

# now, reboot the system
sudo reboot

# after reboot
sudo systemctl daemon-reload
sudo systemctl status NetworkAgent
sudo systemctl stop NetworkAgent
sudo systemctl start NetworkAgent
sudo systemctl status NetworkAgent

tail -f /var/log/NetworkAgent.log

#vi /lib/systemd/system/NetworkAgent.service
#vi /usr/local/src/AgentV2/NetworkAgentV2.py
```
174 changes: 174 additions & 0 deletions AgentAmazonLinux2023/Setup.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,174 @@
#!/bin/bash
set -eu

AGENT='NetworkAgentV2'
DAEMON_DEFAULT_FILE="/etc/default/$AGENT"
DAEMON_DIR='/usr/local/src/AgentV2'
PYTHON_PATH="$DAEMON_DIR/flask/bin"
DAEMON="$DAEMON_DIR/NetworkAgentV2.py"
# Agent script and systemd unit are shared with Amazon Linux 2. Only the install steps differ on AL2023.
DOWNLOAD_URL="https://api.github.com/repos/duplocloud/linuxagent/contents/AgentAmazonLinux2"

DOCKER_OVERRIDE_DIR="/etc/systemd/system/docker.service.d"
DOCKER_OVERRIDE_FILE="$DOCKER_OVERRIDE_DIR/api.conf"

# GitHub redirects raw content to a signed CDN URL, so -L is required. These pin both hops to https.
CURL_HTTPS_OPTS=(--proto '=https' --proto-redir '=https' -fsSL)

if [[ -z "${DOWNLOAD_REF:-}" ]]; then
DOWNLOAD_REF=''
else
DOWNLOAD_REF="?ref=${DOWNLOAD_REF}"
fi

# AL2023 has no amazon-linux-extras. Docker comes from the default repo.
install_dependencies () {
echo "AL2023: installing Docker and base packages"
sudo dnf update -q -y
sudo dnf install -q -y \
docker \
amazon-ecr-credential-helper \
git wget net-tools vim \
gcc \
python3 python3-pip python3-devel \
iptables-devel kernel-headers

sudo usermod -a -G docker ec2-user
mkdir -p ~/.docker
echo '{ "credsStore": "ecr-login" }' > ~/.docker/config.json

sudo mkdir -p "$DOCKER_OVERRIDE_DIR"
sudo tee "$DOCKER_OVERRIDE_FILE" > /dev/null <<EOF
[Service]
ExecStart=
ExecStart=/usr/bin/dockerd -H fd:// -H tcp://0.0.0.0:4243 --containerd=/run/containerd/containerd.sock
EOF

sudo systemctl daemon-reload
sudo systemctl enable docker
sudo systemctl restart docker
sudo systemctl status docker --no-pager || true
sudo docker info || true
}

# Strip the Amazon ECS container agent if present. The pinned AL2023 base does not ship it, and the ECS
# agent lacks the role to start anyhow. Insurance if source_ami_filter is ever loosened to a variant.
remove_ecs_agent () {
echo "AL2023: removing any Amazon ECS agent leftovers"
sudo systemctl disable --now ecs 2>/dev/null || true
sudo systemctl mask ecs 2>/dev/null || true
sudo dnf -q -y remove ecs-init || true
sudo rm -rf /var/lib/ecs /etc/ecs /var/log/ecs
for image in amazon/amazon-ecs-agent amazon/amazon-ecs-pause; do
sudo docker image rm -f "$image:latest" 2>/dev/null || true
sudo docker image rm -f "$image:0.1.0" 2>/dev/null || true
done
sudo docker rm -f ecs-agent 2>/dev/null || true
}

agent_install () {
echo "AL2023: installing NetworkAgent in Python 3 venv"

if [[ -f "$DAEMON_DEFAULT_FILE" ]]; then
sudo rm $DAEMON_DEFAULT_FILE
sudo touch $DAEMON_DEFAULT_FILE
fi

echo "DAEMON=$DAEMON" | sudo tee --append $DAEMON_DEFAULT_FILE > /dev/null
echo "DAEMON_DIR=$DAEMON_DIR" | sudo tee --append $DAEMON_DEFAULT_FILE > /dev/null
echo "PYTHON_PATH=$PYTHON_PATH" | sudo tee --append $DAEMON_DEFAULT_FILE > /dev/null
cat $DAEMON_DEFAULT_FILE

cd "$DAEMON_DIR"
python3 -m venv flask
flask/bin/pip install --upgrade pip
# Install all agent dependencies in a single pip invocation so the
# resolver picks a globally-consistent set. Sequential pip installs
# silently downgrade urllib3 from 2.x to 1.26.x because botocore's
# transitive constraint is only seen on the boto3 install call.
flask/bin/pip --trusted-host pypi.python.org install \
flask \
requests \
python-pytun \
python-iptables \
docker \
boto3

cd "$DAEMON_DIR"
curl "${CURL_HTTPS_OPTS[@]}" \
-H "Accept: application/vnd.github.v3.raw" \
-o NetworkAgentV2.py "$DOWNLOAD_URL/NetworkAgentV2.py$DOWNLOAD_REF"
chmod a+x NetworkAgentV2.py
ls -alt "$DAEMON_DIR"

cd /lib/systemd/system
sudo curl "${CURL_HTTPS_OPTS[@]}" \
-H "Accept: application/vnd.github.v3.raw" \
-o NetworkAgent.service "$DOWNLOAD_URL/NetworkAgent.service$DOWNLOAD_REF"
ls -alt NetworkAgent.service

sudo systemctl daemon-reload
sudo systemctl enable NetworkAgent.service
# Do not start NetworkAgent.service here. Starting it during the
# Packer bake disrupts iptables/networking and severs the SSM/SSH
# session, which surfaces as `Bad exit status: -1`. The unit is
# enabled and will start automatically on first boot.
}

get_os_type () {
if [[ -f /etc/os-release ]]; then
. /etc/os-release
OS=$ID
VER=$VERSION_ID
else
OS=$(uname -s)
VER=$(uname -r)
fi
}

get_os_type
echo "Detected OS=$OS VER=$VER"
if [[ "$OS" != "amzn" ]] || [[ "$VER" != "2023" ]]; then
echo "WARNING: this script targets Amazon Linux 2023; detected $OS $VER"
fi

echo "--------------------------OS=$OS VER=$VER--------------------------"
echo "Step 1: install Docker and base packages"
echo "--------------------------OS=$OS VER=$VER--------------------------"
install_dependencies

echo "==========================="
echo " Docker installed "
echo "==========================="

echo "--------------------------OS=$OS VER=$VER--------------------------"
echo "Step 1b: remove Amazon ECS agent if present"
echo "--------------------------OS=$OS VER=$VER--------------------------"
remove_ecs_agent

echo "--------------------------OS=$OS VER=$VER--------------------------"
echo "Step 2: ensure agent directory exists"
echo "--------------------------OS=$OS VER=$VER--------------------------"
sudo mkdir -p "$DAEMON_DIR"
sudo chown -R "${USER:-ec2-user}" "$DAEMON_DIR"
ls -alt "$DAEMON_DIR"

echo "--------------------------OS=$OS VER=$VER--------------------------"
echo "Step 3: stop any running $AGENT"
echo "--------------------------OS=$OS VER=$VER--------------------------"
if pgrep -f "$DAEMON" > /dev/null; then
echo "$AGENT is running, killing"
sudo pkill -9 -f "$DAEMON" || true
else
echo "$AGENT is not running"
fi

echo "--------------------------OS=$OS VER=$VER--------------------------"
echo "Step 4: install NetworkAgent"
echo "--------------------------OS=$OS VER=$VER--------------------------"
cd "$DAEMON_DIR"
sudo rm -rf NetworkAgentV2.py flask
agent_install

echo "AL2023 setup complete"
exit 0
15 changes: 15 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,18 @@
## 2026-08-19

### Added
- Enabled the Amazon Linux 2023 x86_64 and arm64 Packer builders, along with `AgentAmazonLinux2023/Setup.sh` and a README for it. These sources were added commented-out in 2024-02, so the Amazon Linux 2023 builders claimed by the 2024-02-07 entry below have never run until now.
- Pinned the Amazon Linux 2023 `source_ami_filter` to `al2023-ami-2023.*`, which excludes the ECS-optimized and minimal AMI variants that the looser `al2023-ami-*` pattern also matches.
- Added a per-builder SSH username check to `gen-native-images.sh`. An unmapped builder name previously reused the previous row's username instead of failing.

### Changed
- Merged the generated native-image rows by `Name` instead of replacing every `Docker-Duplo*` row. A build scoped with `only_builders` now leaves the Amazon Linux 2, Ubuntu, and GovCloud rows intact.
- Pinned all third-party GitHub Actions to commit SHAs, and bumped `actions/checkout` to v7, `actions/upload-artifact` to v7, and `actions/download-artifact` to v8.
- Switched the commercial `Packer AWS Role` step to OIDC alone, dropping the static credential inputs that were resolving to empty strings.
- Skipped the GCP credential step unless `only_builders` names a `googlecompute` builder. It had been authenticating on the `all` path, which excludes those builders.
- Treated an empty `only_builders` the same as `all`. A cleared input previously reached packer with no filter, building every source including the GCP ones.
- Added `amazon-ebs.amazonlinux-2-arm64` to the Amazon Linux OS-update step, which had listed only the x86_64 builder.

## 2024-08-13

### Changed
Expand Down
Loading