Skip to content

[#1087] Copy the secret volume on every start of the Docker image, and run the server as PID 1 after the bootstrap - #1100

Open
vharseko wants to merge 4 commits into
OpenIdentityPlatform:masterfrom
vharseko:issue-1087-docker-secret-volume
Open

vharseko wants to merge 4 commits into
OpenIdentityPlatform:masterfrom
vharseko:issue-1087-docker-secret-volume

Conversation

@vharseko

@vharseko vharseko commented Sep 24, 2026 •

Copy link
Copy Markdown
Member

Problem

Change

opendj-packages/opendj-docker/run.sh:

  • copy_secrets copies the key* / trust* files of SECRET_VOLUME that differ from those in ./data/config. Each file is written next to its target, set to mode 0600 and renamed over it, so the server never reads a half copied file. It succeeds only when it copied something, and sync_secrets repeats it until a pass finds nothing left to copy (at most 5 passes): a Secret updated between the copy of keystore and that of keystore.pin would otherwise leave a keystore of one version next to the password of the other.
  • start_server copies the volume, starts watch_secrets in the background when SECRET_VOLUME_REFRESH (default 60 s, 0 disables) is set, logs that the volume is copied on start only when the value is not a whole number of seconds above 0 (1m, 30.5), and execs start-ds --nodetach. Both the first start and every later start end there.
  • On the first start, after the bootstrap, the server setup started is stopped (bin/stop-ds) and started again through start_server. The server is PID 1 in both cases (Docker image: a freshly bootstrapped container ignores SIGTERM and is killed without stopping the server #1085). The stop comes before the .bootstrap-complete marker is written: stopped after it, the health check could report the bootstrap's server healthy just before it went down, and the next dsconfig then failed with "Server Connection Closed" (found in [#1085] Stop the server cleanly on SIGTERM in a container that bootstrapped its instance #1098). stop-ds exits 0 when the server is not running, so a custom BOOTSTRAP that leaves it stopped still works.

README.md gets a "Certificates" section: what the volume holds, that with the default OPENDJ_SSL_OPTIONS the key entry may have any alias (setup binds no handler to one; with --certNickname the key has to be under that alias), when it is copied, that a new keystore password takes effect on the next restart like a new certificate, and a cert-manager setup (spec.keystores.pkcs12 plus a projected volume mapping keystore.p12 → keystore and the password → keystore.pin). The "Health check" section recommends docker run --init: the server is PID 1, and a JVM does not reap the processes a timed-out probe leaves behind. A PEM kubernetes.io/tls Secret cannot be used as is: OpenDJ reads keystores, and the Alpine image has no openssl for a conversion. SECRET_VOLUME_REFRESH is added to the variables table and to both Dockerfiles.

.github/workflows/build.yml gets a "Docker test secret volume" step in both docker jobs. It mounts a keystore at SECRET_VOLUME and checks that:

  • LDAPS serves the mounted certificate from the first start;
  • PID 1 is java;
  • the copied keystore.pin has mode 0600;
  • a renewed keystore reaches the data volume within a minute while the server runs;
  • docker stop takes under 30 s and exits with 143;
  • after a restart the renewed certificate is served;
  • a keystore renewed while no container runs is served by a new container on the same data volume with SECRET_VOLUME_REFRESH=0, so it can only have been copied by the start over the existing instance;
  • with SECRET_VOLUME_REFRESH=0, a keystore renewed while the server runs is not copied.

The existing "Docker test" step in both docker jobs ran stop-ds, an offline import-ldif / rebuild-index and start-ds inside the container. That only worked while PID 1 was tail: with the server as PID 1, stop-ds stops the container, and the step failed with container ... is not running. It now imports online, checks that docker stop -t 60 stops the container in under 50 s and not with exit code 137, restarts it, rebuilds the indexes online and checks the 10000 entries. The step is word for word the one in #1098, which makes the same change for #1085, so the workflow merges cleanly whichever of the two lands first. The offline import path is no longer covered there: it cannot run inside a container whose PID 1 is the server.

A certificate copied while the server runs is served from the next restart. The server side, loading a changed key store file without a restart, is #1095 (PR #1101). Once that lands, the background copy is picked up without a restart.

Verification

Images built from openidentityplatform/opendj:latest and :alpine with this run.sh (and the master health check), a JKS keystore + keystore.pin mounted read-only at /var/secrets/opendj, and SECRET_VOLUME_REFRESH=5:

check before (latest) after (latest and alpine)
first start, secret v1 self-signed certificate CN=secret-v1
PID 1 after the bootstrap bash java
docker stop of the first start whole timeout, exit 137 3-5 s, exit 143
secret replaced with v2 while running not copied copied to the volume by the watcher
restart after v2 still CN=secret-v1 CN=secret-v2
no secret volume self-signed self-signed, clean stop
PKCS12 (openssl legacy and default encoding) named keystore, alias certificate - served
mode of the copied keystore / keystore.pin - 0600 (BusyBox cp recreates the temp file, hence the explicit chmod)

openssl x509 -subject is called with -nameopt RFC2253: the runner's OpenSSL 3.0.13 prints CN = secret-v1 by default, 3.2+ prints CN=secret-v1.

The CI step, extracted from the workflow and run locally the way Actions runs it (bash -eo pipefail), passes on both images with this run.sh (211 s and 247 s), and each of its last two checks goes red against a mutant:

run.sh latest alpine
this PR passes passes
restart branch execs start-ds without copying red at CN=secret-v3 red at CN=secret-v3
watcher guard -ge 0 (a busy loop for 0) red at ! cmp -s red at ! cmp -s

Against the published openidentityplatform/opendj:latest with the old run.sh the step fails at the first check (the self-signed certificate is served). A keystore without a friendlyName (alias 1, as cert-manager's go-pkcs12 writes it) is served as well: setup binds no connection handler to an alias.

All three docker steps of the workflow ("Docker test", "Docker test custom password", "Docker test secret volume"), extracted and run the same way against both images with the run.sh of the previous round, pass (the secret volume step has had two more checks since, with the times above; the other two steps do not mount a secret volume):

image Docker test custom password secret volume
latest 97 s (docker stop 9 s, exit 143) 43 s 63 s
alpine 113 s (docker stop 9 s, exit 143) 54 s 75 s

The old "Docker test" step fails against this run.sh at import-ldif --offline (the container is gone after stop-ds), as the first CI run of this PR showed.

Found while evaluating the Helm chart proposal (discussion #1079).

Fixes #1087
Fixes #1085

@vharseko

Copy link
Copy Markdown
Member Author

@maximthomas, a round before your review: the first CI run of this PR failed in build-docker and build-docker-alpine, and the failure was caused by the PR. Fixed in 481572a.

What failed. The existing "Docker test" step runs stop-ds, an offline import-ldif / rebuild-index and start-ds inside the container. With this PR the server is PID 1 from the first start on, so stop-ds stopped the whole container and the next docker exec failed with container ... is not running. The step only ever worked because PID 1 was tail -f /dev/null (#1085).

What changed.

Verified. All three docker steps of the workflow, run locally the way Actions runs them (bash -eo pipefail) against latest and alpine with this run.sh, pass: "Docker test" 97 s / 113 s (docker stop 9 s, exit 143), "custom password" 43 s / 54 s, "secret volume" 63 s / 75 s.

The PR description is updated to match.

@vharseko
vharseko force-pushed the issue-1087-docker-secret-volume branch from 481572a to 631d048 Compare September 25, 2026 06:56
@vharseko

Copy link
Copy Markdown
Member Author

@maximthomas, rebased onto the current master (2ba918a) as 631d048, with no change to the fix itself.

The merge of #1091 made the second commit (Test the secret volume of the Docker image in CI) conflict in build.yml: both add a step after Docker test custom password in each docker job. Both steps are kept, Docker test arbitrary uid first and then Docker test secret volume. The first and the third commit applied cleanly and are unchanged.

What #1091 changes for this PR: its Docker test arbitrary uid step restarts a container under an arbitrary uid in group 0, so it also runs the start_server path of the new run.sh. copy_secrets writes into data/config, which that uid can write since #1091 gives /opt/opendj to group 0 with g=u.

@maximthomas maximthomas left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

praise: the start sequence is right where #1087 and #1085 broke it.

  • ./bin/stop-ds runs before .bootstrap-complete is written (run.sh:134-140), so the health check never reports the bootstrap's server healthy just before it goes down.
  • copy_secrets writes each file next to its target with mktemp + chmod 600 + mv -f (run.sh:56): the server never reads a half-copied keystore, and the key is readable by its user only.
  • Both roads end in one start_server (run.sh:73-83), so the first start and every later one copy the volume the same way.

issue (blocking): the secret-volume step greps CN=secret-v1, but the runner's openssl x509 -subject prints CN = secret-v1.

.github/workflows/build.yml:588, :594, :605, :803, :809, :820

Both docker jobs run on ubuntu-latest (ubuntu-24.04), whose /usr/bin/openssl is 3.0.13. There get_nameopt() defaults to XN_FLAG_ONELINE (apps/lib/apps.c:198-200 at openssl-3.0.13), which includes XN_FLAG_SPC_EQ, so served() returns subject=CN = secret-v1. grep -q "CN=secret-v1" exits 1 under bash -eo pipefail, and build-docker and build-docker-alpine go red at the first certificate check, whatever LDAPS serves. The no-space form is the 3.2+ default (openssl-3.2.0 apps.c:189-192), which is what a Homebrew openssl prints locally. The step has not run in CI yet: run 36013203998 stopped at "Docker test".

          served() { echo | openssl s_client -connect 127.0.0.1:1637 2>/dev/null | openssl x509 -noout -subject -nameopt RFC2253; }

Or: grep -Eq 'CN ?= ?secret-v1' at each check. Same change in both jobs.


issue (blocking): the cert-manager recipe gives a keystore whose key has alias 1, not server-cert, so the LDAPS handler disables itself.

opendj-packages/opendj-docker/README.md:43, :66-83

Setup's self-signed options set the LDAPS handler's ssl-cert-nickname to server-cert (SecurityOptions.java:37, :104). cert-manager builds keystore.p12 with go-pkcs12 Encode, which puts only a localKeyID on the key bag and no friendlyName, and the JDK loads such a store (as PKCS12 or as JKS) with aliases=[1]. Once run.sh copies it over config/keystore, LDAPConnectionHandler.createSSLContext (:1330-1344) logs ERR_KEYSTORE_DOES_NOT_CONTAIN_ALIAS, drops the alias and disables the handler: a pod that follows the recipe serves no LDAPS, and the image's HEALTHCHECK, which probes LDAPS, never passes. CI misses this because its keystore() uses -alias server-cert. The same gap is in the prose at :43: nothing says which alias the key entry needs.

# the Certificate
spec:
  secretName: opendj-tls
  keystores:
    jks:
      create: true
      alias: server-cert
      passwordSecretRef: { name: opendj-keystore-password, key: password }
---
# the pod template of the StatefulSet
volumes:
  - name: secrets
    projected:
      sources:
        - secret:
            name: opendj-tls
            items: [{ key: keystore.jks, path: keystore }]
        - secret:
            name: opendj-keystore-password
            items: [{ key: password, path: keystore.pin }]

Also state at :43 that the key must be under alias server-cert. keystores.jks.alias needs cert-manager 1.15+. The other option is to set the LDAPS handler's ssl-cert-nickname to the alias the keystore actually has.


issue (non-blocking): nothing tests the copy on the restart road: the watcher has already copied v2 before docker stop.

.github/workflows/build.yml:598-605, :813-820; opendj-packages/opendj-docker/run.sh:92-100

The step waits at :598 until the watcher (SECRET_VOLUME_REFRESH=5) has copied keystore v2 into data/config, then stops and starts the container. At docker start, copy_secrets finds the files equal and copies nothing. A mutant whose [ -d ./data/config ] branch runs BASE's bare exec ./bin/start-ds --nodetach still serves CN=secret-v2 and stays green in both jobs. That is the restart-road half of #1087 ("later starts exec'ed the server before the copy"), and this step is its only test. Pin: recreate the container on the same data volume with the watcher off and a keystore renewed while it was down (after the -nameopt fix above):

          docker rm -f test_secret
          keystore secret-v3
          docker run -d --memory="512m" -e SECRET_VOLUME_REFRESH=0 -p 127.0.0.1:1637:1636 --name=test_secret \
            -v "$SECRETS":/var/secrets/opendj:ro -v test_secret_data:/opt/opendj/data "$IMAGE"
          healthy
          grep -q "CN=secret-v3" <<< "$(served)"

Pin: the restart branch without copy_secrets serves v2, and grep -q "CN=secret-v3" goes red.


suggestion (non-blocking): no test runs with SECRET_VOLUME_REFRESH=0, so nothing pins the -gt 0 guard.

opendj-packages/opendj-docker/run.sh:77; .github/workflows/build.yml:591, :806

SECRET_VOLUME_REFRESH appears in CI only as =5. If the guard is dropped, or turned into -ge, 0 becomes while sleep 0; do copy_secrets; done: a busy loop, where the README says 0 means "copies them on start only". Continue in the REFRESH=0 container from the previous comment. The negation goes inside sh -c because bash -e ignores a failing ! cmd:

          keystore secret-v4
          sleep 15
          docker exec test_secret sh -c '! cmp -s /var/secrets/opendj/keystore /opt/opendj/data/config/keystore'

Pin: with the guard dropped or -ge 0, the busy loop copies v4 within the 15 s and the docker exec goes red.


suggestion (non-blocking): a SECRET_VOLUME_REFRESH that is not a whole number turns the watcher off and logs nothing.

opendj-packages/opendj-docker/run.sh:75-79

[ "$SECRET_VOLUME_REFRESH" -gt 0 ] 2>/dev/null returns 2 for 1m, 60s and 30.5, all of which sleep would accept, and the error goes to /dev/null. The log line at :75 is printed either way. An operator who writes 1m gets no refresh and no hint of it.

    if [ "$SECRET_VOLUME_REFRESH" -gt 0 ] 2>/dev/null; then
      watch_secrets &
    elif [ "$SECRET_VOLUME_REFRESH" != 0 ]; then
      echo "SECRET_VOLUME_REFRESH=$SECRET_VOLUME_REFRESH is not a whole number of seconds, the secret volume is copied on start only"
    fi

suggestion (non-blocking): if the keystore password changes while the server runs, the server keeps the old PIN in memory, and it no longer opens the copied keystore.

opendj-packages/opendj-docker/run.sh:65-68

FileBasedKeyManagerProvider sets keyStorePIN only at init or on a config change (:90), but getKeystore() reads the keystore file again on every call (:129-137). After the watcher copies a keystore with a new password, a dsconfig change to the LDAPS, StartTLS or HTTPS handler rebuilds the SSL context from keystore v2 with PIN v1. The change is rejected (ERR_FILE_KEYMANAGER_CANNOT_LOAD) until the next restart. Serving is not affected. This cannot happen with the recipe's fixed passwordSecretRef. Two options: the watcher skips a pass in which keystore.pin differs from the one in data/config (the copy at the next start applies it, and that is when it takes effect anyway), or the README says that a new keystore password needs a restart.


suggestion (non-blocking): on the first start the JVM is now PID 1, and it does not reap orphans left by a health probe that timed out.

opendj-packages/opendj-docker/run.sh:142, :82; opendj-packages/opendj-docker/Dockerfile:83

The HEALTHCHECK is in shell form and ends in || exit 1, so sh forks ldapsearch, and _client-script.sh forks java. When a probe runs past --timeout=30s, Docker sends SIGKILL to the exec'd sh only. The rest of the chain is reparented to PID 1, and a JVM never calls waitpid, so each orphan becomes a zombie. At BASE, bash on tail -f /dev/null reaped them on the first lifetime. The restart road already had java as PID 1. Each zombie costs one pid slot, and Kubernetes does not use this HEALTHCHECK. One README line would do: recommend docker run --init, which puts a PID 1 there that reaps and forwards SIGTERM.


suggestion (non-blocking): keystore and keystore.pin are compared and renamed one at a time, so one pass can mix two versions of the Secret.

opendj-packages/opendj-docker/run.sh:52-56

If the projected volume's ..data swap lands between the two cmp calls during a password rotation, data/config holds keystore v1 with pin v2 until the next pass. Today this can bite only a start that straddles the swap. It matters once #1101 makes an in-run copy live: compare the whole key*/trust* set first, and copy only when a second compare agrees.

@vharseko

vharseko commented Sep 25, 2026 •

Copy link
Copy Markdown
Member Author

@maximthomas, thanks for the review. Round 4 is in c099b70. Seven of the eight points are taken; the cert-manager alias one did not reproduce, details below.

CN=secret-v1 vs CN = secret-v1 (blocking): fixed. Confirmed in ubuntu:24.04: OpenSSL 3.0.13 prints subject=CN = secret-v1, and with -nameopt RFC2253 it prints subject=CN=secret-v1. My local runs used OpenSSL 3.6.3, which is why the step passed here. served() now passes -nameopt RFC2253 in both jobs.

The cert-manager keystore with alias 1 (blocking): not taken, the LDAPS handler is not bound to server-cert in this image. The image runs setup with OPENDJ_SSL_OPTIONS="--generateSelfSignedCertificate". For SELF_SIGNED_CERTIFICATE, Installer.addCertificateArguments (Installer.java:1589-1594) passes only -k and -t to ConfigureDS, with no -c. So the LDAPS and LDAP handlers get no ssl-cert-nickname at all. SELF_SIGNED_CERT_ALIAS is the alias setup gives the generated key, not a value written to a handler. With an empty nickname set, LDAPConnectionHandler.createSSLContext takes the aliases.isEmpty() branch (:1324-1327) and uses the key manager as it is. ERR_KEYSTORE_DOES_NOT_CONTAIN_ALIAS is never reached.

Measured: I built a PKCS12 with openssl pkcs12 -export and no -name, so there was no friendlyName and only a localKeyID, as go-pkcs12 writes it. keytool -list shows it as 1, PrivateKeyEntry. I mounted it at SECRET_VOLUME with this run.sh. The container turned healthy, LDAPS served subject=CN=noalias-v1, and the error log had no alias message. In a fresh instance of the image, config.ldif has ds-cfg-ssl-cert-nickname only on the Administration Connector (admin-cert) and the Crypto Manager (ads-certificate), not on the LDAPS or LDAP handler. The earlier check in the PR description also stands: a PKCS12 with alias certificate is served.

What I did take from it is the gap in the prose. README.md now says that with the default options the key entry may have any alias, and that if OPENDJ_SSL_OPTIONS sets --certNickname, the key has to be under that alias. The recipe keeps keystores.pkcs12, so it does not need cert-manager 1.15+.

The restart road was not tested (non-blocking): taken. After the restart with v2, the step now stops and removes the container. It then renews the keystore to v3 while no container runs, creates a new container on the same data volume with SECRET_VOLUME_REFRESH=0, and checks that LDAPS serves CN=secret-v3. With the watcher off, only the copy on the restart road can bring v3 in.

SECRET_VOLUME_REFRESH=0 (non-blocking): taken, in the same container: keystore v4 is written, and 15 s later docker exec test_secret sh -c '! cmp -s …' checks that it was not copied.

Both pins were checked against mutants, running the step locally the way Actions does (bash -eo pipefail) on latest and alpine:

run.sh latest alpine
this round passes, 211 s passes, 247 s
restart branch exec ./bin/start-ds --nodetach without the copy red at grep -q "CN=secret-v3" red at grep -q "CN=secret-v3"
guard -ge 0 red at ! cmp -s red at ! cmp -s

SECRET_VOLUME_REFRESH that is not a whole number (non-blocking): taken as you suggested, with the message "is not a whole number of seconds above 0", so that -5 is covered too. I checked it on 60, 5, 0, 1m, 30.5 and -5: only 0 stays silent.

A new keystore password while the server runs (non-blocking): taken as a README sentence. "The same holds for a new keystore password: the server keeps the one it started with." With #1101, a handshake after the change reads the PIN again (currentKeyManager()), so the new password is picked up there. Only getKeyManagers(), which a dsconfig change of a handler calls, still loads the store with the PIN in memory until the next handshake. I noted that on #1101: #1101 (comment)

PID 1 and orphaned probes (non-blocking): taken. The Health check section of the README now recommends docker run --init (init: true in Compose).

keystore and keystore.pin of two versions in one pass (non-blocking): taken. copy_secrets now succeeds only when it copied a file. A new sync_secrets repeats passes until one finds nothing left to copy (at most 5), and both the start and the watcher use it. A pass that straddles the ..data swap copies keystore v1 and pin v2. The next pass sees that the keystore differs and copies v2, and the pass after that is empty. I checked this outside Docker with a cp that swaps the source files right after the keystore is copied: one copy_secrets leaves keystore=v1 pin=v2, and sync_secrets leaves keystore=v2 pin=v2.

The PR description is updated to match.

Edited: the first version said that #1101 reloads the store with the same keyStorePIN. It does not: it reads the PIN again. Only the narrower getKeyManagers() case above remains.

…the Docker image, and run the server as PID 1 after the bootstrap

Keystores and truststores from SECRET_VOLUME were copied only on the first start, and
only after setup had already started the server, so the mounted certificate was served
from the first restart on and a renewed one never reached an instance on a persistent
volume. They are now copied before every start, each file renamed over its target with
mode 0600, and checked again every SECRET_VOLUME_REFRESH seconds while the server runs.

On the first start the server setup started is stopped and started again with exec, as
every later start does, so the server is PID 1 and stops cleanly on SIGTERM.

The README documents the volume and a cert-manager setup that mounts its PKCS12
keystore under the names the image expects.

Fixes OpenIdentityPlatform#1087
Fixes OpenIdentityPlatform#1085
…e in CI

A step in both docker jobs mounts a keystore at SECRET_VOLUME and checks that LDAPS
serves it from the first start, that the server is PID 1 and stops on SIGTERM, that a
renewed keystore reaches the data volume while the server runs and is served after a
restart, and that the copied PIN file is readable by the server's user only.
…alth check may probe it, and import online in the Docker test

The server is PID 1 of the container from the first start on, so stop-ds in
"Docker test" stopped the container and the offline import that followed had
nothing to run in. The step now imports online and checks that docker stop
stops the server cleanly, as OpenIdentityPlatform#1098 does. The server setup started is stopped
before the bootstrap marker is written, so that the health check never reports
it healthy just before it goes down.
…h the watcher off, and read the served subject in one format

The runner's OpenSSL 3.0.13 prints the subject as `CN = secret-v1`, so the
secret-volume step now asks for `-nameopt RFC2253`. The step also renews the
keystore while no container runs and checks that a new container on the same
data volume, with SECRET_VOLUME_REFRESH=0, serves it, and that with 0 a
keystore renewed while the server runs is not copied.

run.sh repeats the copy until a pass finds nothing left to copy, so that a
Secret updated in the middle of a pass cannot leave a keystore of one version
next to the password of the other, and it says so when SECRET_VOLUME_REFRESH is
not a whole number of seconds above 0. The README says that the key may have
any alias with the default options, that a new keystore password takes effect
on the next restart, and recommends `docker run --init`.
@vharseko
vharseko force-pushed the issue-1087-docker-secret-volume branch from c099b70 to 4abcc52 Compare September 25, 2026 09:46
@vharseko

Copy link
Copy Markdown
Member Author

@maximthomas, rebased onto the current master (67eb249) as 4abcc52, with no change to the fix itself.

The merge of #1094 made two commits conflict:

  • run.sh (first commit, Copy the secret volume on every start…): [#1084] Keep the root password out of the log and off the command line when a Docker container joins replication #1094 adds a removal of the password file of replicate.sh from /dev/shm right where this commit adds the secret volume functions. Both are kept, the removal first. It still runs before anything else, and replicate.sh still runs after the bootstrap and before the stop-ds that hands over to start_server.
  • build.yml (second commit, Test the secret volume of the Docker image in CI): both add a step after Docker test arbitrary uid in each docker job. Both steps are kept, Docker test replication first and then Docker test secret volume.

The third and the fourth commit applied cleanly and are unchanged.

@maximthomas maximthomas left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

praise: Round 1's blocking points are settled, and the new REFRESH=0 run pins the copy on an existing instance.

  • served() now pipes through openssl x509 -noout -subject -nameopt RFC2253, so CN=secret-vN matches on the runner's OpenSSL 3.0 too.
  • The REFRESH=0 container, created again on test_secret_data after keystore secret-v3, pins the start-time copy over an existing instance (your mutant table: red at CN=secret-v3).
  • The README's alias paragraph is right, and my round-1 issue on the cert-manager alias is withdrawn: Installer.addCertificateArguments passes no nickname for SELF_SIGNED_CERTIFICATE, and ConfigureDS then removes ds-cfg-ssl-cert-nickname from the LDAP/LDAPS/HTTP/JMX handlers, so a key under alias 1 is served.

issue (non-blocking): The ERR trap of "Docker test secret volume" does not run for a failure inside healthy() or keystore().

.github/workflows/build.yml:627, :630-638, :901, :904-912

Actions runs the step as bash -eo pipefail without -E. A function does not inherit the ERR trap, and errexit ends the shell from inside the function. So a container that never turns healthy (timeout exits 124), or a failed keytool/mv, fails the step without the docker logs test_secret the trap is there to print. I checked this locally: bash -eo pipefail -c 'trap "echo ERR-TRAP-RAN" ERR; h() { bash -c "exit 124"; }; h' exits 124 and prints nothing, and with set -E the trap prints. Your other docker steps call timeout at top level, so their traps do run.

set -E
trap 'code=$?; echo "::group::container logs (test_secret)"; docker logs test_secret 2>&1 || true; echo "::endgroup::"; exit $code' ERR

issue (non-blocking): If the keystore password changes while the server runs, the next dsconfig change to the LDAPS handler shuts LDAPS down. The new README sentence does not warn about this.

opendj-packages/opendj-docker/run.sh:79-83, opendj-packages/opendj-docker/README.md:64-65

watch_secrets is on by default and copies keystore and keystore.pin into data/config while the server runs. FileBasedKeyManagerProvider reads the PIN only at init (:90), but it reloads the keystore file on every getKeystore() (:129-138). A dsconfig modify of the LDAPS handler goes through LDAPConnectionHandler2.isConfigurationAcceptable:584 → createSSLContext. There containsAtLeastOneKey swallows the wrong-password failure, and disableAndWarnIfUseSSL sets enabled = false on the live handler before the change is refused. run() then stops the listener, and LDAPS stays closed until a restart. The flag flip during the dry run is an older server bug, but before this PR nothing in the image rewrote data/config under a running server. You can fix this in either of two ways. One: the watcher leaves a changed password to the next start, which is when it takes effect anyway. Two: the README says to restart before any change to a TLS connection handler.

pins_unchanged() {
  local src
  for src in "$SECRET_VOLUME"/*.pin; do
    [ -f "$src" ] || continue
    cmp -s "$src" "./data/config/$(basename -- "$src")" || return 1
  done
}

watch_secrets() {
  while sleep "$SECRET_VOLUME_REFRESH"; do
    pins_unchanged && sync_secrets
  done
}

suggestion (non-blocking): The secret-volume step never changes keystore.pin and never mounts a truststore. So the sync_secrets repeat, the trust* glob and the skip of unchanged files are not pinned.

.github/workflows/build.yml:630-636, :904-910; opendj-packages/opendj-docker/run.sh:56-76

Three run.sh mutants pass both images' steps:

  • sync_secrets() { copy_secrets; }. One pass copies both changed files.
  • Dropping "$SECRET_VOLUME"/trust* from :56.
  • Deleting cmp -s "$src" "$dst" && continue at :59. Every start and every 5 s tick then copies identical bytes five times, and no check looks at that.

A truststore in the volume pins the glob. Counting the copy log line pins the skip. The repeat needs a Secret that changes in the middle of a pass, which CI cannot produce on demand. Not run here (no Docker daemon).

keystore() {
  # ... as now, then:
  docker run --rm --entrypoint keytool -v "$SECRETS":/secrets "$IMAGE" -exportcert -rfc -alias server-cert \
    -keystore /secrets/keystore -storepass changeit -file /secrets/cert.pem
  rm -f "$SECRETS/truststore.new"
  docker run --rm --entrypoint keytool -v "$SECRETS":/secrets "$IMAGE" -importcert -noprompt -alias ca \
    -file /secrets/cert.pem -keystore /secrets/truststore.new -storetype JKS -storepass changeit
  mv -f "$SECRETS/truststore.new" "$SECRETS/truststore"
  printf changeit > "$SECRETS/truststore.pin"
  rm -f "$SECRETS/cert.pem"
}
# after the first `healthy`:
docker exec test_secret cmp -s /var/secrets/opendj/truststore /opt/opendj/data/config/truststore
sleep 12
test "$(docker logs test_secret 2>&1 | grep -c '^Copied keystore from the secret volume$')" -eq 1

Pin: the trust* mutant goes red at cmp, and the mutant without the skip goes red at the count (5 or more instead of 1).


suggestion (non-blocking): Every CI keystore puts its key under alias server-cert, so nothing in CI pins the README's "the key entry may have any alias".

.github/workflows/build.yml:631, :905; opendj-packages/opendj-docker/README.md:50-52

server-cert is also the name setup gives its own key. Suppose a regression binds the handlers to that name, for example a -a server-cert in the SELF_SIGNED_CERTIFICATE case or a --certNickname server-cert in the default OPENDJ_SSL_OPTIONS. CI would still serve CN=secret-v1..v3. A cert-manager keystore.p12 (alias 1) would lose LDAPS. The "alias 1 is served" result in the PR body came from a local run.

docker run --rm --entrypoint keytool -v "$SECRETS":/secrets "$IMAGE" -genkeypair -alias "$1" \

Pin: with a per-version alias, the handler-binding mutant fails at CN=secret-v1. If you also take the truststore pin above, use -alias "$1" in its -exportcert.


suggestion (non-blocking): The SECRET_VOLUME_REFRESH=0 check waits 15 s, and the default interval is 60 s. A mutant that treats 0 as "use the default" passes.

.github/workflows/build.yml:663-664, :937-938

Take [ "$SECRET_VOLUME_REFRESH" -gt 0 ] 2>/dev/null || SECRET_VOLUME_REFRESH=60 added after run.sh:47. Its watcher ticks at t0+60 s, and the ! cmp runs about 35-45 s after t0. So only the busy-loop mutant from your table (-ge 0) is caught. Asking the container whether a watcher exists answers this at once: with 0 no sleep process runs, and with any watcher one nearly always does. Keep the ! cmp for the busy loop. Not run here.

docker exec test_secret sh -c '! grep -sqx sleep /proc/[0-9]*/comm'

suggestion (non-blocking): Only a health probe landing in a window of about 1 s pins the rule that the bootstrap's server stops before .bootstrap-complete is written.

opendj-packages/opendj-docker/run.sh:150-156; .github/workflows/build.yml:527-528

Swap the two and the HEALTHCHECK (test -f marker && ldapsearch, every 5 s during the start period) reports healthy only if it starts after the touch and finishes before stop-ds sends SIGTERM. Even then, "Docker test" goes red only if its 10 s poll reaches dsconfig create-backend while the foreground server is still starting. So a regression would come back as the same flake as in #1098.

Pin: check the order on the instance rather than through a race. For example, assert that .bootstrap-complete is not older than the first "The Directory Server is now stopped" line in logs/errors. Or log a line after the touch and assert that it follows stop-ds's output in docker logs.


suggestion (non-blocking): The exit status of ./bin/stop-ds is ignored. If the bootstrap's server takes more than 200 s to stop, exec start-ds fails with "already running" and ends the container.

opendj-packages/opendj-docker/run.sh:150

stop-ds waits for logs/server.pid in WaitForFileDelete, and that wait gives up after DirectoryServer.DEFAULT_TIMEOUT (200 s). The file is deleted only when the JVM exits. After a timeout, start_server execs start-ds, whose start check sees the lock still held and exits, and PID 1 goes with it. Not run: nothing I read bounds the stop time of a freshly bootstrapped server, and this machine has no Docker daemon. Failing with the reason says what happened:

./bin/stop-ds || { echo "The server the bootstrap started did not stop (stop-ds exited $?)"; exit 1; }

nitpick (non-blocking): A SECRET_VOLUME_REFRESH with surrounding whitespace passes the -gt 0 guard, but the first sleep then fails. The values 00 and +0 get the "not a whole number" message.

opendj-packages/opendj-docker/run.sh:80, :91-94

[ "60 " -gt 0 ] and [ " 60" -gt 0 ] are true, and sleep "1 " exits 1 with "invalid time interval". So the loop ends at once, and the only trace is sleep's error line. 00 and +0 do switch the watcher off, but the log calls them invalid.

if [[ $SECRET_VOLUME_REFRESH =~ ^[0-9]+$ ]] && [ "$SECRET_VOLUME_REFRESH" -gt 0 ]; then
  watch_secrets &
elif ! [[ $SECRET_VOLUME_REFRESH =~ ^0+$ ]]; then
  echo "SECRET_VOLUME_REFRESH=$SECRET_VOLUME_REFRESH is not a whole number of seconds above 0, the secret volume is copied on start only"
fi

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

2 participants