[#1085] Stop the server cleanly on SIGTERM in a container that bootstrapped its instance by vharseko · Pull Request #1098 · OpenIdentityPlatform/OpenDJ

vharseko · 2026-09-24T14:06:44Z

Problem

On the first start of a container (empty data volume) run.sh did not exec the server: setup had started it in the background, and the script ended in tail -f /dev/null with bash as PID 1. A PID 1 without a SIGTERM handler never receives the signal, so docker stop / pod termination waited out the whole timeout and then SIGKILLed everything; the server was never stopped cleanly.

Change

opendj-packages/opendj-docker/run.sh

Once the bootstrap (and the replication setup, if any) is through, the server that setup started is stopped with ./bin/stop-ds; the existing exec ./bin/start-ds --nodetach then starts it as PID 1, exactly as on a restart. stop-ds exits 0 when the server is not running, so a custom BOOTSTRAP that leaves it stopped still works.
The stop comes before the keystores are copied from SECRET_VOLUME and before the .bootstrap-complete marker (Report the OpenDJ container healthy only once its bootstrap has succeeded #898) is written. Stopping after the marker let the health check report the bootstrap's server healthy just before it went down - the next dsconfig then failed with "Server Connection Closed" (seen while testing this change).
The status | grep Started + tail -f /dev/null block is gone.
When stop-ds fails, run.sh says so and still tries the start: a server that is still stopping makes start-ds fail on the server lock, and one that stopped late lets it run.

A first start now has a short window between the bootstrap's server going down and the final one coming up. The health check is not affected: the marker is written after the stop, so only the server that keeps running can report healthy.

opendj-packages/opendj-docker/bootstrap/replicate.sh

A replica started together with its master can reach it in that window, and its one dsreplication enable then failed with ERROR_CONNECTING, leaving it unhealthy and unreplicated for good. The tools are now tried again every 10 s for up to 5 minutes: enable only when it exits 8 (ERROR_CONNECTING: mostly it could not connect or bind to one of the two servers, which it checks before it changes anything, but it also exits 8 when another server of the topology cannot be reached after it has written to the first two; any other failure is final, as a second enable of a base DN already replicated exits 5; a wrong root DN or password also exits 8 and so fails only after the 5 minutes), and initialize, initialize-all and the dsconfig on the master after any failure. A failed enable ends the script instead of going on to initialize.

.github/benchmark/compare-opendj.sh, .github/workflows/benchmark.yml

Both waited only for the server to answer, then seeded ou=People. That landed in the restart window and failed silently (ldapadd || true), and JMeter then bound as users that did not exist: 69729 / 74409 / 56464 BIND | 49 on JE, PDB and Build in this PR's first CI run, against 3 on master. Both now wait for healthy where the image has a health check, still search the base entry (older images have a health check without the marker), and wait up to 5 minutes, the health check's start period.
After the ldapadd, both check that ou=People is there, and stop (compare-opendj.sh, with the container log saved) or fail the step (benchmark.yml) when it is not, so a seed that missed no longer leaves the job green.

opendj-packages/opendj-docker/README.md

The health check section says that on a first start the server is restarted once, so waiting only for the port is not enough, and that a replica tries a master it cannot connect to again for up to 5 minutes - but still fails when the master stops while dsreplication enable is writing to it.
With OPENDJ_REPLICATION_TYPE=srs the directory server replicas are to be started one at a time, each once the previous one is healthy: each replica pushes its data (initialize-all) to the replicas connected at that moment, and one that is restarting at the end of its own first start misses the push and comes back in a bad generation ID while it reports healthy. CI has no srs job, so this is read, not run.

.github/workflows/build.yml (Docker test in both docker jobs)

The step used to run stop-ds, an offline import-ldif/rebuild-index and start-ds inside the container. That only worked because of this bug: with the server as PID 1, stop-ds stops the container. It now imports online, then checks that docker stop -t 60 stops the container in under 50 s, not with exit code 137, and that the error log gained a "The Directory Server is now stopped" record (the server run in the foreground reports its shutdown only there). It then restarts the container, waits for it to be healthy, rebuilds the indexes online and checks the 10000 entries.
The offline import path is no longer covered here: it cannot run inside a container whose PID 1 is the server.
Right after docker run the step waits for the .bootstrap-complete marker and checks that the error log already holds a "The Directory Server is now stopped" record. That pins the stop before the marker: the server logs the record before its JVM exits, and stop-ds returns only once server.pid is gone at JVM exit.
In the replication part (from [#1084] Keep the root password out of the log and off the command line when a Docker container joins replication #1094), the master is taken off the network while the simple replica sleeps before its first dsreplication, and connected again once the replica logs "dsreplication enable exited with 8, trying again". The replica then has to become healthy and receive a change made on the master. The sdsr replica's enable is pinned the same way.
Once both replicas have the change, replicate.sh is run once more on the simple replica: its enable of a base DN already replicated has to end it with exit 5, without "trying again" and without "initializing replication". That pins both that only exit 8 is tried again and that a failed enable ends the script.

Testing

The Docker test step, run locally against the released images with this run.sh and the branch's ENV BOOTSTRAP_COMPLETE / HEALTHCHECK laid over them:

image	`docker stop -t 60`	exit code	step
`run.sh` of `master` (control)	62 s	137	fails
this change, `latest`	8-10 s	143	passes (twice)
this change, `alpine`	9 s	143	passes

The stop-before-marker check, run locally the same way (with a longer timeout, as the bootstrap takes about 4.5 minutes under emulation): passes on this change, fails with the stop-ds block moved after touch "$BOOTSTRAP_COMPLETE".

The replication retry, run locally the same way over the #1094 image: with this replicate.sh the replica's enable exited with 8 while the master was off the network, connected on the next try, and the replica became healthy and received the change; with the replicate.sh of master (no retry) the replica logged "Replication setup failed" and never became healthy.

The replication part of the step as it now stands (build-docker job, extracted from build.yml, only the names and timeouts changed), run locally over the round-3 image with this replicate.sh: passes in 6 minutes, both disconnects included. The second replicate.sh exits 5 on this change, and fails the check with retry any on the simple enable (it tries again and runs into timeout 90) and with its || exit removed (it goes on to initialize and exits 0). An sdsr replica whose enable is not wrapped in retry never logs "trying again" while the master is off the network, and logs "Replication setup failed".

The new benchmark wait, run locally followed by ldapadd people.ldif: ou=People was there after the container became healthy both on this change (98 s) and on the released openidentityplatform/opendj:latest (55 s). The old wait did not lose the entry in the one local run: the window is a race, and the CI numbers above are the failing side.

Out of scope

SIGTERM is still not delivered while the bootstrap itself runs (PID 1 is bash until the exec), and a bootstrap interrupted halfway leaves data/config behind, so the next start takes the restart branch. Not verified; left for a separate issue.

setup.sh opens the admin port before create-backend, so a replica that reaches its master during that part of the master's bootstrap gets "no suffix to replicate" rather than a connect error, which is not tried again. This predates this change (read, not run).

…alth check may probe it, and import online in the Docker test The server is PID 1 of the container from the first start on, so stop-ds in "Docker test" stopped the container and the offline import that followed had nothing to run in. The step now imports online and checks that docker stop stops the server cleanly, as OpenIdentityPlatform#1098 does. The server setup started is stopped before the bootstrap marker is written, so that the health check never reports it healthy just before it goes down.

maximthomas

praise: The server is now PID 1 on the first start as well, and the stop sits where the health check cannot see it.

opendj-packages/opendj-docker/run.sh:88 stops the bootstrap's server and :107 execs start-ds --nodetach, so a first start and a restart end in the same process tree.
The stop runs before touch "$BOOTSTRAP_COMPLETE" (run.sh:103), so the HEALTHCHECK (Dockerfile:78) never reports the bootstrap's server healthy just before it goes down.
The Docker test step now requires docker stop -t 60 to finish in under 50 s with an exit code other than 137 (.github/workflows/build.yml:535-540), which is exactly the #1085 symptom.

issue (blocking): On a first start the server now comes up, goes down and comes up again, and the in-repo benchmark seeds its users into that gap.

opendj-packages/opendj-docker/run.sh:67-107, .github/benchmark/compare-opendj.sh:53-74, .github/workflows/benchmark.yml:180-193

setup.sh starts the server and creates the base entry, so wait_dj returns as soon as the bootstrap's server answers. run.sh:88 then stops that server, and the ldapadd -f people.ldif ... || true runs during the stop/start window: it fails without an error, and JMeter then binds as users that do not exist. This PR's build-docker job (run 36010414848) logs 69729 / 74409 / 56464 BIND | 49 | Invalid Credentials on the JE, PDB and Build images. On the base commit, master (run 36001081168) logs 0 / 3 / 0. The job stays green, so the benchmark numbers mean nothing from now on. benchmark.yml has the same wait and ldapadd || true, followed by a dsconfig (:193) with no || true. The one readiness signal that stays correct across the restart is the container's health status.

wait_dj() {  # healthy once the image has a HEALTHCHECK; the first start restarts the server
  for _ in $(seq 1 90); do
    case "$(docker inspect -f '{{if .State.Health}}{{.State.Health.Status}}{{end}}' opendj-bench 2>/dev/null)" in
      healthy) return 0 ;;
      "") ldapsearch -x -H ldap://localhost:1389 -D "cn=Directory Manager" -w password \
            -b "$BASEDN" -s base dn >/dev/null 2>&1 && return 0 ;;
    esac
    sleep 2
  done
  return 1
}

Or: the same loop in benchmark.yml:180-186, with the container opendj.

suggestion (non-blocking): No test checks that stop-ds runs before the .bootstrap-complete marker is written.

opendj-packages/opendj-docker/run.sh:88, :103, .github/workflows/build.yml:526-528

Suppose stop-ds is moved after the touch. The Docker test step then fails only when a 30 s health probe lands in the 1-3 s between the marker and the stop. Otherwise it sees the same sequence as the head, so that mutant passes in most runs (traced, not run). The stop/restart checks at :535-544 do not see the order, because the baseline count is read after the bootstrap's stop in both orders.

# right after `docker run` in both Docker test steps: the bootstrap's server is down before the marker exists
timeout 3m bash -c 'until docker exec test sh -c "test -f \"\$BOOTSTRAP_COMPLETE\"" 2>/dev/null; do sleep 0.2; done'
test "$(docker exec test grep -c "The Directory Server is now stopped" /opt/opendj/data/logs/errors)" -ge 1

Pin: the head always passes, because the stop record exists whenever the marker does. The mutant fails unless the 0.2 s poll misses the stop window. This assumes setup itself never stops the server: setup.sh has no stop-ds and no --doNotStart. Not run.

suggestion (non-blocking): The exit status of the new ./bin/stop-ds is ignored.

opendj-packages/opendj-docker/run.sh:88

stop-ds exits non-zero when the server is running but logs/server.pid is missing (stop-ds:183-185), or when WaitForFileDelete gives up on a slow shutdown (stop-ds:168-173). In that case the marker is still written, start-ds cannot take the server lock, and the container stops without a line saying why. Before this change the bootstrap's server stayed up. The failure path was read, not run, and it is rare.

./bin/stop-ds || { BOOTSTRAPPED=false; echo "Could not stop the server started by the bootstrap, this container will not report itself healthy"; }

vharseko · 2026-09-25T08:27:10Z

Thanks. Round 2 is 1803490, rebased onto the current master (2ba918a, after #1091; no conflicts).

issue (blocking), the benchmark seeds into the restart gap: agreed. The job logs you quoted show it: 69729 / 74409 / 56464 BIND | 49 on JE, PDB and Build, and 56464 in the alpine job, against 3 on master.

compare-opendj.sh wait_dj and the wait in benchmark.yml now wait for healthy where the image has a health check, and search the base entry in both cases. The search stays because an older image (5.1.2, before Report the OpenDJ container healthy only once its bootstrap has succeeded #898) has a health check without the marker, which can pass before its bootstrap is done.
Both waits now last 5 minutes (150 × 2 s), the health check's start period. healthy arrives later than "the port answers" used to: after the restart, and up to one 30 s probe interval after the marker.
The README's health check section now says that on a first start the server is restarted once, so a client that waits only for the port can have its first requests fail.

Run locally with the new wait_dj taken from the script, then ldapadd people.ldif, waiting for healthy, and a search for ou=People. The latest + this run.sh overlay waited 98 s, the released openidentityplatform/opendj:latest (health check without the marker) 55 s, and ou=People was there in both. The old wait did not lose the entry in the one local run (its ldapadd landed before the stop). The gap is a race, and the CI logs above are the red side.

suggestion, no test pins the stop before the marker: taken, in both Docker test steps, right after docker run. It cannot flake on the head: the server logs "The Directory Server is now stopped" and closes its log publishers before the JVM exits (DirectoryServer.java:4344-4347), server.pid is removed only at JVM exit (deleteOnExit), and stop-ds returns only once server.pid is gone. So the record is written before the marker can be.

Run locally with the two lines as they are in build.yml (only the timeout raised, since the bootstrap takes about 4.5 minutes under emulation here): the head passes, and the mutant with the stop-ds block moved after touch "$BOOTSTRAP_COMPLETE" fails.

suggestion, the exit status of stop-ds: taken in part. run.sh now says "Could not stop the server started by the bootstrap, starting OpenDJ may fail" when stop-ds fails, and still tries the start. I did not add BOOTSTRAPPED=false. When WaitForFileDelete gives up after 200 s on a slow shutdown that does finish before start-ds runs, the server comes up normally, but without the marker the container would never report healthy. When the old server is still running, exec start-ds --nodetach fails on the server lock, reports that another instance is already running, and the container stops, so the failure is already logged (read, not run).

…alth check may probe it, and import online in the Docker test The server is PID 1 of the container from the first start on, so stop-ds in "Docker test" stopped the container and the offline import that followed had nothing to run in. The step now imports online and checks that docker stop stops the server cleanly, as OpenIdentityPlatform#1098 does. The server setup started is stopped before the bootstrap marker is written, so that the health check never reports it healthy just before it goes down.

maximthomas

praise: The round-1 blocker is fixed with the one signal that survives the restart, and the stop-before-marker order is now pinned.

.github/benchmark/compare-opendj.sh:58-69 and .github/workflows/benchmark.yml:185-193 wait for healthy before seeding. run.sh:107 writes the marker only after the stop at run.sh:90, so no seed can land between the two servers.
.github/workflows/build.yml:529-530 and :710-711 wait for the marker and require the "The Directory Server is now stopped" record to be in the log already, in both docker jobs.

issue (blocking): A replica started together with its master runs its only dsreplication enable while the master is restarting, and never tries again.

opendj-packages/opendj-docker/run.sh:89-111, opendj-packages/opendj-docker/bootstrap/replicate.sh:26-51, :85-116

On a first start the master now stops the bootstrap's server (run.sh:90) and execs start-ds (run.sh:111), so port 4444 is down for the stop plus the start: about 3 s plus 10-16 s in CI's container logs. A replica that bootstrapped at the same time runs replicate.sh, which does sleep 5 and then one dsreplication enable --host1 $MASTER_SERVER. That call connects once and returns ERROR_CONNECTING on failure (ReplicationCliMain.java:3605-3611), and it lands in the window whenever both containers start together: Compose without condition: service_healthy, or the openshift template's StatefulSet scaled past 1 (OrderedReady, no readinessProbe). run.sh:75 then withholds the marker, so the replica never reports healthy and has no replication. Because ./data/config now exists, every restart takes run.sh:36-54, which never runs replicate.sh again; only wiping the volume recovers it. Before this PR the bootstrap's server stayed up, and the same start succeeded. The path is traced through the code; how often the lockstep start happens has not been measured.

# replicate.sh — the master stops and restarts its server once at the end of its own
# first start (run.sh), so a call that lands in that gap cannot connect: try again for
# up to 5 minutes.
retry() {
  for _ in $(seq 1 30); do
    "$@" && return 0
    echo "$1 failed, trying again in 10 s"
    sleep 10
  done
  return 1
}

# in each OPENDJ_REPLICATION_TYPE branch:
retry /opt/opendj/bin/dsreplication enable ... || exit 1
retry /opt/opendj/bin/dsreplication initialize ... || exit 1

Or: a README line saying a replica must be started only after its master is healthy. That is weaker, because it leaves the openshift template exposed.

suggestion (non-blocking): Nothing fails when the benchmark seed misses, so no check covers the new wait.

.github/benchmark/compare-opendj.sh:81-83, .github/workflows/benchmark.yml:185-195

After the wait, compare-opendj.sh continues past WARN: ... not ready in time and past ldapadd ... || true. benchmark.yml leaves its loop with no message when the server never gets ready, and its ldapadd is also || true; both behave as on master. If the port-only wait comes back, every job stays green, and the round-1 numbers (69729 / 74409 / 56464 BIND | 49) appear only in the step summary.

# compare-opendj.sh, after the ldapadd at :82-83
# (A_VER="$(bench_one ...)" under set -euo pipefail stops the script)
ldapsearch -x -H ldap://localhost:1389 -D "cn=Directory Manager" -w password \
  -b "ou=People,$BASEDN" -s base dn >/dev/null 2>&1 \
  || { echo "ERROR: $image: ou=People was not seeded" >&2; return 1; }

# benchmark.yml, after the ldapadd at :194-195
ldapsearch -x -H ldap://localhost:1389 -D "cn=Directory Manager" -w password \
  -b "ou=People,$BASEDN" -s base dn >/dev/null

Pin: this passes at this head, since ou=People is there once the container is healthy. With the health test removed from the wait, it fails in every run whose seed lands in the restart window.

nitpick (non-blocking): The new wait comments say an add made during the restart "is lost". In fact the add fails.

.github/benchmark/compare-opendj.sh:53-57, .github/workflows/benchmark.yml:180-184

An add that the bootstrap's server completed survives the stop. stop-ds returns only once server.pid is gone at JVM exit, after the backends are closed, and nothing replaces the backend before start-ds. A request sent during the window gets connection refused or closed, and || true hides that. This is what the README hunk says: "can have its first requests fail".

# first: on a first start the server the bootstrap started answers, then is stopped and
# started again, and a request sent in between fails. An older image's health check

…ontainer that bootstrapped its instance On the first start run.sh kept bash as PID 1 behind `tail -f /dev/null`, so SIGTERM was never delivered and the container was killed at the end of the stop timeout without stopping the server. The server that setup started is now stopped once the bootstrap is through, before the health marker is written, and started again with exec, so it is PID 1 as on every restart. The Docker test stopped and started the server inside the container, which only worked because of this; it now imports online, checks that `docker stop` stops the server cleanly, and checks the restarted container. Fixes OpenIdentityPlatform#1085

…re seeding the benchmark, and check that the bootstrap's server is down before the marker On a first start the server the bootstrap started is stopped and started again, so waiting for it to answer is no longer enough: the benchmark seeded ou=People into that gap and ran against a directory without its users. Both benchmark waits now require a healthy container where the image has a health check, still search the base entry, which older images need, and last as long as the health check's start period. The Docker test steps check that the bootstrap's server has logged its stop by the time the marker is written. run.sh says so when stop-ds fails, and still tries the start.

…ng at the end of its first start, and fail the benchmark when its seed missed A replica started together with its master reached it while the master stopped the server its bootstrap started and started it again, and its one dsreplication enable failed with ERROR_CONNECTING. replicate.sh now runs enable again when it exits 8, the only code it returns before changing anything, and initialize, initialize-all and the dsconfig on the master again after any failure, every 10 s for up to 5 minutes. A failed enable ends the script instead of running initialize. Both Docker test steps take the master off the network while the replica sleeps before its first try, and bring it back once the replica says it tries again. The benchmark checks that ou=People was seeded, and stops when it was not.

vharseko · 2026-09-25T12:00:35Z

Thanks. Round 3 is df6b632, rebased onto the current master (d30ff78, after #1094 and #1097; no conflicts, range-diff shows both earlier commits unchanged).

issue (blocking), a replica started together with its master: agreed for docker run and Compose. replicate.sh (the #1094 version, with the password file) now wraps its tools in a retry that tries again every 10 s for up to 5 minutes:

dsreplication enable is tried again only when it exits 8 (ERROR_CONNECTING). It connects to both servers before it changes anything, so that code means nothing was written. Any other failure is final. An enable that got further may have replicated the base DN already, and a second one then exits 5 (REPLICATION_CANNOT_BE_ENABLED_ON_BASEDN, ReplicationCliMain.java:3629-3634). So the snippet's retry on any failure would spend the 5 minutes and fail anyway.
initialize, initialize-all and the dsconfig set-replication-server-prop on the master are tried again after any failure. They can be repeated, and an initialize can also land in the gap after an enable that reached the bootstrap's server just before its stop.
A failed enable now ends the script with its exit code instead of going on to initialize.
The README's health check section says a replica tries a master it cannot connect to again for up to 5 minutes.

The openshift template is not affected: it passes OPENDJ_REPLICATION_TYPE only to the template description, not to the container's env, so run.sh:77 never runs replicate.sh there.

Pinned in both Docker test steps (the replication part from #1094). After docker run of the simple replica, the step waits for "Will sleep for a bit", takes the master off the network (docker network disconnect), waits for "dsreplication enable exited with 8, trying again", and connects the master again with --alias dj-master. The replica then has to become healthy and receive the change made on the master, as before.

Run locally with the branch's run.sh and replicate.sh laid over the #1094 image (only the timeouts raised, since a bootstrap takes 4.5 to 8.5 minutes here):

head: enable exited with 8 while the master was off the network and connected on the next try. The replica was healthy 714 s after its start, and the change reached it.
mutant, replicate.sh of master (no retry): its one enable and the initialize after it failed with "Connect Error", run.sh logged "Replication setup failed", the replica never became healthy within 15 minutes, and the change did not reach it. The new step fails there at its wait for the "trying again" line.

One gap remains, and it is older than this PR: setup.sh opens port 4444 before create-backend. A replica that reaches the master during that part of the master's bootstrap gets "no suffix to replicate" rather than a connect error, and that exit code is not tried again (read, not run). I left it out of this round.

suggestion, nothing fails when the seed misses: taken, with one change to the snippet. In compare-opendj.sh, bench_one checks ou=People,$BASEDN after the ldapadd. When the entry is missing, it saves docker logs to $out.docker.log, removes the container and returns 1, which stops the script through A_VER="$(bench_one ...)" under set -e. Without the first two steps the log would be lost. In benchmark.yml the same search fails the step with ::error::ou=People was not seeded and prints the container log.

nitpick, "is lost": taken, in both comments: "a request sent in between fails".

…alth check may probe it, and import online in the Docker test The server is PID 1 of the container from the first start on, so stop-ds in "Docker test" stopped the container and the offline import that followed had nothing to run in. The step now imports online and checks that docker stop stops the server cleanly, as OpenIdentityPlatform#1098 does. The server setup started is stopped before the bootstrap marker is written, so that the health check never reports it healthy just before it goes down.

maximthomas

praise: The round-3 commit fixes the round-2 blocker where it happens and pins the fix with a real disconnect.

retry in replicate.sh:57-70 retries dsreplication enable only on exit 8 and the repeatable tools on any failure. || exit at :89, :117 and :148 stops a failed enable from going on to initialize.
build.yml:604-608 takes the master off the network around the replica's first try and requires "exited with 8, trying again". At the base that wait times out.
Both benchmark seeds now fail on a missing ou=People: compare-opendj.sh through return 1 under set -e, and benchmark.yml through || { ...; false; }.

issue (non-blocking): With srs, a DS replica that restarts right after its own replicate.sh can miss a later replica's initialize-all. It then stays in BAD_GEN_ID while it reports healthy.

opendj-packages/opendj-docker/bootstrap/replicate.sh:121, opendj-packages/opendj-docker/run.sh:94

Each srs DS pushes its own data with initialize-all --hostname $MYHOSTNAME. The push goes to the DSs connected at that moment (ReplicationDomain.java:1716) and fails only when there are none (:1623-1628). Take one RS and three DS started together. DS1 pushes, then stops at run.sh:94 for the new stop/start. DS3's push lands in that window and succeeds against DS2 alone, so retry any never fires. The RS then takes DS3's generation ID. DS1 comes back with its own data and generation ID and goes to BAD_GEN_ID (DataServerHandler.java:134-138). Its replicate.sh does not run again and the marker is already written. The IDs differ whenever the entries were added over LDAP (setup.sh:92-105, ldapmodify of bootstrap/data, random entryUUID). At the base, tail -f kept DS1's server up and it received the push. Not run: this needs one RS and three DS containers, and CI has no srs job.

With `OPENDJ_REPLICATION_TYPE=srs`, start the directory server replicas one at a time, each once the
previous one is healthy: every replica pushes its data to the replicas connected at that moment,
and one that is restarting at the end of its own first start misses it.

Or: before writing the marker, check that the domain is not in BAD_GEN_ID.

issue (non-blocking): dsreplication enable can exit 8 after it has written configuration. The comment at replicate.sh:52-54 and the PR description say it cannot.

opendj-packages/opendj-docker/bootstrap/replicate.sh:52-54, opendj-server-legacy/src/main/java/org/opends/server/tools/dsreplication/ReplicationCliMain.java:6486-6509

Exit 8 does come before any write for the two admin connections (:3605-3611). updateConfiguration then writes the ADS, configureServer and the first domain, and reconnects to every server in the topology cache (:6494). An LdapException there is thrown as ERROR_CONNECTING (:6505-6509). Suppose a third server is up when the cache loads and gone by the loop, for example an earlier replica inside its own run.sh:94 stop/start. Then enable exits 8 after those writes, and retry 8 runs a partly applied enable again. Whether that second enable completes or exits 5 is not traced.

# dsreplication enable exits 8 (ERROR_CONNECTING) when it cannot connect or bind to one of the two
# servers, which it checks before it changes anything, and also when another server of the topology
# cannot be reached after it has written to the first two. Only that exit code is tried again: an
# enable that failed otherwise may have replicated the base DN already, and a second one then fails.

suggestion (non-blocking): Nothing pins that only exit 8 is retried, or that a failed enable ends replicate.sh.

opendj-packages/opendj-docker/bootstrap/replicate.sh:80, :89, .github/workflows/build.yml:604-608, :840-844

The only CI road makes enable fail with 8 and then succeed. Change retry 8 to retry any at :80 and the job still prints the same line and stays green. Remove || exit at :89 and the job never reaches it. The pin below runs a second enable of the already replicated base DN once test_replica is healthy. With retry any, the output gets "trying again". Without || exit, it gets "initializing replication". The pin relies on a second enable exiting non-zero (5 per ReplicationCliMain.java:3629-3634). Under the step's set -e a failing ! grep would not stop the step, hence the ifs.

out=$(docker exec -e BASE_DN=dc=example,dc=com -e ROOT_USER_DN="cn=Directory Manager" test_replica \
  timeout 90 /opt/opendj/bootstrap/replicate.sh 2>&1) && { echo "a second enable succeeded"; exit 1; }
if grep -q "trying again" <<<"$out"; then echo "a non-8 enable failure was retried"; exit 1; fi
if grep -q "initializing replication" <<<"$out"; then echo "initialize ran after a failed enable"; exit 1; fi

suggestion (non-blocking): The sdsr enable retry and the retry any on initialize have no case that can fail.

.github/workflows/build.yml:609-610, opendj-packages/opendj-docker/bootstrap/replicate.sh:94, :133, :152

test_replica_sdsr starts only once the master and test_replica are healthy, so its enable and its initialize succeed on the first try. Dropping retry 8 at :133, or retry any at :94/:152, leaves both docker jobs green. The same disconnect as for test_replica pins the sdsr enable. An initialize case would need the disconnect to land between "initializing replication" and the tool's connect, which is racy.

timeout 5m bash -c 'until docker logs test_replica_sdsr 2>&1 | grep -q "Will sleep for a bit"; do sleep 0.2; done'
docker network disconnect test_replication test_master
timeout 2m bash -c 'until docker logs test_replica_sdsr 2>&1 | grep -q "dsreplication enable exited with 8, trying again"; do sleep 1; done'
docker network connect --alias dj-master test_replication test_master

suggestion (non-blocking): A failed bind also exits 8, so a replica with the wrong ROOT_PASSWORD is retried for 5 minutes.

opendj-packages/opendj-docker/bootstrap/replicate.sh:63, opendj-server-legacy/src/main/java/org/opends/server/tools/dsreplication/ReplicationCliMain.java:3752-3768

createAdministrativeConnection collects every LdapException from new ConnectionWrapper(...), invalid credentials included, and enableReplication returns ERROR_CONNECTING for all of them (:3605-3611). A misconfigured replica ends up unhealthy, which is right. It just gets there after about 5 minutes of "dsreplication enable exited with 8, trying again in 10 s". Saying so next to the retry is enough:

# A wrong root DN or password also exits 8, so it fails only once the 5 minutes are over.

question (non-blocking): Is an enable that is still running when the master stops meant to be covered by the README's "so that window does not fail its replication setup"?

opendj-packages/opendj-docker/README.md:35-37

Suppose the master's stop-ds lands while the replica's enable is connected and writing. The write fails as ERROR_UPDATING_ADS (14), or as 11 or 17 in the other phases. That code is not retried, || exit ends replicate.sh, and the replica never becomes healthy. The comment at replicate.sh:52-54 states this choice, so as README wording this is minor. If the sentence is meant as a guarantee, it is an uncovered road of the round-2 blocker. Not run: it needs a master stopped mid-enable.

A replica set up with `MASTER_SERVER` tries a master it cannot connect to again, every 10 s for up
to 5 minutes, so a master that is down when the replica reaches it does not fail its replication
setup; one that stops while `dsreplication enable` is writing to it still does.

nitpick (non-blocking): After the 30th failure, retry still logs "trying again in 10 s" and sleeps before it returns.

opendj-packages/opendj-docker/bootstrap/replicate.sh:60-69

retry() {
  local on=$1 rc i
  shift
  for i in $(seq 1 30); do
    "$@" && return 0
    rc=$?
    if [ "$on" != any ] && [ "$rc" -ne "$on" ] || [ "$i" -eq 30 ]; then
      return $rc
    fi
    echo "$(basename "$1") $2 exited with $rc, trying again in 10 s"
    sleep 10
  done
}

… replica tries again, and document what the retry does not cover replicate.sh - The comment on retry no longer says that exit 8 means nothing was written: dsreplication enable also exits 8 when another server of the topology cannot be reached after it has written to the first two. It also says that a wrong root DN or password exits 8, so it fails only once the 5 minutes are over. - retry returns after its 30th failure instead of saying it will try again and sleeping another 10 s. README.md - A master that stops while dsreplication enable is writing to it still fails the replica's setup; the retry covers only a master that cannot be reached. - With OPENDJ_REPLICATION_TYPE=srs the directory server replicas are to be started one at a time: each pushes its data to the replicas connected at that moment, and one restarting at the end of its own first start misses it and stays in a bad generation ID. build.yml (Docker test replication, both docker jobs) - The sdsr replica's enable is pinned the same way as the simple one's: the master is taken off the network until the replica logs that it tries again. - replicate.sh is run once more on the healthy simple replica. Its enable of a base DN already replicated has to end it with exit 5, with no "trying again" and no "initializing replication".

vharseko · 2026-09-25T14:29:42Z

Round 4 is 76bdaf0, on the same master (d30ff78). All seven points are taken.

issue, srs replica restarting during a later replica's initialize-all: agreed. The generation ID is a checksum of objectclass, sn, cn and entryuuid, and EntryUUIDPlugin draws a random UUID for an LDAP add (a name-based one only on import), so replicas loaded from bootstrap/data do get different IDs. A BAD_GEN_ID check cannot go before the marker: the marker is written before start-ds, and the server that would be in BAD_GEN_ID is not up yet. So I took the README paragraph: with srs, start the directory server replicas one at a time, each once the previous one is healthy. It is also in the PR description, marked read, not run.

issue, exit 8 after writes: agreed. configureToReplicateBaseDN configures the first server's domain, then connects to every server of the topology cache and turns an LdapException there into ERROR_CONNECTING (ReplicationCliMain.java:6505-6509). A server the cache could not read at load time is not in that loop, so it takes one that goes away between the two, as you describe. The comment now uses your wording, and so does the PR description, which had the same wrong claim. I kept retry 8: whatever the second enable does, it is no worse than no retry, since a failure other than 8 is final either way.

suggestion, pin "only 8" and || exit: taken, in both docker jobs, as one check instead of the snippet's three: replicate.sh run once more on test_replica must exit 5 and log neither "trying again" nor "initializing replication". I checked the 5 first: for a base DN that both servers already replicate, checkSuffixesForEnableReplication finds it FULLY_REPLICATED and empties the list, and enableReplication then returns REPLICATION_CANNOT_BE_ENABLED_ON_BASEDN before anything is written. Run locally on a healthy topology with the mutants copied into the replica: head exits 5; retry any at the simple enable logs "dsreplication enable exited with 5, trying again" and ends in timeout 90 (124); without || exit it exits 0 after "initializing replication". The alpine timeout exits 143, not 124, when it expires. The check only asks for 5, so this makes no difference.

suggestion, pin the sdsr retry: taken, with your snippet, in both jobs. Taking the master off the network also cuts test_replica's replication, so I ran the whole replication part locally to see whether the later "change reaches the replicas" check (1 minute) still passes. It does: the step passed in 6 minutes. An sdsr replica without retry on its enable never logs "trying again" in the 2 minutes, and logs "Replication setup failed".

I left the initialize retry without a pin, as you suggest: a pin there would be racy.

suggestion, wrong password exits 8: taken, the comment line next to retry.

question, the README sentence: no, it is not meant as a guarantee. It now uses your wording: a master that is down when the replica reaches it does not fail the setup; one that stops while dsreplication enable is writing to it still does.

nitpick, the 30th failure: taken, with your retry: after the 30th failure it returns at once.

vharseko added bug docker labels Sep 24, 2026

vharseko requested a review from maximthomas September 24, 2026 14:06

vharseko mentioned this pull request Sep 25, 2026

[#1087] Copy the secret volume on every start of the Docker image, and run the server as PID 1 after the bootstrap #1100

Open

maximthomas requested changes Sep 25, 2026

View reviewed changes

vharseko force-pushed the issue-1085-docker-sigterm branch from f6cb09d to 1803490 Compare September 25, 2026 08:27

vharseko added CI benchmark docs labels Sep 25, 2026

vharseko requested a review from maximthomas September 25, 2026 08:28

maximthomas requested changes Sep 25, 2026

View reviewed changes

vharseko added 3 commits September 25, 2026 12:55

vharseko force-pushed the issue-1085-docker-sigterm branch from 1803490 to df6b632 Compare September 25, 2026 12:00

vharseko added the replication label Sep 25, 2026

vharseko requested a review from maximthomas September 25, 2026 12:01

maximthomas approved these changes Sep 25, 2026

View reviewed changes

vharseko requested a review from maximthomas September 25, 2026 14:29

Navigation Menu

Uh oh!

[#1085] Stop the server cleanly on SIGTERM in a container that bootstrapped its instance - #1098

[#1085] Stop the server cleanly on SIGTERM in a container that bootstrapped its instance#1098
vharseko wants to merge 4 commits into
OpenIdentityPlatform:masterfrom
vharseko:issue-1085-docker-sigterm

vharseko commented Sep 24, 2026 •

edited

Loading

Uh oh!

maximthomas left a comment

Uh oh!

vharseko commented Sep 25, 2026

Uh oh!

maximthomas left a comment

Uh oh!

vharseko commented Sep 25, 2026

Uh oh!

maximthomas left a comment

Uh oh!

vharseko commented Sep 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

vharseko commented Sep 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Change

Testing

Out of scope

Uh oh!

maximthomas left a comment

Choose a reason for hiding this comment

Uh oh!

vharseko commented Sep 25, 2026

Uh oh!

maximthomas left a comment

Choose a reason for hiding this comment

Uh oh!

vharseko commented Sep 25, 2026

Uh oh!

maximthomas left a comment

Choose a reason for hiding this comment

Uh oh!

vharseko commented Sep 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

vharseko commented Sep 24, 2026 •

edited

Loading