Docker image: joining replication is one-shot, fixed-master and unchecked, which a StatefulSet cannot rely on

[vharseko]

Problem

The container joins a replication topology only once, only through a single MASTER_SERVER, and without checking that the master is ready (replicate.sh, run.sh#L72-L78). That is enough for a hand-started pair, but not for a StatefulSet, which is where an orchestrator (discussion #1079) would use it:

1. No readiness check. The script waits a fixed sleep 5 ("This is hacky....", L26-L30) and then runs dsreplication enable once. If the master is still bootstrapping, or merely listening before its BASE_DN backend exists, the join fails and is never retried. Since Report the OpenDJ container healthy only once its bootstrap has succeeded #898 the container is then unhealthy, but only until it restarts: a restart on the same volume takes the ./data/config branch, which writes the health marker right after upgrade -n (run.sh#L44-L54). The failed join thus turns into a healthy, unreplicated server. In Kubernetes this happens by itself: a liveness probe on the marker fails, the kubelet restarts the container, and the failure disappears from view.
2. Only on the first bootstrap. replicate.sh runs only when ./data/config does not exist. A restart never checks or repairs membership.
3. The master cannot rejoin. A server decides it is the master when MASTER_SERVER appears in its own /etc/hosts (L16-L20). In a StatefulSet every pod points at <name>-0. If <name>-0 loses its volume, it bootstraps an empty instance, sees itself in /etc/hosts and skips replication. The others still hold it in their topology, but it is now an unreplicated empty server behind the same Service. (The grep is also an unanchored regex: opendj-1 matches opendj-10, and . matches any character.)
4. Leaving is not handled. Nothing runs dsreplication disable when a server goes away (scale down). The rest of the topology keeps it in cn=admin data and in their replication-server lists.

Expected

The join becomes a background step next to the server, not part of the bootstrap:

• The server is PID 1 on every start (exec start-ds --nodetach, see Docker image: a freshly bootstrapped container ignores SIGTERM and is killed without stopping the server #1085). The join runs in the background next to it, on every start, not only on the first one.
• Peers come from a list, not a single master. For example REPLICATION_PEERS=host1,host2,.... A chart can derive it from the StatefulSet ordinals (<sts>-0 … <sts>-N-1.<headless svc>), so it needs no registry. MASTER_SERVER stays as a one-element list for compatibility. The server recognises itself by exact comparison with hostname -f, not by grep /etc/hosts.
• Membership is checked, not inferred from the exit code. Exit code 5 of dsreplication enable is not "already enabled": it is REPLICATION_CANNOT_BE_ENABLED_ON_BASEDN (ReplicationCliReturnCode.java#L46), returned whenever no suffix is left to enable (ReplicationCliMain.java#L3628-L3635). That covers both an already replicated suffix and a BASE_DN not found on one of the servers (L4225-L4236), which is exactly the race of item 1. So the step decides from the result instead: the local replication domain for BASE_DN exists under cn=domains,cn=Multimaster Synchronization,cn=Synchronization Providers,cn=config, and the server is registered in cn=admin data.
• Initialization follows what the volume went through, not whether BASE_DN has entries. Every fresh volume has entries: setup.sh imports the base entry (ADD_BASE_ENTRY) or SAMPLE_DATA. By the same token, "the first peer that answers a search of BASE_DN" says nothing about which peer holds the data. The generation ID is computed from the entry count and the objectclass/sn/cn/entryuuid of the first 1000 entries (LDAPReplicationDomain.java#L5802-L5837). If the seed holds data, the IDs differ, and the joined replica sits in "bad generation ID" without replicating, yet healthy. If both hold only the imported base entry, the IDs match, initialize is skipped, and whatever reaches the seed later by import-ldif (which produces no changes to replay) never reaches the replica. Instead, dsreplication initialize runs when this volume was bootstrapped and has not been initialized from the topology yet (a marker in ./data written after a successful initialize), and the step cross-checks the local generation ID against the peer's.
• Readiness of a populated server does not wait for its peers. A server whose replication domain is already configured is ready as soon as it serves. Only an empty volume waits for the join. Otherwise a restart of the whole cluster deadlocks under OrderedReady: -0 waits for its peers, and the StatefulSet does not start -1 until -0 is ready.
• The seed is a rule, not "when the server is alone". On the first deployment -0 finds no peers because none exist yet; after all pods went down at once, a -0 with an empty volume finds none either, which is item 3 again. Proposal: only the first entry of REPLICATION_PEERS may seed, and only after the retries are exhausted. The residual risk (every pod down, and -0 lost its volume) is documented.
• Retries are bounded and configurable (count and interval) instead of sleep 5, and every attempt has its own timeout: dsreplication enable can hang indefinitely (build.yml#L302-L304). The health marker is written only once the join succeeded, or by the seed.
• Leaving is handled by the survivors, not by preStop. Kubernetes runs preStop on every termination — rolling update, node drain, kubectl delete pod — and it cannot tell a scale-down from a restart. dsreplication disable --disableAll there would take the server out of the topology on every image update, and pods terminating together (drain, Parallel) would write cn=admin data concurrently. disable also removes references only on the peers it can reach (ReplicationCliMain.java#L5449-L5486), and OpenDJ 4 has no subcommand to remove a dead server. So a survivor removes every server that is registered but no longer in REPLICATION_PEERS: its replication-server values on each remaining server (dsconfig) and its entries in cn=admin data. It runs as a step of the join or as a Job of the chart.
• No hardcoded ports: replicate.sh uses 4444 and 8989 instead of $ADMIN_PORT and a replication-port variable.
• No password on a command line (--adminPasswordFile, --bindPasswordFile1/2), see Docker image: replicate.sh prints its environment, ROOT_PASSWORD included, to the container log #1084.
• srs, sdsr and rg are either kept as the one-shot path they are today or deprecated; the design above covers simple.
• CI covers it. The "Docker test" steps run a single container only. The change adds two containers on one network, a restart of both, and the case "the seed lost its volume".

Prior art

Gluu ran a design along these lines for years on its OpenDJ 4 fork, with a shared peer registry instead of StatefulSet ordinals (gluufederation/opendj, chart in GluuFederation/cloud-native-edition). The image's source repository is no longer public; the scripts are in /app/scripts of gluufederation/opendj:4.5.5-1.

• entrypoint.sh does exec start-ds -N and starts ldap_replicator.py in the background.
• Each pod registers itself in a shared peer list (register_peer.py).
• The replicator joins through any registered peer that already holds data, and accepts exit code 5 of dsreplication enable (which, as above, also hides a missing BASE_DN). It runs initialize only when a sentinel entry is missing locally, and retries 10 × 10 s by default.
• The chart's preStop runs deregister_peer.py, which calls dsreplication disable --disableAll. cleanup_repl_config.py removes a dead server from cn=admin data and from the replication-server lists.

Their April 2024 write-up is the other half of the lesson. They withdrew multi-cluster (multi-region) replication in Kubernetes:

• changes that never synced and had to be forced by hand;
• replication lag growing under autoscaling at 200–300 req/s, so pods could not shut down with unsent changes;
• recoveries that needed a whole region stopped.

They keep OpenDJ for a single multi-AZ cluster. So the scope here is one cluster, and no autoscaling.

Notes

The pod/peer naming belongs to the chart; the retry loop, peer list, idempotence and removal of servers no longer listed belong to the image, so plain docker run users get them too. The change builds on #1085 (server as PID 1 on every start).