Replication: no test writes to several replicas under load and then checks that they hold the same data

[vharseko]

Problem

Nothing in CI writes to several replicas at once under load, and nothing checks afterwards that the replicas hold the same data.

What exists:

• The Test replication step of build.yml chains three combined DS/RS servers with dsreplication enable and initialize. It then counts the uid=user.* entries on the new replica. There are no writes after the initialization, no load, no standalone replication servers, and no comparison beyond an entry count.
• benchmark.yml measures a single server against OpenLDAP. Replication is not involved.
• The unit and integration tests exercise replication paths one at a time, in-process, and mostly with a handful of changes.

That leaves the failure users report untested. Two examples:

• Missing Changes count is piling up during peak load testing with OpenDJ 4.9.4 #534 / discussion Missing Changes count is piling up during peak load testing with OpenDJ 4.9.4 #538: 4 load-balanced directory servers and 2 standalone replication servers under 500 connections / 1300 TPS piled up about 2.5 million missing changes in 2 hours. It was closed without a cause.
• Gluu's April 2024 write-up describes changes "that would not sync in" and had to be forced by hand (see Add an official Helm chart so OpenDJ can be deployed on any Kubernetes cluster #1079). Their fork is a snapshot of this repository from September 2020.

Several classes of defects fixed since would each have shown up in such a run as replicas that disagree:

• a change recorded as applied which is not in the data: Replication replay records a failed operation as applied: the ServerState advances past the change and the assured ack reports success #889, A change which never applied is recorded as replayed when server-error-result-code is set to NO_OPERATION or SUCCESS #953, A failed entryUUID search reads as a deleted entry, and conflict resolution records the change as replayed #956, Disabling a replication domain can drop a change a replay thread is applying #908, A replica which reconnects after a restore can be declared up to date and never sent the change it was restored past (ReSyncTest.testResyncAfterRestore) #963;
• a change left owned by a replay thread which is gone, never applied until a restart: Replication: a change whose replay throws is left owned by a thread which is gone #922, Replication: an Error in a replay kills a replay thread the pool never replaces #923, Replication: a change parked as a dependency is left owned by a thread which never comes back to it #954, Replication: a change parked as a dependency is left owned by a replay thread the pool stopped #986;
• a replica's position lost or not written: A ServerState update landing during a save is marked saved and never written to disk #916, Disabling a domain can persist an empty ds-sync-state and lose the replica's position #951, An exception from a state write kills the checkpointer and hangs server shutdown #952;
• data applied but not indexed, so searches on one replica miss entries: Replicated objectClass value additions are not written to the objectClass equality index #1021, and probably DS Replication Syncing Entries Successfully But Not Returning Search Results With Specific Search Options On A Given Server #363;
• catch-up stopping after the first chunk: Replication not catching up properly if there is many pending changes #141 (2020).

All of them were found by reading code or by in-process tests failing intermittently, not by a test that asserts convergence.

missing-changes reaching 0 is not enough of a check. The first class above advances the ServerState past a change which never reached the data. dsreplication status then shows a replica which is fully caught up and wrong.

Proposal

A separate workflow, not a step of every PR build, since it runs for tens of minutes. Triggers: workflow_dispatch, a nightly schedule, and before a release. It builds the server once and runs a topology on one runner:

1. Topology like Missing Changes count is piling up during peak load testing with OpenDJ 4.9.4 #534. Four directory servers without replication servers (--noReplicationServer) and two standalone replication servers (--onlyReplicationServer), all on ports below 32768 (CI: the third replication server is given ports inside the ephemeral range, so its setup fails at random #1031). A second variant uses combined DS/RS servers, the default most users run.
2. Load on every directory server at once, for a fixed duration. Use modrate, addrate (with its delete phase) and a modify of the same entries from several servers, so that conflicts and same-DN ordering (Replication of modify operations on same DN may be replayed out of order #139) are exercised. Use searchrate alongside to keep the work queues busy.
3. Disturbances during the load, one per variant:
   • restart a directory server;
   • kill -9 a replication server and start it again;
   • stop a directory server for longer than one catch-up chunk (Replication not catching up properly if there is many pending changes #141);
   • dsreplication disable one server, then enable and re-initialize it, which is the scale-in/scale-out a Kubernetes chart will do (Add an official Helm chart so OpenDJ can be deployed on any Kubernetes cluster #1079, Docker image: joining replication is one-shot, fixed-master and unchecked, which a StatefulSet cannot rely on #1086).
4. Quiesce and wait, with a bound, until missing-changes is 0 for every server in cn=Replication,cn=monitor. Record how long that took and the peak missing-changes / approximate-delay during the load, and publish them in the job summary to spot regressions.
5. Assert convergence on the data, not on the counters:
   • export-ldif every replica, drop the attributes which legitimately differ (ds-sync-hist, ds-sync-state and the like), sort, and diff. Any difference fails the run, and the diff is uploaded as an artifact.
   • The same indexed searches on every replica ((objectClass=…), equality on modified attributes) return the same DNs. This catches Replicated objectClass value additions are not written to the objectClass equality index #1021-class index drift, which an export does not show.
   • No ds-sync-conflict entries other than those the load deliberately provokes.
   • The error logs of all servers hold no SYNC error, and no replay give-up (Make the replay retry budget of a replication domain configurable instead of a constant with a test-only setter #901, Replication: the replay retry warning is logged once per delivery, with nothing bounding it once the give-up budget is raised #942).
6. On failure, upload every instance's logs/, config/ and changelogDb state. That is what Missing Changes count is piling up during peak load testing with OpenDJ 4.9.4 #534 could not get from its reporter.

Start with the plain variant (1, 2, 4, 5), then add the disturbances one at a time. Each disturbance that reproduces a known fixed defect on a build without its fix is a useful check that the test actually bites.

Notes

• The JE backend first. The same topology with the JDBC backend (OutOfMemory error initializing replication using jdbc postgres backend #708) is a natural second matrix axis.
• The load tools and the setup commands are the ones the Windows smoke test and the replication step already use, so no new tooling is needed.

Replication defects fixed so far

For reference, every closed replication defect (68 issues), grouped by what a convergence run would have seen. All fixes are merged. The "Released" column says whether a release carries the fix: most of them are on master only, after 5.1.2 (2026-07-17). The 2020 fixes (#139, #141, #143) are also in Gluu's fork, and none of the later ones are.

A change recorded as applied, or never sent, while the data does not hold it (silent divergence) (8)

Issue	Fixed by	Closed	Released
#139 Replication of modify operations on same DN may be replayed out of order	#142	2020-09-09	yes (2020)
#889 Replication replay records a failed operation as applied: the ServerState advances past the change and the assured ack reports success	#892	2026-09-07	master
#953 A change which never applied is recorded as replayed when server-error-result-code is set to NO_OPERATION or SUCCESS	#980	2026-09-16	master
#956 A failed entryUUID search reads as a deleted entry, and conflict resolution records the change as replayed	#968	2026-09-15	master
#908 Disabling a replication domain can drop a change a replay thread is applying	#945	2026-09-10	master
#963 A replica which reconnects after a restore can be declared up to date and never sent the change it was restored past (ReSyncTest.testResyncAfterRestore)	#1015	2026-09-21	master
#927 Replication: AddMsg.addAttribute() swallows every exception, dropping the ds-sync-conflict marker	#975	2026-09-11	master
#1051 A Relax Rules change on a replicated suffix is not replicated and logs an internal error	#1053	2026-09-24	master

A change stuck and never applied until a restart (5)

Issue	Fixed by	Closed	Released
#141 Replication not catching up properly if there is many pending changes	029beee	2020-09-09	yes (2020)
#922 Replication: a change whose replay throws is left owned by a thread which is gone	#958	2026-09-12	master
#923 Replication: an Error in a replay kills a replay thread the pool never replaces	#958	2026-09-12	master
#954 Replication: a change parked as a dependency is left owned by a thread which never comes back to it	#988	2026-09-18	master
#986 Replication: a change parked as a dependency is left owned by a replay thread the pool stopped	#988	2026-09-18	master

The replica's position or domain state lost, stale or not written (6)

Issue	Fixed by	Closed	Released
#916 A ServerState update landing during a save is marked saved and never written to disk	#948	2026-09-10	master
#951 Disabling a domain can persist an empty ds-sync-state and lose the replica's position	#970	2026-09-11	master
#952 An exception from a state write kills the checkpointer and hangs server shutdown	#977	2026-09-15	master
#924 Replication: enable() clears "disabled" after starting the session, where disable() sets it before stopping one	#971	2026-09-09	master
#966 A failed backend disable leaves a replication domain disabled for the life of the server	#969	2026-09-11	master
#943 Replication: applyConfigurationChange() publishes a domain configuration it may then report as failed	#959	2026-09-11	master

Data applied but not indexed (searches miss entries on some replicas) (1)

Issue	Fixed by	Closed	Released
#1021 Replicated objectClass value additions are not written to the objectClass equality index	#1022	2026-09-23	master

Shutdown and scale-in: the offline announcement and changes still queued (8)

Issue	Fixed by	Closed	Released
#900 The ReplicaOfflineMsg grace period is spent on the first message of the process: DSRSShutdownSync latches its timestamp and never resets it	#919	2026-09-07	master
#917 The ReplicaOfflineMsg grace period ends at the first forward, so a second replication server can be left unaware of the offline replica	#947	2026-09-10	master
#918 A ReplicaOfflineMsg still queued behind an uncommitted change is recorded as sent, and the shutdown then waits its whole grace period for nothing	#946	2026-09-09	master
#949 A ReplicaOfflineMsg the broker refuses is still recorded as sent: ReplicationDomain.publish() discards the outcome	#976	2026-09-11	master
#950 A ReplicaOfflineMsg forwarded before it is recorded leaves a pending announcement nothing will clear, and the shutdown waits out its grace period	#978	2026-09-15	master
#983 A peer replication server whose handshake is in flight is killed by the shutdown's own interrupt, and never told the replica went offline	#987	2026-09-16	master
#1014 A ReplicaOfflineMsg which cannot be encoded for a peer is still recorded as forwarded: Session.publish() drops it below protocol V8	#1019	2026-09-23	master
#1029 A ReplicaOfflineMsg sent to a directory server on the catch-up path takes a send-window permit the session never gets back	#1034	2026-09-22	master

Total update, initialization and session restarts (12)

Issue	Fixed by	Closed	Released
#730 initializeRemote() leaks import/export context on failed validation: domain permanently rejects total updates (InitOnLineTest CI cascade)	#731	2026-07-11	5.1.2
#735 Replication: ServerHandler.abortStart rolls back a concurrently-adopted generation ID on cross-connect abort (GenerationIdTest.testMultiRS flake)	#736	2026-07-16	5.1.2
#861 dsreplication enable hangs initializing registration information when adding a third replica; CI Test replication step has no timeout	#864	2026-08-13	master
#868 Flaky total update termination: the import/export context is released after the initialize task is notified	#869	2026-08-19	master
#925 Replication: a session restart request is consumed before the restart runs	#981	2026-09-16	master
#926 Replication: ExternalChangelogDomain restarts the session outside serviceStateLock and the session generation	#974	2026-09-14	master
#967 An import which disables a replication domain can deadlock against a write to that domain's configuration entry	#972	2026-09-11	master
#995 dsreplication initialize fails hard when a total-update request gets no answer: the peers-not-found retry can never fire	#996	2026-09-16	master
#1039 Replication: a total update whose session stops before the DoneMsg ends as a finished import, with the exporter's generationId over partial data	#1044	2026-09-22	master
#1040 Replication: the session restart a configuration change asks for cuts a total update into this replica	#1042	2026-09-22	master
#1041 Replication: the owner read of a session restart and the listener's claim of a remote-initiated import share no lock	#1045	2026-09-23	master
#1048 Replication: a session restart run by the thread which released a change cuts the export the session carries	#1049	2026-09-24	master

Connections, handshakes, TLS and diagnostics (8)

Issue	Fixed by	Closed	Released
#143 Connection issues towards one RS may prevent others from working properly	#144	2020-09-09	yes (2020)
#728 Replication port stolen by TCP self-connect: RS silently starts without listener (testSafeDataLevelOne CI flake)	#729	2026-07-11	5.1.2
#821 Aborted DS handshake leaves a dead DataServerHandler registered in ReplicationServerDomain	#838	2026-08-04	master
#905 Need to setup the CA certificate instead of self signed certificate for Replication Port 8989	#906	2026-09-03	master
#911 Replication drops connections and gives up on peers without logging a cause	#935	2026-09-11	master
#912 setup cannot provision the ads-truststore from an existing PKI	#984	2026-09-16	master
#913 ssl-cert-nickname declares component-restart but the crypto manager reads it once at startup	#979	2026-09-15	master
#1017 Replication: a peer registered under another address is dialled on every connect pass, and each attempt logs ERR_DUPLICATE_REPLICATION_SERVER_ID unthrottled	#1020	2026-09-23	master

Changelog database (5)

Issue	Fixed by	Closed	Released
#802 FileChangelogDB.initializeDB() swallows ChangelogException: a replication server with an unreadable changelog starts anyway	#805	2026-08-04	master
#813 FileChangelogDB.getOrCreateReplicaDB() races shutdownDB(): a replica DB created during shutdown is never released	#820	2026-08-05	master
#816 FileChangelogDB.removeDomain() throws NullPointerException when it races shutdownDB()	#827	2026-08-04	master
#818 An empty domain map is left behind by a replica DB creation which bails out	#830	2026-08-05	master
#819 FileReplicaDB leaks its log reference when its constructor cannot read the CSN limits	#833	2026-08-04	master

Other replay and conflict handling (5)

Issue	Fixed by	Closed	Released
#710 Replication server re-sends updates from changelog catch-up with the original assured flag	#714	2026-07-11	5.1.2
#901 Make the replay retry budget of a replication domain configurable instead of a constant with a test-only setter	#944	2026-09-10	master
#928 Replication: the permissive-modify check dereferences a null entry DN when the DN does not parse	#973	2026-09-11	master
#942 Replication: the replay retry warning is logged once per delivery, with nothing bounding it once the give-up budget is raised	#982	2026-09-17	master
#955 NPE in solveNamingConflict(ModifyDNOperation) when both the moved entry and its new parent are gone	#965	2026-09-10	master

Tests only (flakes and missing pins, no product change of their own) (10)

Issue	Fixed by	Closed	Released
#744 Flaky test: ChangelogBackendTestCase.searchInChangeNumberModeOnOneSuffixMultipleTimes fails when both change batches get identical CSNs	#745	2026-07-17	5.1.2
#792 Flaky test: AssuredReplicationServerTest#testSafeDataFromRS — RS silently starts without a listener when the replication port is briefly occupied	#795	2026-07-31	master
#800 Replication tests pin their listen ports for the whole class and rebind them hundreds of times	#803	2026-08-03	master
#815 ReplicationServerLoadBalancingTest asserts the RS side DS count before the RS registers the DS	#822	2026-08-04	master
#841 Flaky InitOnLineTest: initializeExportMultiSS times out, leaked replication server port fails initializeImport	#845	2026-08-05	master
#909 No test for the change a stopped replay thread hands back to the replication server	#941	2026-09-09	master
#910 No test for a ModifyDN conflict solved while server-error-result-code is one of the conflict codes	#938	2026-09-08	master
#939 Five of the six codes in CONFLICT_RESULT_CODES are never exercised: pin isServerFailure() directly	#960	2026-09-09	master
#1036 The #958 hand-out tests assert which thread replays a parked change, which getNextUpdate() does not guarantee	#1037	2026-09-19	master
#1038 Replication: the session lock and generation of #974 are pinned by no test, and neither is the restart a configuration change does run	#1043	2026-09-22	master