[#1080] Give back the send-window permit of an update the writer drops, and count only what it sends

[vharseko]

Fixes #1080

The accounting

ServerHandler.take() charges a permit of the send window for every update it hands to the writer
(:994), before ServerWriter.isUpdateMsgFiltered() decides whether the update is sent at all.
The only release is the WindowMsg the peer sends for what it received (updateWindow()), and a
dropped update never reaches the peer, so every drop cost the session a permit for good. The peer
gives its credit back in halves of its window, so once more than half of the permits are gone it
can no longer be sent enough to give any back: the writer sits in acquirePermitInSendWindow()
until the session is re-established, and nothing is logged. The replication server never probes a
peer for credit.

Where the drops come from

ReplicationServerDomain.put() applies the same status and generation id checks before it queues
an update, so the drops come from the changelog: the catch-up of a peer which connects behind reads
its whole backlog there, and the writer drops every record of it.

• A directory server in BAD_GEN_ID_STATUS or under a total update leaves that status on a new
  session - a new handler, a new semaphore - which is why [#1029] Send a directory server only the updates it gives send-window credit for #1034 left those arms alone.
• A peer replication server does not: ReplicationServerDomain.resetGenerationId() only calls
  rsHandler.setGenerationId(), and a TopologyMsg updates the generation id of the handler in
  place (ReplicationServerHandler.java:614). A peer which connected with another generation id
  and a backlog of more than half a window is left, once the two agree, with a session over which
  nothing can be sent.
• The protocol version guard of A ReplicaOfflineMsg which cannot be encoded for a peer is still recorded as forwarded: Session.publish() drops it below protocol V8 #1014 drops one ReplicaOfflineMsg per replica which goes offline,
  over a session to an older peer which is not re-established for it either.

The change

• ServerWriter.run() gives the permit back - ServerHandler.releasePermitInSendWindow() - in the
  one branch every filter goes through, rather than at each return true.
• take() counted the update as sent as well - sent-updates and the assured counters of the
  monitor entry of the handler - before the writer dropped it. The count moves to
  ServerHandler.countSentUpdate(), which the writer calls only for what it publishes.

Taking the permit only after the filter was the other option in the issue. It is not taken: the
filter reads the status of the peer when the update is written, and take() decides the assured
flag of an update re-read from the changelog after the wait for the permit on purpose (the comment
in take() says why), so both would have had to move with it.

A writer which is shut down leaves its wait without a permit, so a release on that path may give
back one it never took. The window of a session which is going away is not used again; the javadoc
of releasePermitInSendWindow() says so.

Tests

FilteredUpdateSendWindowTest (new):

• aPeerWhichAgreesOnTheGenerationIdAfterItsBacklogWasDroppedIsSentTheNextChange - a real peer
  replication server takes another generation id before it connects
  (ReplicationServerDomain.changeGenerationId()), so its catch-up of a whole window of backlog is
  dropped; it then takes the generation id of the writer over the same session (a TopologyMsg), and
  the next change must reach it, with sent-updates at 1 and the window at W - 1. This is the
  stall of the issue.
• aDirectoryServerInBadGenerationIdStatusKeepsItsSendWindowAndIsSentNothing - the window of a
  directory server in BAD_GEN_ID_STATUS stays whole after its catch-up, and nothing is counted as
  sent to it.

ReplicationServerShutdownSyncTest.onlyAPeerWhichCanDecodeTheMessageIsToldTheReplicaWentOffline
now also asserts the window and sent-updates of the peer, which covers the #1014 arm.

Run	Result
Without the fix	3 failures: window 7 of 10 (DS), 98 of 99 (V7 peer), the peer never receives the change
Mutant: no release	all 3 red; the peer test reports "the send window of its session is at 0"
Mutant: dropped updates counted as sent	all 3 red on sent-updates (3, 11, 2)
With the fix	FilteredUpdateSendWindowTest 2/2, ReplicationServerShutdownSyncTest 21/21, ReplicaOfflineMsgCatchUpTest 2/2
org/opends/server/replication/**	3820 tests, 0 failures, 0 errors, 0 skipped
javadoc (package, JDK 11)	green; a broken link planted in take() fails it

A test broker takes the credit of its own send window only while it receives, so the replication
server the publishing brokers talk to advertises a window of 100; the window of 10 is the one the
peer under test advertises.

[maximthomas]

praise: The permit goes back in the one place every drop passes through, and the stall is reproduced with a real peer.

• ServerWriter.run() calls handler.releasePermitInSendWindow() at the top of the filtered branch (ServerWriter.java:123), so every arm of isUpdateMsgFiltered() gives its permit back without a release per return true.
• countSentUpdate() is called only on the publish branch (ServerWriter.java:132), so sent-updates no longer counts the dropped backlog.
• FilteredUpdateSendWindowTest.aPeerWhichAgreesOnTheGenerationIdAfterItsBacklogWasDroppedIsSentTheNextChange drives a real peer replication server through the in-place generation-id change of #1080. It passes at the head, 2/2 for the class, along with both rows of ReplicationServerShutdownSyncTest#onlyAPeerWhichCanDecodeTheMessageIsToldTheReplicaWentOffline.