Unhandled ObjectDisposedException in TcpServer.HandleConnection after request timeout

[Kowjako]

Description

A RAFT node using the built-in TCP transport can terminate because of an unhandled ObjectDisposedException after request cancellation or timeout.

The exception is thrown while TcpServer.HandleConnection attempts to determine whether an OperationCanceledException was caused by a timeout.

Environment

• DotNext.Net.Cluster: 6.6.1
• DotNext.Threading: 6.6.1
• .NET: 10
• Runtime: Linux Docker container
• Transport: built-in TCP transport
• Cluster size: 3 RAFT members

Configuration:

var configuration = new RaftCluster.TcpConfiguration(listenEndpoint)
{
    PublicEndPoint = publicEndpoint,
    LowerElectionTimeout = 150,
    UpperElectionTimeout = 300,
    ColdStart = coldStart,
    LoggerFactory = loggerFactory,
    ConfigurationStorage = configurationStorage
};

Scenario

Start a three-member RAFT cluster.
Let member 1 become the leader.
Stop member 1.
Member 2 is elected as the new leader.
Member 2 successfully replicates and commits a log entry.
Shortly afterwards, member 2 terminates with an unhandled exception inside the DotNext TCP transport.
The issue appears to occur when processing a request takes longer than RequestTimeout, but interrupting a connection during request processing may also trigger it.

Actual behavior:

Unhandled exception. System.ObjectDisposedException: The CancellationTokenSource has been disposed.

at DotNext.Threading.CancellationTokenMultiplexer.Scope.get_Token()
at DotNext.Threading.MultiplexedCancellationTokenSource.CausedByTimeout(...)
at DotNext.Net.Cluster.Consensus.Raft.NetworkTransport.ConnectionOriented.Tcp.TcpServer.HandleConnection(Socket remoteClient)
at System.Threading.Tasks.Task.<>c.b__124_1(Object state)
at System.Threading.ThreadPoolWorkQueue.Dispatch()

Suspected Cause

In TcpServer.HandleConnection, timeoutSource is disposed in the finally block:

timeoutSource = multiplexer.Combine(receiveTimeout, lifecycleToken);

try
{
    await ProcessRequestAsync(
        messageType,
        protocol,
        timeoutSource.Token).ConfigureAwait(false);
}
finally
{
    await timeoutSource.DisposeAsync().ConfigureAwait(false);
}

The outer catch then accesses the disposed scope:

catch (OperationCanceledException e)
{
    if (e.CausedByTimeout(timeoutSource))
        logger.RequestTimedOut(clientAddress, e);
}

CausedByTimeout accesses CancellationTokenMultiplexer.Scope.Token, but the underlying CancellationTokenSource may already be disposed. This results in another exception being thrown from the exception handler.

Workaround

Explicitly increasing RequestTimeout reduces the probability of the crash:

RequestTimeout = TimeSpan.FromSeconds(10)

However, this does not fully solve the problem because a timeout or lifecycle cancellation can still trigger the same race condition.

[Kowjako]

Here is fix #294 @sakno

[Kowjako]

Here is commit which introduced regression -> 5667548 @sakno

[sakno]

Hi @Kowjako , thanks for your report. I'll take a look as soon as possible.