Skip to content

[BUG] A timeout is never reported as a timeout #612

Description

@bburda

Bug report

Steps to reproduce

  1. Run an aggregator with a peer. On the peer, define an entity with an operation that takes
    longer than aggregation.timeout_ms (default 2000 ms).
  2. POST /api/v1/components/<id>/operations/<op>/executions on the aggregator, then send the
    same request directly to the owning peer.
  3. On a single gateway, execute an operation whose own work takes longer than
    service_call_timeout_sec (default 10 s).
  4. Request a fanned-out collection such as GET /api/v1/faults while one peer is slow enough to
    exceed the fan-out budget.

Expected behavior

A request that ran out of time says so. It returns a timeout status, or at least a message that
names the budget that was exceeded. A timeout can be told apart from a peer that is really down,
and a partial answer can be told apart from a complete one.

Actual behavior

Three paths, three different ways of not reporting a timeout.

1. Peer forward: reported as an unavailable peer.

aggregator  -> 502 in 2.002822 s
  {"error_code": "vendor-error",
   "message": "Peer '<name>' at http://<host>:<port> is unavailable",
   "vendor_code": "x-medkit-peer-unavailable"}
owning peer -> 200 in 4.936800 s   (the full operation result)

The peer was alive at that moment. Probed from inside the aggregator container at the same time:
/api/v1/health returned 200 and /api/v1/components/<id> returned 200. The message reports a
peer as down while that peer answers the same request in 4.9 s.

2. Fan-out: reported as success. A peer that times out during a fanned-out collection gives
HTTP 200 with x-medkit.partial = true and x-medkit.failed_peers. No status change, no
error code, no message.

3. Local operation: reported as a generic failure. An operation that exceeds
service_call_timeout_sec returns 500 with "message": "Service call failed".

Environment

  • ros2_medkit version: main (a268379), gateway vendor_info.version 0.6.0, SOVD API 1.0.0
  • ROS 2 distro: Jazzy
  • OS: Ubuntu 24.04

Additional information

Why the peer forward cannot tell the two cases apart. httplib::Result::operator bool() is
a null check (src/vendored/cpp_httplib/httplib.h:1053). The forward path branches on !result
at src/core/aggregation/peer_client.cpp:637 and never reads result.error(). So
Error::ConnectionTimeout, Error::Read (read timeout), Error::Connection (refused) and
Error::Canceled all produce the same 502 with x-medkit-peer-unavailable and "is unavailable".

Why the fan-out path loses the failure completely. The error string is built at
src/core/aggregation/peer_client.cpp:741 and then dropped. PeerResult
(src/aggregation/aggregation_manager.cpp:677-681) has no error field, and the per-peer task
only records a boolean (aggregation_manager.cpp:719-723). A timeout, a refused connection, a
peer returning 500, a body over the size limit and unparseable JSON all look the same here.

The local operation path. The transport already builds a useful message,
"Service call timed out (10000ms): /path/to/service"
(src/ros2/transports/ros2_service_transport.cpp:116-127). The handler puts it in
parameters.details but returns 500 with the fixed top-level message "Service call failed"
(src/http/handlers/operation_handlers.cpp:705-707). A client has to match on the text in
details to find out it was a timeout.

The action send-goal path does not build that message at all. "Send goal timed out"
(src/ros2/transports/ros2_action_transport.cpp:214) names neither the action nor the budget,
and maps to 500 with x-medkit-ros2-action-unavailable and "Action execution failed"
(operation_handlers.cpp:675-677).

There is already a precedent in the tree. #576 made this case for the action cancel path and
#593 fixed it. ERR_NOT_RESPONDING is reserved for exactly this ("no response from the
underlying entity in time", include/ros2_medkit_gateway/core/http/error_codes.hpp:150-154) and
is returned with 504 in one place only: the cancel path,
src/http/handlers/operation_handlers.cpp:282. Nothing in the aggregation layer or the operation
execute path uses it. This issue asks for the same treatment on the three paths that were out of
scope for #576.

Two related facts.

  • There is no separate budget for operation execute. PeerClient holds one int timeout_ms_,
    and both construction sites pass config_.timeout_ms
    (src/aggregation/aggregation_manager.cpp:219,275). It is applied as both the connect and the
    read timeout (src/core/aggregation/peer_client.cpp:813-814). So a synchronous ROS 2 service
    call on a peer gets 2 s end to end, while the peer's own budget for the same call is
    service_call_timeout_sec, default 10 s. The value itself is [BUG] Aggregator times out fanning out large resource payloads to peers #528; the point here is that one
    budget covers both a metadata read and real work.
  • The write timeout is never set, so it stays at the cpp-httplib default of 5 s
    (src/vendored/cpp_httplib/httplib.h:41-46) and does not follow aggregation.timeout_ms.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions