feat(storage): add range read latency metrics and trace annotations - #16353
Conversation
There was a problem hiding this comment.
Code Review
This pull request introduces telemetry and metrics collection for asynchronous reader connections in the Google Cloud Storage C++ client, adding ReaderConnectionTelemetry to record queue, network, and internal overhead latencies using OpenTelemetry. It updates various tracing classes to propagate bucket names and record read latency metrics, fixes a locking issue in ReadRange, and adds unit tests. The reviewer feedback focuses on enforcing repository style guide rules, specifically replacing absl::string_view with std::string_view, avoiding auto where it obscures domain objects, StatusOr, function return types, or primitive types, improving header includes, and using more idiomatic checks for std::chrono::steady_clock::time_point initialization.
|
/gemini review |
There was a problem hiding this comment.
Code Review
This pull request introduces 'ReaderConnectionTelemetry' to record read latency metrics and trace events, integrating it into 'AsyncReaderConnectionTracing' and 'ObjectDescriptorReaderTracing' while passing the bucket name through the tracing pipeline. It also fixes a mutex locking issue in 'ReadRange'. The review feedback recommends avoiding a process-global static 'ReadLatencyMetrics' to respect client-specific 'MeterProvider' configurations, and suggests adding defensive checks to ensure timestamps are monotonically ordered before recording latency.
9a3fcc9 to
3c6a274
Compare
| GOOGLE_CLOUD_CPP_INLINE_NAMESPACE_BEGIN | ||
|
|
||
| #ifdef GOOGLE_CLOUD_CPP_STORAGE_WITH_OTEL_METRICS | ||
| ReaderConnectionTelemetry::ReaderConnectionTelemetry() { |
There was a problem hiding this comment.
Calling GetMeter and CreateDoubleHistogram dynamically on the hot path acquires internal SDK locks, which might cause severe lock contention and slow down the client under high concurrency.
| void ReaderConnectionTelemetry::RecordMetrics(std::string const& bucket_name, | ||
| double p1, double p2, | ||
| double p3) const { | ||
| if (metrics_.queue_hist) |
There was a problem hiding this comment.
Passing opentelemetry::context::Context{} explicitly forces an empty context when recording the metric.
Because the context is empty, OpenTelemetry cannot extract the Trace ID and Span ID. This breaks "exemplars," meaning these metrics will lose their correlation to the active trace spans.
| if (!absl::holds_alternative<Status>(result)) { | ||
| auto const& payload = absl::get<storage::ReadPayload>(result); | ||
|
|
||
| if (auto const* payload = |
There was a problem hiding this comment.
You are calling internal::DetachOTelContext(oc) before you call metrics.RecordRead(...).
This means the trace context is removed from the active thread before the metric is recorded, which guarantees that trace exemplars will fail to generate.
Key changes in this PR: