Skip to content

fix: handled MetricAgent plugin for multi matrix#4338

Open
Hardikl wants to merge 13 commits into
mainfrom
hl_sg
Open

fix: handled MetricAgent plugin for multi matrix#4338
Hardikl wants to merge 13 commits into
mainfrom
hl_sg

Conversation

@Hardikl

@Hardikl Hardikl commented Jul 2, 2026

Copy link
Copy Markdown
Contributor
image

Added supporting test cases as well.

Copilot AI review requested due to automatic review settings July 2, 2026 10:21
@cla-bot cla-bot Bot added the cla-signed label Jul 2, 2026
@Hardikl

Hardikl commented Jul 2, 2026

Copy link
Copy Markdown
Contributor Author

Validated with below template changes,

name:   Prometheus
query:  prometheus
object: # leave blank to disable prefixing exported metrics with object name

schedule:
  - data: 3m

counters:
  - node_cpu_seconds_total                                                => storagegrid_node_cpu_seconds_total
  - storagegrid_content_buckets_and_containers
  - storagegrid_content_objects
  - storagegrid_ilm_awaiting_client_objects
  - storagegrid_ilm_awaiting_total_objects
  - storagegrid_ilm_objects_processed
  - storagegrid_ilm_scan_objects_per_second
  - storagegrid_metadata_queries_average_latency_milliseconds
  - storagegrid_network_received_bytes
  - storagegrid_network_transmitted_bytes
  - storagegrid_node_cpu_utilization_percentage
  - storagegrid_private_ilm_awaiting_delete_objects
  - storagegrid_private_load_balancer_storage_request_body_bytes_bucket
  - storagegrid_private_load_balancer_storage_request_count
  - storagegrid_private_load_balancer_storage_request_time
  - storagegrid_private_load_balancer_storage_rx_bytes
  - storagegrid_private_load_balancer_storage_tx_bytes
  - storagegrid_private_s3_total_requests
  - storagegrid_s3_data_transfers_bytes_ingested
  - storagegrid_s3_data_transfers_bytes_retrieved
  - storagegrid_s3_operations_failed
  - storagegrid_s3_operations_successful
  - storagegrid_s3_operations_unauthorized
  - storagegrid_storage_utilization_data_bytes
  - storagegrid_storage_utilization_metadata_allowed_bytes
  - storagegrid_storage_utilization_metadata_bytes
  - storagegrid_storage_utilization_total_space_bytes
  - storagegrid_storage_utilization_usable_space_bytes

plugins:
  - JoinRest
  - MetricAgent:
      compute_metric:
        - storagegrid_storage_utilization_used_percent PERCENT storagegrid_storage_utilization_data_bytes storagegrid_storage_utilization_total_space_bytes

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates the MetricAgent plugin so compute_metric rules can operate when the required operands are split across multiple matrices (instead of assuming a single object-keyed matrix), and adjusts/adds unit tests to cover the multi-matrix scenario.

Changes:

  • Changed MetricAgent.actions to accept the full dataMap (map[string]*matrix.Matrix) rather than a single *matrix.Matrix.
  • Updated computeMetrics to select operand matrices per rule/operand (enabling multi-matrix support).
  • Updated the single-matrix test to run through Run(), and added a new multi-matrix test case.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File Description
cmd/poller/plugin/metricagent/parse_rules.go Updates action function signature to accept the full matrix map.
cmd/poller/plugin/metricagent/metric_agent.go Passes dataMap through actions and updates compute logic for multi-matrix operands.
cmd/poller/plugin/metricagent/metric_agent_test.go Renames/updates existing test to call Run() and adds a multi-matrix test.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread cmd/poller/plugin/metricagent/metric_agent_test.go Outdated
Comment thread cmd/poller/plugin/metricagent/metric_agent.go
Copilot AI review requested due to automatic review settings July 2, 2026 11:46

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

Comment thread cmd/poller/plugin/metricagent/metric_agent.go Outdated
Comment thread cmd/poller/plugin/metricagent/metric_agent.go
Comment thread cmd/poller/plugin/metricagent/metric_agent_test.go
Comment thread cmd/poller/plugin/metricagent/metric_agent.go Outdated
@Hardikl

Hardikl commented Jul 9, 2026

Copy link
Copy Markdown
Contributor Author

Validated with this plugin changes,

plugins:
  - JoinRest
  - MetricAgent:
      compute_metric:
        - storagegrid_storage_utilization_used_percent1 PERCENT storagegrid_storage_utilization_data_bytes storagegrid_private_s3_total_requests
        - storagegrid_storage_utilization_used_percent2 PERCENT storagegrid_storage_utilization_data_bytes storagegrid_storage_utilization_total_space_bytes

Logs:

harvest % ./bin/poller -p sg-tudor --promPort 13004 -o Prometheus 
time=2026-07-09T14:47:32.646+05:30 level=INFO source=poller.go:244 msg=Init Poller=sg-tudor logLevel=INFO configPath=harvest.yml cwd=/Users/hardikl/Library/CloudStorage/OneDrive-NetAppInc/Documents/Harvest/harvest version="harvest version 26.07.09-1 (commit 3ab04ff6) (build date 2026-07-09T14:44:35+0530) darwin/amd64" fips=false options="&{Poller:sg-tudor Daemon:false Debug:false PromPort:13004 Config:harvest.yml HomePath: LogPath:/var/log/harvest/ LogFormat:plain LogLevel:2 LogToFile:false Version:26.07.09 Hostname:hardikl-mac-1 Collectors:[] Objects:[Prometheus] Profiling:0 Asup:false IsTest:false ConfPath:conf ConfPaths:[conf]}"
time=2026-07-09T14:47:32.647+05:30 level=INFO source=poller.go:283 msg="started in foreground" Poller=sg-tudor pid=23165
time=2026-07-09T14:47:35.049+05:30 level=INFO source=poller.go:1944 msg="Cluster info" Poller=sg-tudor remote="{Name:SG5712FP Model: UUID:4cca45b6-1e5a-4c15-b3a6-e29598de9d44 Version:12.0.0 Release: Serial: IsSanOptimized:false IsDisaggregated:false ZAPIsExist:false ZAPIsChecked:false HasREST:false HasRESTPerf:false IsClustered:false}"
time=2026-07-09T14:47:37.812+05:30 level=INFO source=helpers.go:113 msg="best-fit template" Poller=sg-tudor collector=StorageGrid:Prometheus path=conf/storagegrid/11.6.0/storagegrid_metrics.yaml v=12.0.0 jitter=none
time=2026-07-09T14:47:40.244+05:30 level=INFO source=poller.go:420 msg="Autosupport scheduled" Poller=sg-tudor asupSchedule=24h
time=2026-07-09T14:47:40.244+05:30 level=INFO source=poller.go:430 msg="poller start-up complete" Poller=sg-tudor
time=2026-07-09T14:47:40.246+05:30 level=INFO source=httpd.go:51 msg="server listen" Poller=sg-tudor exporter=prometheus1 url=http://:13004/metrics
time=2026-07-09T14:47:40.281+05:30 level=INFO source=poller.go:661 msg="updated status" Poller=sg-tudor collectors.up=1 collectors.total=1 exporters.up=1 exporters.total=1
time=2026-07-09T14:47:40.294+05:30 level=INFO source=poller.go:1757 msg=Metadata Poller=sg-tudor remote.name=SG5712FP remote.version=12.0.0 version="harvest version 26.07.09-1 (commit 3ab04ff6) (build date 2026-07-09T14:44:35+0530) darwin/amd64" mem.liveHeapMB=1 mem.heapMB=1 mem.heapGoalMB=4 mem.rssMB=19 mem.maxRssMB=19 uptimeSeconds=7
time=2026-07-09T14:47:42.323+05:30 level=WARN source=storagegrid.go:169 msg="no instances on storagegrid" Poller=sg-tudor collector=StorageGrid:Prometheus metric=storagegrid_ilm_awaiting_client_objects
time=2026-07-09T14:47:50.619+05:30 level=WARN source=metric_agent.go:136 msg="computeMetrics: skipping compute metric calculation as instance labels are not matching for given metrics" Poller=sg-tudor plugin=StorageGrid:MetricAgent object=Prometheus metric=storagegrid_storage_utilization_used_percent1 "given metrics"="[storagegrid_storage_utilization_data_bytes storagegrid_private_s3_total_requests]"
time=2026-07-09T14:47:50.619+05:30 level=WARN source=metric_agent.go:136 msg="computeMetrics: skipping compute metric calculation as instance labels are not matching for given metrics" Poller=sg-tudor plugin=StorageGrid:MetricAgent object=Prometheus metric=storagegrid_storage_utilization_used_percent1 "given metrics"="[storagegrid_storage_utilization_data_bytes storagegrid_private_s3_total_requests]"
time=2026-07-09T14:47:50.619+05:30 level=WARN source=metric_agent.go:136 msg="computeMetrics: skipping compute metric calculation as instance labels are not matching for given metrics" Poller=sg-tudor plugin=StorageGrid:MetricAgent object=Prometheus metric=storagegrid_storage_utilization_used_percent1 "given metrics"="[storagegrid_storage_utilization_data_bytes storagegrid_private_s3_total_requests]"
time=2026-07-09T14:47:50.628+05:30 level=INFO source=collector.go:629 msg=Collected Poller=sg-tudor collector=StorageGrid:Prometheus apiMs=8954 bytesRx=300814 calcMs=0 exportMs=8 instances=1069 instancesExported=1069 metrics=1069 metricsExported=1072 numCalls=33 parseMs=0 pluginInstances=0 pluginMs=1418 pollMs=10373 renderedBytes=270358 zBegin=1783588660247
image

Copilot AI review requested due to automatic review settings July 9, 2026 09:25

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

Comment thread cmd/poller/plugin/metricagent/metric_agent.go Outdated
Comment thread cmd/poller/plugin/metricagent/metric_agent.go Outdated
Comment thread cmd/poller/plugin/metricagent/metric_agent.go Outdated
Copilot AI review requested due to automatic review settings July 9, 2026 10:48

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

cmd/poller/plugin/metricagent/metric_agent.go:155

  • If an operand metric isn't found, this appends err (typically nil from the preceding Atoi failure) and breaks out without setting skipMetric, so the derived metric can be written with a partial/incorrect result and the missing-metric details are lost. Append a concrete error and mark skipMetric before breaking.
					} else {
						metricNotFound = append(metricNotFound, err)
						break
					}

Copilot AI review requested due to automatic review settings July 9, 2026 10:52

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

Comment thread cmd/poller/plugin/metricagent/metric_agent.go Outdated
Comment thread cmd/poller/plugin/metricagent/metric_agent_test.go
Comment thread cmd/poller/plugin/metricagent/metric_agent_test.go
Copilot AI review requested due to automatic review settings July 9, 2026 13:15

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

cmd/poller/plugin/metricagent/metric_agent.go:155

  • When an operand metric is missing, this appends err to metricNotFound, but err here is the strconv.Atoi parse error for the metric name (or nil) rather than a "missing metric" error. Record a concrete error tied to the missing metric so the warning at the end is actionable.
					} else {
						metricNotFound = append(metricNotFound, err)
						break
					}

Comment thread cmd/poller/plugin/metricagent/metric_agent.go Outdated
Comment thread cmd/poller/plugin/metricagent/metric_agent.go Outdated
@cgrinds

cgrinds commented Jul 14, 2026

Copy link
Copy Markdown
Collaborator

I agree with these review comments from Sonnet 4.6

sgMatrix / sgIndex embed a StorageGrid-specific name in a general plugin

The MetricAgent plugin isn't StorageGrid-specific, but these variable names suggest it is. Names like isMultiMatrix and instanceIndex would better convey intent.

The instance key convention (metricName + "-" + index) is unexplained

The key parsing via strings.LastIndex(iKey, "-") only works because StorageGrid's collector always constructs keys as metricName + "-" + strconv.Itoa(i) (confirmed at storagegrid.go:219). There's no comment noting this contract. A future maintainer reading only metric_agent.go would have no idea why the index is extracted by splitting on the last -.

@Hardikl

Hardikl commented Jul 15, 2026

Copy link
Copy Markdown
Contributor Author

I agree with these review comments from Sonnet 4.6

sgMatrix / sgIndex embed a StorageGrid-specific name in a general plugin

The MetricAgent plugin isn't StorageGrid-specific, but these variable names suggest it is. Names like isMultiMatrix and instanceIndex would better convey intent.

The instance key convention (metricName + "-" + index) is unexplained

The key parsing via strings.LastIndex(iKey, "-") only works because StorageGrid's collector always constructs keys as metricName + "-" + strconv.Itoa(i) (confirmed at storagegrid.go:219). There's no comment noting this contract. A future maintainer reading only metric_agent.go would have no idea why the index is extracted by splitting on the last -.

handled.

Copilot AI review requested due to automatic review settings July 15, 2026 10:23

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (1)

cmd/poller/plugin/metricagent/metric_agent.go:194

  • This warning logs an empty message and uses a field key containing spaces/colon, which makes the structured log harder to query. Consider using a clear message and a stable field name.
	if len(metricNotFound) > 0 {
		a.SLogger.Warn("", slog.Any("computeMetrics: errors for metric not found", metricNotFound))
	}

Comment thread cmd/poller/plugin/metricagent/metric_agent.go Outdated
Comment thread cmd/poller/plugin/metricagent/metric_agent.go
Copilot AI review requested due to automatic review settings July 15, 2026 10:47

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

Comment thread cmd/poller/plugin/metricagent/metric_agent.go Outdated
Copilot AI review requested due to automatic review settings July 15, 2026 10:54

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

Comment thread cmd/poller/plugin/metricagent/metric_agent.go
Comment thread cmd/poller/plugin/metricagent/metric_agent.go Outdated
Copilot AI review requested due to automatic review settings July 15, 2026 12:36

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

cmd/poller/plugin/metricagent/metric_agent.go:200

  • computeMetrics accumulates metricNotFound errors, but currently only logs them (with an empty log message) and then returns nil. This prevents Run from surfacing missing-metric failures via its errors.Join return, making misconfigured compute_metric rules easy to miss (and metricNotFound construction effectively unused).
	if len(metricNotFound) > 0 {
		a.SLogger.Warn("", slog.Any("computeMetrics: errors for metric not found", metricNotFound))
	}
	return nil

}
}
if m[0] == nil {
return errs.New(errs.ErrMissingMetric, "matrix not found for metric "+r.metricNames[0])

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should here we use continue? a missing matrix for one rule shouldn't abort all remaining rules

if val, ok := metricVal.GetValueFloat64(otherInstance); ok {
v = val
} else {
continue

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this be

skipMetric = true
break

like other checks?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

MetricAgent crashes with nil pointer dereference on StorageGRID Prometheus collector

4 participants