CloudStack Prometheus memory/cpu used value is actually the allocated value

[soreana]

ISSUE TYPE

• Bug Report

COMPONENT NAME

monitoring, prometheus

CLOUDSTACK VERSION

Probably the older versions as well
4.17,
4.18,
main

SUMMARY

I've noticed that prometheus exporter and cloudmonkey reports the different values for memory usage and allocated. See the output for more clarifications:

As you can see in the node01 output the cloudstack_host_memory_usage_mibs_total is 512.00 mibs which is equals to 536870912/1024/1024=512 in cmk output for memoryallocated. The same is true for mhz in cpu.

Memory

root@mgt01:~# curl http://127.0.0.1:9595/metrics | grep cloudstack_host_memory_usage_mibs_total 
cloudstack_host_memory_usage_mibs_total{zone="Zone1",hostname="node01",ip="10.1.2.5",overprovisioningfactor="1.0",filter="used",dedicated="0",tags=""} 512.00
cloudstack_host_memory_usage_mibs_total{zone="Zone1",hostname="node01",ip="10.1.2.5",overprovisioningfactor="1.0",filter="total",dedicated="0",tags=""} 2907.71
cloudstack_host_memory_usage_mibs_total{zone="Zone1",hostname="node02",ip="10.1.2.6",overprovisioningfactor="1.0",filter="used",dedicated="0",tags=""} 1024.00
cloudstack_host_memory_usage_mibs_total{zone="Zone1",hostname="node02",ip="10.1.2.6",overprovisioningfactor="1.0",filter="total",dedicated="0",tags=""} 2907.70
cloudstack_host_memory_usage_mibs_total{zone="Zone1",filter="allocated"} 1536.00
root@mgt01:~# cmk list hostsmetrics filter=memoryallocated,memoryused,memorytotal,name
{
  "count": 2,
  "host": [
    {
      "memoryallocated": 1073741824,
      "memorytotal": 3048947712,
      "memoryused": 1249058816,
      "name": "node02"
    },
    {
      "memoryallocated": 536870912,
      "memorytotal": 3048960000,
      "memoryused": 1262129152,
      "name": "node01"
    }
  ]
}

CPU

root@mgt01:~# curl http://127.0.0.1:9595/metrics | grep cloudstack_host_cpu_usage_mhz_total
cloudstack_host_cpu_usage_mhz_total{zone="Zone1",hostname="node01",ip="10.1.2.5",overprovisioningfactor="1.0",filter="used",tags=""} 500.00
cloudstack_host_cpu_usage_mhz_total{zone="Zone1",hostname="node01",ip="10.1.2.5",overprovisioningfactor="1.0",filter="total",tags=""} 4398.00
cloudstack_host_cpu_usage_mhz_total{zone="Zone1",hostname="node02",ip="10.1.2.6",overprovisioningfactor="1.0",filter="used",tags=""} 500.00
cloudstack_host_cpu_usage_mhz_total{zone="Zone1",hostname="node02",ip="10.1.2.6",overprovisioningfactor="1.0",filter="total",tags=""} 4400.00
cloudstack_host_cpu_usage_mhz_total{zone="Zone1",filter="allocated"} 1000.00
root@mgt01:~# cmk list hostsmetrics filter=cpuallocatedghz,cpuusedghz,cputotalghz,name
{
  "count": 2,
  "host": [
    {
      "cpuallocatedghz": "0.50 Ghz",
      "cputotalghz": "4.40 Ghz",
      "cpuusedghz": "0.06 Ghz",
      "name": "node02"
    },
    {
      "cpuallocatedghz": "0.50 Ghz",
      "cputotalghz": "4.40 Ghz",
      "cpuusedghz": "0.05 Ghz",
      "name": "node01"
    }
  ]
}

STEPS TO REPRODUCE

1. Enable the prometheus if it hasn't been already enabled using the "prometheus.exporter.enable" global setting.
2. Restart the mgt server if you enabled the prometheus.
3. Compare listhostsmetrics api call output with the prometheus output using the above commands.

EXPECTED RESULTS

prometheus output matches the cmk outputs

ACTUAL RESULTS

prometheus output DOESN'T match the cmk outputs

[soreana]

@weizhouapache I'm going to make a pr for it.

[DaanHoogland]

relates to #7591 , don't you think @soreana ?

[soreana]

@DaanHoogland Not exactly, I've checked that.

What described in #7591 is different. It says that when the overcommitting changed the CloudStack doesn't change the value stored in the capacity table. That is why the prometheus returns different value.

[DaanHoogland]

fixed by #7629