What causes "Value cannot be negative" in cloudwatch agent logs? - amazon-cloudwatch

Question

I see the following log entries in /var/log/amazon/amazon-cloudwatch-agent/amazon-cloudwatch-agent.log
2021-10-06T13:18:00Z W! Value cannot be negative: -1
2021-10-06T13:18:00Z W! Value cannot be negative: -1
2021-10-06T13:18:00Z W! Value cannot be negative: -1
Is there anyway to find out what is the source of this -1 value?

Answer 1

I don't think there is a good way to find the actual source in the general case. That's why I opened an issue in aws/amazon-cloudwatch-agent.
But there are some guesses that you can do:
First, check if you are using collectd GenericJMX to collect the NonHeapMemoryUsage attribute from java.lang:type=Memory MBean, that is part of the usual collectd's GenericJMX example configuration. The NonHeapMemoryUsage is a composite attribute with committed,init, max and used. And max is always -1.
You can verify this using jmx-term cli utility:
java -jar jmxterm-1.0.2-uber.jar
$>open localhost:8993
$>get -d java.lang -b java.lang:type=Memory NonHeapMemoryUsage
#mbean = java.lang:type=Memory:
NonHeapMemoryUsage = {
committed = 312860672;
init = 7667712;
max = -1;
used = 304634560;
};
If this is the case you can use collectd Chains functionality to "drop" the TypeInstance=nonheap-max, by adding the following to your collectd.conf:
<Chain "PostCache">
# only show system and user
<Rule "jmx_memory">
<Match "regex">
Plugin "GenericJMX"
Type "jmx_memory"
</Match>
<Match "regex">
TypeInstance "nonheap-max"
Invert true
</Match>
# If a jmx_memory metric and it's not the nonheap-max then send it to the amazon-cloudwatch-agent 25826
<Target "write">
Plugin network
</Target>
</Rule>
# Default target drop all metrics
Target "stop"
</Chain>
Second, if it not the NonHeapMemoryUsage is likely to be any other collectd metric, you can drop all metrics with negative values with collectd's <Match "Value"> although I don't recommend it:
<Chain "PostCache">
# only show system and user
<Rule "drop negative">
<Match "value">
Min 0
Satisfy "All"
</Match>
# send positive values to amazon-cloudwatch-agent 25826
<Target "write">
Plugin network
</Target>
</Rule>
# Default target drop all metrics
Target "stop"
</Chain>