vROps 8.x: Alert triggered for potentially stale objects
About one month ago Pure Storage released Management Pack (MP) 3.0.0 for VMware’s vRealize Operations Manager (vROps) 8.x suite. A short time after the release several customers began reporting that vROps was triggering a new alert in their environment that they haven’t seen in previous versions of vROps or the Pure Storage MP.
What made things even more complicated was that the issue was reported at random times throughout the day and it was not associated with every object in the environment. This of course intrigued me and thus the investigation began!
Objects are not receiving data from adapter instance
As noted in the paragraph header here, the alert being triggered in the environment was: “Objects are not receiving data from adapter instance.”
If you click on the image above you will note that vROps is alerting that 7 objects are not receiving data from “the adapter”. The adapter in this instance would be a specific array defined within the Pure Storage Management Pack.
After investigating the issue it was noted that not all objects are reported as “not receiving data”, rather only specific types of objects:
FlashArray FC Ports
FlashAray Volume Snapshots
FlashArray Protection Groups
FlashArray Protection Group Snapshots
Reviewing the data showed that these 4 particular groups of objects do not have any metrics associated with them by design (such as performance or capacity metrics). They are simply added to the adapter when they are discovered by the MP and then verified whether or not they still exist on all subsequent runs. Since there are no metrics associated with these objects then there is nothing to update during each new collection.
It is because of this that the alert gets triggered within vROps. If we look at the definition of the alert it states the following:
Wait Cycle: 1
Cancel Cycle: 1
This symptom is set to true when:
Base object exhibits all of the following symptoms.
1. Object is an adapter instance.
2. Objects are not receiving data from adapter instance.
So if vROps finds that an object is not updated after 2 cycles (as it waits for one cycle) then this alert will be generated.
This then led to the question: Why did we not see this issue in vROps 7.5 and earlier?
The alert was available and defined in vROps 7.5 and earlier and the objects were treated the same way in previous versions of the Pure Storage Management pack. So it doesn’t make sense why this alert was suddenly appearing.
As a result a ticket was opened with VMware to investigate this in parallel with their team. A short while later it was discovered that there was a change in vROps 8.x that led to this behavior and how it handles objects with only properties and no metrics.
Now that the issue has been identified a fix can be implemented to prevent the issue from happening in future versions. As it stands now there are several ways in which a fix can be implemented that are carefully being weighed by both Pure Storage and VMware. It is important that the fix is released as quickly as possible but also ensure that it doesn’t have any negative side effects.
Once that decision has been made, and the fix implemented, I will update the blog with the version of Pure Storage Management Pack or vROps it will be provided in.
Workaround
The only known workaround at this time is to disable the alert. The most important thing to keep in mind with this choice is that disabling the alert is a global option. This means that the alert is not only disabled for the Pure Storage Management Pack but any others that you have installed within vROps too. If this is not a concern for you in a short term, and the alerts are causing too much noise, then disabling the alert can be done by following the VMware KB: “Objects are not receiving data from adapter instance” alerts in vRealize Operations Manager 6.x (2149137).
Note: The KB above references 6.x but the steps are the same in 8.x. Additionally, the issue referenced in the KB is a different issue from the one described here. It just contains the clearest steps to disabling this particular alert.
I hope this helps someone who may run across this issue and feel free to spread the word to those you think might need it!
-jhop