How can we help?

High CPU Utilization on Dell and Cisco SG Switch Stacks

Follow

Auvik has received a few reports of high CPU utilization on switches from Dell and Cisco Small Business that are running in stacked mode. A smaller subset of these cases have also reported device instability until one of the following actions was performed:

  1. Unmanaging the switch stack device in Auvik, or
  2. Turning off the Auvik collector.

With such instability, you may see fatal errors contained within the device’s syslog that resemble something like the following:

2147473283 YYYY-MM-DD HH:MM:SS Emergency %SNMP-F-FTLERR: SNMP Package: SnmpCen Routine: 24 Location: 10 Error: Bad value returned by application: Range error: variable = 2664, field = rlPethPsePortOper PowerLimit, value = 2264783668 ***** FATAL ERROR ****** (warning: SYSLOGG_log_fatal should be used to log fatal errors, not SYSLOGG_log) Reporting Task: SNMP. Software Version: 1.3.0.62 (date 02-May-2013 time 14:56:31) 0x16acd4 0x167514 0x730324 0x4de590 0x4dee84 0x4df0a4 0x4df444 0x494918 0x49528c 0x477aa0 0x47c798 0x47d318 0x4626d4 0x44a130 0x4551b0 0x45e278 0x45e2f0 0x4600b8 0x46017c 0x491b2c 0x677080 0x121d9c ***** END OF FATAL ERROR ***

Background

When stacked, Dell PowerConnect and Cisco SG 500 switches initialize their SNMP interface tables (such as ifTable) for the theoretical maximum number of stack members and ports that would be available in a maxed out configuration.

This leads to the population of 900 to 1000 interfaces within the respective tables. The data leads Auvik to believe there are that many interfaces within the switch stack. So upon every interface polling cycle (by default, every minute), Auvik sends a query to your switches asking for interface stats like bits per second, packet errors, and packet discards. Such a query for that many interfaces proves challenging for the switch.

Short-term workaround

For the device to populate the correct number of physical and logical interfaces within the switch stack, disable the stacking feature on the switch. This allows Auvik to query the correct number of interfaces per switch chassis. The downside is the loss of features and easy provisioning that comes with setting up switches in a stacked arrangement.

Long-term

Auvik is looking at building additional capability into the collector to gracefully handle these device idiosyncrasies. We’ll mention this capability in the release notes once it has been deployed into production.

If you have any questions about this issue, please contact Auvik Support.

Have more questions? Submit a request

Comments

Powered by Zendesk