How can we help?

How do I debug using the Auvik collector?

Follow

The Auvik collector is equipped with a Linux shell that can capture data about your network, devices, or collector to help Auvik diagnose issues.

If you run any of the following commands, send the resulting information to Auvik support for data analysis, or include it on a support ticket that needs follow-up or resolution.

How to enable the collector shell

Depending on the collector you have running, you may need to complete different actions for enabling the shell.

Virtual collector (OVA or scripted)

  1. From the virtual machine window for the collector, click onto the console.
  2. Select Option 6 and enter y to enter the bash shell from the collector menu.
  3. Type sudo /usr/share/agent/bin/agent_shell

Windows service

  1. From the start menu, search for Command Prompt (cmd).
  2. Right-click and select Run as administrator.
  3. From the command prompt window, change the directory to the install location. For example: cd c:\auvik
  4. Type AuvikAgentService -shell

Docker Image

Using docker compose

If your docker image was deployed using docker compose, the following steps will leverage the configuration to start an agent shell process.

  1. From the host machine, open a terminal window
  2. Change directories to the location of the docker compose configuration file
  3. Run the following command to start the agent shell, replacing <SERVICE_NAME> with the service name defined within the docker compose file:
docker compose exec -it <SERVICE_NAME> /var/auvik/bin/agent -x

Note: The <SERVICE_NAME> will be dependent on the deployment configuration. The service name will be based on the entry within the docker compose file, specifically under the services section.

See the following docker section regarding docker compose service definitions: Services top-level element

Using docker

If your docker image was not deployed using docker compose, or you do not have access to the docker compose configuration, then the following docker commands can be used to start an agent shell process.

  1. From the host machine, open a terminal window
  2. Run the following docker command:
 docker exec -it <container name/id> /var/auvik/bin/agent -x

Note: The <container name/id> will be dependent on the deployment. If using docker compose for the deployment, this will be the container_name used in the docker-compose.yaml file.

If no name was specified during the configuration, a docker will set a default name. This can be found using docker ps command to list all running containers. Once the correct container has been identified, the container id OR the container name can be used in the above command.

How to diagnose discovery scanning

Where are all the devices?

Things to be answered with these diagnostics: What does the health check ping say about this device? Does it find the device? If so, what is the ping latency? If it doesn’t find the device, it's unlikely the Auvik collector will find the device no matter what we do.

Pick a device with a known IP address that you know is currently online. Next, try a scan of the subnet from the diagnostic shell. This will serve as a baseline for scan performance.

For example:

auvik > scan 10.0.40.0 24
      Scan result for 10.0.40.0/24
      10.0.40.1
      10.0.40.110
      10.0.40.123
      10.0.40.198
auvik > ping 10.0.40.110
      10.0.40.110 responded to ping
auvik > portOpen 10.0.40.110 80
      10.0.40.110:80 port is not open
auvik > portScan 10.0.40.110 80
       No ports found to be open

Try repeating the scan a couple of times to see if hosts magically start appearing. Contact Auvik support for further analysis of what you’re seeing.

Is a scan killing the network?

There are two parameters that control how fast Auvik sends out ICMP packets:

ScanService.max-simultaneous-ping

ScanService.simultaneous-send-delay

Both default to 10, meaning we send at most 10 packets within any particular 10ms window. Increasing the delay or decreasing the maximum number of sends will slow the scan down.

How to diagnose discovery services

SNMP

The most common answers have to do with devices not being configured correctly, or incorrect credentials being used. The diagnostic shell has a few debug commands that can help to understand SNMP issues:

snmpcred sets the credentials that will be used by the other commands.

For SNMP v1/2c:

snmpcred community private
configures snmp v2c community authentication with community string "private"
snmpcred version v1 community private
configures snmp v1 community authentication with community string "private"

For SNMP v3, with the auth protocol of MD5:

snmpcred user V3User auth md5 
md5password priv des des56password
configures snmp v3 USM authentication with authentication hash algorithm MD5 and authentication passphrase "md5password", and privacy cipher DES and privacy passphrase "des56password"
snmpcred user V3User auth md5 
md5password priv desede desedepassword
configures snmp v3 USM authentication with authentication hash algorithm MD5 and authentication passphrase "md5password", and privacy cipher 3DES-EDE and privacy passphrase "desedepassword"
snmpcred user V3User auth md5 
md5password priv aes192 aes192password
configures snmp v3 USM authentication with authentication hash algorithm MD5 and authentication passphrase "md5password", and privacy cipher AES-192 and privacy passphrase "aes192password"
snmpcred user V3User auth md5 
md5password priv aes256 aes265password
configures snmp v3 USM authentication with authentication hash algorithm MD5 and authentication passphrase "md5password", and privacy cipher AES-256 and privacy passphrase "aes265password"

For SNMP v3, with the auth protocol of SHA:

snmpcred user V3User auth sha 
shapassword priv des despassword
configures snmp v3 USM authentication with authentication hash algorithm SHA and authentication passphrase "shapassword", and privacy cipher DES and privacy passphrase "despassword"
snmpcred user V3User auth sha 
shapassword priv desede desedepassword
configures snmp v3 USM authentication with authentication hash algorithm SHA and authentication passphrase "md5password", and privacy cipher 3DES-EDE and privacy passphrase "desedepassword"
snmpcred user V3User auth sha 
shapassword priv aes192 aes192password
configures snmp v3 USM authentication with authentication hash algorithm SHA and authentication passphrase "md5password", and privacy cipher AES-192 and privacy passphrase "aes192password"
snmpcred user V3User auth sha 
shapassword priv aes256 aes265password
configures snmp v3 USM authentication with authentication hash algorithm SHA and authentication passphrase "md5password", and privacy cipher AES-256 and privacy passphrase "aes256password"

Setting Auvik shell credentials for Cisco devices running SNMPv3 privacy above AES-128: Cisco uses what they call 3-DES key extension as part of the protocol; therefore to account for this, you'll need to add a "w3des" suffix to the privacy parameter e.g - when using AES-256, you would specify: priv aes256w3des aespassword

If snmpcred isn't invoked at all, or is invoked without any arguments, the default configuration is used: snmp v2c community authentication with community string "public".

snmpget attempts to fetch one or more OIDs from the host at a specified IP address using an SNMP GET request. If only an IP address is specified it fetches the OID sysDescr.0. To fetch other OIDs, any number of numeric OID identifiers can be specified following the IP address. Note that it doesn’t support symbolic OID names like the net-snmp tools do.

snmpgetnext is like snmpget except it uses an SNMP GETNEXT request.

snmpgetbulk is like snmpget except it uses an SNMP GETBULK request.

SNMP has no "authentication failed" response, so a failed authentication is generally indistinguishable from "host down or not present".

For example:

auvik> snmpcred user V3User auth md5 md5password priv des des56password
user {
security_name: "V3User"
auth_protocol: SNMP_AUTH_MD5
auth_passphrase: "md5password"
privacy_protocol: SNMP_PRIVACY_DES
privacy_passphrase: "des56password"
}

auvik> snmpget 10.0.40.1
[1.3.6.1.2.1.1.1.0]: snmpString
(Cisco IOS Software, 2801 Software (C2801-ADVENTERPRISEK9-M),
Version 12.4(17), RELEASE SOFTWARE (fc1)

Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2007 by Cisco Systems, Inc.
Compiled Fri 07-Sep-07 17:15 by prod_rel_team)

auvik> snmpget 10.0.40.1 1.3.6.1.2.1.1.5.0
[1.3.6.1.2.1.1.5.0]: snmpString
(CISCO_2801.auvik.local)


auvik> snmpget 10.0.40.2
snmp request returned no result

auvik>

snmpwalk attempts to retrieve all OIDS from a specified host that have a common prefix. If only an IP address is specific, snmpwalk will fetch all OIDs starting with 1.3.6.1.2.1.1. To fetch more or different OIDs, a different prefix can be specified as an additional parameter.

For example:

auvik> snmpwalk 10.0.40.3 1.3.6.1.2.1.1

[1.3.6.1.2.1.1.1.0]: snmpString(SonicWALL NSA 2400 (SonicOS Enhanced 5.8.1.14-68o))
[1.3.6.1.2.1.1.2.0]: oid([1.3.6.1.4.1.8741.1])
[1.3.6.1.2.1.1.3.0]: long(0x348e7c13e90)
[1.3.6.1.2.1.1.4.0]: snmpString(Alex Hoff)
[1.3.6.1.2.1.1.5.0]: snmpString(Auvik's SonicWall 2400)
[1.3.6.1.2.1.1.6.0]: snmpString()
[1.3.6.1.2.1.1.7.0]: int(0x4f)

SSH

The diagnostic shell contains a simple SSH tester. This tester opens a connection to the specified remote system, sends the command to be executed, and displays all output received in the next five seconds.

For example:

auvik> ssh 10.0.40.1 auvik testpassword show ver

ssh session established and authenticated
ssh channel created
CISCO_2801> show ver
Cisco IOS Software, 2801 Software (C2801-ADVENTERPRISEK9-M),
Version 12.4(17), RELEASE SOFTWARE (fc1)

Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2007 by Cisco Systems, Inc.
Compiled Fri 07-Sep-07 17:15 by prod_rel_team
ROM: System Bootstrap, Version 12.4(13r)T, RELEASE SOFTWARE (fc1)
CISCO_2801 uptime is 48 weeks, 3 hours, 26 minutes
System returned to ROM by Reload Command
System image file is "flash:c2801-adventerprisek9-mz.124-17.bin"

HTTP

The diagnostic shell contains a simple HTTP tester. It will attempt to fetch any URL, although note that the given URL must have at least a slash after the hostname to be considered valid by the remote system.

For example:

auvik> http http://10.0.40.1/

http_message {
response {
  status: 401
  entity {
    content_type {
      main_type: "application"
      sub_type: "octet-stream"
    }
    data: "401 Unauthorized\r\n"
  }
  headers {
    key: "Date"
    value: "Mon, 30 May 2016 15:06:03 GMT"
  }
  headers {
    key: "Server"
    value: "cisco-IOS"
  }
  headers {
    key: "Connection"
    value: "close"
  }
  headers {
    key: "Accept-Ranges"
    value: "none"
  }
  headers {
    key: "WWW-Authenticate"
    value: "Basic realm=\"level_15_access\""
  }
  protocol: HTTP_1_1
}
}

How to capture and send collector logs

The agent-shell provides fuctionality for the user to upload logs from the host machine to a secure AWS S3 bucket controlled by Auvik. By using the copylogs command, the agent shell can automatically grab all collector associated log files within its log directory. These files are typically anything which has a file extension of .log or .log.N.gz (the latter being older files which have been automatically rotated and compressed).

auvik> copylogs
      ActorTable.log
      AuvikAgent.log
      AuvikAgent.log.0.gz
      AuvikAgent.log.1.gz
      AuvikAgent.log.2.gz
      AuvikWatchdog.log
      HealthCheck.log
       logs uploaded as https://s3.amazonaws.com/auvik-agent-logs/auviktestsite/52e6555f-fe26-4274-a194-57076c96f23d/2016.10-master-/logs-20161028T170258.zip

The URL for the log zip file can then be added to the relevant support ticket.

Was this article helpful?
16 out of 20 found this helpful
Have more questions? Submit a request