What the… : Weirdness/Oddities Encountered in the Past Week(ish) #1

I have decided to start writing down some of the weird networking things I encounter in my day job. I am hoping that it helps others fix issues in their own environments; or at least give you a chuckle. 🙂

/31 Gotchas on Cisco\Viptela Equipment

  • Weirdness: I ran into this last fall & then promptly forgot about it. /31 subnet masks are supported on the Cisco (formerly Viptela) SD-WAN gear and have worked great for me for the past 18 months. Except if you are using vEdge 5000s. vManage will let you configure the interfaces with a /31 and the devices will accept the config, but they will not pass any traffic.
  • Investigation: I haven’t seen this issue on vEdge 100s, 1000s, or any converted ISRs. Just on vEdge 5000s
  • Fix: Changed the subnets to /30s and everything works. I can ping it now, so it has to be rock solid…

Routing issues for specific PI IPv4 address space

  • Weirdness: Thanks to a predecessor in the 90s at $DayJob, we have a good amount of provider independent IPv4 space (especially for a mid-sized company). We ran into an issue where some users trying to access a SaaS application from one location (and one PI block) would get constant timeouts. If those same users used their cell phones, the application responded immediately. To cause further confusion, users at a second location (and second PI block) could access the application without issue.
  • Investigation: Using WinMTR we tracked the path from our location (in the USA) to the SaaS application (hosted in Sweden). It appeared that somewhere in the UK we started having 50-80% packet loss. Performing this same test from the second location (and the adjoining /24 block), we did not see the same issue.
  • Fix (sorta): Using a centralized traffic data policy on our Cisco SD-WAN equipment, we took the SaaS provider’s /17 network and set it to route out of a different Internet circuit at the first location, which was not using our PI address space. As soon as this policy was pushed to our vSmart(s) and vEdge(s), the webpage started responding immediately.
vManage Data Policy Rule
vManage Data Policy Rule

IP Phones dropping out

  • Weirdness: Two call center employees reported that the screens on their Cisco IP Phone went dark momentarily and they were kicked out of Cisco Finesse multiple times over a 2 hour period.
  • Investigation: Looking at the switch logs, the two associated switch ports did not log any up/down events, nor did they log a PoE removal/granted. Also, the two switch ports were on separate switches in the switch stack. Talking to the users, the phones were not going through a full reboot process. The screens were just going dark and then coming back. For one user, we replaced their patch cable & then complete phone, but they still had the issue. But, while going through the switch logs, we came across another switch port; again, on an entirely different switch in the stack; that was logging frequent PoE events.
14:37:12.163 UTC: %ILPOWER-5-POWER_GRANTED: Interface Gi3/0/27: Power granted (3750SWITCH-3)
14:37:12.691 UTC: %ILPOWER-5-IEEE_DISCONNECT: Interface Gi3/0/27: PD removed (3750SWITCH-3)
14:37:28.778 UTC: %ILPOWER-5-IEEE_DISCONNECT: Interface Gi3/0/27: PD removed (3750SWITCH-3)
14:37:45.254 UTC: %ILPOWER-5-IEEE_DISCONNECT: Interface Gi3/0/27: PD removed (3750SWITCH-3)
14:38:01.553 UTC: %ILPOWER-5-POWER_GRANTED: Interface Gi3/0/27: Power granted (3750SWITCH-3)
14:38:01.905 UTC: %ILPOWER-5-IEEE_DISCONNECT: Interface Gi3/0/27: PD removed (3750SWITCH-3)
14:38:18.462 UTC: %ILPOWER-5-IEEE_DISCONNECT: Interface Gi3/0/27: PD removed (3750SWITCH-3)
14:38:34.702 UTC: %ILPOWER-5-POWER_GRANTED: Interface Gi3/0/27: Power granted (3750SWITCH-3)
14:38:34.828 UTC: %ILPOWER-5-IEEE_DISCONNECT: Interface Gi3/0/27: PD removed (3750SWITCH-3)
14:38:50.791 UTC: %ILPOWER-5-IEEE_DISCONNECT: Interface Gi3/0/27: PD removed (3750SWITCH-3)
  • Fix: We went to the desk that was logging the PoE events & there we found that the user had NO IP PHONE (DUN, DUN, DUUUUNNN). They were using a softphone on their PC and their PC was plugged into the logged port. They were also using a 25 foot patch cable to plug their PC in the 5 feet they needed to reach the wall port. We replaced the cable with a 7 foot patch cable and the switch stopped logging the PoE events & the other users stopped reporting the issues with their phones.

Runt Frame – Firepower Quick Tip – Management Interface & SNMP/Syslog

Runt frames are going to be some quick tips that I run into in my day to day life as a network engineer.

So, lets say that your preparing to migrate your firewalls to some shiny new ASAs or Firepowers running FTD mode, even though the Internet has tried to warn you off… (Reddit – Firepower Rant Part 1 & Reddit – Firepower Rant Part 2)

As part of your initial setup, you start to configure SNMP & Syslog, but to your horror you find that the system does not allow you to source the traffic from the management interface! It wants you to use a standard data interface, but you can’t activate any of those until you’re ready to complete the migration!

There is a workaround. But it’s not the cleanest.

You can use the “diagnostic” interface. This is a logical interface that shares the the physical management interface (at least on the ASA 5500-Xs). So put an IP address on the diagnostic interface (must be the same subnet used on the management interface), and then manually add the diagnostic interface to the SNMP settings under Platform Settings in FMC.

“But Justin,” you say, “It still doesn’t work!”

Yep. I ran into that myself. The diagnostic interface doesn’t utilize the default gateway that is configured on the management interface. You have to manually add routes for traffic from the diagnostic interface to your SNMP management stations. You can repeat this process if you want to do the same for Syslog traffic.


References