WarrenTyson.com | TysonConsulting.LLC
Troubleshooting
Vendors: Cisco, Microsoft, VMWare etc...
Authored By: Warren Tyson
Tyson TAC Cases
لا تنزعج إذا أنقلبت حياتك رأساً على عقب ، فكيف تعرف أن الجانب الذي اعتد ت عليه ، أفضل من ذلك الجانب الذي سوفيأتي. - جلال الدين الرومي
Cisco ACI APIC's unable to discover each other due to Kubernetes/Docker Subnet overlap
Date:11/2022
Vendor: Cisco
Technology: Cisco ACI
Description Below:
Observed Error
If there is a 3 APIC Cluster all the APICs will come up but APIC2 and APIC3 will not be able to fully sync with APIC1 without getting a configuration replication error. When the configuration is commited on APIC1, the subsequent configuration changes will not be replicated to APIC's 2 and 3.
Error Configuration
During the intial bootstrapping of the APIC's if an engineer uses subnet 172.17.0.0/16 as the infrastructure subnet then the Cisco APIC's will not be able to fully discover all the APIC's in the cluster.
Resolution
The resolution is use a different infrastructure subnet outside of the 172.17.0.0/16 which is the default kubernetes subnet used for kubernetes to integrate with Cisco ACI.
Additional Details
There are mulitple ways to resolve this issue. You can alternatively change the default subnet default Kubernetes subnet or alternatively change the infrastructure subnet during the APIC bootstrapping process.
Related Links
- https://learn.microsoft.com/en-us/azure/aks/configure-kubenet
- https://medium.com/patrik-bego/docker-networking-practical-examples-23900904486e
ZTP Testing with iPXE/Bash Scripting
Date:4/2022
Vendor: Cisco
Technology: IOS XR Automation/Bash Scripting/PXE
Description Below:
Observed Error
8201 XR router not able to be upgraded through ZTP process
Error Configuration
A bug was identified in the code version 7.3.3. Code version 7.3.3 sends the wrong configuraton string to the router. The expected string is xr-config but the router sent the string exr-config.
Resolution
The long-term solution is to upgrade the 8201 XR to the latest version code, but the temporary workaround to resolve the issue in the short-term was to change the bash script to use string exr-config
Additional Details
This case was related to scripting an automation script using bash sripting. The script consisted of creating multiple bash functions that were called from the main bash script. The functions included a upgrade and downgrade function along with a version function which compared the current running version of code to the desired code version and if the code’s differed then the ugprade process was performed. Upon completion of the upgrade then the router would be reboot and then the script would run on the upgraded version of code. The code version would now be on the desired version of code and then a final configuration would be applied to the device to provide the router with it’s Day 0 configuration. The key componenents to the ZTP process is to have a router that supports PXE boot or more specifically iPXE and a Red Hat Linux server running DHCP(dhcpd) and HTTP(httpd). The bug was found during the setup related to the DHCP config string that was sent to the DHCP server. Instead of sending DHCP string exr-config the router sent string xr-config which was not indicated in the release notes for code version 7.3.3 for 8201 XR. Another interesting part of this implementation was that the DHCP server was coded with if/else statements and the bash script was actually passed to the router through the DHCP process telling the routing which folder on the Linux server where the router could located the script file to run in order to upgrade the system and find it’s configuration file.
Related Links
- ZTP configuration example 1
- https://iosxr-lab-ciscolive.github.io/LTRSPG-2414-cleur2019/wkinstructions/2019-02-01-step-3-ios-xr-ztp-bash-and-python-hooks/
Exchange 2010 Outlook Anywhere replaces RPC static configuration
Date:2/2023
Vendor: Microsoft
Technology: Outlook/Windows Server 2008 R2/Exchange 2010
Description Below:
Observed Error
Port 443 and Port 135 along with port range 6005-59530 was required for external Outlook connectivity, but Microsoft documentation says that only TCP Port 443 is required.
Error Configuration
TCP Ports 443, 135, and 6005-59530 were permitted through the edge firewall.
Resolution
Configure Outlook Anywhere
Additional Details
443 should be the only port required for Outlook Anywhere. There is a bug with the outlook app on IPhone. The Outlook App requires port 443 be open on the edge firewall even if you only plan to the use the Outlook APP internally only. Proper configuration for Outlook Anywhere resolves the issue of port 135 being opened externally on the edge firewall. You have to use a different mobile app to connect to your internal exchange server if port 443 is denied externally to the mail server.
Related Links
- https://practical365.com/how-to-configure-exchange-server-2010-outlook-anywhere/
- https://social.technet.microsoft.com/wiki/contents/articles/864.exchange-2010-client-access-server-configure-static-rpc-ports.aspx
- https://www.xfinity.com/support/articles/list-of-blocked-ports
Dell PowerEdge T610 - Memory Module Installation Order
Date:4/2023
Vendor: Dell
Technology: Server/Compute memory configuration
Description Below:
Observed Error
Server reports invalid memory configuration error in the POST Operation(Unsupported memory configuration).
Error Configuration
Memory slots A1, A2, A3, and A4 are populated with 4 memory modules and both A5 and A6 are empty.
Resolution
Per Dell's documentation: "memory modules are installed in the numeric order of the sockets beginning with A1 or B1."
Additional Details
So you can't install memory modules in sequential order from A1-A6, you have to install the memory modules in the order of sockets from socket 1 to 3. So you have to populate DIMM slots A1 and A4 first, and then A2 and A5 and then A3 and A6. So the DIMM order is actually A1, A4, A2, A5, A3, and A6. So if I have 2 memory sticks then A1 and A4 get populated, and if I have 3 memory sticks then A1, A4 and A2 get populated.
Related Links
- https://www.dell.com/community/PowerEdge-Hardware-General/T610-mixing-16GB-RDIMMS-and-4GB-RDIMMS/td-p/4180494
- PowerEdge T610 Owner's manual
- https://www.ebay.com/sch/i.html?_from=R40&_trksid=p2047675.m570.l1313&_nkw=192GB+12x+16GB+PC3L-8500R+RDIMM+Dell+PowerEdge+T610&_sacat=0
No Spanning Tree(STP) due to MLAG, VPC Inconsistency
Date:9/2023
Vendor: Arista
Technology: Multi-Chassis Link Aggregation - (MLAG)
Description Below:
Observed Error
Spanning Tree Protocol does not exist or nonoperational for MLAG VLAN
Error Configuration
Only one of the MLAG Peers has the layer 2 VLAN configured.
Resolution
Configure the layer 2 VLAN on each MLAG Peer
Additional Details
Use the show mlag command to verify consistency checks for the MLAG trunk interfaces and for the allowed vlans on the MLAG interfaces.
Related Links
- Troubleshoot VPC inconsistencies
- https://www.arista.com/en/um-eos/eos-multi-chassis-link-aggregation#xx1152438
Warren Tyson