Network Troubleshooting

About the Picture

The picture you selected was taken in the lab setup located at the Department of Transportation Communications Building. Please forgive the mess, it was inherited and I have not taken time to clean it up since I am often on the run to the next fix. Covering two Statewide Networks has been a difficult task.

My current networks https://www.ok.gov/cio/documents/SolicitationConnectOklahomaAppendix1.pdf


Layer 1 - Physical Layer (PHY)

Missing any of the following tests when preparing your physical layer may lead to problems in the future.

Physical Mediums

Wireless

  1. Conduct a Path (Elevation/Obstruction) Analysis.
  2. Conduct a Frequency Analysis of the path and expected coverage area (AKA Spectrum Sweep).
  3. Determine frequency band and assignment.
  4. Verify grounding compliance.
  5. Conduct a path interference analysis (Intentional Radiators, Unintentional Radiators and Incidental Radiators).
  6. Calculate availability (MTBF and MTTR).
  7. Align antennas.
  8. Determine Equivalent Isotropic Radiated Power (EIRP).
  9. Determine Antenna Gain.
  10. (Optional) Determine theoretical max transmission.
  11. Test power transmission with power meter for carrier microwave.
  12. Conduct signal loss and interference analysis using spectrum analyzer or AirCheck testers.
  13. Calculate path loss (attenuation) over coverage area.
  14. For non-path coverage based connection perform signal coverage testing and produce heat-maps.
  15. Run throughput test with IPerf/JPerf.
  16. Perform simulation (load) testing.
  17. Perform in-motion testing.
  18. Stress testing https://tools.ietf.org/html/rfc2544 in a non-production environment.

Fiber Optics

  1. Proper splicing technique IAW with ITU-T
  2. Clean connectors.
  3. Scope connectors.
  4. (Optional) Determine theoretical max transmission.
  5. Test power transmission with power meter
  6. View loss analysis or breaks with a Visual Fault Locator (VFL) or with Optical Time Domain Reflectometer (OTDR).
  7. Calculate path loss (attenuation).
  8. Perform simulation (load) throughput testing using IPerf/JPerf .
  9. Stress testing https://tools.ietf.org/html/rfc2544 in a non-production environment.

Twisted Pair Copper (UTP/STP)

  1. CompTIA 568C compliant connectorization for RJ45/CAT level or T1/E1 RJ48C
  2. (Optional) Determine theoretical max transmission.
  3. (Optional) Check for breaks with tone/pulse generator with toner probe and/or Time Domain Reflectometer (TDR).
  4. Conduct acceptance/verification network transmission tests with cable tester.
  5. (Optional) Calculate voltage loss using a breakout box and multimeter (DMM).
  6. Calculate path loss (attenuation).
  7. Determine cross-talk (NEXT, FEXT) and return loss (RL).
  8. For T1 circuits test with Bit Error Rate Tester (BERT)
  9. Verify grounding compliance.
  10. Conduct a path interference analysis (Intentional Radiators, Unintentional Radiators and Incidental Radiators).
  11. Calculate availability (MTBF and MTTR).
  12. Perform simulation (load) throughput testing using IPerf/JPerf .
  13. Stress testing https://tools.ietf.org/html/rfc2544 in a non-production environment.

Coax

  1. Compliant connectorization for RG-6, RG-58, RG-59
  2. (Optional) Determine theoretical max transmission.
  3. (Optional) Check for breaks with tone/pulse generator with toner probe and/or Time Domain Reflectometer (TDR).
  4. (Optional) Check for distance to fault using Frequency Domain Relectometer (FDR).
  5. Conduct signal loss test using a spectrum analyzer.
  6. (Optional) Calculate voltage loss using a breakout box and multimeter (DMM).
  7. Calculate path loss (attenuation).
  8. Verify grounding compliance.
  9. Conduct a path interference analysis (Intentional Radiators, Unintentional Radiators and Incidental Radiators).
  10. Calculate availability (MTBF and MTTR).
  11. Perform simulation (load) throughput testing using IPerf/JPerf .
  12. Stress testing https://tools.ietf.org/html/rfc2544 in a non-production environment.

Serial (RS-232/RS-422)

  1. Determine connection pinout.
  2. Determine Data Communications Equipment (DCE)/Data Terminal Equipment (DTE)
  3. Serial Loopback Testing
  4. Determine connection settings.
  5. Connection testing using Hyperterminal or RS-232 program.
  6. Perform simulation testing.

Parallel


Functions

Transmission Modes

Negotiation

Bit Rate

Point to Point (P2P)

Point to Multi-point (P2MP)

Network Topology

Modulation

Line Coding

Bit Sync

Signal Flow Controls

Circuit Switching

Multiplexing

Carrier Sense

Collision Detection

Signal Processing

Forward Error Correction

Channel Coding

Layer 2 - DataLink Layer

Layer 3 - Network Layer

Layer 4 - Transport Layer

Layer 5 - Session Layer

Layer 6 - Presentation Layer

Layer 7 - Application Layer

https://www.nanog.org/


Troubleshooting MPLS

Troubleshooting VoIP/RoIP

General

  1. Scope: Endpoint or endpoints.
  2. Time dependent failure?
  3. Receive Packet Loss or Receive Packet Discard.
    • Packet loss or errors?
    • Duplex mismatch?
    • Network Congestion?
      • QoS enabled
      • Flow Analyzed
    • Transient flapping link?
    • Overloaded Control Unit or DSP?
    • High Jitter?
    • Packets out of order because route change?
  4. Monitor with Software
  5. Call quality. Mean Opinion Score (MOS)

Connection

RTP and RTCP https://tools.ietf.org/html/rfc1889

RTP FAQ http://www.cs.columbia.edu/~hgs/rtp/faq.html#transport

http://www.nojitter.com/post/240164279/troubleshooting-voip-packet-loss

http://www.voiptroubleshooter.com/basics/index.html

http://www.voipmechanic.com/aboutus.htm

QoS

http://www.cisco.com/c/en/us/td/docs/ios/solutions_docs/qos_solutions/QoSVoIP/QoSVoIP.html

https://touchtone.net/Business-Phone/Cisco-VoIP-QoS.pdf

https://infoproducts.alcatel-lucent.com/html/0_add-h-f/93-0267-HTML/7X50_Advanced_Configuration_Guide/QoS_arch.html

Call Quality

http://davidhoglund.typepad.com/integra_systems_inc_david/files/Finding_Fixing_VoIP_Call_Quality_Issues.pdf

https://www.netcor.de/download.php?file=1587

Tools

https://wiki.wireshark.org/VoIP_calls

https://wiki.wireshark.org/RTP_statistics

https://www.voip-info.org/wiki/view/How+To+Debug+and+Troubleshoot+VOIP

https://www.voipmonitor.org/

Proprietary

https://www.shoretel.com/sites/default/files/ShoreTel%20Premises%20Services%20Data%20Network%20Best%20Practices.pdf

http://www.w5dpd.org/Manuals/NXU-2/OverView/NXU2.pdf

http://www.psicompany.com/man-prod-info/Raytheon-JPS/Control-Equipment/NXU-2A/NXU-2A-Point-Multipoint-Multicast-Note.pdf

http://www.alcatelunleashed.com/viewtopic.php?t=24065

http://www.alcatelunleashed.com/viewtopic.php?t=16548

http://www.alcatelunleashed.com/viewtopic.php?t=16197

http://download.skype.com/share/business/guides/skype-connect-troubleshooting-guide.pdf

Troubleshooting SIP

PBX Software

Structured Cabling