Networking: Connectivity Troubleshooting Guide

Step 1: DNS Resolution
Step 2: Basic Connectivity
Step 3: HTTP/HTTPS Connectivity
Step 4: Port Listening Check
Step 5: Port Accessibility Testing
Step 6: Network Route Tracing
Step 7: Advanced DNS Troubleshooting
Step 8: Process and Port Analysis
Step 9: Routing Table Inspection
Step 10: Authenticated API Uploads
Step 11: Firewall and Gateway Identification
Step 12: Internal DNS Server Verification
Step 13: Latency and Packet Loss Analysis
Quick Reference

Network debugging can be complex and time-consuming. This guide provides a systematic approach to troubleshooting network connectivity issues, from basic connectivity checks to advanced packet analysis.

Use these tools in order: start with basic connectivity tests, then move to more specific diagnostics as needed.

Step 1: DNS Resolution

First, verify that hostnames can be resolved to IP addresses. DNS issues are a common source of connectivity problems.

# Basic DNS lookup
nslookup <host>

# Example output showing successful resolution
nslookup rhino
# Server:         192.168.1.1
# Address:        192.168.1.1#53
#
# Name:   rhino.localdomain
# Address: 192.168.1.168

Step 2: Basic Connectivity

Confirm the target server is online and reachable from your source machine.

# Test basic network connectivity
ping <host>

# Test SSH connectivity
ssh <host>

# One-line SSH test without hanging
timeout 5 ssh -o ConnectTimeout=5 user@host "echo Connection successful" || echo "Connection failed"

nc -vz localhost 22

Step 3: HTTP/HTTPS Connectivity

Test web service endpoints to see detailed connection information.

# Verbose curl shows full connection details
curl -v https://example.com:443

# Test specific port
curl -v telnet://example.com:8080

# Include response headers only
curl -I https://example.com

Step 4: Port Listening Check

Verify which ports are listening on the target machine.

# Check if specific port is listening (modern Linux)
ss -tuln | grep :8080

# Alternative using lsof
lsof -i :8080

# Scan all ports on target (use responsibly)
nmap -p- TARGET

# Scan specific common ports
nmap -p 22,80,443,8080 TARGET

Step 5: Port Accessibility Testing

Use iperf3 to test if ports are reachable and measure bandwidth. This helps identify firewall blocks or bandwidth issues.

# On the server - start iperf3 in server mode
iperf3 -s -p 5201

# On the client - test connection to server
iperf3 -c <server-ip> -p 5201

# Example output showing successful connection
iperf3 -c rhino -p 5201
# Connecting to host rhino, port 5201
# [ ID] Interval           Transfer     Bitrate
# [  7]   0.00-1.00   sec  20.6 MBytes   172 Mbits/sec

# Test UDP instead of TCP
iperf3 -c <server-ip> -p 5201 -u

# Test multiple ports sequentially
for port in {5201..5210}; do
    echo "Testing port $port..."
    iperf3 -c <server-ip> -p $port -t 3
done

Step 6: Network Route Tracing

Identify the network path and potential bottlenecks between source and destination.

# Show all network hops to destination
traceroute <host>

# Example output - direct local network connection
traceroute rhino
# traceroute to rhino.localdomain (192.168.1.168), 64 hops max, 40 byte packets
#  1  rhino.localdomain (192.168.1.168)  3.722 ms  5.200 ms  3.759 ms

# Use ICMP instead of UDP (may bypass some firewalls)
traceroute -I <host>

# Specify maximum number of hops
traceroute -m 10 <host>

Step 7: Advanced DNS Troubleshooting

Use dig for detailed DNS information beyond basic nslookup.

# Basic dig query
dig google.com

# Query specific DNS server (useful for testing DNS server changes)
dig @8.8.8.8 example.com

# Show full resolution path from root servers
dig +trace example.com

# Query specific record types
dig example.com MX    # Mail servers
dig example.com TXT   # TXT records
dig example.com AAAA  # IPv6 addresses

# Short answer only
dig +short example.com

Step 8: Process and Port Analysis

Identify which process is using a specific port.

# Show all network connections
lsof -i
# find out what host to look at
ip addr show
# for instance bond-, lo0
sudo tcpdump -nni bond- host 192.168.1.1 and port 5567

# Show what's using a specific TCP port
lsof -i TCP:22

# Show what's listening on localhost
lsof -i TCP@127.0.0.1

# Alternative using netstat (older systems)
netstat -tunapl | grep :8080

# Show process using port with ss
ss -tulpn | grep :8080

Step 9: Routing Table Inspection

View your system’s routing configuration.

# Show routing table
route -n

# Modern alternative
ip route show

# Show network interfaces and MAC addresses
ip link show

# Show detailed interface information
ip addr show

Step 10: Authenticated API Uploads

Example of uploading files to authenticated services like Artifactory.

# Upload file with bearer token authentication
curl -H "Authorization: Bearer <Token>" \
     -T myfile.tar.gz \
     "https://artifactory/artifactory/<reponame>/path/to/file.tar.gz"

# Upload with basic auth
curl -u username:password \
     -T myfile.tar.gz \
     "https://artifactory/artifactory/<reponame>/path/to/file.tar.gz"

Step 11: Firewall and Gateway Identification

Identify network gateways and firewalls in your traffic path.

# View routing table to identify gateways
ip route

# Example routing table output:
# DEFAULT GATEWAY - all non-local traffic goes here
# default via 10.0.0.1 dev eth0 proto dhcp metric 100

# LOCAL LAN ROUTE - direct connection, no gateway
# 10.0.0.0/24 dev eth0 proto kernel scope link src 10.0.0.10 metric 100

# SPECIFIC SUBNET ROUTE - traffic to 172.16.x.x goes through 10.0.0.254
# This 10.0.0.254 is likely a firewall or router
# 172.16.0.0/16 via 10.0.0.254 dev eth0

# Test connectivity through the gateway
ping 10.0.0.254

# Trace route to destination behind firewall
traceroute 172.16.0.1

# Reverse DNS lookup on gateway
nslookup 10.0.0.254

# Check ARP table for MAC address
arp -n | grep 10.0.0.254

# Scan the gateway to identify device type
nmap -sV 10.0.0.254
# Output may indicate if it's a firewall, router, or other device

Step 12: Internal DNS Server Verification

Verify connectivity to internal DNS servers and their resolution capabilities.

# Check SSSD configuration for IPA servers
less /etc/sssd/sssd.conf

# Test DNS server reachability
nslookup url.ipa.net
# Example output: 10.0.0.0

# Query specific internal domain using the IPA server
dig @10.0.0.0 artifactory.internal.net

# Verify internal domain resolution
dig @10.0.0.0 +short artifactory.internal.net

Step 13: Latency and Packet Loss Analysis

Use mtr (My Traceroute) for continuous monitoring of network performance.

# Run mtr with report mode
# -r: report mode, -w: wide output, -n: no DNS resolution
# -i: interval in seconds, -c: number of pings
mtr -rwn -i 2 -c 5 <domain or ip>

# Example with DNS resolution
mtr -rw google.com

# Real-time interactive mode (exit with 'q')
mtr google.com

# Output columns explained:
# Loss%: Packet loss percentage
# Snt: Packets sent
# Last: Last packet latency (ms)
# Avg: Average latency (ms)
# Best: Best (lowest) latency (ms)
# Wrst: Worst (highest) latency (ms)
# StDev: Standard deviation of latency

Quick Reference

Tool	Purpose	Example
`nslookup`	DNS lookup	`nslookup example.com`
`dig`	Advanced DNS queries	`dig +trace example.com`
`ping`	Basic connectivity	`ping -c 4 example.com`
`traceroute`	Route tracing	`traceroute example.com`
`mtr`	Continuous route analysis	`mtr -rw example.com`
`curl`	HTTP/HTTPS testing	`curl -v https://example.com`
`ss`	Socket statistics	`ss -tuln`
`lsof`	Process port usage	`lsof -i :8080`
`nmap`	Port scanning	`nmap -p 80,443 example.com`
`iperf3`	Bandwidth testing	`iperf3 -c example.com`