In k8s, it’s often necessary to inspect the network communication of Pods, such as checking the connection status to an external service like MariaDB on port 3306 or ensuring a server inside the Pod is listening correctly. The netstat command is commonly used for this purpose. However, challenges arise when netstat is not installed in the container or when there is no direct access to the Pod. This post will guide you through an efficient method to achieve this.
1. Checking Pod network interface
Finding the Pod IP
First, you need to find the IP address of the Pod in question. Use the following command:
kubectl get pod -A -o wide | grep ns
Example output:
ns ns-8467c87dff-7dqcl 1/1 Running 1 2d4h 172.17.252.196 node-01 <none> <none>
ns ns-8467c87dff-8tcsw 1/1 Running 1 2d4h 172.17.252.174 node-02 <none> <none>
ns ns-8467c87dff-czgfg 1/1 Running 0 10h 172.17.252.233 node-03 <none> <none>
Checking the ARP Table
Next, find the ARP entry for the Pod’s IP to determine the MAC address:
ip neigh show | grep 172.17.252.196
Example output:
172.17.252.196 dev k8br0 lladdr 6e:34:4f:52:03:27 REACHABLE
Inspecting the FDB Table
With the MAC address from the ARP table, locate the forwarding database (FDB) entry to identify the egress interface:
bridge fdb show | grep "6e:34:4f:52:03:27"
Example output:
6e:34:4f:52:03:27 dev veth86ae2978 master k8br0
Checking the Egress Interface
Finally, verify the egress interface details:
ip link show veth86ae2978
Example output:
49: veth86ae2978@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master k8br0 state UP mode DEFAULT group default
Checking Pod Sessions with netstat
Finding the Container Name
First, locate the container running in the Pod:
crictl ps -name "ns"
Example output:
CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID
7fb491b4f182f node-01:30012/infra/ns@sha256:... 11 hours ago Running ns 1 9b8a0084bae48
2. Identifying the Container PID
Next, find the PID of the container process:
crictl inspect 7fb491b4f182f | grep pid
Example output:
"pid": 40870,
3. Executing netstat Inside the Pod
Use nsenter to run netstat in the container’s namespace:
nsenter -t 40870 -n netstat -anopt
Example output:
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name Timer
tcp6 0 0 :::8080 :::* LISTEN 40952/ns.bin off (0.00/0/0)
tcp6 0 0 :::443 :::* LISTEN 40952/ns.bin off (0.00/0/0)
tcp6 0 0 172.17.252.196:443 172.17.252.0:35942 TIME_WAIT - timewait (43.85/0/0)
4. Detailed Session Information with conntrack
To get detailed session information, use conntrack:
shCopy codensenter -t 40870 -n conntrack -L
Example output:
conntrack v1.4.5 (conntrack-tools): 1 flow entries have been shown.
tcp 6 5 CLOSE src=172.17.252.0 dst=172.17.252.196 sport=58160 dport=443 src=172.17.252.196 dst=172.17.252.0 sport=443 dport=58160 [ASSURED] mark=0 use=1
Conclusion
By following these steps, you can effectively inspect and troubleshoot Pod network communications in k8s. This method provides a reliable alternative when direct access or specific tools like netstat are unavailable inside the container.
For further reading and detailed explanations, you can refer to the k8s official documentation and other resources on DevOps with Kubernetes (Production-Grade Container Orchestration) (DevOps with Kubernetes).
