Enough About tcpdump To Be Dangerous
At some point one may need to look at a network packet trace to help diagnose what is going on. Since this seems to be a fairly mysterious art and yet one that is useful some brief notes may be in order.
The tcpdump documented here is for OpenBSD 7.4. Other tcpdump may vary in the flags and features offered, and interface names vary. Check the fine manual for details.
First up, there are various graphical interfaces that make it easier to click and drill into a packet trace. Wireshark is typical here. The next question might be "but I cannot run wireshark on my server!" Excuses here include wayland not allowing remote X11, or not wanting to (or not being able to) install wireshark. tshark may be another option.
Anyways, a simple solution to wireshark's unavailability is to capture raw packets to a file, and then transfer that file somewhere that does have wireshark. The -s size option may need tuning to be as large as possible, but not so large that too many packets are lost. This may also depend on the system, how busy it is, etc.
Ideally only match the minimum necessary to avoid the capture file growing too large.
Note that regulations (SOX PCI DSS HIPAA blah blah) may frown on taking production data into development environments. A packet trace could capture problematic information.
Match Not Your Own Traffic
Do not capture the packets of your own connection, especially if your connection is active, perhaps because tcpdump is spewing all sorts of information which in turn generates packets that tcpdump matches and shows. This is something like sending speaker output back into the microphone. Avoid it by excluding your own connection from the packet trace.
Disabling DNS lookups in tcpdump may also cut down on traffic generated.
It may be prudent to limit the number of packets captured so that the trace or output file does not spiral out of control. Remove this option if you do need to run a long packet trace, like when you know it's a rare issue you're trying to find, and after you've tuned the dump to only match exactly what you want and you've confirmed that there is sufficient disk space available for the trace to happen.
So the above boils down to using "-c 42 -n" to limit the captured packets and to turn off DNS, plus a filter to exclude your own traffic. Here my SSH traffic actual goes over wireguard so that's what an unguarded tcpdump picks up. In other cases it may be SSH traffic, TCP/22 by default.
You may want to start with a very small packet count and increase it after you get the expression dialed in.
An easy way to exclude your traffic from the capture is to exclude your IP address. 104.207.156.138 is the server here (thrig.me), which I probably should have forged to some example IP (see RFC 3330) like I did for the client IP.
Another method is to narrow in on the protocol in question on the assumption that your SSH connection does not use that protocol.
Some protocols (DNS, in particular) use both TCP and UDP, while others only use TCP (Gemini) or UDP (DHCP). For DNS one thus needs to log both TCP and UDP, while others only need one of the other. Also there may be ICMP packets related to other network traffic that you might want to log; "tcp port 1965" would exclude ICMP traffic. In that case you may want to include ICMP traffic, or maybe restrict the capture to everything from a remote IP address, minus your own SSH connection.
The extra spaces are only to make the expression stand out more (the shell will remove them). You can do a lot more with tcpdump expressions, and wireshark has an expression language as well.
Protocol Inspection
Wireshark and other tools are able to decode various protocols. SSL and such may be tricky to decode, because they are encrypted, and if anyone could decrypt it to see what is going on it would not be very good encryption. (Governments try to include "Clipper Chips" or suchlike so they (but somehow never the bad guys) can decrypt all the traffic. This was not, is not, and will not be a good idea.) Anyways, if you have suitable private key material encrypted traffic may be be able to be decrypted. This is sometimes necessary when debugging a problem; another way is to log more things in the client and server software, but that may not be possible if the client or server software do not let you log something. Or maybe it's a black box from the Redmond Barrows—who knows what it is doing? And who can afford a support contract with them?
The tcpdump of a gemini request and response will not be very interesting; syn/syn+ack/ack and then a bunch of back and forths until the fins. The interesting part is the contents of the packets, which other tools can decode from the packet trace.
Bothsidesism
A packet trace may need to be taken on both sides of a problem; too often one side will claim "it's your firewall!" but then refuse to take a packet trace to show what their side sees. Maybe there's a load balancer in the middle that is forging packets? Or some firewall that everyone has forgotten about? With evidence from both sides of a connection attempt, any such forgery will be more apparent, or maybe the problem is on your side but only the information from the other end helps point you to what the issue is.
Firewall logging may also help show what is going on, but that's a different story.
tags #tcpdump #ssl