How to debug packet loss?


How to debug packet loss?



I wrote a C++ application (running on Linux) that serves an RTP stream of about 400 kbps. To most destinations this works fine, but some destinations expericence packet loss. The problematic destinations seem to have a slower connection in common, but it should be plenty fast enough for the stream I'm sending.

Since these destinations are able to receive similar RTP streams for other applications without packet loss, my application might be at fault.

I already verified a few things: - in a tcpdump, I see all RTP packets going out on the sending machine - there is a UDP send buffer in place (I tried sizes between 64KB and 300KB) - the RTP packets mostly stay below 1400 bytes to avoid fragmentation

What can a sending application do to minimize the possibility of packet loss and what would be the best way to debug such a situation ?


How to grep lines having specific format

1:

How to make shell output redirect (>) write while script is still running?
Don't send out packets in big bursty chunks. C++: Help with cin difference between Linux and Windows. Monitor memory usage of child process The packet loss is usually caused by slow routers with limited packet buffer sizes. php cli script hangs with no messagesThe slow router might be able to handle 1 Mbps just fine if it has time to send out say, 10 packets before receiving ananother 10, although if the 100 Mbps sender side sends it a big chunk of 50 packets it has no choice although to drop 40 of them.. Can't Install libcurl PHP on Ubuntu Linux Try spreading out the sending so this you write only what is necessary to write in each time period. How to generate one large dependency map for the whole project that builds with makefiles?If you have to write one packet every fifth of a second, did it this way instead of writing 5 packets per second.. Check linux distro name

2:

netstat has several usefull option to debug the situation.. First one is netstat -su (dump UDP statistics):.
dima@linux-z8mw:/media> netstat -su                                                       IcmpMsg:                                                                                      InType3: 679     InType4: 20     InType11: 548     OutType3: 100 Udp:     12945 packets received     88 packets to unknown port received.     0 packet receive errors     13139 packets sent     RcvbufErrors: 0     SndbufErrors: 0 UdpLite:     InDatagrams: 0     NoPorts: 0     InErrors: 0     OutDatagrams: 0     RcvbufErrors: 0     SndbufErrors: 0 IpExt:     InNoRoutes: 0     InTruncatedPkts: 0     InMcastPkts: 3877     OutMcastPkts: 3881     InBcastPkts: 0     OutBcastPkts: 0     InOctets: 7172779304     OutOctets: 785498393     InMcastOctets: 525749     OutMcastOctets: 525909     InBcastOctets: 0     OutBcastOctets: 0 
Notice "RcvbufErrors" and "SndbufErrors". Additional option is to monitor receive and send UDP buffers of the process:.
dima@linux-z8mw:/media> netstat -ua Active Internet connections (servers and established) Proto Recv-Q Send-Q Local Address           Foreign Address         State udp        0      0 *:bootpc                *:* udp        0      0 *:40134                 *:* udp        0      0 *:737                   *:* udp        0      0 *:mdns                  *:* 
Here you need to look at Recv-Q and Send-Q column of the connection you're interested. If the values high and don't drop to zero, than the process must not handle the load.. You must use these commands on sending and on receiving machine.. Also you must use mtr, which combines traceroute and ping - it pings each hop in route. This may detect a slow hop in your route. Run it on oth machines to check connectivity to the second one..

3:

RTP typically uses UDP, which is inherently lossy. Packets could be lost anywhere between sender and receiver, so local debug will show you nothing useful.. Obvious things to do:.
  • a: Reduce the overall data rate
  • b: Reduce the 'peak' data rate, by sending small packets more often rather than one huge chunk every few seconds. ie, REDUCE your UDP send buffer - maybe even to just 1400 bytes.
  • c: See if you must switch to a TCP variant of RTP.
If all else fails, WireShark is your friend. It will commit you a true picture of how enough data - and when is being sent by your app..

4:

You should try reducing the rate you send packets. A slow connection must mean all sorts of things, and endeavor to send it packets (small or large) at a high rate won't help..

5:

This may not be the answer you want, although if I had packet loss problems I'd try to switch my application to use TCP, and have most worries of packet loss taken off my mind..


62 out of 100 based on 47 user ratings 322 reviews

#