In this blog post, I'll explain what is going on, by showing how you can calculate your own latency issues.
The issue raised by Gettys, is that buffer-bloat (too big buffers on the network path) has fundamentally broken Internet broadband connections [2].
As buffer-bloat can introduce enough latency to cripple your line. Basically killing the possibility of interactive and realtime services, being delivered (e.g. by companies) over your broadband connection.
I'm very happy to see that Gettys is bring this issue up again.Back in 2005, I discovered the same issues as Gettys. I wrote my masters thesis[3] about the issue, and even created an Open Source "mitigation" solution the ADSL-optimizer[4]. It seems my solution has not gained a wider use.
The major contribution for my side, is to take the ADSL overhead into account when doing QoS packet shaping. (Everything is in mainline, just use/add the TC options "linklayer adsl" and "overhead", if you already have a Linux box doing QoS on your line.)
The issue is that:
A single TCP upload cause a delay of 1.2 seconds (on a 512 Kbit/s ADSL line)(see thesis[3] page 21).
Lets calculate what is happening, without going into details of why TCP/IP miss-behaves and cause queues to build (details are in my thesis[3]).
Before starting the calculations, here is a beautiful cite by Jim Gettys[1]:
"Large network buffers can be thought of as 'dark buffers', analogous to 'dark matter' in the universe; they are undetectable under many/most circumstances, and you can detect them only by indirect means. Buffers do not cause problems when they are empty. But when they fill they introduce additional latency (and create other problems, possibly very severe) to other traffic sharing the link."
Given the line speed and the delay, we can calculate the buffer size
(this is the bandwidth-delay product). Due to ADSL overhead the
effective bandwidth is actually 454 Kbit/s of the 512 Kbit/s line,
and the measured delay was 1138 ms.
454 Kbit/s * 1138 ms = 64581 bytesThis, corresponds to the TCP window-size. Thus, this is not the maximum buffer-size of the modem. (Use several TCP connection or UDP to find your maximum ping RTT, and calc your buffer size).
Where does the delay come from?!
The delay consists of different components, the important one in our case is the transmission delay (combined with the packets in queue).
The transmission delay of a 1500 bytes (MTU) packet is:
1500 bytes / 454 Kbit/s = 26.34 ms
Thus, the experienced delay is the time it takes to empty the packets in the queue, which is greatly dependend on the line speed.
E.g. 64000 bytes / 454 Kbit/s = 1127 ms.
With a RTT delay of 150 ms, your interactive SSH connection will feel sluggish.
(A side note on ADSL is that; the processing delay in the ADSL modem can get as large as 60 ms, and is caused by the interleaving depth, but its "fortunately" a constant fixed delay on the path)
Increasing the bandwidth, will reduce the latency, but its not the
solution. Besides, ADSL technology is often limited to a 1024 Kbit/s
upstream link.
The Point:
"ISPs SHOULD configure the buffer size based upon the link bandwidth"
I have a feeling that the ISP just configure a default queue size, and tune the queue size based upon max throughput on their largest product.
The line I did my measurements on, I could see delay on 3.3 sec, thus a buffer-bloat size of 187332 bytes (454 Kbit/s * 3300 ms), or 125 packets at 1500 bytes (MTU). Simply crazy!
For more details on formulas and calculation see thesis page 19 to 27.
Links:
[1] Jim Gettys: introducing-the-criminal-mastermind-bufferbloat
[2] Jim Gettys: whose-house-is-of-glasse-must-not-throw-stones-at-another[3] http://www.adsl-optimizer.dk/thesis/
[4] http://www.adsl-optimizer.dk/
[5] http://en.wikipedia.org/wiki/Jim_Gettys
http://netoptimizer.blogspot.com/2010/12/buffer-bloat-calculations.html
