[needs-packaging] Bufferbloat: Ubuntu doesn't come with bufferbloat fixes

Bug #940541 reported by Emilio Wuerges
This bug affects 5 people
Affects Status Importance Assigned to Milestone
Simon Iremonger

Bug Description

The problem is better explained here:

Demonstration can be seen here:

A simple mitigation (not a real solution) I is this:

Create a file: /etc/network/if-up.d/buffer_unbloat

With the contents:
[ "$IFACE" != "lo" ] || exit 0
ifconfig $IFACE txqueuelen 10

Robert Roth (evfool)
tags: added: needs-packaging
Revision history for this message
Brian Murray (brian-murray) wrote :

*** This is an automated message ***

This bug is tagged needs-packaging which identifies it as a request for a new package in Ubuntu. As a part of the managing needs-packaging bug reports specification, https://wiki.ubuntu.com/QATeam/Specs/NeedsPackagingBugs, all needs-packaging bug reports have Wishlist importance. Subsequently, I'm setting this bug's status to Wishlist.

summary: - Bufferbloat: Ubuntu doesn't come with bufferbloat fixes
+ [needs-packaging] Bufferbloat: Ubuntu doesn't come with bufferbloat
+ fixes
Changed in ubuntu:
importance: Undecided → Wishlist
Revision history for this message
Emilio Wuerges (wuerges) wrote :

If someone tell me how, I can build a candidate package.

Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in ubuntu:
status: New → Confirmed
Revision history for this message
Elijah Lynn (elijah-lynn) wrote :

Can you explain more about how this mitigates bufferbloat, I suspect I have this issue on Ubuntu 12.04 and it gets pretty bad sometimes.

Revision history for this message
Emilio Wuerges (wuerges) wrote :

Here is a little big introduction from the source itself:

The bufferbloat problem arises from the fact that the delay in a TCP network under congestion is exponential to the size of the buffers. For that, in 1993 RED was devised: http://en.wikipedia.org/wiki/Random_early_detection
It tries to keep the size of the buffers small by randomly (and thus fairly) dropping packages .

The problem is that people didn't like to have their packages dropped and manufactures built routers and network cards with huge buffers and over time each layer of abstraction on the network added its own buffer. So today we have tons of buffers that we aren't even aware of.

This "fix" I sent mitigates the problem of buffer bloat by reducing the max size of the linux buffer. It does nothing about the underlying buffers on the hardware but simple testing shows that it reduces the delay in a local area network by a lot.

Revision history for this message
Emilio Wuerges (wuerges) wrote :

It probably also reduces the delay on the internet too, it is just harder to test.

Testing in a lan can be done by doing a SSH copy of a huge file from/to another pc in the lan in one terminal and in another terminal you do a ping to the same or to another address.

Changing the size of the txqueuelen of the network interface in use affects the delay.
For most fun, try doing it in a wireless network.

Revision history for this message
Emilio Wuerges (wuerges) wrote :

There is more stuff that can be done, according to this:

Also, TCP algorithm might also be at fault:

Revision history for this message
Manoj Iyer (manjo) wrote :

I talked to DaveTaht at UDS and we tested the effects of turning on bufferbloat fixes on wifi on my laptop and I saw very encouraging results. He gave me access to a script that needs to be added to /etc/networking/if-up.d/ that will set up wifi wlanXXX. The script could be modified such that it works for wlanXXX and ethXXX devs.

Changed in ubuntu:
assignee: nobody → Stéphane Graber (stgraber)
importance: Wishlist → High
milestone: none → ubuntu-13.04
Revision history for this message
Manoj Iyer (manjo) wrote :

This script could be modified slightly to handle wlan and eth devices.

Revision history for this message
Steve Langasek (vorlon) wrote :

Has this been discussed with kernel upstream? In the first instance, such changes should be applied in the kernel defaults, rather than overriding in userspace.

Revision history for this message
Dave Taht (dave-taht) wrote :

The bufferbloat project has been at the problem for a great deal of time now. As of Linux 3.6, a great deal of the bufferbloat problem on ethernet has been resolved, with the incorporation (and device driver support) of BQL (linux 3.3), fq_codel (linux 3.5), and TCP small queues (3.6)

At linux plumbers in august, there seemed to be consensus that one day fq_codel could replace pfifo_fast, but it needs a bit more work and testing in the field. Many of the core netdev people are running it, there are also tests going on within several large companies. Work on improving wifi further is beginning. So, seeking wider deployment, by default, will get us to where it could ultimately be turned on by default. Certainly the benefits of the code as it stands now are *quite* noticible under many workloads particularly on desktops and more than a few on servers.

Latency under load last year: http://www.teklibre.com/~d/bloat/pfifo_fast_vs_sfq_qfq_linear.png
Linux 3.6 PFIFO_FAST: https://github.com/tohojo/netperf-wrapper/blob/master/sample_plots/100m_pfifo_fast.png
Linux 3.6 FQ_CODEL: https://github.com/tohojo/netperf-wrapper/blob/master/sample_plots/100m_fq_codel.png

Kamal Mostafa, for example, did some testing at canonical earlier this year and described his reaction in a kernel meeting...


I am maintaining a simple script that turns it on, that perhaps could be shipped by default for the desktop configuration of ubuntu.


Regardless I encourage people to try fq_codel out, run it day and day out, as we do, and see if they notice a difference. They will. Particularly gamers.

Some backing reading material for the work:

Van Jacobson's presentation on codel and fq_codel: http://recordings.conf.meetecho.com/Recordings/watch.jsp?recording=IETF84_TSVAREA&chapter=part_3
Linux Plumbers BQL: http://linuxplumbers.ubicast.tv/videos/byte-queue-limits/
Linux Plumbers fq_codel: http://linuxplumbers.ubicast.tv/videos/codel-and-fq_codel-fighting-the-delays/

TCP small queues in LWN: http://lwn.net/Articles/507065/


I note that one of my other "causes" at the moment, is that the enormous improvements in network performance require, as a base, BQL support in the underlying device driver. The majority of intel ethernet devices are BQL-enabled, arm is lacking...


For the ongoing work into fixing routers, and cpe, which suffer far worse than servers and desktops do - see the cerowrt project.

Revision history for this message
Kamal Mostafa (kamalmostafa) wrote :

FYI, we're now shipping the new iproute package with fq_codel support bits in Ubuntu Raring as of iproute (20121001-1ubuntu1) I have also pushed that package to the bufferbloat PPA for 12.04 and 12.10: https://launchpad.net/~kamalmostafa/+archive/bufferbloat

Changed in ubuntu:
assignee: Stéphane Graber (stgraber) → nobody
Changed in ubuntu:
assignee: nobody → Simon Iremonger (ubuntu-iremonger)
Revision history for this message
Simon Iremonger (ubuntu-iremonger) wrote :

A lot of comments here seem outdated, however things have moved on in a good-way!

Taking reference of https://www.bufferbloat.net/projects/bloat/wiki/Linux_Tips/ -- would now seem that many of the bufferbloat fixes HAVE been implemented --

Upstream kernel changes have now introduced tcp_sack tcp_dsack seemingly.
Systemd, interestingly-enough, now enables fq_codel as default qdisc.
[See LP #1436945 ].

The remaining piece of this bufferbloat puzzle on desktop distributions, seems to be
turning on tcp_ecn fully so that it actually gets used on outgoing TCP connections
(with adaptive fallback now supported).

See related Issue: #1773157 -- Please support this and mark as affecting you as you see fit!
Also: https://github.com/systemd/systemd/issues/9087

Hope that helps!

Revision history for this message
Simon Iremonger (ubuntu-iremonger) wrote :

Launchpad issue link didn't come out -- LP #1773157

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.