Service status

Insights

Making one download fast: a TCP buffer story

Our Teraco server had a 10 Gbps port, yet one download from London ran at about 170 Mbps. The cause was one Linux setting, and the fix more than doubled it.

  • The GOLDNET team
  • Published
  • 5 min read
  • Network

On 10 Oct 2026 we had a puzzle. One of our servers in Teraco's Durban data centre sits on a 10 Gbps network port, next to multi-gigabit IP transit and a 10 Gbps port into the NAPAfrica Durban internet exchange. Yet some downloads to it were slow. Not broken, just slow, and only sometimes.

This is the story of how we found the cause, and why it matters to anyone who runs a Linux server, on our cloud or anywhere else.

First, rule out the obvious

We started with the pipes. Was anything full or faulty?

  • The server's network card ran at 10 Gbps with no errors to speak of.
  • The transit link had plenty of headroom at the evening peak.
  • The internet exchange port was lightly used.
  • Our edge router's processor was mostly idle, and nothing on the path was shaping traffic.

Then a multi-stream speed test from the server to two local Durban test servers: about 3 to 4 Gbps down, with no loss and well under a millisecond of delay. The network was fine. So why would a single download crawl?

The clue: one stream versus many

We downloaded a large test file from London, over one connection. It ran at about 170 Mbps. One stream from New York: about 110 Mbps. Yet eight streams at once from a French server reached over 400 Mbps.

When many connections together are fast but each one alone is slow, the limit isn't the pipe. It's something that caps each connection.

Why distance caps a single connection

TCP, the protocol behind almost every download, doesn't send a whole file at once. It sends a chunk, waits for the other side to say "got it", and keeps a limited amount of data in flight. How much can be in flight is set by the receive window: in effect, how much the receiving computer has said it's ready to accept.

That gives a simple rule:

Tip: The most one connection can carry is roughly its window divided by the round-trip time.

From Durban, a round trip to London takes something like 170 milliseconds. To fill even 1 Gbps at that distance, about 21 MB has to be on the way at any moment. Locally, with a round trip under a millisecond, a small window is plenty. Over long distances, the window decides everything.

The culprit: a default from another era

Linux tunes the window itself, but only up to a ceiling, set by the net.ipv4.tcp_rmem setting. On this server the ceiling was the distribution's default: about 6 MB. Linux also keeps part of that buffer for its own bookkeeping, so the usable window is smaller still.

At 150 to 250 milliseconds, that caps one connection at roughly 130 to 170 Mbps, whatever the link can do. Exactly what we saw.

The default makes sense for a laptop or a small virtual machine. It doesn't for a server with a 10 Gbps port talking to the other side of the world.

The fix

We raised the ceilings with one small file, /etc/sysctl.d/90-goldnet-tcp.conf:

net.core.rmem_max = 67108864
net.core.wmem_max = 67108864
net.ipv4.tcp_rmem = 4096 131072 67108864
net.ipv4.tcp_wmem = 4096 65536 33554432
net.core.netdev_max_backlog = 16384
net.ipv4.tcp_mtu_probing = 1

and applied it with:

sysctl -p /etc/sysctl.d/90-goldnet-tcp.conf

That lets the receive window grow to 64 MB (and the send buffer to 32 MB) when a connection needs it. Linux still starts small and only grows the window for connections that can use it, so a server with thousands of light connections doesn't suddenly use more memory.

The result

Same tests, one connection each, ten seconds each, before and after:

Download fromBefore (6 MB ceiling)After (64 MB ceiling)
Londonabout 170 Mbpsabout 440 Mbps
New Yorkabout 110 Mbpsabout 275 Mbps

A longer 30-second London download then averaged about 235 Mbps, and held around 270 Mbps for most of the test. A look at the connection's internals confirmed that the receiver was no longer the limit.

Every virtual machine has its own kernel and its own copy of the setting, so the same day we checked our other servers too. They had the same 6 MB default, and we started rolling the same file out to them.

What's left, and what isn't worth chasing

After the fix, a single long-distance connection is limited by things outside our control:

  • The far end. Many download servers cap each connection on purpose. A server that gives 300 Mbps per connection to everyone won't give you more.
  • Loss on the long path. TCP backs off when packets are lost. Over thousands of kilometres, even a little loss keeps a single connection from filling a huge window.

So the way to go faster still is more connections, not more tuning. Download managers, aria2c -x 8, package tools and most streaming services already open several connections in parallel.

What this means for you

If you run a Linux server, on GOLDNET Cloud or anywhere else, check the setting:

sysctl net.ipv4.tcp_rmem

If the last number is around 6 MB (6291456) and your server moves big files to or from far away, the file above is a safe, well-tested change. On a GOLDNET Cloud server you control the kernel settings, so you can apply it yourself.

If you're testing your home internet, remember that one download from overseas is capped by the same physics, whatever your plan, and that the device's own settings matter too. Windows, macOS, Android and iOS tune their windows well for home use. For a fair test, use a nearby server: see How to run a fair speed test.

And the bigger lesson, for us: a fast pipe is only as fast as the settings at each end. Measure each layer before you blame the network, or before you buy a bigger one.

All insights

Talk to a person, not a call centre

Our teams in Gqeberha and Durban answer Monday to Friday, 09:00 to 18:00. Our WhatsApp assistant answers any time, and the network is monitored around the clock.