Tuning a TCP Connection

I wrote this a long time ago so I don't remember the source or really the intent. I think it was about trying to maximize the throughput when running iperf over TCP.

TCP Background

The TCP Congestion Window

  • Larger window implies higher throughput
  • The max allowed is based on the buffer space the kernel allocates for each socket
  • Each socket has a default buffer size
  • Sender and receiver can both adjust the size

Optimal Buffer Size

  • Too small - Sender will be throttled
    • More likely if the Sender is faster than the receiver
  • Too big - Receiver might be overloaded and packets will be dropped
    • If the receiver has lots of memory, less likely to happen

Changing the Buffer Size

  • TCP chooses the smaller of both sides of the transaction.
  • Set the receiver high and let the sender negotiate

Picking a Buffer Size

The Maximum Throughput

If the network isn't congested the throughput is characterized by TCP buffer size and network latency.

\begin{equation*} max \; throughput \gets \frac{buffer \; size}{latency} \end{equation*}

Bandwidth Delay Product

This is a rule of thumb to calculate the optimal buffer size.

\begin{equation*} bdp \gets bottleneck \; bandwidth \times round \; trip \; time \end{equation*}
  • Get the rtt from ping.
  • Get bottleneck bandwidth from the theoretical maximum rate of the slowest link.

Iperf Tuning

Iperf Warnings

  • Iperf can set the buffer size up to a point – but the OS sets the upper limit on window sizes.
  • If the request is too high, iperf will use the maximum allowed and gives a warning.

Parallel Streams

  • Iperf lets you run multiple parallel sessions using the -P flag.
  • If the aggregate (SUM) is greater than a single stream, this is an indication that something is wrong – most likely one of:
    • The TCP window is too small
    • The OS implementation has bugs
    • The network has problems

Maximum Transmission Unit (MTU)

  • the most effective way to set it is if both hosts support Path MTU Discovery and set it themselves.
  • iperf's -m flag displays what mss is being used
  • mss: Maximum Segment Size – the maximum TCP segment size
\begin{equation*} mss \gets MTU - protocol \; headers \end{equation*}
  • Using -m is mainly to watch for the warning that the node lacks path mtu discovery