What TCP guarantees, and what it costs

A lesson from Wire Protocols & Applied Cryptography. Play it above, or read it through below.

• TCP delivers bytes. It does not preserve your message boundaries.

  The common read: A socket write is a message send, so three writes are three arrivals and the receiver's read loop sees the messages I sent.

  TCP is a stream of bytes, and your write calls define no boundaries in it: RFC 9293 guarantees "no correlation between the boundaries of TCP segments sent and received and the boundaries of the read or write buffers of user application data", so the stack may merge, split and re-chunk your writes anywhere on the path. UDP does not re-chunk: a datagram arrives whole or not at all, which is why the three UDP sends in the first screen arrived as three.

  So over TCP, framing is the application's job: a length prefix, a delimiter, or a self-describing encoding. A handler that assumes one read is one message passes on localhost and in staging, then fails the week a proxy is added between the services; the ticket says "the proxy is corrupting our JSON", and the packet capture shows every byte arriving intact, because the reader split one message across two reads.

  A length prefix is an encoding decision: it makes the stream parseable and authenticates nothing. The length is a number an attacker can choose, so check it against a maximum before you buffer that many bytes, or you replace a framing bug with a memory bug.

• A new connection costs round trips first. Bandwidth matters later.

  The common read: A fast server on a high-bandwidth link means a fast first response, so when a small response is slow the bottleneck is the server or the link.

  A fresh connection spends its first round trip on SYN, SYN-ACK and the ACK that may carry your first byte, so nothing you wrote reaches the server before one full round trip; RFC 7413's Fast Open saves "up to one full round-trip time (RTT)" by putting data on the SYN, but it is Experimental and some paths drop such SYNs.

  Then the sender gets an initial window, not the link rate: RFC 6928, also Experimental, sets it to ten times the maximum segment size, capped at 14600 bytes and floored at two segments, and RFC 5681 grows it by about one segment per acknowledgement, a doubling per round trip. Below about fourteen kilobytes you pay one round trip whatever the response size; above it you pay one more per doubling, and the link rate plays no part.

  A pooled connection skips the handshake, and keeps its grown window only if it sent data within the last retransmission timeout (RTO). After a longer idle period RFC 5681 says the sender SHOULD cut the window to at most the initial window, and Linux does by default (tcp_slow_start_after_idle = 1), halving it for each RTO of idle time, so a connection idle for a second or more starts again at the initial window.

  A ticket reading "the API is slow from Europe" gets a cache in front of the API and reopens next quarter, when someone notices p50 was always fine and p99 follows the pool's eviction rate. The fix is fewer new connections, and it has a cost the second half of this lesson covers: a pool holds sockets that can be dead while reporting themselves open.

• In-order delivery has a cost. QUIC narrowed its scope and did not remove it.

  The common read: HTTP/2 multiplexing removed head-of-line blocking, so one stalled response no longer holds up the others on that connection.

  HTTP/2 multiplexed requests onto one connection at the HTTP layer, but underneath is still one ordered byte stream: a receiver missing byte N may not deliver byte N+1, whichever request it belongs to, even though N+1 sits complete and correct in a kernel buffer. The HTTP/3 specification says so about HTTP/2 over TCP: "a lost or reordered packet causes all active transactions to experience a stall regardless of whether that transaction was directly impacted by the lost packet."

  So HTTP/2 kept head-of-line blocking: in-order delivery is the feature, and head-of-line blocking is its cost when a packet is lost. QUIC changes the scope, not the existence: RFC 9000 says QUIC "does not provide any means of ensuring ordering between bytes on different streams", so a loss on stream A stops stream A while stream B keeps delivering, and stream A's own later bytes still wait for the retransmission.

  That last clause explains a common disappointing migration: a large sequential download moved to HTTP/3 measures identical, and the team files a ticket against their CDN. RFC 9308 recommends putting data that must stay ordered on one stream and data that must not block on separate streams, so splitting data across streams is now your design decision.

• Nagle and delayed ACK are each correct. Combined, each waits for the other.

  The common read: The 40 ms stall is a TCP constant, so a fixed round delay on small request and response traffic means I have found the delayed-ACK timer.

  Nagle holds a small write while earlier data is unacknowledged, the 1984 answer to terminal sessions spending forty-one bytes per keystroke; delayed acknowledgement holds an ACK briefly so it can ride on data going the other way. Both are correct and on by default, and an application that writes a header, then a body, then waits for a reply creates the workload where each waits for the other.

  RFC 9293, a 2022 standard, concedes the problem in an appendix: "TCP is used by many applications that have a request-response style of communication, where the combination of the Nagle algorithm and delayed acknowledgments can result in poor application performance." RFC 9293 bounds the ACK delay only from above, "MUST be less than 0.5 seconds"; Linux fixes TCP_DELACK_MIN at HZ/25 and TCP_DELACK_MAX at HZ/5, 40 ms and 200 ms; this machine's macOS exposes only net.inet.tcp.delayed_ack = 3, a mode selector.

  So the quoted 40 ms is a Linux minimum remembered as a protocol constant, and the bug gets closed as flaky: it reproduces on the Linux build agent and not on the developer's laptop. Coalescing header and body into one write costs a buffer copy and removes the interaction; TCP_NODELAY costs you Nagle's protection, so avoiding tinygrams becomes your application's job.

• Nothing happening is not an event.

  The common read: A dead peer breaks the socket, so a connection my pool still lists as open is a connection something is still alive on the other end of.

  TCP sends nothing when nothing happens: RFC 9293 says a half-open connection "will automatically become reset if an attempt is made to send data in either direction", so detection is an action you take and never an event you receive. Killing the server PROCESS does not show this, because its kernel closes the socket for it: in five runs here, with the peer killed at 100 ms, the client's socket ended between 102 and 104 ms without the client writing a byte.

  The silent case needs the HOST or the path to disappear: a laptop that slept, a NAT box that evicted an idle mapping, a load balancer idle timeout. Your pool still reports the socket open, and the first request after the idle period gets a timeout instead of an error. Linux's tcp_retries2 defaults to 15, which the kernel documentation says yields a "hypothetical timeout of 924.6 seconds" and calls "a lower bound for the effective timeout": at least fifteen minutes.

  Keepalive does not help by default: RFC 9293 requires it off and at an interval "no less than two hours"; this machine has always_keepalive at 0 and keepidle at 7200000 ms. A ticket saying "the first call after lunch fails" gets a retry, which holds a worker for fifteen minutes. Send an application heartbeat more often than the shortest idle timeout on the path: it costs bytes per idle connection and proves what kernel keepalive cannot, that the peer's application is running.

• Handshake packets are small. Connecting proved nothing about size.

  The common read: Connectivity is binary, so a connection that established and moved some bytes has proved the path works.

  The handshake succeeds because its packets are small; path MTU discovery then sends a full-size do-not-fragment packet and waits for an ICMP "too big" reply, which many firewalls drop. The sender retransmits the oversized packet until its retransmission timeout gives up, which RFC 2923 described in 2000 as "a connection that connects and starts to transfer, then eventually terminates after 15 minutes with zero bytes transfered".

  That is close to Linux's retransmission timeout from earlier, though RFC 2923 does not name the timer. Tunnels lower the path MTU, so a VPN is where you will meet this: small GETs answer and every large POST hangs after its headers. RFC 4821's replacement probes with larger packets and learns from its own acknowledgements; Linux ships it for TCP as tcp_mtu_probing at 0, disabled, and RFC 8899 defines it for datagram transports such as QUIC.

  QUIC "MUST NOT be used if the network path cannot support a maximum datagram size of at least 1200 bytes", pads its first datagram to 1200, and has the server discard anything smaller. QUIC's initial congestion window is limited to 14,720 bytes against TCP's 14,600, and RFC 9002 attributes the 120 bytes to UDP's 8-byte header against TCP's 20-byte one: RFC 6928's analysis recalculated. BCP 145 makes UDP senders run their own congestion control, so ask what a new transport moves onto you.