What TCP buys you, and what it costs

A lesson from Wire Protocols & Applied Cryptography. Play it above, or read it through below.

• TCP moves bytes. It never agreed to move your messages.

  The common read: A socket write is a message send, so three writes are three arrivals and the receiver's read loop sees the messages I sent.

  Nothing preserved your boundaries.

  TCP is specified as a stream of bytes, and the specification says out loud that your write calls have no standing in it: RFC 9293 guarantees "no correlation between the boundaries of TCP segments sent and received and the boundaries of the read or write buffers of user application data", which is a sentence about your code rather than about the network, and it licenses the stack to merge, split and re-chunk whatever you handed it, anywhere on the path, for any reason, forever.

  UDP is the other contract and it is worth naming as an absence rather than as a feature: there is no re-chunking, because there is nothing to re-chunk, a datagram arrives whole or not at all, which is why the three sends TCP fused arrived as three messages on the probe's control run.

  So framing is yours, in every TCP protocol you will ever write, and the only real decision is which one you took deliberately: a length prefix, a delimiter, or a self-describing encoding. The version of this that reaches a pager is a hand-rolled protocol whose data handler assumes one read is one message.

  It passes every test against localhost, passes staging, and then cuts a message in half the week a proxy lands between the two services, and it arrives as "the payment service is corrupting our JSON", filed against the proxy by an engineer holding a packet capture that shows every byte leaving intact and arriving intact, which is the frustrating part, because the bytes were never wrong.

  Take the price along with the fix. A length prefix is an ENCODING decision: it buys parseability and it buys nothing else, it authenticates nothing, and the length you just read is a number an attacker chose, so it is bounded before it is allocated against, or you have traded a framing bug for a memory one.

• The first bill is round trips. Bandwidth arrives late.

  The common read: A fast server on a fat pipe means a fast first response, so when a small response is slow the bottleneck is the server or the link.

  Bandwidth is not a term in the model yet.

  A fresh connection spends its first round trip on SYN, SYN-ACK and the ACK that may carry your first byte, so nothing you wrote has reached the server until one full round trip has elapsed, and RFC 7413's Fast Open exists to buy exactly that back, saving "up to one full round-trip time (RTT)" at the price of being an Experimental extension that puts data on a SYN, which some paths on the Internet drop.

  Then the second bill arrives, and it is the one seniors forget. The sender does not get the link, it gets an initial window: RFC 6928, also Experimental, sets it to ten times the maximum segment size, capped at 14600 bytes and floored at two segments, and RFC 5681 grows it by roughly one segment per acknowledgement, which is a doubling per round trip.

  Below about fourteen kilobytes the size of your response is irrelevant and you pay one round trip; above it you pay one more round trip per doubling, and the link rate has still not been consulted about anything. That is why the two facts a reader holds separately, "cold requests are slow" and "connection pooling helped", are the same fact: the pool is not saving a handshake, it is inheriting a window somebody else already paid for.

  It arrives as "the API is slow from Europe", filed against the API, closed by putting a cache in front of it, and reopened the next quarter by someone who notices that p50 was always fine and that p99 tracks the pool's eviction rate rather than anything the API does.

  The prescription is not a wider pipe, it is fewer new connections, and it has a price that the second half of this lesson charges in full: a pool is a set of sockets that can be dead while reporting themselves open, so buying back the round trips means taking on the job of noticing.

• In order is the bill. QUIC re-scoped it and never deleted it.

  The common read: HTTP/2 multiplexing removed head-of-line blocking, so one stalled response no longer holds up the others on that connection.

  HTTP/2 never touched this. Multiplexing put many requests onto one connection at the HTTP layer, and the layer underneath is still a single ordered byte stream, so a receiver missing byte N may not hand byte N+1 to the application no matter which request N+1 belongs to, and the bytes it is holding back are not late or damaged, they are complete, correct, and sitting in a kernel buffer.

  HTTP/3's specification states it against its own predecessor and the sentence is worth reading slowly, because it names the part that surprises people: over TCP, "a lost or reordered packet causes all active transactions to experience a stall regardless of whether that transaction was directly impacted by the lost packet." Ordering and head-of-line blocking are not two properties, they are one property named twice, once as a feature in the sales copy and once as an incident in the postmortem.

  QUIC changes the SCOPE of that property rather than its existence: RFC 9000 states that QUIC "does not provide any means of ensuring ordering between bytes on different streams", so a loss on stream A stops stream A, and stream B keeps moving, and stream A's own later bytes wait exactly as long as they always did.

  That last clause is the one nobody draws, and it is where the migration disappoints: a team moves to HTTP/3 for a large sequential download, measures it, finds it identical, and files a ticket against their CDN for not enabling the feature they are already using. Take the corresponding job with the win.

  RFC 9308 tells you to exploit the scoping deliberately, putting data that must be ordered on the same stream and data that must not block on separate ones, which means the partition is now a design decision you own, and no library will make it for you.

• Nagle is correct. Delayed ACK is correct. Together they wait.

  The common read: The 40 ms stall is a TCP constant, so a fixed round delay on small request and response traffic means I have found the delayed-ACK timer.

  Neither algorithm is malfunctioning. Nagle holds a small write while any previously sent data is still unacknowledged, which was the 1984 answer to a terminal session spending forty-one bytes to move one keystroke. Delayed acknowledgement holds an ACK briefly, hoping to send it back attached to data going the other way, which saves a packet on any workload where data does flow the other way.

  Both are correct, both are on by default, and an application that writes a header and then writes a body and then waits for a reply hands them the exact workload where each is waiting for the other to move first.

  The current TCP standard concedes the point in an appendix rather than in a footnote, verbatim: "TCP is used by many applications that have a request-response style of communication, where the combination of the Nagle algorithm and delayed acknowledgments can result in poor application performance." A 2022 standard telling you that its own 1984 optimisation is bad for the workload you are shipping is not folklore, and the number you will quote about it is where the real damage is.

  RFC 9293 bounds the ACK delay only from above, requiring that it "MUST be less than 0.5 seconds"; Linux's own header fixes TCP_DELACK_MIN at HZ/25 and TCP_DELACK_MAX at HZ/5, which is 40 ms and 200 ms; and this machine's macOS exposes no millisecond at all, only net.inet.tcp.delayed_ack = 3, a mode selector.

  So the famous 40 ms is a Linux floor memorised as a protocol constant, which is why this bug is closed as flaky: it is reproduced on the Linux build agent, cannot be reproduced on the developer's laptop, and the difference between the two machines is treated as noise rather than as the evidence. Price the two fixes honestly.

  Coalescing the header and the body into one write costs a buffer copy and removes the interaction outright; setting TCP_NODELAY costs you Nagle's protection, which means avoiding tinygrams is now a thing your application does on purpose.

• Nothing happening is not an event.

  The common read: A dead peer breaks the socket, so a connection my pool still lists as open is a connection something is still alive on the other end of.

  TCP sends nothing when nothing happens. That is the whole mechanism, and everything expensive about a stale connection follows from it: RFC 9293 says a half-open connection "will automatically become reset if an attempt is made to send data in either direction", and the load-bearing words are the conditional ones, because detection is an action you take and never an event you receive.

  This is why the demonstration everybody reaches for proves the opposite of what it is quoted as proving. Kill the server PROCESS and the host's kernel is still there to close the socket on its behalf, which is exactly what happened on this machine: five runs, the peer killed at 100 ms, the client's socket ending between 102 and 104 ms, and the client never wrote a byte.

  The silent case needs the HOST or the path to disappear rather than the process: a laptop that slept, a NAT box that evicted an idle mapping, a load balancer with an idle timeout that is in nobody's runbook. Then nothing is generated by anyone, your pool reports the socket open because open is the last thing anybody told it, and the first request after the quiet period buys a timeout instead of an error.

  That timeout is not folklore either, it is a number you can look up: Linux's tcp_retries2 defaults to 15, and the kernel documentation states the arithmetic itself, a "hypothetical timeout of 924.6 seconds" that it calls "a lower bound for the effective timeout", which is fifteen and a half minutes of a request that is going nowhere.

  Keepalive is the wrong instrument and its defaults say so: RFC 9293 requires it to default to off and to default to an interval "no less than two hours", and this machine agrees, always_keepalive at 0 and keepidle at 7200000 ms. It arrives as "the first call after lunch always fails", filed against the HTTP client library, closed by adding a retry, and reopened when the retry starts costing fifteen minutes of a worker.

  Send an application-level heartbeat on an interval shorter than the shortest idle timeout between you and the peer, and pay for it in bytes on every idle connection, in exchange for the one thing a kernel keepalive can never show you: that the peer's application is alive and its event loop is still turning.

• Handshake packets are small. Connecting proved nothing about size.

  The common read: Connectivity is binary, so a connection that established and moved some bytes has proved the path works.

  It connected because the packets that connect are small. Path MTU discovery is how a sender learns the largest packet the path will carry, and it works by sending a full-size packet marked do-not-fragment and waiting for a router to send back an ICMP message saying it was too big, so the entire mechanism depends on a diagnostic packet surviving an Internet full of firewalls configured to drop exactly that class of traffic.

  When it does not survive, the sender never learns, retransmits the same oversized packet forever, and produces the failure RFC 2923 described in 2000: "a connection that connects and starts to transfer, then eventually terminates after 15 minutes with zero bytes transfered". Those fifteen minutes are the retransmit ladder from two screens ago, seen from twenty-six years earlier.

  Tunnels lower the path MTU, so a VPN is where you will meet this, and it arrives as "uploads are broken on the VPN", filed against the upload endpoint, whose small GETs answer perfectly while every large POST hangs with the headers sent and nothing after them.

  The replacement that does not depend on ICMP exists and is off: RFC 8899 has the transport probe with progressively larger packets and confirm delivery with its own acknowledgements instead of trusting ICMP, and Linux ships tcp_mtu_probing at 0, disabled.

  QUIC deleted the failure mode by refusing to start below the size: it "MUST NOT be used if the network path cannot support a maximum datagram size of at least 1200 bytes", it pads its very first datagram to 1200 and has the server discard anything smaller, so the smallest packet on the connection is already the probe.

  Read that as the shape of every trade in this lesson rather than as a feature list, because nothing was actually deleted: QUIC's initial congestion window is limited to 14,720 bytes where TCP's from S5 is limited to 14,600, and RFC 9002 attributes that whole 120-byte difference, in its own sentence, to UDP's 8-byte header against TCP's 20-byte one, so it is RFC 6928's analysis with one term re-priced rather than a new cost model, and BCP 145 hands every UDP sender the congestion control TCP was doing for them.

  Read a new transport the way you would read a bill: not for what it removed, but for which line moved into your name.