Writing · August 2026 · 9 min read
Three planes and a €4 box: how MoonLite's netcode holds 480 players without a real server
Design note, August 2026. MoonLite (moonlite.gg) is a multiplayer 3D “after-hours park” that runs in the browser. I built it alone; this is the part of the design I’d defend hardest, and the part I got wrong first.
The problem I was actually solving
Every “lots of people in one 3D room” project I’ve looked at dies the same way, and it isn’t payload size. It’s two things compounding:
- Per-event fan-out is O(n²). One player moves, the server tells everyone, everyone moves, the server tells everyone. At 20 players that’s nothing. At 200 it’s the whole CPU.
- Crowd traffic shares a channel with the thing that must not glitch. In MoonLite that thing is music: players mix audio stems on a shared timeline and everyone hears the same beat at the same instant. If an emote storm can delay a mix command, the product is broken.
So the constraint I wrote down on day one was: per-event server work must be bounded by the tick rate, never by how hard clients send. Everything below falls out of that sentence.
Three planes that never share a buffer
Traffic is split into three planes with different guarantees. They go over the same WebSocket, but they are encoded, rate-limited and fanned out completely separately.
Control plane. Reliable, ordered, rare. JSON. “Stem 3’s gain is now 0.7”, “the session tempo is 124”, “this player owns that item”. Server-authoritative: the relay applies the message to the room’s session state and broadcasts the result. Per-player token bucket of 30 messages per second, which nobody legitimate gets near.
Presence plane. Lossy, binary, batched. Each client sends its position at a fixed 12 Hz, independent of frame rate, as an 8-byte frame: a type byte, int16 x, int16 z, a one-byte heading (256 directions is plenty for a voxel avatar) and int16 y for jumps. The world is 500 m square, so int16 gives ~8 mm precision, which nobody can see. The server doesn’t forward these. It collects them and, on its own 15 Hz tick (66 ms budget), emits one snapshot per player: 8 bytes of header plus 9 bytes per visible neighbour. “Visible” is decided by an area-of-interest spatial hash: 40 m radius, capped at 80 neighbours. Beyond that you simply don’t receive someone’s position. Clients interpolate between snapshots, so 15 Hz looks smooth at 60 fps.
After the first keyframe, snapshots are deltas: only players whose quantized state changed are included, with a full keyframe every 30 ticks (2 s) as a self-heal. When I added deltas I measured the unchanged data they removed: about 10.8 KB/s per client in a full room. That’s the cost of “just send everything” that nobody notices until they pay for egress.
Reaction plane. Emotes, jumps, waves. Lossy, batched per tick, 2 bytes each. Past the interest radius they collapse into aggregate counts (“14 people waved over there”), so a crowd cheering is one small message, not a thousand.
Everything in the presence and reaction planes has a 256 KB backpressure limit per socket: if a client can’t keep up, its lossy traffic is dropped, never queued. The control plane is never dropped. That asymmetry is the whole point.
The protocol has about 62 message types with hand-written Reader/Writer codecs and round-trip tests for every one of them. There’s one relay-side trick I’m fond of: when a client frame just needs to be re-tagged with the sender’s id and forwarded, spliceRelay patches the bytes in place instead of decoding and re-encoding.
Mini-games: no frames on the wire
The same principle governs the twelve attractions (chess, 8-ball, bowling, pinball, go-karts and so on). None of them stream state. Each is a pure, deterministic step(state, inputs) function in a shared package that the browser and the relay both import. The wire carries tick-stamped input operations and, for late joiners, a snapshot. Whack-a-mole’s entire late-join state is 54 bytes; the mole schedule is a pure function of seed, hole and tick, so there’s nothing else to send. The rides are even cheaper: a roller-coaster seat’s position is a function of a networked startedAt timestamp and the shared clock, so a ride costs zero bandwidth while it runs.
The side benefit is that the sims are trivially testable, and the tests have earned their keep: they caught a wrong Catmull-Rom coefficient in the kart track (the curve oscillated, and the depot sat half a metre off the centreline), a pinball ball that a slingshot kept re-energising forever, and a “toddler at the wheel” case where random steering input could wedge a kart into the barriers. The kart test literally flails the wheel for 45 simulated seconds and asserts the kart never escapes the track.
What the numbers say
I wrote a load bot that speaks the real binary protocol (guest join, 12 Hz random-walk movement, occasional reactions) and pointed it at a production-identical box: a Hetzner CX22, 2 vCPU / 4 GB, about €4 a month, reached through the actual Cloudflare and Caddy path.
| Load | Room shape | Worst tick (budget 66 ms) | Drops |
|---|---|---|---|
| 200 bots | 10 lobby rooms | ~1.5 ms | 0 |
| 200 bots | 1 event room, all moving | ~26 ms (relay flipped itself to “draining”) | 0 |
| 350 bots | 1 event room, all moving | 48–60 ms, loop max 135 ms. The knee. | 0 |
| 500 bots | 24 lobby rooms (full box) | 1.5–2.5 ms at 480/480 players | 20 over-cap refused cleanly |
Two conclusions, both of which changed the product:
- Lobbies are room-slot-bound, not CPU-bound. 480 players across 24 rooms leaves ~25× tick headroom. The box runs out of configured room slots long before it runs out of CPU.
- One all-active event room is good for ~300 players (more with realistic idling). I’d originally planned 1,000-player events. The data said a single room of 1,000 needs a different class of machine, and that one cheap box per ~500 players is the better shape. So I capped rooms (20 lobby, 100 event) and designed for many cheap boxes instead of one big one.
The relay watches itself: every 5 s it publishes whether it’s accepting (hot = load average per core > 0.8, or worst room tick EMA > 25 ms, or RSS > 1.2 GB, with hysteresis). A tiny director on the edge polls every box and places new players on the fullest accepting one, so small populations meet instead of scattering. When a box is hot it simply stops taking new rooms and keeps serving the ones it has. There is no live room migration and there never will be; rooms are transient, so “drain, don’t migrate” is both simpler and kinder to players. In a local run I pushed 800 bots into one event room: the drain reflex fired under real overload, and all 800 were still served with zero drops.
A real-player test with 24 browser tabs under maximum manual activity showed the other side of the ledger: a full 20-player lobby averaged 1.07 ms per tick, worst 11.8 ms, and the VPS CPU sat under 15%, most of it Caddy doing TLS rather than simulation.
The part I got wrong: media over P2P
Audio stems are the heavy bytes, and the rule from the start was media bytes never travel the relay. My first design moved them peer-to-peer: WebRTC data channels, perfect negotiation with glare rollback, a want/have/chunk protocol, a swarm with backpressure. I built all of it and shipped it for avatar skins. Three days later I deleted it.
Three reasons, in order of how much they hurt:
- NAT traversal rarely worked for my actual players. CGNAT and symmetric NATs on mobile carriers meant most pairs fell back anyway.
- It leaked peer IP addresses. In a social product where strangers share a room, that’s a security and privacy problem I’d have to document and defend.
- It bought nothing. The fallback was Cloudflare R2, which has zero egress cost. Once R2 carried every miss, P2P saved money I wasn’t spending.
The chain is now local IndexedDB cache → R2, keyed by content id (the first 16 bytes of the SHA-256 of the cleaned file, re-encoded from PCM so metadata can’t leak). A stem is a file fetched once per listener, then each client plays its local copy on the shared clock; it is never a 1→N stream. Transfers are gated on the 14 m listening radius, not the 40 m control radius, so you only download what you’ll actually hear.
The lesson I took: the thing to optimise was never bandwidth. It was what the server is allowed to do per event. Once that was bounded, the cheap box was enough, and the clever P2P layer was complexity I got to throw away.
What I’d tell someone building the same thing
- Decide which traffic may be dropped and which may never be, and give them different pipes. Don’t let the two meet in a queue.
- Quantize early. int16 positions and a fixed input rate made every later decision cheaper.
- Put the simulation in a pure function and test it like a library. Netcode bugs are mostly sim bugs with latency on top.
- Measure on the real box through the real path before buying a bigger one. My “scale” problem turned out to be a room-cap setting.
- Be willing to delete the part you’re proudest of. The P2P layer was the most interesting code in the project, and removing it made the product better.
The protocol, quantization, clock sync and all the game sims are open source: github.com/lectorsalamanca/moonlite-netcode. The load bot and these results: github.com/lectorsalamanca/ws-loadbot.
