A neural compressor spends most of its life on one job: assign a short code to the next byte given everything before it. Cross-entropy is the right score for that job. Train it well and the easy structure disappears first — the runs, the templates, the stretches that were never really random. What remains is sparse. A few positions still cost a lot of bits. The rest of the file is already cheap.
That is the saturation point. Not failure. Saturation.
More of the same stops paying
The reflex past that point is familiar: a larger model, a longer context, another pass on the same objective. Sometimes it moves the needle. Often it does not, because the leftover cost is not "the network has not seen enough of this distribution." It is that some local structure is not sequential prediction at all. It is a table lookup, a checksum, a transform, a rewrite that either is exact or is nothing.
Spending more gradient on those positions is how you burn compute without buying capacity. The plant still pays for every bit that leaves the encoder. If the net cannot name those bits cheaply, another layer on the same loss rarely can either.
The residual is an engineering surface
Treat the high-cost positions as a catalog, not a mood. Where they sit. What they cost in bits. What sits around them. Sort by cost. The distribution is usually uneven: most of the file is already well predicted; a thin set of residuals carries most of what is left.
That catalog is the brief for a second kind of work. Beside the predictor, keep a small set of exact operators the decoder can run for free or near-free: lookups, arithmetic that does not guess, verified rewrites that reconstruct the input byte for byte. The network's job becomes deciding when to step outside its own distribution and which operator to call — not inventing entropy that is not there.
Two proofs stay in force. Every candidate still has to round-trip exactly. Decode still has to be cheap enough to place where the data already lives. An operator that wins ratio and loses either proof is not capacity. It is a lab result.
Capacity is the stack
Compression is capacity only if the smaller representation is the one you keep and the one you read. A saturated predictor is not the end of that argument. It is the moment the stack has to widen: prediction for what prediction owns, and exact, cheap machinery for what it does not.
The lead still has to be allowed to move. Next year's operators, next year's selection policy, same portable decode path. Size the hall for the uncompressed residual and you will overbuild for bits a short exact step could have removed. Measure those bits first. Then decide what the plant still needs to carry.