Keep It Simple, Scale

Time and again I'm reminded of Rich Sutton's Bitter Lesson¹: the observation that scale always seems to eventually creep up and supersede heuristics, handcrafted ingenuity and inductive biases. While Sutton wrote his essay in the context of AI and ML, what's interesting is that this isn't just a philosophy from and about machine learning and function approximation. The core principles have come up before, and are applicable to a wide range of settings.

Introduction from the ML Perspective:#

In machine learning, the Bitter Lesson manifests through the lens of excess risk decomposition in function approximation.

Let $f^\star$ be the best possible predictor, $f_{\mathcal H}$ the best one inside your model class $\mathcal H$ , and $\hat f$ the one you actually fit from $n$ samples:

\underbrace{R(\hat f)-R(f^\star)}_{\text{excess risk}}=\underbrace{R(f_{\mathcal H}) -R(f^\star)}_{\text{approximation error}}+\underbrace{R(\hat f) -R(f_{\mathcal H})}_{\text{estimation error}}.

Growing $\mathcal H$ shrinks approximation error as the truth is more likely to live inside a bigger class. However, this potentially inflates estimation error as there's a bigger space to search and fit, requiring more data to pin it down. Adding data shrinks estimation error.

Handcrafting a clever space of functions to search is one way to keep approximation error low whilst also keeping the data requirements low. However, restricting the space also limits how expressive the function can be.

The Bitter Lesson that Sutton highlights is that scaling up our data and compute and searching in a more general class of functions is a better use of our time.

A Cosmic Treasure Hunt#

A cute (but imperfect) metaphor is to picture the search for $f^\star$ as hunting for treasure in outer space, and you are the captain of the spaceship.

The model class is the region you commit to searching — a single moon, a planet, a whole galaxy. Your fuel is the data and compute at your disposal: they determine how thoroughly you can comb whatever region you picked.

Improvements to your spaceship would be analogous to optimization improvements such as RMSNorm, AdamW, SWA, etc. These help reduce variance and better navigate the roughness of space.

There are two primary modes of failure.

(1) Approximation error: Searching a galaxy that doesn't contain the treasure.

(2) Estimation error: Searching a galaxy that contains the treasure, but insufficient fuel to comb it.

The whole lesson is in the asymmetry of returns: it's easier to scale up your fuel and search broadly than to research exactly where to look.

MSE and Experiments#

Classical mean-squared error decomposition of an estimator has an eerily similar look:

\mathrm{MSE}(\hat\theta_N) =\underbrace{\big(\mathbb E[\hat\theta_N]-\theta\big)^2}_{\text{bias}^2} +\underbrace{\mathrm{Var}(\hat\theta_N)}_{\propto \text{ 1/N}}.

Why is this relevant?

Because in experimentation settings (assuming niceties such as i.i.d. draws, no novelty effects, and so on), it would be ideal to make a call on the results of an A/B test as fast as possible for increased velocity. This usually means variance reduction, as variance determines the statistical power and the sample size needed.

Re-arranging helps us put into context the bias-variance trade off for a given estimator:

\mathrm{Var}(\hat\theta_N) = \mathrm{MSE}(\hat\theta_N) - \mathrm{bias}^2

Picking an estimator with more bias could help with variance reduction (though keep in mind that MSE itself also changes).

A canonical move then is to switch to estimators that trade a little bias for a lot of variance reduction. This is often a great deal (hello, regularization and Bayesian priors)! Alternatively, you can get clever and reduce variance without touching bias (e.g. CUPED, control variates, stratified sampling, etc).

While they're all great tools, implementing them in large systems and organizations carries a cost that doesn't appear in the decomposition: infrastructure changes, data pipelines rewiring, and the stakeholder alignment required to ship it. And when KPI accountability is on the line, bias is often a non-starter.

I've found that the simplest lever is usually the least technical one: ask for more time and more samples.

The incremental bidder in auctions#

Perhaps my favorite manifestation of this principle appears in auctions.

Revenue matters. When you're running billions of auctions, squeezing more out of each is a real priority. Aside from predictive model calibration and core bidding product improvements, the next biggest lever for the marketplace is to optimize the expected revenue of auctions by tuning reserve prices as shown by Myerson (1981)².

However, Bulow–Klemperer (1996)³ puts an interesting spin on that perspective. For a single item with $n$ bidders drawn i.i.d. from a regular distribution, the plain second-price auction with no reserve and $n+1$ bidders earns at least as much as the revenue-optimal auction with $n$ bidders.

\mathbb{E}[\text{Rev}_{\text{SPA}}(n+1)] \ge \mathbb{E}[\text{Rev}_{\text{Optimal}}(n)].

One more bidder in the simplest auction beats the smartest possible second-price auction without it. This isn't to say there's no utility in reserve price optimization; but if you are able to produce a more frictionless experience for advertisers to onboard and stick around, that'll probably have a greater long-term compounding effect.

Not a silver bullet#

As highlighted in the examples above, scaling sample sizes and even strategic agents can prove to be low-cost and effective solutions. However, sometimes you genuinely can't scale. The data, compute, or time just isn't there.