Measuring the Frontier: How Figures of Merit Shape Innovation

Abstract

When a scientific or technical community confronts a new problem, it tends toward a shared figure of merit, such as a benchmark score, performance proxy, or model-evaluation task, so that scattered inventors can compare ideas and search the problem together. This paper argues that such a metric is a shared search representation: a simplification device for a problem landscape that no one can see whole. Yet because the communities that build these measures run on a currency of recognition, a metric that makes the innovation process comparable is also the channel through which attention and status flow. As a result, the coupling of metric to reward arises on its own, before any principal or third party decides that it should, because legibility is the reward channel. Over time, the representation can stop being what the community relies on to search with, instead serving as the target that it searches for. Furthermore, the metric can become embedded as a field paradigm, turning a subjective construct into a seemingly objective dimension of progress. Unlike metrics used in social settings, even if the community is not gaming the metric, the honest optimization of this proxy can narrow a field’s collective search, foreclosing the other directions that are not captured. I develop the mechanisms through which this occurs and illustrate them with the LINPACK benchmark. Taken together, because innovation communities largely run on recognition, the figure of merit that a field uses to explore the frontier ends up shaping where the frontier goes.

Keywords: collective search, problem representation, technological paradigms, metrics, benchmarks, innovation, attention, science of science