Non-Standard Errors

58 Pages Posted: 8 Dec 2021

Multiple version iconThere are 3 versions of this paper

Date Written: 2021


In statistics, samples are drawn from a population in a data-generating process (DGP). Standard errors measure the uncertainty in sample estimates of population parameters. In science, evidence is generated to test hypotheses in an evidence-generating process (EGP). We claim that EGP variation across researchers adds uncertainty: non-standard errors. To study them, we let 164 teams test six hypotheses on the same sample. We find that non-standard errors are sizeable, on par with standard errors. Their size (i) co-varies only weakly with team merits, reproducibility, or peer rating, (ii) declines significantly after peer-feedback, and (iii) is underestimated by participants.

Keywords: non-standard errors, multi-analyst approach, liquidity

JEL Classification: C120, C180, G100, G140

Suggested Citation

Menkveld, Albert J., Non-Standard Errors (2021). CESifo Working Paper No. 9453, Available at SSRN: or

Albert J. Menkveld (Contact Author)

Vrije Universiteit Amsterdam ( email )

De Boelelaan 1105
Amsterdam, 1081HV
+31 20 5986130 (Phone)
+31 20 5986020 (Fax)

Do you have negative results from your research you’d like to share?

Paper statistics

Abstract Views
PlumX Metrics