The Impact of Effect Size Heterogeneity on Meta-Analysis: A Monte Carlo Experiment

Tinbergen Institute Discussion Paper No. 07-052/3

28 Pages Posted: 17 Jul 2007

See all articles by Mark J. Koetse

Mark J. Koetse

Free University of Amsterdam - Department of Spatial Economics

Raymond J.G.M. Florax

Purdue University; VU University Amsterdam - Department of Spatial Economics; Tinbergen Institute

Henri L. F. de Groot

VU University Amsterdam - Department of Spatial Economics; Tinbergen Institute

Date Written: July 2007

Abstract

In this paper we use Monte Carlo simulation to investigate the impact of effect size heterogeneity on the results of a meta-analysis. Specifically, we address the small sample behaviour of the OLS, the fixed effects regression and the mixed effects meta-estimators under three alternative scenarios of effect size heterogeneity. We distinguish heterogeneity in effect size variance, heterogeneity due to a varying true underlying effect across primary studies, and heterogeneity due to a non-systematic impact of omitted variable bias in primary studies. Our results show that the mixed effects estimator is to be preferred to the other two estimators in the first two situations. However, in the presence of random effect size variation due to a non-systematic impact of omitted variable bias, using the mixed effects estimator may be suboptimal. We also address the impact of sample size and show that meta-analysis sample size is far more effective in reducing meta-estimator variance and increasing the power of hypothesis testing than primary study sample size.

Keywords: Effect size heterogeneity, meta-analysis, Monte Carlo simulation, fixed effects regression estimator, mixed effects estimator

JEL Classification: C12, C15, C40

Suggested Citation

Koetse, Mark J. and Florax, Raymond J.G.M. and de Groot, Henri L.F., The Impact of Effect Size Heterogeneity on Meta-Analysis: A Monte Carlo Experiment (July 2007). Available at SSRN: https://ssrn.com/abstract=1001159 or http://dx.doi.org/10.2139/ssrn.1001159

Mark J. Koetse (Contact Author)

Free University of Amsterdam - Department of Spatial Economics ( email )

De Boelelaan 1105
Amsterdam, 1081 HV
Netherlands
+31(0)20 598 6168 (Phone)
+31(0)20 598 6004 (Fax)

Raymond J.G.M. Florax

Purdue University ( email )

West Lafayette, IN 47907-1145
United States
(765) 494-4300 (Phone)
(765) 494-9176 (Fax)

HOME PAGE: http://web.ics.purdue.edu/~rflorax/

VU University Amsterdam - Department of Spatial Economics ( email )

De Boelelaan 1105
1081HV Amsterdam
Netherlands
+31 20 444 6092 (Phone)
+31 20 444 6004 (Fax)

Tinbergen Institute

Burg. Oudlaan 50
Rotterdam, 3062 PA
Netherlands

Henri L.F. De Groot

VU University Amsterdam - Department of Spatial Economics ( email )

De Boelelaan 1105
Amsterdam, 1081 HV
Netherlands
+31 20 598 6168 (Phone)
+31 20 598 6004 (Fax)

Tinbergen Institute

Gustav Mahlerplein 17
Amsterdam, 1082 MS
Netherlands

Here is the Coronavirus
related research on SSRN

Paper statistics

Downloads
87
Abstract Views
1,023
rank
304,669
PlumX Metrics