misc-models.RdBooks and articles about meta-analysis often describe and discuss the difference between the so-called ‘fixed-effects model’ and the ‘random-effects model’ (e.g., Cooper et al., 2009). The former term is (mostly) avoided throughout the documentation of the metafor package. The term ‘equal-effects model’ is used instead, since it more concretely describes the main assumption underlying this model (i.e., that the underlying true effects/outcomes are homogeneous, or in other words, that they are all equal to each other). The terms ‘common-effect(s) model’ or ‘homogenous-effect(s) model’ have also sometimes been used in the literature to describe this model and are equally descriptive.
Moreover, the term ‘fixed-effects model’ creates a bit of a conundrum. When authors use this term, they are really typically referring to the equal-effects model. There is however another type of model, the ‘real’ fixed-effects model, that is different from the equal-effects model, but now we would need to invent (unnecessarily) a different term to refer to this model. Some have done so or tried to make a distinction between the ‘fixed-effect model’ (without the s!) and the ‘fixed-effects model’, but this subtle difference in terminology is easily overlooked/missed. Using the term ‘equal-effects model’ avoids this confusion and is more informative.
However, the question then remains what the real fixed-effects model is all about. The purpose of this page is to describe this model and to contrast it with the well-known random-effects model.
Assume we have a set of \(i = 1, \ldots, k\) independent studies and let \(y_i\) denote the observed value of the effect size or outcome measure in the \(i\text{th}\) study. Let \(\theta_i\) denote the corresponding (unknown) true effect/outcome, such that \[y_i \mid \theta_i \sim N(\theta_i, v_i).\] In other words, the observed effect sizes or outcomes are assumed to be unbiased and normally distributed estimates of the corresponding true effects/outcomes with sampling variances equal to \(v_i\). The \(v_i\) values are assumed to be known.
The fixed-effects model is simply given by \[y_i = \theta_i + \varepsilon_i,\] where the \(\theta_i\) values are the (fixed) true effects/outcomes of the \(k\) studies. Therefore, the model ‘conditions’ on the true effects/outcomes and provides a conditional inference about the \(k\) studies included in the meta-analysis.
When using weighted estimation (the default in rma.uni when method="FE"), this implies that the fitted model provides an estimate of \[\bar{\theta}_w = \frac{\sum_{i=1}^k w_i \theta_i}{\sum_{i=1}^k w_i},\] that is, the weighted average of the true effects/outcomes in the \(k\) studies, with weights equal to \(w_i = 1/v_i\).
As an example, consider the meta-analysis by Bangert-Drowns et al. (2004) on the effectiveness of writing-to-learn interventions on academic achievement. The dataset (dat.bangertdrowns2004) includes the observed standardized mean differences (variable yi) and the corresponding sampling variances (variable vi) of 48 studies that have examined such an intervention. We can fit a fixed-effects model to these data with:
# copy data into 'dat'
dat <- dat.bangertdrowns2004
# fit a fixed-effects model
res <- rma(yi, vi, data=dat, method="FE")
res
# Fixed-Effects Model (k = 48)
#
# I^2 (total heterogeneity / total variability): 56.12%
# H^2 (total variability / sampling variability): 2.28
#
# Test for Heterogeneity:
# Q(df = 47) = 107.1061, p-val < .0001
#
# Model Results:
#
# estimate se zval pval ci.lb ci.ub
# 0.1656 0.0269 6.1499 <.0001 0.1128 0.2184The Q-test suggests that the underlying true standardized mean differences are heterogeneous \((Q(\text{df}=47) = 107.11, p < .0001).\) Therefore, if we believe this to be true, then the value shown under estimate is an estimate of the inverse-variance weighted average of the true standardized mean differences of these 48 studies (i.e., \(\hat{\bar{\theta}}_w = 0.17\)).
One can also employ an unweighted estimation method (by setting weighted=FALSE in rma.uni), which provides an estimate of the unweighted average of the true effects/outcomes in the \(k\) studies, that is, an estimate of \[\bar{\theta}_u = \frac{\sum_{i=1}^k \theta_i}{k}.\]
Returning to the example, we then find:
# fit a fixed-effects model using unweighted estimation
res <- rma(yi, vi, data=dat, method="FE", weighted=FALSE)
res
# Fixed-Effects Model (k = 48)
#
# I^2 (total heterogeneity / total variability): 56.12%
# H^2 (total variability / sampling variability): 2.28
#
# Test for Heterogeneity:
# Q(df = 47) = 107.1061, p-val < .0001
#
# Model Results:
#
# estimate se zval pval ci.lb ci.ub
# 0.2598 0.0380 6.8366 <.0001 0.1853 0.3343Therefore, the value shown under estimate is now an estimate of the unweighted average of the true standardized mean differences of these 48 studies (i.e., \(\hat{\bar{\theta}}_u = 0.26\)).
For weighted estimation, one could also choose to estimate \(\bar{\theta}_w\), where the \(w_i\) values are user-defined weights (via argument weights in rma.uni). Hence, using inverse-variance weights or unit weights (as in unweighted estimation) are just special cases. It is up to the user to decide to what extent \(\bar{\theta}_w\) is a meaningful parameter to estimate (regardless of the weights used).
For example, we could use the sample sizes of the studies as weights:
# fit a fixed-effects model using the sample sizes as weights
res <- rma(yi, vi, data=dat, method="FE", weights=ni)
res
# Fixed-Effects Model (k = 48)
#
# I^2 (total heterogeneity / total variability): 56.12%
# H^2 (total variability / sampling variability): 2.28
#
# Test for Heterogeneity:
# Q(df = 47) = 107.1061, p-val < .0001
#
# Model Results:
#
# estimate se zval pval ci.lb ci.ub
# 0.1719 0.0269 6.3802 <.0001 0.1191 0.2248We therefore obtain an estimate of the sample-size weighted average of the true standardized mean differences of these 48 studies (i.e., \(\hat{\bar{\theta}}_w = 0.17\)). Since the sample sizes and the inverse sampling variances are highly correlated (cor(dat$ni, 1/dat$vi) yields 0.999), the results are almost identical to the ones we obtained earlier using inverse-variance weighting.
The random-effects model does not condition on the true effects/outcomes. Instead, the \(k\) studies included in the meta-analysis are assumed to be a random sample from a larger population of studies. In rare cases, the studies included in a meta-analysis are actually sampled from a larger collection of studies. More typically, all efforts have been made to find and include all relevant studies providing evidence about the phenomenon of interest and hence the population of studies is a hypothetical population of an essentially infinite set of studies comprising all of the studies that have been conducted, that could have been conducted, or that may be conducted in the future. We assume that \(\theta_i \sim N(\mu, \tau^2)\), that is, the true effects/outcomes in the population of studies are normally distributed with \(\mu\) denoting the average true effect/outcome and \(\tau^2\) the variance of the true effects/outcomes in the population (\(\tau^2\) is therefore often referred to as the amount of ‘heterogeneity’ in the true effects/outcomes). The random-effects model can also be written as \[y_i = \mu + u_i + \varepsilon_i,\] where \(u_i \sim N(0, \tau^2)\) and \(\varepsilon_i \sim N(0, v_i)\). The fitted model provides estimates of \(\mu\) and \(\tau^2\). Consequently, the random-effects model provides an unconditional inference about the average true effect/outcome in the population of studies (from which the \(k\) studies included in the meta-analysis are assumed to be a random sample).
Fitting a random-effects model to the example data yields:
# fit a random-effects model (note: method="REML" is the default)
res <- rma(yi, vi, data=dat)
res
# Random-Effects Model (k = 48; tau^2 estimator: REML)
#
# tau^2 (estimated amount of total heterogeneity): 0.0499 (SE = 0.0197)
# tau (square root of estimated tau^2 value): 0.2235
# I^2 (total heterogeneity / total variability): 58.37%
# H^2 (total variability / sampling variability): 2.40
#
# Test for Heterogeneity:
# Q(df = 47) = 107.1061, p-val < .0001
#
# Model Results:
#
# estimate se zval pval ci.lb ci.ub
# 0.2219 0.0460 4.8209 <.0001 0.1317 0.3122The value shown under estimate is now an estimate of the average true standardized mean difference of studies in the population of studies from which the 48 studies included in this dataset have come (i.e., \(\hat{\mu} = 0.22\)).
When using weighted estimation in the context of a random-effects model, the model is fitted with weights equal to \(w_i = 1/(\tau^2 + v_i)\), with \(\tau^2\) replaced by its estimate (the default in rma.uni when method is set to one of the possible choices for estimating \(\tau^2\)). One can also choose unweighted estimation in the context of the random-effects model (weighted=FALSE) or specify user-defined weights (via weights), although the parameter that is estimated (i.e., \(\mu\)) remains the same regardless of the estimation method and weights used (as opposed to the fixed-effect model, where the parameter estimated is different for weighted versus unweighted estimation or when using different weights than the standard inverse-variance weights). Since weighted estimation with inverse-variance weights is most efficient, it is usually to be preferred for random-effects models (while in the fixed-effect model case, we must carefully consider whether \(\bar{\theta}_w\) or \(\bar{\theta}_u\) is the more meaningful parameter to estimate).
Contrary to what is often stated in the literature, it is important to realize that the fixed-effects model does not assume that the true effects/outcomes are homogeneous (i.e., that \(\theta_i\) is equal to some common value \(\theta\) in all \(k\) studies). In other words, the fixed-effects model provides perfectly valid inferences under heterogeneity, as long as one is restricting these inferences to the set of studies included in the meta-analysis and one realizes that the model does not provide an estimate of \(\theta\) or \(\mu\), but of \(\bar{\theta}_w\) or \(\bar{\theta}_u\) (depending on the estimation method used).
However, such inferences are conditional on the included studies. It is therefore not permissible to generalize those inferences beyond the set of studies included in a meta-analysis (or doing so requires ‘extra-statistical’ arguments). In contrast, a random-effects model provides unconditional inferences and therefore allows a generalization beyond the set of included studies, although the population of studies to which we can generalize is typically only vaguely defined (since the included studies are not a proper random sample from a specified sampling frame). Instead, we simply must assume that the included studies are a representative sample of some population and it is to that population to which we are generalizing.
Leaving aside this issue, the above implies that there is nothing wrong with fitting both the fixed- and random-effects models to the same data, since these models address inherently different questions (i.e., what was the average effect in the studies that have been conducted and are included in this meta-analysis versus what is the average effect in the larger population of studies?).
In the special case that the true effects/outcomes are actually homogeneous (the equal-effects case), the distinction between the fixed- and random-effects models disappears, since homogeneity implies that \(\mu = \bar{\theta}_w = \bar{\theta}_u \equiv \theta\). Therefore, if one belives that the true effects/outcomes are homogeneous, then one can fit an equal-effects model (using weighted estimation), since this will provide the most efficient estimate of \(\theta\) (note that if the true effects/outcomes are really homogeneous but we fit a random-effects model, it can happen that the estimate of \(\tau^2\) is actually larger than 0, which then leads to a loss of efficiency).
However, since there is no infallible method to test whether the true effects/outcomes are really homogeneous or not, a researcher should decide on the type of inference desired before examining the data and choose the model accordingly.
Note that fitting an equal-effects model (with method="EE") yields the exact same output as fitting a fixed-effects model, since the equations used to fit these two models are identical. However, the interpretation of the results is different. If we fit an equal-effects model, we make the assumption that the true effects are homogeneous and, if we believe this assumption to be justified, can interpret the estimate as an estimate of the true effect. On the other hand, if we reject the homogeneity assumption, then we should reject the model altogether. In contrast, if we fit a fixed-effects model, we do not assume homogeneity and instead interpret the estimate as an estimate of the (weighted) average true effect of the included studies.
For further discussions of the distinction between the equal-, fixed-, and random-effects models, see Laird and Mosteller (1990) and Hedges and Vevea (1998).
Cooper, H., Hedges, L. V., & Valentine, J. C. (Eds.) (2009). The handbook of research synthesis and meta-analysis (2nd ed.). New York: Russell Sage Foundation.
Hedges, L. V., & Vevea, J. L. (1998). Fixed- and random-effects models in meta-analysis. Psychological Methods, 3(4), 486–504. https://doi.org/10.1037/1082-989X.3.4.486
Laird, N. M., & Mosteller, F. (1990). Some statistical methods for combining experimental results. International Journal of Technology Assessment in Health Care, 6(1), 5–30. https://doi.org/10.1017/S0266462300008916
Viechtbauer, W. (2010). Conducting meta-analyses in R with the metafor package. Journal of Statistical Software, 36(3), 1–48. https://doi.org/10.18637/jss.v036.i03