The function provides (cluster) robust tests and confidence intervals of the model coefficients for objects of class "rma".

robust(x, cluster, ...)

# S3 method for rma.uni
robust(x, cluster, adjust=TRUE, digits, ...)
# S3 method for rma.mv
robust(x, cluster, adjust=TRUE, digits, ...)

## Arguments

x an object of class "rma.uni" or "rma.mv". a vector to specify a clustering variable to use for constructing the sandwich estimator of the variance-covariance matrix. logical to specify whether a small-sample correction should be applied to the variance-covariance matrix. integer to specify the number of decimal places to which the printed results should be rounded (if unspecified, the default is to take the value from the object). other arguments.

## Details

The function constructs a (cluster) robust estimate of the variance-covariance matrix of the model coefficients based on a sandwich-type estimator and then computes tests and confidence intervals of the model coefficients.

Tests of individual coefficients and confidence intervals are based on a t-distribution with $$n-p$$ degrees of freedom is used, while the omnibus test statistic uses an F-distribution with $$m$$ and $$n-p$$ degrees of freedom, where $$n$$ is the number of clusters, $$p$$ denotes the total number of model coefficients (including the intercept if it is present), and $$m$$ denotes the number of coefficients tested (in the omnibus test).

When adjust=TRUE (the default), the (cluster) robust estimate of the variance-covariance matrix is multiplied by the factor $$n/(n-p)$$, which serves as a small-sample adjustment that tends to improve the performance of the method when the number of clusters is small.

For even better small-sample adjustments (Pustejovsky & Tipton, 2018), see the clubSandwich package, which nicely works together with the metafor package (see ‘Examples’).

## Value

An object of class "robust.rma". The object is a list containing the following components:

beta

estimated coefficients of the model.

se

robust standard errors of the coefficients.

zval

test statistics of the coefficients.

pval

corresponding p-values.

ci.lb

lower bound of the confidence intervals for the coefficients.

ci.ub

upper bound of the confidence intervals for the coefficients.

vb

robust variance-covariance matrix of the estimated coefficients.

QM

test statistic of the omnibus test of moderators.

QMp

corresponding p-value.

...

The results are formatted and printed with the print.robust.rma function.

## Note

The variable specified via cluster is assumed to be of the same length as the data originally passed to the rma.uni or rma.mv function. Any subsetting and removal of studies with missing values as done when fitting the original model is also automatically applied to the variable specified via cluster.

The idea of the robust (sandwich-type) estimator for models with unspecified heteroscedasticity can be traced back to Eicker (1967), Huber (1967), and White (1980). Hence, the method in general is often referred to as the Eicker-Huber-White method. Some small-sample improvements to the method are described by MacKinnon and White (1985). The extension to the cluster robust estimator can be found in Froot (1989) and Williams (2000). Cameron and Miller (2015) provide an extensive overview of cluster robust methods. Sidik and Jonkman (2005, 2006) introduced robust methods in the meta-analytic context for standard random/mixed-effects models. The use of the cluster robust estimator for multivariate/multilevel meta-analytic models is described in Hedges, Tipton, and Johnson (2010).

## Author

Wolfgang Viechtbauer wvb@metafor-project.org http://www.metafor-project.org

## References

Cameron, A. C., & Miller, D. L. (2015). A practitioner's guide to cluster-robust inference. Journal of Human Resources, 50(2), 317--372. https://doi.org/10.3368/jhr.50.2.317

Eicker, F. (1967). Limit theorems for regressions with unequal and dependent errors. In L. M. LeCam & J. Neyman (Eds.), Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability (pp. 59--82). Berkeley: University of California Press.

Froot, K. A. (1989). Consistent covariance matrix estimation with cross-sectional dependence and heteroskedasticity in financial data. Journal of Financial and Quantitative Analysis, 24(3), 333--355. https://doi.org/10.2307/2330815

Hedges, L. V., Tipton, E., & Johnson, M. C. (2010). Robust variance estimation in meta-regression with dependent effect size estimates. Research Synthesis Methods, 1(1), 39--65. https://doi.org/10.1002/jrsm.5

Huber, P. (1967). The behavior of maximum-likelihood estimates under nonstandard conditions. In L. M. LeCam & J. Neyman (Eds.), Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability (pp. 221--233). Berkeley: University of California Press.

MacKinnon, J. G., & White, H. (1985). Some heteroskedasticity-consistent covariance matrix estimators with improved finite sample properties. Journal of Econometrics, 29(3), 305--325. https://doi.org/10.1016/0304-4076(85)90158-7

Pustejovsky, J. E., & Tipton, E. (2018). Small-sample methods for cluster-robust variance estimation and hypothesis testing in fixed effects models. Journal of Business & Economic Statistics, 36(4), 672--683. https://doi.org/10.1080/07350015.2016.1247004

Sidik, K., & Jonkman, J. N. (2005). A note on variance estimation in random effects meta-regression. Journal of Biopharmaceutical Statistics, 15(5), 823--838. https://doi.org/10.1081/BIP-200067915

Sidik, K., & Jonkman, J. N. (2006). Robust variance estimation for random effects meta-analysis. Computational Statistics & Data Analysis, 50(12), 3681--3701. https://doi.org/10.1016/j.csda.2005.07.019

White, H. (1980). A heteroskedasticity-consistent covariance matrix estimator and a direct test for heteroskedasticity. Econometrica, 48(4), 817--838. https://doi.org/10.2307/1912934

Williams, R. L. (2000). A note on robust variance estimation for cluster-correlated data. Biometrics, 56(2), 645--646. https://doi.org/10.1111/j.0006-341x.2000.00645.x

Viechtbauer, W. (2010). Conducting meta-analyses in R with the metafor package. Journal of Statistical Software, 36(3), 1--48. https://doi.org/10.18637/jss.v036.i03

rma.uni, rma.mv

## Examples

### copy data from Konstantopoulos (2011) into 'dat'
dat <- dat.konstantopoulos2011

### multilevel random-effects model
res <- rma.mv(yi, vi, random = ~ 1 | district/school, data=dat)
res
#>
#> Multivariate Meta-Analysis Model (k = 56; method: REML)
#>
#> Variance Components:
#>
#>             estim    sqrt  nlvls  fixed           factor
#> sigma^2.1  0.0651  0.2551     11     no         district
#> sigma^2.2  0.0327  0.1809     56     no  district/school
#>
#> Test for Heterogeneity:
#> Q(df = 55) = 578.8640, p-val < .0001
#>
#> Model Results:
#>
#> estimate      se    zval    pval   ci.lb   ci.ub
#>   0.1847  0.0846  2.1845  0.0289  0.0190  0.3504  *
#>
#> ---
#> Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
#>
### results based on the sandwich method
robust(res, cluster=dat$district) #> #> Number of outcomes: 56 #> Number of clusters: 11 #> Outcomes per cluster: 3-11 (mean: 5.09, median: 4) #> #> Model Results: #> #> estimate se tval pval ci.lb ci.ub #> 0.1847 0.0845 2.1859 0.0537 -0.0036 0.3730 . #> #> --- #> Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 #> ### illustrate use of the clubSandwich package together with metafor # \dontrun{ require(clubSandwich) #> Loading required package: clubSandwich#> Registered S3 method overwritten by 'clubSandwich': #> method from #> bread.mlm sandwichcoef_test(res, vcov="CR2", cluster=dat$district)
#>     Coef. Estimate     SE t-stat d.f. p-val (Satt) Sig.
#> 1 intrcpt    0.185 0.0845   2.19 9.86        0.054    .# }

### copy data from Berkey et al. (1998) into 'dat'
dat <- dat.berkey1998

### construct list with the variance-covariance matrices of the observed outcomes for the studies
V <- lapply(split(dat[c("v1i", "v2i")], dat$trial), as.matrix) ### construct block diagonal matrix V <- bldiag(V) ### fit multivariate model res <- rma.mv(yi, V, mods = ~ outcome - 1, random = ~ outcome | trial, struct="UN", data=dat) res #> #> Multivariate Meta-Analysis Model (k = 10; method: REML) #> #> Variance Components: #> #> outer factor: trial (nlvls = 5) #> inner factor: outcome (nlvls = 2) #> #> estim sqrt k.lvl fixed level #> tau^2.1 0.0327 0.1807 5 no AL #> tau^2.2 0.0117 0.1083 5 no PD #> #> rho.AL rho.PD AL PD #> AL 1 0.6088 - no #> PD 0.6088 1 5 - #> #> Test for Residual Heterogeneity: #> QE(df = 8) = 128.2267, p-val < .0001 #> #> Test of Moderators (coefficients 1:2): #> QM(df = 2) = 108.8616, p-val < .0001 #> #> Model Results: #> #> estimate se zval pval ci.lb ci.ub #> outcomeAL -0.3392 0.0879 -3.8589 0.0001 -0.5115 -0.1669 *** #> outcomePD 0.3534 0.0588 6.0057 <.0001 0.2381 0.4688 *** #> #> --- #> Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 #> ### results based on sandwich method robust(res, cluster=dat$trial)
#>
#> Number of outcomes:   10
#> Number of clusters:   5
#> Outcomes per cluster: 2
#>
#> Test of Moderators (coefficients 1:2):
#> F(df1 = 2, df2 = 3) = 41.6138, p-val = 0.0065
#>
#> Model Results:
#>
#>            estimate      se     tval    pval    ci.lb    ci.ub
#> outcomeAL   -0.3392  0.1010  -3.3597  0.0437  -0.6605  -0.0179  *
#> outcomePD    0.3534  0.0675   5.2338  0.0136   0.1385   0.5683  *
#>
#> ---
#> Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
#>
### illustrate use of the clubSandwich package together with metafor
# \dontrun{
require(clubSandwich)
coef_test(res, vcov="CR2", cluster=dat\$trial)
#>       Coef. Estimate     SE t-stat d.f. p-val (Satt) Sig.
#> 1 outcomeAL   -0.339 0.0892  -3.80 3.81       0.0208    *
#> 2 outcomePD    0.353 0.0582   6.08 3.79       0.0044   **# }