dat.assink2016.RdResults from 17 studies on the association between recidivism and mental health in delinquent juveniles.
dat.assink2016The data frame contains the following columns:
| study | numeric | study id number |
| esid | numeric | effect size within study id number |
| id | numeric | row id number |
| yi | numeric | standardized mean difference |
| vi | numeric | corresponding sampling variance |
| pubstatus | numeric | published study (0 = no; 1 = yes) |
| year | numeric | publication year of the study (approximately mean centered) |
| deltype | character | type of delinquent behavior in which juveniles could have recidivated (either general, overt, or covert) |
The studies included in this dataset (which is a subset of the data used in Assink et al., 2015) compared the difference in recidivism between delinquent juveniles with a mental health disorder and a comparison group of juveniles without a mental health disorder. Since studies differed in the way recidivism was defined and assessed, results are given in terms of standardized mean differences, with positive values indicating a higher prevalence of recidivism in the group of juveniles with a mental health disorder.
Multiple effect size estimates could be extracted from most studies (e.g., for different delinquent behaviors in which juveniles could have recidivated), necessitating the use of appropriate models/methods for the analysis. Assink and Wibbelink (2016) illustrate the use of multilevel meta-analysis models for this purpose.
The year variable is not constant within study 3, as this study refers to two different publications using the same data.
Assink, M., & Wibbelink, C. J. M. (2016). Fitting three-level meta-analytic models in R: A step-by-step tutorial. The Quantitative Methods for Psychology, 12(3), 154–174. https://doi.org/10.20982/tqmp.12.3.p154
Assink, M., van der Put, C. E., Hoeve, M., de Vries, S. L. A., Stams, G. J. J. M., & Oort, F. J. (2015). Risk factors for persistent delinquent behavior among juveniles: A meta-analytic review. Clinical Psychology Review, 42, 47–61. https://doi.org/10.1016/j.cpr.2015.08.002
psychology, criminology, standardized mean differences, multilevel models, cluster-robust inference
### copy data into 'dat' and examine data
dat <- dat.assink2016
head(dat, 9)
#>
#> study esid id yi vi pubstatus year deltype
#> 1 1 1 1 0.9066 0.0740 1 4.5 general
#> 2 1 2 2 0.4295 0.0398 1 4.5 general
#> 3 1 3 3 0.2679 0.0481 1 4.5 general
#> 4 1 4 4 0.2078 0.0239 1 4.5 general
#> 5 1 5 5 0.0526 0.0331 1 4.5 general
#> 6 1 6 6 -0.0507 0.0886 1 4.5 general
#> 7 2 1 7 0.5117 0.0115 1 1.5 general
#> 8 2 2 8 0.4738 0.0076 1 1.5 general
#> 9 2 3 9 0.3544 0.0065 1 1.5 general
#>
### load metafor package
library(metafor)
### fit multilevel model
res <- rma.mv(yi, vi, random = ~ 1 | study/esid, data=dat)
res
#>
#> Multivariate Meta-Analysis Model (k = 100; method: REML)
#>
#> Variance Components:
#>
#> estim sqrt nlvls fixed factor
#> sigma^2.1 0.1879 0.4334 17 no study
#> sigma^2.2 0.1120 0.3347 100 no study/esid
#>
#> Test for Heterogeneity:
#> Q(df = 99) = 809.4611, p-val < .0001
#>
#> Model Results:
#>
#> estimate se zval pval ci.lb ci.ub
#> 0.4268 0.1184 3.6038 0.0003 0.1947 0.6589 ***
#>
#> ---
#> Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
#>
### use cluster-robust inference methods
robust(res, cluster=study, clubSandwich=TRUE)
#>
#> Multivariate Meta-Analysis Model (k = 100; method: REML)
#>
#> Variance Components:
#>
#> estim sqrt nlvls fixed factor
#> sigma^2.1 0.1879 0.4334 17 no study
#> sigma^2.2 0.1120 0.3347 100 no study/esid
#>
#> Test for Heterogeneity:
#> Q(df = 99) = 809.4611, p-val < .0001
#>
#> Number of estimates: 100
#> Number of clusters: 17
#> Estimates per cluster: 1-22 (mean: 5.88, median: 5)
#>
#> Model Results:
#>
#> estimate se¹ tval¹ df¹ pval¹ ci.lb¹ ci.ub¹
#> 0.4268 0.1186 3.5971 15.45 0.0025 0.1745 0.6791 **
#>
#> ---
#> Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
#>
#> 1) results based on cluster-robust inference (var-cov estimator: CR2,
#> approx t-test and confidence interval, df: Satterthwaite approx)
#>
### LRTs for the variance components
res0 <- rma.mv(yi, vi, random = ~ 1 | study/esid, data=dat, sigma2=c(0,NA))
anova(res0, res)
#>
#> df AIC BIC AICc logLik LRT pval QE
#> Full 3 153.2654 161.0508 153.5180 -73.6327 809.4611
#> Reduced 2 214.0678 219.2581 214.1928 -105.0339 62.8024 <.0001 809.4611
#>
res0 <- rma.mv(yi, vi, random = ~ 1 | study/esid, data=dat, sigma2=c(NA,0))
anova(res0, res)
#>
#> df AIC BIC AICc logLik LRT pval QE
#> Full 3 153.2654 161.0508 153.5180 -73.6327 809.4611
#> Reduced 2 233.1313 238.3215 233.2563 -114.5656 81.8658 <.0001 809.4611
#>
### examine some potential moderators via meta-regression
rma.mv(yi, vi, mods = ~ pubstatus, random = ~ 1 | study/esid, data=dat)
#>
#> Multivariate Meta-Analysis Model (k = 100; method: REML)
#>
#> Variance Components:
#>
#> estim sqrt nlvls fixed factor
#> sigma^2.1 0.1712 0.4138 17 no study
#> sigma^2.2 0.1128 0.3359 100 no study/esid
#>
#> Test for Residual Heterogeneity:
#> QE(df = 98) = 702.5110, p-val < .0001
#>
#> Test of Moderators (coefficient 2):
#> QM(df = 1) = 1.8444, p-val = 0.1744
#>
#> Model Results:
#>
#> estimate se zval pval ci.lb ci.ub
#> intrcpt 0.8117 0.3056 2.6565 0.0079 0.2128 1.4106 **
#> pubstatus -0.4474 0.3294 -1.3581 0.1744 -1.0930 0.1983
#>
#> ---
#> Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
#>
rma.mv(yi, vi, mods = ~ year, random = ~ 1 | study/esid, data=dat)
#>
#> Multivariate Meta-Analysis Model (k = 100; method: REML)
#>
#> Variance Components:
#>
#> estim sqrt nlvls fixed factor
#> sigma^2.1 0.1351 0.3675 17 no study
#> sigma^2.2 0.1128 0.3359 100 no study/esid
#>
#> Test for Residual Heterogeneity:
#> QE(df = 98) = 673.5617, p-val < .0001
#>
#> Test of Moderators (coefficient 2):
#> QM(df = 1) = 5.4633, p-val = 0.0194
#>
#> Model Results:
#>
#> estimate se zval pval ci.lb ci.ub
#> intrcpt 0.4257 0.1040 4.0947 <.0001 0.2219 0.6294 ***
#> year -0.0421 0.0180 -2.3374 0.0194 -0.0773 -0.0068 *
#>
#> ---
#> Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
#>
dat$deltype <- relevel(factor(dat$deltype), ref="general")
rma.mv(yi, vi, mods = ~ deltype, random = ~ 1 | study/esid, data=dat)
#>
#> Multivariate Meta-Analysis Model (k = 100; method: REML)
#>
#> Variance Components:
#>
#> estim sqrt nlvls fixed factor
#> sigma^2.1 0.1899 0.4358 17 no study
#> sigma^2.2 0.0847 0.2910 100 no study/esid
#>
#> Test for Residual Heterogeneity:
#> QE(df = 97) = 761.8270, p-val < .0001
#>
#> Test of Moderators (coefficients 2:3):
#> QM(df = 2) = 14.9760, p-val = 0.0006
#>
#> Model Results:
#>
#> estimate se zval pval ci.lb ci.ub
#> intrcpt 0.4702 0.1180 3.9858 <.0001 0.2390 0.7015 ***
#> deltypecovert -0.7297 0.1923 -3.7941 0.0001 -1.1066 -0.3527 ***
#> deltypeovert -0.2219 0.1392 -1.5939 0.1110 -0.4948 0.0510
#>
#> ---
#> Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
#>
rma.mv(yi, vi, mods = ~ year + deltype, random = ~ 1 | study/esid, data=dat)
#>
#> Multivariate Meta-Analysis Model (k = 100; method: REML)
#>
#> Variance Components:
#>
#> estim sqrt nlvls fixed factor
#> sigma^2.1 0.1493 0.3863 17 no study
#> sigma^2.2 0.0853 0.2920 100 no study/esid
#>
#> Test for Residual Heterogeneity:
#> QE(df = 96) = 610.2644, p-val < .0001
#>
#> Test of Moderators (coefficients 2:4):
#> QM(df = 3) = 19.2399, p-val = 0.0002
#>
#> Model Results:
#>
#> estimate se zval pval ci.lb ci.ub
#> intrcpt 0.4656 0.1071 4.3461 <.0001 0.2556 0.6756 ***
#> year -0.0380 0.0183 -2.0773 0.0378 -0.0738 -0.0021 *
#> deltypecovert -0.7094 0.1914 -3.7069 0.0002 -1.0845 -0.3343 ***
#> deltypeovert -0.2040 0.1385 -1.4725 0.1409 -0.4755 0.0675
#>
#> ---
#> Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
#>
### assume that the effect sizes within studies are correlated with rho=0.6
V <- vcalc(vi, cluster=study, obs=esid, data=dat, rho=0.6)
round(V[dat$study %in% c(1,2), dat$study %in% c(1,2)], 4)
#> [,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8] [,9]
#> [1,] 0.0740 0.0326 0.0358 0.0252 0.0297 0.0486 0.0000 0.0000 0.0000
#> [2,] 0.0326 0.0398 0.0263 0.0185 0.0218 0.0356 0.0000 0.0000 0.0000
#> [3,] 0.0358 0.0263 0.0481 0.0203 0.0239 0.0392 0.0000 0.0000 0.0000
#> [4,] 0.0252 0.0185 0.0203 0.0239 0.0169 0.0276 0.0000 0.0000 0.0000
#> [5,] 0.0297 0.0218 0.0239 0.0169 0.0331 0.0325 0.0000 0.0000 0.0000
#> [6,] 0.0486 0.0356 0.0392 0.0276 0.0325 0.0886 0.0000 0.0000 0.0000
#> [7,] 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0115 0.0056 0.0052
#> [8,] 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0056 0.0076 0.0042
#> [9,] 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0052 0.0042 0.0065
### fit multilevel model using this approximate V matrix
res <- rma.mv(yi, V, random = ~ 1 | study/esid, data=dat)
res
#>
#> Multivariate Meta-Analysis Model (k = 100; method: REML)
#>
#> Variance Components:
#>
#> estim sqrt nlvls fixed factor
#> sigma^2.1 0.0807 0.2841 17 no study
#> sigma^2.2 0.1545 0.3931 100 no study/esid
#>
#> Test for Heterogeneity:
#> Q(df = 99) = 745.4385, p-val < .0001
#>
#> Model Results:
#>
#> estimate se zval pval ci.lb ci.ub
#> 0.3678 0.0965 3.8097 0.0001 0.1786 0.5570 ***
#>
#> ---
#> Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
#>
### use cluster-robust inference methods
robust(res, cluster=study, clubSandwich=TRUE)
#>
#> Multivariate Meta-Analysis Model (k = 100; method: REML)
#>
#> Variance Components:
#>
#> estim sqrt nlvls fixed factor
#> sigma^2.1 0.0807 0.2841 17 no study
#> sigma^2.2 0.1545 0.3931 100 no study/esid
#>
#> Test for Heterogeneity:
#> Q(df = 99) = 745.4385, p-val < .0001
#>
#> Number of estimates: 100
#> Number of clusters: 17
#> Estimates per cluster: 1-22 (mean: 5.88, median: 5)
#>
#> Model Results:
#>
#> estimate se¹ tval¹ df¹ pval¹ ci.lb¹ ci.ub¹
#> 0.3678 0.0970 3.7924 14.53 0.0019 0.1605 0.5750 **
#>
#> ---
#> Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
#>
#> 1) results based on cluster-robust inference (var-cov estimator: CR2,
#> approx t-test and confidence interval, df: Satterthwaite approx)
#>
### use a correlation of 0.7 for effect sizes corresponding to the same type of
### delinquent behavior and a correlation of 0.5 for effect sizes corresponding
### to different types of delinquent behavior
V <- vcalc(vi, cluster=study, type=deltype, obs=esid, data=dat, rho=c(0.7, 0.5))
### fit multilevel model using this approximate V matrix
res <- rma.mv(yi, V, random = ~ 1 | study/esid, data=dat)
res
#>
#> Multivariate Meta-Analysis Model (k = 100; method: REML)
#>
#> Variance Components:
#>
#> estim sqrt nlvls fixed factor
#> sigma^2.1 0.0704 0.2654 17 no study
#> sigma^2.2 0.1508 0.3883 100 no study/esid
#>
#> Test for Heterogeneity:
#> Q(df = 99) = 840.9174, p-val < .0001
#>
#> Model Results:
#>
#> estimate se zval pval ci.lb ci.ub
#> 0.3618 0.0933 3.8794 0.0001 0.1790 0.5446 ***
#>
#> ---
#> Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
#>
### use cluster-robust inference methods
robust(res, cluster=study, clubSandwich=TRUE)
#>
#> Multivariate Meta-Analysis Model (k = 100; method: REML)
#>
#> Variance Components:
#>
#> estim sqrt nlvls fixed factor
#> sigma^2.1 0.0704 0.2654 17 no study
#> sigma^2.2 0.1508 0.3883 100 no study/esid
#>
#> Test for Heterogeneity:
#> Q(df = 99) = 840.9174, p-val < .0001
#>
#> Number of estimates: 100
#> Number of clusters: 17
#> Estimates per cluster: 1-22 (mean: 5.88, median: 5)
#>
#> Model Results:
#>
#> estimate se¹ tval¹ df¹ pval¹ ci.lb¹ ci.ub¹
#> 0.3618 0.0938 3.8567 14.34 0.0017 0.1611 0.5626 **
#>
#> ---
#> Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
#>
#> 1) results based on cluster-robust inference (var-cov estimator: CR2,
#> approx t-test and confidence interval, df: Satterthwaite approx)
#>