Function to compute the fail-safe N (also called a file drawer analysis).

fsn(x, vi, sei, subset, data, type, alpha=.05, target,
    method, exact=FALSE, verbose=FALSE, digits, ...)

Arguments

x

a vector with the observed effect sizes or outcomes or an object of class "rma".

vi

vector with the corresponding sampling variances (ignored if x is an object of class "rma").

sei

vector with the corresponding standard errors (note: only one of the two, vi or sei, needs to be specified).

subset

optional (logical or numeric) vector to specify the subset of studies that should be used for the calculation (ignored if x is an object of class "rma").

data

optional data frame containing the variables given to the arguments above.

type

optional character string to specify the type of method to use for the calculation of the fail-safe N. Possible options are "Rosenthal" (the default when x is a vector with the observed effect sizes or outcomes), "Orwin", "Rosenberg", or "General" (the default when x is an object of class "rma"). Can be abbreviated. See ‘Details’.

alpha

target alpha level for the Rosenthal, Rosenberg, and General methods (the default is .05).

target

target average effect size or outcome for the Orwin and General methods.

method

optional character string to specify the model fitting method for type="General" (if unspecified, either "REML" by default or the method that was used in fitting the "rma" model). See rma.uni for options.

exact

logical to specify whether the general method should be based on exact (but slower) or approximate (but faster) calculations.

verbose

logical to specify whether output should be generated on the progress of the calculations for type="General" (the default is FALSE).

digits

optional integer to specify the number of decimal places to which the printed results should be rounded.

...

other arguments.

Details

The function can be used to calculate the ‘fail-safe N’, that is, the minimum number of studies averaging null results that would have to be added to a given set of \(k\) studies to change the conclusion of a meta-analysis. If this number is small (in relation to the actual number of studies), then this indicates that the results based on the observed studies are not robust to publication bias (of the form assumed by the method, that is, where a set of studies averaging null results is missing). The method is also called a ‘file drawer analysis’ as it assumes that there is a set of studies averaging null results hiding in file drawers, which can overturn the findings from a meta-analysis. There are various types of methods that are all based on the same principle, which are described in more detail further below. Note that the fail-safe N is not an estimate of the number of missing studies, only how many studies must be hiding in file drawers for the findings to be overturned.

One can either pass a vector with the observed effect sizes or outcomes (via x) and the corresponding sampling variances via vi (or the standard errors via sei) to the function or an object of class "rma". When passing a model object, the model must be a model without moderators (i.e., either an equal- or a random-effects model).

Rosenthal Method

The Rosenthal method (type="Rosenthal") calculates the minimum number of studies averaging null results that would have to be added to a given set of studies to reduce the (one-tailed) combined significance level (i.e., p-value) to a particular alpha level, which can be specified via the alpha argument (.05 by default). The calculation is based on Stouffer's method for combining p-values and is described in Rosenthal (1979). Note that the method is primarily of interest for historical reasons, but the other methods described below are more closely aligned with the way meta-analyses are typically conducted in practice.

Orwin Method

The Orwin method (type="Orwin") calculates the minimum number of studies averaging null results that would have to be added to a given set of studies to reduce the (unweighted or weighted) average effect size / outcome to a target value (as specified via the target argument). The method is described in Orwin (1983). When vi (or sei) is not specified, the method is based on the unweighted average of the effect sizes / outcomes; otherwise, the method uses the inverse-variance weighted average. If the target argument is not specified, then the target value will be equal to the observed average effect size / outcome divided by 2 (which is entirely arbitrary and will always lead to a fail-safe N number that is equal to \(k\)). One should really set target to a value that reflects an effect size / outcome that would be considered to be practically irrelevant. Note that if target has the opposite sign as the actually observed average, then its sign is automatically flipped.

Rosenberg Method

The Rosenberg method (type="Rosenberg") calculates the minimum number of studies averaging null results that would have to be added to a given set of studies to reduce the significance level (i.e., p-value) of the average effect size / outcome (as estimated based on an equal-effects model) to a particular alpha level, which can be specified via the alpha argument (.05 by default). The method is described in Rosenberg (2005). Note that the p-value is calculated based on a standard normal distribution (instead of a t-distribution, as suggested by Rosenberg, 2005), but the difference is typically negligible.

General Method

This method is a generalization of the methods by Orwin and Rosenberg (see Viechtbauer, 2024). By default (i.e., when target is not specified), it calculates the minimum number of studies averaging null results that would have to be added to a given set of studies to reduce the significance level (i.e., p-value) of the average effect size / outcome (as estimated based on a chosen model) to a particular alpha level, which can be specified via the alpha argument (.05 by default). The type of model that is used in the calculation is chosen via the method argument. If this is unspecified, then a random-effects model is automatically used (using method="REML") or the method that was used in fitting the "rma" model (see rma.uni for options). Therefore, when setting method="EE", then an equal-effects model is used, which yields (essentially) identical results as Rosenberg's method.

If target is specified, then the method calculates the minimum number of studies averaging null results that would have to be added to a given set of studies to reduce the average effect size / outcome (as estimated based on a chosen model) to a target value (as specified via the target argument). As described above, the type of model that is used in the calculation is chosen via the method argument. When setting method="EE", then an equal-effects model is used, which yields (essentially) identical results as Orwin's method with inverse-variance weights.

The method uses an iterative algorithm for calculating the fail-safe N, which can be computationally expensive especially when N is large. By default, the method uses approximate (but faster) calculations, but when setting exact=TRUE, the method uses exact (but slower) calculations. The difference between the two is typically negligible. If N is larger than \(10^7\), then the calculated number is given as >1e+07.

Value

An object of class "fsn". The object is a list containing the following components (some of which may be NA if they are not applicable to the chosen method):

type

the type of method used.

fsnum

the calculated fail-safe N.

est

the average effect size / outcome based on the observed studies.

tau2

the estimated amount of heterogeneity based on the observed studies.

pval

the p-value of the observed results.

alpha

the specified target alpha level.

target

the target average effect size / outcome.

est.fsn

the average effect size / outcome when combining the observed studies with those in the file drawer.

tau2

the estimated amount of heterogeneity when combining the observed studies with those in the file drawer.

pval

the p-value when combining the observed studies with those in the file drawer.

...

some additional elements/values.

The results are formatted and printed with the print function.

Note

If the significance level of the observed studies is already above the specified alpha level or if the average effect size / outcome of the observed studies is already below the target average effect size / outcome, then the fail-safe N value is zero.

References

Rosenthal, R. (1979). The "file drawer problem" and tolerance for null results. Psychological Bulletin, 86(3), 638–641. https://doi.org/10.1037/0033-2909.86.3.638

Orwin, R. G. (1983). A fail-safe N for effect size in meta-analysis. Journal of Educational Statistics, 8(2), 157–159. https://doi.org/10.3102/10769986008002157

Rosenberg, M. S. (2005). The file-drawer problem revisited: A general weighted method for calculating fail-safe numbers in meta-analysis. Evolution, 59(2), 464–468. https://doi.org/10.1111/j.0014-3820.2005.tb01004.x

Viechtbauer, W. (2010). Conducting meta-analyses in R with the metafor package. Journal of Statistical Software, 36(3), 1–48. https://doi.org/10.18637/jss.v036.i03

Viechtbauer, W. (2024). A fail-safe N computation based on the random-effects model. Annual Meeting of the Society for Research Synthesis Methodology, Amsterdam, The Netherlands. https://www.wvbauer.com/lib/exe/fetch.php/talks:2024_viechtbauer_srsm_fail_safe_n.pdf

See also

regtest for the regression test, ranktest for the rank correlation test, trimfill for the trim and fill method, tes for the test of excess significance, and selmodel for selection models.

Examples

### calculate log risk ratios and corresponding sampling variances
dat <- escalc(measure="RR", ai=tpos, bi=tneg, ci=cpos, di=cneg, data=dat.bcg)

### fit equal-effects model
rma(yi, vi, data=dat, method="EE")
#> 
#> Equal-Effects Model (k = 13)
#> 
#> I^2 (total heterogeneity / total variability):   92.12%
#> H^2 (total variability / sampling variability):  12.69
#> 
#> Test for Heterogeneity:
#> Q(df = 12) = 152.2330, p-val < .0001
#> 
#> Model Results:
#> 
#> estimate      se      zval    pval    ci.lb    ci.ub      
#>  -0.4303  0.0405  -10.6247  <.0001  -0.5097  -0.3509  *** 
#> 
#> ---
#> Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
#> 

### fail-safe N computations
fsn(yi, vi, data=dat)
#> 
#> Fail-safe N Calculation Using the Rosenthal Approach
#> 
#> Observed Significance Level: <.0001
#> Target Significance Level:   0.05
#> 
#> Fail-safe N: 598
#> 
fsn(yi, data=dat, type="Orwin", target=log(0.95)) # target corresponds to a 5% risk reduction
#> 
#> Fail-safe N Calculation Using the Orwin Approach
#> 
#> Average Effect Size: -0.7407
#> Target Effect Size:  -0.0513
#> 
#> Fail-safe N: 175
#> 
fsn(yi, vi, data=dat, type="Orwin", target=log(0.95)) # Orwin's method with 1/vi weights
#> 
#> Fail-safe N Calculation Using the Orwin Approach
#> 
#> Average Effect Size: -0.4303
#> Target Effect Size:  -0.0513
#> 
#> Fail-safe N: 97
#> 
fsn(yi, vi, data=dat, type="General", target=log(0.95), method="EE") # like Orwin's method
#> 
#> Fail-safe N Calculation Using the General Approach
#> 
#> Average Effect Size:         -0.4303 (with file drawer: -0.0509)
#> Observed Significance Level:  <.0001 (with file drawer:  0.0003)
#> Target Effect Size:          -0.0513
#> 
#> Fail-safe N: 97
#> 
fsn(yi, vi, data=dat, type="Rosenberg")
#> 
#> Fail-safe N Calculation Using the Rosenberg Approach
#> 
#> Average Effect Size:         -0.4303
#> Observed Significance Level:  <.0001
#> Target Significance Level:    0.05
#> 
#> Fail-safe N: 370
#> 
fsn(yi, vi, data=dat, type="General", method="EE") # like Rosenberg's method
#> 
#> Fail-safe N Calculation Using the General Approach
#> 
#> Average Effect Size:         -0.4303 (with file drawer: -0.0146)
#> Observed Significance Level:  <.0001 (with file drawer:  0.0503)
#> Target Significance Level:    0.05
#> 
#> Fail-safe N: 370
#> 
fsn(yi, vi, data=dat, type="General") # based on a random-effects model
#> 
#> Fail-safe N Calculation Using the General Approach
#> 
#> Average Effect Size:         -0.7145 (with file drawer: -0.2112)
#> Amount of Heterogeneity:      0.3132 (with file drawer:  0.4214)
#> Observed Significance Level:  <.0001 (with file drawer:  0.0543)
#> Target Significance Level:    0.05
#> 
#> Fail-safe N: 26
#> 
fsn(yi, vi, data=dat, type="General", target=log(0.95)) # based on a random-effects model
#> 
#> Fail-safe N Calculation Using the General Approach
#> 
#> Average Effect Size:         -0.7145 (with file drawer: -0.0508)
#> Amount of Heterogeneity:      0.3132 (with file drawer:  0.3516)
#> Observed Significance Level:  <.0001 (with file drawer:  0.3109)
#> Target Effect Size:          -0.0513
#> 
#> Fail-safe N: 138
#> 

### fit a random-effects model and use fsn() on the model object
res <- rma(yi, vi, data=dat)
fsn(res)
#> 
#> Fail-safe N Calculation Using the General Approach
#> 
#> Average Effect Size:         -0.7145 (with file drawer: -0.2112)
#> Amount of Heterogeneity:      0.3132 (with file drawer:  0.4214)
#> Observed Significance Level:  <.0001 (with file drawer:  0.0543)
#> Target Significance Level:    0.05
#> 
#> Fail-safe N: 26
#> 
fsn(res, target=log(0.95))
#> 
#> Fail-safe N Calculation Using the General Approach
#> 
#> Average Effect Size:         -0.7145 (with file drawer: -0.0508)
#> Amount of Heterogeneity:      0.3132 (with file drawer:  0.3516)
#> Observed Significance Level:  <.0001 (with file drawer:  0.3109)
#> Target Effect Size:          -0.0513
#> 
#> Fail-safe N: 138
#>