fsn.RdFunction to compute the fail-safe N (also called a file drawer analysis).
fsn(x, vi, sei, subset, data, type, alpha=.05, target,
method, exact=FALSE, verbose=FALSE, digits, ...)a vector with the observed effect sizes or outcomes or an object of class "rma".
vector with the corresponding sampling variances (ignored if x is an object of class "rma").
vector with the corresponding standard errors (note: only one of the two, vi or sei, needs to be specified).
optional (logical or numeric) vector to specify the subset of studies that should be used for the calculation (ignored if x is an object of class "rma").
optional data frame containing the variables given to the arguments above.
optional character string to specify the type of method to use for the calculation of the fail-safe N. Possible options are "Rosenthal" (the default when x is a vector with the observed effect sizes or outcomes), "Orwin", "Rosenberg", or "General" (the default when x is an object of class "rma"). Can be abbreviated. See ‘Details’.
target alpha level for the Rosenthal, Rosenberg, and General methods (the default is .05).
target average effect size or outcome for the Orwin and General methods.
optional character string to specify the model fitting method for type="General" (if unspecified, either "REML" by default or the method that was used in fitting the "rma" model). See rma.uni for options.
logical to specify whether the general method should be based on exact (but slower) or approximate (but faster) calculations.
logical to specify whether output should be generated on the progress of the calculations for type="General" (the default is FALSE).
optional integer to specify the number of decimal places to which the printed results should be rounded.
other arguments.
The function can be used to calculate the ‘fail-safe N’, that is, the minimum number of studies averaging null results that would have to be added to a given set of \(k\) studies to change the conclusion of a meta-analysis. If this number is small (in relation to the actual number of studies), then this indicates that the results based on the observed studies are not robust to publication bias (of the form assumed by the method, that is, where a set of studies averaging null results is missing). The method is also called a ‘file drawer analysis’ as it assumes that there is a set of studies averaging null results hiding in file drawers, which can overturn the findings from a meta-analysis. There are various types of methods that are all based on the same principle, which are described in more detail further below. Note that the fail-safe N is not an estimate of the number of missing studies, only how many studies must be hiding in file drawers for the findings to be overturned.
One can either pass a vector with the observed effect sizes or outcomes (via x) and the corresponding sampling variances via vi (or the standard errors via sei) to the function or an object of class "rma". When passing a model object, the model must be a model without moderators (i.e., either an equal- or a random-effects model).
The Rosenthal method (type="Rosenthal") calculates the minimum number of studies averaging null results that would have to be added to a given set of studies to reduce the (one-tailed) combined significance level (i.e., p-value) to a particular alpha level, which can be specified via the alpha argument (.05 by default). The calculation is based on Stouffer's method for combining p-values and is described in Rosenthal (1979). Note that the method is primarily of interest for historical reasons, but the other methods described below are more closely aligned with the way meta-analyses are typically conducted in practice.
The Orwin method (type="Orwin") calculates the minimum number of studies averaging null results that would have to be added to a given set of studies to reduce the (unweighted or weighted) average effect size / outcome to a target value (as specified via the target argument). The method is described in Orwin (1983). When vi (or sei) is not specified, the method is based on the unweighted average of the effect sizes / outcomes; otherwise, the method uses the inverse-variance weighted average. If the target argument is not specified, then the target value will be equal to the observed average effect size / outcome divided by 2 (which is entirely arbitrary and will always lead to a fail-safe N number that is equal to \(k\)). One should really set target to a value that reflects an effect size / outcome that would be considered to be practically irrelevant. Note that if target has the opposite sign as the actually observed average, then its sign is automatically flipped.
The Rosenberg method (type="Rosenberg") calculates the minimum number of studies averaging null results that would have to be added to a given set of studies to reduce the significance level (i.e., p-value) of the average effect size / outcome (as estimated based on an equal-effects model) to a particular alpha level, which can be specified via the alpha argument (.05 by default). The method is described in Rosenberg (2005). Note that the p-value is calculated based on a standard normal distribution (instead of a t-distribution, as suggested by Rosenberg, 2005), but the difference is typically negligible.
This method is a generalization of the methods by Orwin and Rosenberg (see Viechtbauer, 2024). By default (i.e., when target is not specified), it calculates the minimum number of studies averaging null results that would have to be added to a given set of studies to reduce the significance level (i.e., p-value) of the average effect size / outcome (as estimated based on a chosen model) to a particular alpha level, which can be specified via the alpha argument (.05 by default). The type of model that is used in the calculation is chosen via the method argument. If this is unspecified, then a random-effects model is automatically used (using method="REML") or the method that was used in fitting the "rma" model (see rma.uni for options). Therefore, when setting method="EE", then an equal-effects model is used, which yields (essentially) identical results as Rosenberg's method.
If target is specified, then the method calculates the minimum number of studies averaging null results that would have to be added to a given set of studies to reduce the average effect size / outcome (as estimated based on a chosen model) to a target value (as specified via the target argument). As described above, the type of model that is used in the calculation is chosen via the method argument. When setting method="EE", then an equal-effects model is used, which yields (essentially) identical results as Orwin's method with inverse-variance weights.
The method uses an iterative algorithm for calculating the fail-safe N, which can be computationally expensive especially when N is large. By default, the method uses approximate (but faster) calculations, but when setting exact=TRUE, the method uses exact (but slower) calculations. The difference between the two is typically negligible. If N is larger than \(10^7\), then the calculated number is given as >1e+07.
An object of class "fsn". The object is a list containing the following components (some of which may be NA if they are not applicable to the chosen method):
the type of method used.
the calculated fail-safe N.
the average effect size / outcome based on the observed studies.
the estimated amount of heterogeneity based on the observed studies.
the p-value of the observed results.
the specified target alpha level.
the target average effect size / outcome.
the average effect size / outcome when combining the observed studies with those in the file drawer.
the estimated amount of heterogeneity when combining the observed studies with those in the file drawer.
the p-value when combining the observed studies with those in the file drawer.
some additional elements/values.
The results are formatted and printed with the print function.
If the significance level of the observed studies is already above the specified alpha level or if the average effect size / outcome of the observed studies is already below the target average effect size / outcome, then the fail-safe N value is zero.
Rosenthal, R. (1979). The "file drawer problem" and tolerance for null results. Psychological Bulletin, 86(3), 638–641. https://doi.org/10.1037/0033-2909.86.3.638
Orwin, R. G. (1983). A fail-safe N for effect size in meta-analysis. Journal of Educational Statistics, 8(2), 157–159. https://doi.org/10.3102/10769986008002157
Rosenberg, M. S. (2005). The file-drawer problem revisited: A general weighted method for calculating fail-safe numbers in meta-analysis. Evolution, 59(2), 464–468. https://doi.org/10.1111/j.0014-3820.2005.tb01004.x
Viechtbauer, W. (2010). Conducting meta-analyses in R with the metafor package. Journal of Statistical Software, 36(3), 1–48. https://doi.org/10.18637/jss.v036.i03
Viechtbauer, W. (2024). A fail-safe N computation based on the random-effects model. Annual Meeting of the Society for Research Synthesis Methodology, Amsterdam, The Netherlands. https://www.wvbauer.com/lib/exe/fetch.php/talks:2024_viechtbauer_srsm_fail_safe_n.pdf
### calculate log risk ratios and corresponding sampling variances
dat <- escalc(measure="RR", ai=tpos, bi=tneg, ci=cpos, di=cneg, data=dat.bcg)
### fit equal-effects model
rma(yi, vi, data=dat, method="EE")
#>
#> Equal-Effects Model (k = 13)
#>
#> I^2 (total heterogeneity / total variability): 92.12%
#> H^2 (total variability / sampling variability): 12.69
#>
#> Test for Heterogeneity:
#> Q(df = 12) = 152.2330, p-val < .0001
#>
#> Model Results:
#>
#> estimate se zval pval ci.lb ci.ub
#> -0.4303 0.0405 -10.6247 <.0001 -0.5097 -0.3509 ***
#>
#> ---
#> Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
#>
### fail-safe N computations
fsn(yi, vi, data=dat)
#>
#> Fail-safe N Calculation Using the Rosenthal Approach
#>
#> Observed Significance Level: <.0001
#> Target Significance Level: 0.05
#>
#> Fail-safe N: 598
#>
fsn(yi, data=dat, type="Orwin", target=log(0.95)) # target corresponds to a 5% risk reduction
#>
#> Fail-safe N Calculation Using the Orwin Approach
#>
#> Average Effect Size: -0.7407
#> Target Effect Size: -0.0513
#>
#> Fail-safe N: 175
#>
fsn(yi, vi, data=dat, type="Orwin", target=log(0.95)) # Orwin's method with 1/vi weights
#>
#> Fail-safe N Calculation Using the Orwin Approach
#>
#> Average Effect Size: -0.4303
#> Target Effect Size: -0.0513
#>
#> Fail-safe N: 97
#>
fsn(yi, vi, data=dat, type="General", target=log(0.95), method="EE") # like Orwin's method
#>
#> Fail-safe N Calculation Using the General Approach
#>
#> Average Effect Size: -0.4303 (with file drawer: -0.0509)
#> Observed Significance Level: <.0001 (with file drawer: 0.0003)
#> Target Effect Size: -0.0513
#>
#> Fail-safe N: 97
#>
fsn(yi, vi, data=dat, type="Rosenberg")
#>
#> Fail-safe N Calculation Using the Rosenberg Approach
#>
#> Average Effect Size: -0.4303
#> Observed Significance Level: <.0001
#> Target Significance Level: 0.05
#>
#> Fail-safe N: 370
#>
fsn(yi, vi, data=dat, type="General", method="EE") # like Rosenberg's method
#>
#> Fail-safe N Calculation Using the General Approach
#>
#> Average Effect Size: -0.4303 (with file drawer: -0.0146)
#> Observed Significance Level: <.0001 (with file drawer: 0.0503)
#> Target Significance Level: 0.05
#>
#> Fail-safe N: 370
#>
fsn(yi, vi, data=dat, type="General") # based on a random-effects model
#>
#> Fail-safe N Calculation Using the General Approach
#>
#> Average Effect Size: -0.7145 (with file drawer: -0.2112)
#> Amount of Heterogeneity: 0.3132 (with file drawer: 0.4214)
#> Observed Significance Level: <.0001 (with file drawer: 0.0543)
#> Target Significance Level: 0.05
#>
#> Fail-safe N: 26
#>
fsn(yi, vi, data=dat, type="General", target=log(0.95)) # based on a random-effects model
#>
#> Fail-safe N Calculation Using the General Approach
#>
#> Average Effect Size: -0.7145 (with file drawer: -0.0508)
#> Amount of Heterogeneity: 0.3132 (with file drawer: 0.3516)
#> Observed Significance Level: <.0001 (with file drawer: 0.3109)
#> Target Effect Size: -0.0513
#>
#> Fail-safe N: 138
#>
### fit a random-effects model and use fsn() on the model object
res <- rma(yi, vi, data=dat)
fsn(res)
#>
#> Fail-safe N Calculation Using the General Approach
#>
#> Average Effect Size: -0.7145 (with file drawer: -0.2112)
#> Amount of Heterogeneity: 0.3132 (with file drawer: 0.4214)
#> Observed Significance Level: <.0001 (with file drawer: 0.0543)
#> Target Significance Level: 0.05
#>
#> Fail-safe N: 26
#>
fsn(res, target=log(0.95))
#>
#> Fail-safe N Calculation Using the General Approach
#>
#> Average Effect Size: -0.7145 (with file drawer: -0.0508)
#> Amount of Heterogeneity: 0.3132 (with file drawer: 0.3516)
#> Observed Significance Level: <.0001 (with file drawer: 0.3109)
#> Target Effect Size: -0.0513
#>
#> Fail-safe N: 138
#>