`fsn.Rd`

Function to compute the fail-safe N (also called a file drawer analysis).

```
fsn(x, vi, sei, subset, data, type, alpha=.05, target,
method, exact=FALSE, verbose=FALSE, digits, ...)
```

- x
a vector with the observed effect sizes or outcomes or an object of class

`"rma"`

.- vi
vector with the corresponding sampling variances (ignored if

`x`

is an object of class`"rma"`

).- sei
vector with the corresponding standard errors (note: only one of the two,

`vi`

or`sei`

, needs to be specified).- subset
optional (logical or numeric) vector to specify the subset of studies that should be used for the calculation (ignored if

`x`

is an object of class`"rma"`

).- data
optional data frame containing the variables given to the arguments above.

- type
optional character string to specify the type of method to use for the calculation of the fail-safe N. Possible options are

`"Rosenthal"`

(the default when`x`

is a vector with the observed effect sizes or outcomes),`"Orwin"`

,`"Rosenberg"`

, or`"General"`

(the default when`x`

is an object of class`"rma"`

). Can be abbreviated. See ‘Details’.- alpha
target alpha level for the Rosenthal, Rosenberg, and General methods (the default is .05).

- target
target average effect size or outcome for the Orwin and General methods.

- method
optional character string to specify the model fitting method for

`type="General"`

(if unspecified, either`"REML"`

by default or the method that was used in fitting the`"rma"`

model). See`rma.uni`

for options.- exact
logical to specify whether the general method should be based on exact (but slower) or approximate (but faster) calculations.

- verbose
logical to specify whether output should be generated on the progress of the calculations for

`type="General"`

(the default is`FALSE`

).- digits
optional integer to specify the number of decimal places to which the printed results should be rounded.

- ...
other arguments.

The function can be used to calculate the ‘fail-safe N’, that is, the minimum number of studies averaging null results that would have to be added to a given set of \(k\) studies to change the conclusion of a meta-analysis. If this number is small (in relation to the actual number of studies), then this indicates that the results based on the observed studies are not robust to publication bias (of the form assumed by the method, that is, where a set of studies averaging null results is missing). The method is also called a ‘file drawer analysis’ as it assumes that there is a set of studies averaging null results hiding in file drawers, which can overturn the findings from a meta-analysis. There are various types of methods that are all based on the same principle, which are described in more detail further below. Note that *the fail-safe N is not an estimate of the number of missing studies*, only how many studies must be hiding in file drawers for the findings to be overturned.

One can either pass a vector with the observed effect sizes or outcomes (via `x`

) and the corresponding sampling variances via `vi`

(or the standard errors via `sei`

) to the function or an object of class `"rma"`

. When passing a model object, the model must be a model without moderators (i.e., either an equal- or a random-effects model).

The Rosenthal method (`type="Rosenthal"`

) calculates the minimum number of studies averaging null results that would have to be added to a given set of studies to reduce the (one-tailed) combined significance level (i.e., p-value) to a particular alpha level, which can be specified via the `alpha`

argument (.05 by default). The calculation is based on Stouffer's method for combining p-values and is described in Rosenthal (1979). Note that the method is primarily of interest for historical reasons, but the other methods described below are more closely aligned with the way meta-analyses are typically conducted in practice.

The Orwin method (`type="Orwin"`

) calculates the minimum number of studies averaging null results that would have to be added to a given set of studies to reduce the (unweighted or weighted) average effect size / outcome to a target value (as specified via the `target`

argument). The method is described in Orwin (1983). When `vi`

(or `sei`

) is not specified, the method is based on the unweighted average of the effect sizes / outcomes; otherwise, the method uses the inverse-variance weighted average. If the `target`

argument is not specified, then the target value will be equal to the observed average effect size / outcome divided by 2 (which is entirely arbitrary and will always lead to a fail-safe N number that is equal to \(k\)). One should really set `target`

to a value that reflects an effect size / outcome that would be considered to be practically irrelevant. Note that if `target`

has the opposite sign as the actually observed average, then its sign is automatically flipped.

The Rosenberg method (`type="Rosenberg"`

) calculates the minimum number of studies averaging null results that would have to be added to a given set of studies to reduce the significance level (i.e., p-value) of the average effect size / outcome (as estimated based on an equal-effects model) to a particular alpha level, which can be specified via the `alpha`

argument (.05 by default). The method is described in Rosenberg (2005). Note that the p-value is calculated based on a standard normal distribution (instead of a t-distribution, as suggested by Rosenberg, 2005), but the difference is typically negligible.

This method is a generalization of the methods by Orwin and Rosenberg. By default (i.e., when `target`

is not specified), it calculates the minimum number of studies averaging null results that would have to be added to a given set of studies to reduce the significance level (i.e., p-value) of the average effect size / outcome (as estimated based on a chosen model) to a particular alpha level, which can be specified via the `alpha`

argument (.05 by default). The type of model that is used in the calculation is chosen via the `method`

argument. If this is unspecified, then a random-effects model is automatically used (using `method="REML"`

) or the method that was used in fitting the `"rma"`

model (see `rma.uni`

for options). Therefore, when setting `method="EE"`

, then an equal-effects model is used, which yields (essentially) identical results as Rosenberg's method.

If `target`

is specified, then the method calculates the minimum number of studies averaging null results that would have to be added to a given set of studies to reduce the average effect size / outcome (as estimated based on a chosen model) to a target value (as specified via the `target`

argument). As described above, the type of model that is used in the calculation is chosen via the `method`

argument. When setting `method="EE"`

, then an equal-effects model is used, which yields (essentially) identical results as Orwin's method with inverse-variance weights.

The method uses an iterative algorithm for calculating the fail-safe N, which can be computationally expensive especially when N is large. By default, the method uses approximate (but faster) calculations, but when setting `exact=TRUE`

, the method uses exact (but slower) calculations. The difference between the two is typically negligible. If N is larger than \(10^7\), then the calculated number is given as `>1e+07`

.

An object of class `"fsn"`

. The object is a list containing the following components (some of which may be `NA`

if they are not applicable to the chosen method):

- type
the type of method used.

- fsnum
the calculated fail-safe N.

- est
the average effect size / outcome based on the observed studies.

- tau2
the estimated amount of heterogeneity based on the observed studies.

- pval
the p-value of the observed results.

- alpha
the specified target alpha level.

- target
the target average effect size / outcome.

- est.fsn
the average effect size / outcome when combining the observed studies with those in the file drawer.

- tau2
the estimated amount of heterogeneity when combining the observed studies with those in the file drawer.

- pval
the p-value when combining the observed studies with those in the file drawer.

- ...
some additional elements/values.

The results are formatted and printed with the `print`

function.

If the significance level of the observed studies is already above the specified alpha level or if the average effect size / outcome of the observed studies is already below the target average effect size / outcome, then the fail-safe N value is zero.

Rosenthal, R. (1979). The "file drawer problem" and tolerance for null results. *Psychological Bulletin*, **86**(3), 638–641. https://doi.org/10.1037/0033-2909.86.3.638

Orwin, R. G. (1983). A fail-safe N for effect size in meta-analysis. *Journal of Educational Statistics*, **8**(2), 157–159. https://doi.org/10.3102/10769986008002157

Rosenberg, M. S. (2005). The file-drawer problem revisited: A general weighted method for calculating fail-safe numbers in meta-analysis. *Evolution*, **59**(2), 464–468. https://doi.org/10.1111/j.0014-3820.2005.tb01004.x

Viechtbauer, W. (2010). Conducting meta-analyses in R with the metafor package. *Journal of Statistical Software*, **36**(3), 1–48. https://doi.org/10.18637/jss.v036.i03

```
### calculate log risk ratios and corresponding sampling variances
dat <- escalc(measure="RR", ai=tpos, bi=tneg, ci=cpos, di=cneg, data=dat.bcg)
### fit equal-effects model
rma(yi, vi, data=dat, method="EE")
#>
#> Equal-Effects Model (k = 13)
#>
#> I^2 (total heterogeneity / total variability): 92.12%
#> H^2 (total variability / sampling variability): 12.69
#>
#> Test for Heterogeneity:
#> Q(df = 12) = 152.2330, p-val < .0001
#>
#> Model Results:
#>
#> estimate se zval pval ci.lb ci.ub
#> -0.4303 0.0405 -10.6247 <.0001 -0.5097 -0.3509 ***
#>
#> ---
#> Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
#>
### fail-safe N computations
fsn(yi, vi, data=dat)
#>
#> Fail-safe N Calculation Using the Rosenthal Approach
#>
#> Observed Significance Level: <.0001
#> Target Significance Level: 0.05
#>
#> Fail-safe N: 598
#>
fsn(yi, data=dat, type="Orwin", target=log(0.95)) # target corresponds to a 5% risk reduction
#>
#> Fail-safe N Calculation Using the Orwin Approach
#>
#> Average Effect Size: -0.7407
#> Target Effect Size: -0.0513
#>
#> Fail-safe N: 175
#>
fsn(yi, vi, data=dat, type="Orwin", target=log(0.95)) # Orwin's method with 1/vi weights
#>
#> Fail-safe N Calculation Using the Orwin Approach
#>
#> Average Effect Size: -0.4303
#> Target Effect Size: -0.0513
#>
#> Fail-safe N: 97
#>
fsn(yi, vi, data=dat, type="General", target=log(0.95), method="EE") # like Orwin's method
#>
#> Fail-safe N Calculation Using the General Approach
#>
#> Average Effect Size: -0.4303 (with file drawer: -0.0513)
#> Observed Significance Level: <.0001 (with file drawer: 0.0002)
#> Target Effect Size: -0.0513
#>
#> Fail-safe N: 97
#>
fsn(yi, vi, data=dat, type="Rosenberg")
#>
#> Fail-safe N Calculation Using the Rosenberg Approach
#>
#> Average Effect Size: -0.4303
#> Observed Significance Level: <.0001
#> Target Significance Level: 0.05
#>
#> Fail-safe N: 370
#>
fsn(yi, vi, data=dat, type="General", method="EE") # like Rosenberg's method
#>
#> Fail-safe N Calculation Using the General Approach
#>
#> Average Effect Size: -0.4303 (with file drawer: -0.0146)
#> Observed Significance Level: <.0001 (with file drawer: 0.0500)
#> Target Significance Level: 0.05
#>
#> Fail-safe N: 370
#>
fsn(yi, vi, data=dat, type="General") # based on a random-effects model
#>
#> Fail-safe N Calculation Using the General Approach
#>
#> Average Effect Size: -0.7145 (with file drawer: -0.2172)
#> Amount of Heterogeneity: 0.3132 (with file drawer: 0.4228)
#> Observed Significance Level: <.0001 (with file drawer: 0.0513)
#> Target Significance Level: 0.05
#>
#> Fail-safe N: 26
#>
fsn(yi, vi, data=dat, type="General", target=log(0.95)) # based on a random-effects model
#>
#> Fail-safe N Calculation Using the General Approach
#>
#> Average Effect Size: -0.7145 (with file drawer: -0.0511)
#> Amount of Heterogeneity: 0.3132 (with file drawer: 0.3518)
#> Observed Significance Level: <.0001 (with file drawer: 0.3094)
#> Target Effect Size: -0.0513
#>
#> Fail-safe N: 138
#>
### fit a random-effects model and use fsn() on the model object
res <- rma(yi, vi, data=dat)
fsn(res)
#>
#> Fail-safe N Calculation Using the General Approach
#>
#> Average Effect Size: -0.7145 (with file drawer: -0.2172)
#> Amount of Heterogeneity: 0.3132 (with file drawer: 0.4228)
#> Observed Significance Level: <.0001 (with file drawer: 0.0513)
#> Target Significance Level: 0.05
#>
#> Fail-safe N: 26
#>
fsn(res, target=log(0.95))
#>
#> Fail-safe N Calculation Using the General Approach
#>
#> Average Effect Size: -0.7145 (with file drawer: -0.0511)
#> Amount of Heterogeneity: 0.3132 (with file drawer: 0.3518)
#> Observed Significance Level: <.0001 (with file drawer: 0.3094)
#> Target Effect Size: -0.0513
#>
#> Fail-safe N: 138
#>
```