[R-meta] rma, sandwich correction and very small data sets

Just want to add to this an answer to question 5.

robust() from metafor with adjust=TRUE does use a small-sample adjustment,
but a very simple one. clubSandwich has much more sophisticated adjustments
and I would recommend to use them (CR2 is the recommended one).

Best,
Wolfgang

-----Original Message-----
From: Michael Dewey [mailto:lists at dewey.myzen.co.uk]
Sent: Wednesday, 09 December, 2020 17:47
To: Valeria Ivaniushina; Viechtbauer, Wolfgang (SP)
Cc: R meta
Subject: Re: [R-meta] rma, sandwich correction and very small data sets

Dear Valeria

I think as a general principle you are entitled to do your analysis even
on a small data-set as long as you accept that your results may not be
very precise. There seems to be a general feeling among analysts in the
area in which I work (health) that looking for small study effects is
not worth trying with fewer than ten studies and even with more may well
be uninformative. I am personally rather sceptical about identifying
observations as outliers in the absence of a scientific reason for doing

so.

Michael

On 09/12/2020 15:21, Valeria Ivaniushina wrote:

Dear Wolfgang,

Thank you VERY much!
Thank you for correcting my code -- indeed, random effect on the 1st

level

is totally needed!

A couple more questions, if I may

1. There are too little cases for such a complex data structure, and

it's

a

serious limitation.
But I hope that even if the results may be considered only as

descriptive,

they still point out in the correct direction?
Especially taking into account that all three subsamples show quite

similar

results.
Is it a valid interpretation?

2. Considering that the sample is small (and 3-level!),  I guess that
analysis of outliers would be excessive. Is it right?

3. The same goes for publication bias analysis? (as James points out,

these

tests do not have strong power:
www.jepusto.com/publication/selective-reporting-with-dependent-effects/

)

4. and there is no power for mediation analysis, so I don't have to even
attempt to do it?

5. Estimators question:
  "robust" function in rma is using sandwich-type estimator, and with

adjust

= TRUE it does a small-sample adjustment
In the clubSandwich library there are a bunch of estimators with

different

small sample corrections. They give somewhat different results, some are
very close to "robust" output
Is clubSandwich  CR2 (for example) better than robust.rma?
Or, if CR estimators from clubSandwich are not definitely preferable,

can

I

just use robust.rma?

Best,
Valeria