[R-meta] mean-variance relationships introduces additional heterogeneity, how?
Hi James, Thanks a lot for investing so much effort into my question! Let me ask a quick question regarding the second diagnostic in your post. In your post, you note that *"[Since] there is a strong linear relationship between the two [groups'] means, with a best-fit line that might go through the origin. . . the response ratio might be an appropriate metric."* Could you please elaborate on how this speaks to the appropriateness of LRR over SMD? Luke
On Tue, Nov 2, 2021 at 8:12 AM James Pustejovsky <jepusto at gmail.com> wrote:
HI Luke and listserv, I wrote up some thoughts on the question of using standardized mean differences to analyze outcomes measured as proportions: https://www.jepusto.com/mean-variance-relationships-and-smds/ Thoughts, comments, questions, and critiques welcome. James On Mon, Oct 25, 2021 at 9:07 PM James Pustejovsky <jepusto at gmail.com> wrote:
All I mean is that a skewed distribution or one with large outliers does not necessarily *imply* that a mean-sd relationship exists. It could be the result of one, but skewness might be due to something else (such as selective reporting) instead. I would suggest that a well-behaved effect distribution is desirable and appropriate to the extent that it indicates empirical regularity of the phenomenon you're interested in. A less heterogeneous distribution means that effects are more predictable (at least in the corpus of studies that you're examining). On Mon, Oct 25, 2021 at 8:58 PM Luke Martinez <martinezlukerm at gmail.com> wrote:
I thought the existence of outlying effect estimates under SMD and lack of it under LRR could attest to the existence of heterogeneity-generating artefacts like mean-sd relationships (and/or variation in measurement error) across the studies. If not, then, would you mind commenting on why a more symmetric and well-behaved effect distribution is equated with its appropriateness for a set of summaries (e.g., means & sds) from studies? Luke On Mon, Oct 25, 2021 at 8:47 PM James Pustejovsky <jepusto at gmail.com> wrote:
Responses below. On Mon, Oct 25, 2021 at 4:21 PM Luke Martinez <martinezlukerm at gmail.com> wrote:
Sure, thanks. Along the same lines, if I see that the unconditional distribution of the SMD estimates is multi-modal or right or left skewed (perhaps due to extreme outliers), but the unconditional distribution of the corresponding LRR estimates looks more symmetric and well-behaved, does that also empirically suggest a mean-sd relationship in one or more groups?
I'm not sure that it implies a mean-sd relationship. But I think it does suggest that LRR might be a more appropriate metric.
PS. Is there a reason for exploring the mean-sd relationship specifically in the control group?
No, you could certainly examine the relationships in the treatment group(s) as well.