[R-meta] Dear Wolfgang
Dear James, Thank you so much for these detailed thoughts and suggestions. My co-authors and I find this input extremely helpful. Currently, we are analyzing fishery data across the US, trying to understand how habitat or environmental contexts influence fish productivity by combining both peer-reviewed and agency data. We think our current issue is most related to your first point. We have >80 peer-reviewed papers and 3-4 agency dataset. One major difference between peer-reviewed articles and agency data is that agency data are collected more frequently, across larger areas and different sites, and over a longer time frame (> 20 yr long data accumulated). Also, they do indiscriminately record all catch, whereas peer-reviewed papers often report ones that are more relevant to their research question (one that is more aligned with our research question as well). So, after reading your comments, I envision us applying a stricter rule for the agency data. We have 1000 or so effect sizes combining many peer-reviewed articles, whereas each dataset generates between a whopping 9000-13000 effect sizes if we apply the same inclusion criteria. Our follow-up questions are: 1. Given that we think there is still some value in including the agency data in our analysis, is it reasonable to apply different criteria or rules (more strict) just for the agency data? For example, in our peer-reviewed data, we treat samplings conducted in different areas and years as independent studies and as these effects are matched with discrete measurements for our moderator of interest (say temperature or depth). I am wondering if it is justifiable if we apply a different protocol just for agency data (ex. merging and pooling across all years and sites and just generate a single effect size for each fish species OR only including randomly chosen year or site from the dataset) for the sake of not taking over the entire data with agency ones. 2. One option we were considering was to run our models with and without agency data and report both. However, you pointed out that that model output (including such an abnormally large study) may not be reliable at all if there is such a huge study size differences to begin with. So, my understanding is that this should not be one of our options unless we can significantly reduce the number of agency data being incorporated? 3. Finally, you mentioned analyzing the agency data separately. I have not considered this option but say we have two agency datasets that we combine to run our models. There would be plenty of effect sizes (coming from different sites, years, observers, and fish species) but only two levels of study (data 1, 2). I am unsure if this approach would make any sense, but do you have any additional thoughts on the validity of such models or approaches? Thank you very much for your time, James. I hope my answers and follow-up questions are clear enough. Best regards, JU
From: James Pustejovsky <jepusto at gmail.com>
Sent: Wednesday, December 9, 2020 6:12 PM
To: Ju Lee <juhyung2 at stanford.edu>
Cc: Viechtbauer Wolfgang (SP) <wolfgang.viechtbauer at maastrichtuniversity.nl>; R meta <r-sig-meta-analysis at r-project.org>
Subject: Re: [R-meta] Dear Wolfgang
Sent: Wednesday, December 9, 2020 6:12 PM
To: Ju Lee <juhyung2 at stanford.edu>
Cc: Viechtbauer Wolfgang (SP) <wolfgang.viechtbauer at maastrichtuniversity.nl>; R meta <r-sig-meta-analysis at r-project.org>
Subject: Re: [R-meta] Dear Wolfgang
Hi JU, This is a very interesting question. My personal sense of it is that one should be very cautious when working with such highly imbalanced datasets like this (where some studies contribute just a few effect sizes while others contribute orders of magnitude more). The usual methods for handling dependent effects (such as multi-level meta-analysis, robust variance estimation, etc.) can do strange and wacky things in this situation. As far as I know, their performance has not been evaluated with data structures like this, and my understanding is that the usual statistical theory supporting these methods doesn't necessarily apply very well. To make progress, I would recommend first carefully reviewing your inclusion criteria and checking whether the effect sizes from the agency studies are really comparable and aligned with the effect sizes extracted from the peer-reviewed papers. One possible scenario is that the peer-reviewed papers all only report outcomes on well-validated scales, whereas the agency studies report outcomes on a kitchen sink of different outcomes, including well-validated full scales but also never-validated scales, sub-scales, single items, and assorted other oddities. In this situation, I don't seem any benefit to throwing in all the sub-scales and other chaff, just because it's available. Better to use stricter inclusion criteria and just focus on the sound, validated measures. Another possible scenario is that the peer-reviewed studies report one sort of information, whereas the agency studies report a categorically different sort of information (with little or no overlap in terms of the measures used, scale of the samples, etc.). In this case, it would seem sensible to perform separate syntheses of the peer-reviewed literature and the agency studies, then scratch your head over how to understand differences between the two bodies of evidence. Another possible scenario is that the peer-reviewed studies are all poorly reported (and potentially selectively reported), whereas the agency studies are not just completely reported, but also include information on a wider range of relevant outcomes (e.g., effects over longer follow-up times) than the peer-reviewed stuff. In that case, it seems useful and potentially important to include the broader range of effects from the agency studies. However, it needs to be done with care. In particular, it will be critical to understand the sources of variation *within* the agency studies. You might investigate this by conducting preliminary analyses of the effects from each agency study *on its own*, understanding what the important within-study moderators are, and only then thinking about how to line up the evidence from the agency studies with the evidence from the peer reviewed studies. I'd be curious to hear more about the context you're working in, and which (if any) of these scenarios you find yourself in. I'm also very interested to hear others perspectives on this question. Kind Regards, James > On Dec 9, 2020, at 12:07 PM, Ju Lee <juhyung2 at stanford.edu> wrote: > > ?Dear Wolfgang, > > I hope you are well these days. > > I had some general questions related to the data structure in mixed-effect models. > We are currently working with data extracted from pee-reviewed papers as well as big data extracted from state or agency surveys. > The issue we have is although we are including only 2-3 agency studies, each study can generate up to 1000-9000 effect sizes due to the abundance of data they produced. > > Conversely, the data collected from peer-reviewed articles are much smaller than that perhaps < 800 effect sizes combined. My co-authors want to use those agency data, but I am very concerned that including those data makes sense. > > So the question is: > > 1. Is it even reasonable to consider including such few studies that will pretty much dominate the entire data? > 2. Can common mixed-effect model approaches with study random factor account for such disproportionate contribution of few studies? > > It would be extremely helpful to hear your perspectives. > Thank you > > Best, > JU > > [[alternative HTML version deleted]] > > _______________________________________________ > R-sig-meta-analysis mailing list > R-sig-meta-analysis at r-project.org > https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis