Dear Vivien,
My response is only to the second question (weighting using quality scores):
This is strongly discouraged at least by Cochrane. It is a matter of the
difference between the multidimensional concept of quality and the
unidimensional concept of bias (which we want to avoid). Not every item
of quality leads to bias, and if it does, the direction of bias might be
unclear. Biases may cancel out. Moreover, to downweight a biased study
still leads to bias, thus you might as well throw the study out.
See the Cochrane Handbook http://handbook-5-1.cochrane.org/ , particularly
8.3? Tools for assessing quality and risk of bias
8.3.1 Types of tools
8.3.2 Reporting versus conduct
8.3.3 Quality scales and Cochrane reviews
8.3.4 Collecting information for assessments of risk of bias
8.15.2? Assessing risk of bias from other sources
References:
Sander Greenland and Keith O'Rourke, "On the bias produced by quality
scores in meta-analysis, and a hierarchical view of proposed solutions",
Biostatistics., vol. 2, pp. 463-471, 2001.
Peter J?ni, Douglas G. Altman, and Matthias Egger, "Assessing the
quality of controlled clinical trials", Brit. Med. J., vol. 323, pp.
42-46, 2001.
A. R. Jadad, D. J. Cook, A. Jones, T. P. Klassen, P. Tugwell, M. Moher,
and D. Moher, "Methodology and reports of systematic reviews and
meta-analyses: A comparison of Cochrane reviews with articles published
in paper-based journals.", J. Amer. Med. Assoc., vol. 280, pp. 278-80, 1998.
Emerson JD, Burdick E, Hoaglin DC, Mosteller F, Chalmers TC. An
empirical study of the possible relation of treatment differences to
quality scores in controlled randomized clinical trials. Controlled
Clinical Trials 1990; 11: 339-352.
Schulz KF, Chalmers I, Hayes RJ, Altman DG. Empirical evidence of bias.
Dimensions of methodological quality associated with estimates of
treatment effects in controlled trials. JAMA 1995; 273: 408-412.
Best,
Gerta
Am 19.01.2018 um 18:04 schrieb Vivien Bonnesoeur:
Dear all,
I would need some advice in the way to combine quality score and inverse
variance for weighting studies.
I'm contrasting the infiltration rate between tree plantation and grassland
and also tree plantation and native forest (effect size = Log ROM) to know
if tree plantation on grassland can increase the infiltration and recover
to level of infiltration of native forest.
here is the raw data :
article;trial;Land-use_change;Plantation_N;Plantation_mean;Plantation_sd;Control_N;Control_mean;Control_sd;yi;vi;quality_score
Gaitan2016;1;Plantation-grassland;32;36;41;32;11;7;1.186;0.053;1
Gonzalez2015;2;Plantation-grassland;9;76.6;17.6;2;76.6;37.5;0.000;0.126;0.8
Hoyos2005;3;Plantation-grassland;3;101.3;66;23;2.5;1.6;3.702;0.159;0.5
Hoyos2005;4;Plantation-Native_forest;3;101.3;66;3;225;271;-0.798;0.625;0.5
Moreno2012;5;Plantation-grassland;10;8064;7092;10;5004;7092;0.477;0.278;0.3
Moreno2012;6;Plantation-Native_forest;10;8064;7092;10;34092;7092;-1.442;0.082;0.3
Sadeghian2001;7;Plantation-grassland;12;210;120;16;30;27;1.946;0.078;1
Sadeghian2001;8;Plantation-Native_forest;12;210;120;16;760;439;-1.286;0.048;1
Zimmerman2007;9;Plantation-grassland;30;514;137;30;3;4;5.144;0.062;0.8
Zimmerman2007;10;Plantation-Native_forest;30;514;137;30;135;51;1.337;0.007;0.6
If I just used the inverse variance-covariance weighting (to account for
dependency between the reuse of some plantations) :
model1 = rma.mv
(yi,V,mods=~Land-use_change-1,method="REML",slab=article,random=~factor(trial)|article,data=ma.infilt)
I end with a lot of weight to the studies where there is a reuse of the
plantation. Actually, those weight are really different from the inverse
vi. For example
Gaitan2016 : weight(model1) = 2.98% ; weight from inverse vi = 7.8%
Hoyos2005.1 :weight(model1) = 9.01% ; weight from inverse vi = 2.6%
Hoyos2005.2 :weight(model1) = 7.67% ; weight from inverse vi = 0.66%
Zimmerman2007.1 :weight(model1) = 13.6% ; weight from inverse vi = 6.7%
Zimmerman2007.2 :weight(model1) = 13.9% ; weight from inverse vi = 58%
Here I have a first question :
-is there a way to reduce the weight of studies where the plantation is
reused for contrasting with 2 different control? It seems to be an
artificial over-weighting decision to me?
Besides, some studies with a low quality score have stronger weights than
studies with high quality score. To combine the quality score and the
inverse variance in study weighting, my try is to use the weight from the
model1 and to multiply it with the quality score in this way :
model2=rma.mv
(yi,V,mods=~Land-use_change-1,W=(ma.infilt$quality_score*weights(model1))/sum(ma.infilt$quality_score*weights(model1)),method="REML",slab=article,random=~factor(trial)|article,data=ma.infilt)
It gives more satisfactory weigths since the studies with very low quality
score have now a small contribution to the grand mean.
I would like to know however if this way of combining the quality and
inverse variance weighting is sound theoretically and won't be rejected by
reviewer as a "critical flaw"
Best regards
Dr. rer. nat. Gerta R?cker, Dipl.-Math.
Institute of Medical Biometry and Statistics,
Faculty of Medicine and Medical Center - University of Freiburg
Stefan-Meier-Str. 26, D-79104 Freiburg, Germany
Phone: +49/761/203-6673
Fax: +49/761/203-6680
Mail: ruecker at imbi.uni-freiburg.de
Homepage: https://portal.uni-freiburg.de/imbi/persons/ruecker?set_language=en