[R-meta] Meta-analysis of single group attitude scores - R-SIG-meta-analysis

Tue, Jan 16, 2018 7:26 AM #

Dear Michael and Wolfgang,

Thank you both very much for your helpful comments.
I have been testing the two methods suggested by Wolfgang on some fictional data to see where the differences lie, and I don?t fully understand the resulting sampling variances.

Fictional data of 2 studies:
Both are attitudes on a 7-point scale (neutral point = 4, range = 6) with n = 100.
Both studies have a mean score of 5. 
Study 1 SD = 1
Study 2 SD = 2.

With the first idea outlined by Wolfgang, the sampling variances are 0.015 (Study 1) and 0.01125 (Study 2).
With the second idea, the sampling variances are 0.000277 (Study 1) and 0.00111 (study 2).

So using the second idea, Study 2 results in a larger sampling variance, and therefore a less precise measurement, and inverse variance weighting results in a lower weight for this study. This seems to make sense as the SD for Study 2 is higher than for Study 1. 

However, using the first idea, the sampling variance of Study 2 is actually lower, which suggests that even though the SD of that study is higher, the study is more precise. 

Am I interpreting the results wrong? Or could it be that the first idea already incorporates the inverse variance calculation?

Thank you for your help!

Best wishes,
Tommy
===============
Dr Tommy van Steen
Research Associate in Psychology
University of Bath

On 26 Nov 2017, at 10:54, Viechtbauer Wolfgang (SP) <wolfgang.viechtbauer at maastrichtuniversity.nl> wrote:

I agree that ideally one would want to use the raw mean (or raw mean minus neutral point) as the outcome measure. However, if studies differ in terms of the number of answering possibilities, then the raw values are not really comparable.

Two ideas:

1) Divide by the SD. So you then compute d = (mean - neutral point) / SD. Then the (large-sample) sampling variance can be estimated with:

1/n + d^2/(2*n)

2) Divide by the possible range (not the observed one!). So you then compute d = (mean - neutral point) / range. Then the sampling variance can be estimated with:

SD^2 / (n * range^2)

Best,
Wolfgang

-----Original Message-----
From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces at r-
project.org] On Behalf Of Michael Dewey
Sent: Friday, 24 November, 2017 15:02
To: Tommy van Steen; r-sig-meta-analysis at r-project.org
Subject: Re: [R-meta] Meta-analysis of single group attitude scores

Dear Tommy

If you subtract the neutral point fr each study then this will only make
a difference if the studies have different neutral points but it would
not harm things anyway.

Dividing by the standard deviation seems unnecessary to me, and
potentially harmful. Why not use the standard error of each mean? Then
you would get the proper weights and an estimate of heterogeneity.

Michael

On 24/11/2017 12:26, Tommy van Steen wrote:

Dear all,

I have a question about my meta-analytic approach in a project that I?m

working on.

In this project, we are conducting a meta-analysis of attitudes. The

study outcome variable is the mean attitude score on a scale (often a
Likert-scale type survey or single item).

I am trying to figure out what the best way is to conduct a meta-

analysis based on this type of data as the data comes from single groups
and studies vary in the number of answering possibilities on the Likert-
scales.

My thought would be to transform the study mean into a z-score by

subtracting the neutral score (e.g. ?3' in a 5-point scale study) from
the study mean and divide the outcome by the standard deviation of the
study mean. This way, the z-score reflects whether the attitude was
positive (f the z-score is positive) or negative (if the z-score is
negative), with 0 being neutral. I would then use this z-score as the
effect size for my meta-analysis.

Based on this, I have three questions:
1. Is this a sensible option?
2. If so, how should I calculate the study weights? (As z-scores

typically have an SD of 1 if I understand correctly.)

3. How would I run the meta-analysis based on these z-scores? (Simply

loading the scores as effect sizes in an SMD meta-analysis using the
metafor-package seems odd perhaps?)

Thank you very much for your time and thoughts.

With kind regards,
Tommy van Steen

Viechtbauer Wolfgang (STAT)

Tue, Jan 16, 2018 9:34 AM #

Hi Tommy,

The sampling variance of a 'd' value (as in the present context, but this also applies to 'd' values in the more usual context where two independent groups are being compared) is only very indirectly a function of the SD of the measurements. In fact, the SD only affects the 'd' values itself, which is part of the computation of the sampling variance, but it does not enter the equation of the sampling variance in any other form. This might not be very intuitive, but that's just how it is.

Generally though, the effect of the 'd' value on the sampling variance is relatively minor compared to how strongly it is affected by the sample size. For example, if n had been 200 for the first study, then we would get a sampling variance of 0.0075, which is much smaller than that of the second study. Now that indeed makes a lot of intuitive sense.

Best,
Wolfgang

-----Original Message-----
From: Tommy van Steen [mailto:tommyvansteen at yahoo.com] 
Sent: Tuesday, 16 January, 2018 16:27
To: Viechtbauer Wolfgang (SP)
Cc: Michael Dewey; r-sig-meta-analysis at r-project.org
Subject: Re: [R-meta] Meta-analysis of single group attitude scores

Dear Michael and Wolfgang,

Thank you both very much for your helpful comments.
I have been testing the two methods suggested by Wolfgang on some fictional data to see where the differences lie, and I don?t fully understand the resulting sampling variances.

Fictional data of 2 studies:
Both are attitudes on a 7-point scale (neutral point = 4, range = 6) with n = 100.
Both studies have a mean score of 5.?
Study 1 SD = 1
Study 2 SD = 2.

With the first idea outlined by Wolfgang, the sampling variances are 0.015 (Study 1) and 0.01125 (Study 2).
With the second idea, the sampling variances are 0.000277 (Study 1) and 0.00111 (study 2).

So using the second idea, Study 2 results in a larger sampling variance, and therefore a less precise measurement, and inverse variance weighting results in a lower weight for this study. This seems to make sense as the SD for Study 2 is higher than for Study 1.?

However, using the first idea, the sampling variance of Study 2 is actually lower, which suggests that even though the SD of that study is higher, the study is more precise.?

Am I interpreting the results wrong? Or could it be that the first idea already incorporates the inverse variance calculation?

Thank you for your help!

Best wishes,
Tommy
===============
Dr Tommy van Steen
Research Associate in?Psychology
University of Bath

On 26 Nov 2017, at 10:54, Viechtbauer Wolfgang (SP) <wolfgang.viechtbauer at maastrichtuniversity.nl> wrote:

I agree that ideally one would want to use the raw mean (or raw mean minus neutral point) as the outcome measure. However, if studies differ in terms of the number of answering possibilities, then the raw values are not really comparable.

Two ideas:

1) Divide by the SD. So you then compute d = (mean - neutral point) / SD. Then the (large-sample) sampling variance can be estimated with:

1/n + d^2/(2*n)

2) Divide by the possible range (not the observed one!). So you then compute d = (mean - neutral point) / range. Then the sampling variance can be estimated with:

SD^2 / (n * range^2)

Best,
Wolfgang

Tommy van Steen

Wed, Jan 17, 2018 9:16 AM #

Hi Wolfgang,

Thanks for explaining this, it?s much clearer to me now.

Best wishes,
Tommy
===============
Dr Tommy van Steen
Research Associate in Psychology
University of Bath

On 16 Jan 2018, at 17:34, Viechtbauer Wolfgang (SP) <wolfgang.viechtbauer at maastrichtuniversity.nl> wrote:

Hi Tommy,

The sampling variance of a 'd' value (as in the present context, but this also applies to 'd' values in the more usual context where two independent groups are being compared) is only very indirectly a function of the SD of the measurements. In fact, the SD only affects the 'd' values itself, which is part of the computation of the sampling variance, but it does not enter the equation of the sampling variance in any other form. This might not be very intuitive, but that's just how it is.

Generally though, the effect of the 'd' value on the sampling variance is relatively minor compared to how strongly it is affected by the sample size. For example, if n had been 200 for the first study, then we would get a sampling variance of 0.0075, which is much smaller than that of the second study. Now that indeed makes a lot of intuitive sense.

Best,
Wolfgang

-----Original Message-----
From: Tommy van Steen [mailto:tommyvansteen at yahoo.com] 
Sent: Tuesday, 16 January, 2018 16:27
To: Viechtbauer Wolfgang (SP)
Cc: Michael Dewey; r-sig-meta-analysis at r-project.org
Subject: Re: [R-meta] Meta-analysis of single group attitude scores

Dear Michael and Wolfgang,

Thank you both very much for your helpful comments.
I have been testing the two methods suggested by Wolfgang on some fictional data to see where the differences lie, and I don?t fully understand the resulting sampling variances.

Fictional data of 2 studies:
Both are attitudes on a 7-point scale (neutral point = 4, range = 6) with n = 100.
Both studies have a mean score of 5. 
Study 1 SD = 1
Study 2 SD = 2.

With the first idea outlined by Wolfgang, the sampling variances are 0.015 (Study 1) and 0.01125 (Study 2).
With the second idea, the sampling variances are 0.000277 (Study 1) and 0.00111 (study 2).

So using the second idea, Study 2 results in a larger sampling variance, and therefore a less precise measurement, and inverse variance weighting results in a lower weight for this study. This seems to make sense as the SD for Study 2 is higher than for Study 1. 

However, using the first idea, the sampling variance of Study 2 is actually lower, which suggests that even though the SD of that study is higher, the study is more precise. 

Am I interpreting the results wrong? Or could it be that the first idea already incorporates the inverse variance calculation?

Thank you for your help!

Best wishes,
Tommy
===============
Dr Tommy van Steen
Research Associate in Psychology
University of Bath

On 26 Nov 2017, at 10:54, Viechtbauer Wolfgang (SP) <wolfgang.viechtbauer at maastrichtuniversity.nl> wrote:

I agree that ideally one would want to use the raw mean (or raw mean minus neutral point) as the outcome measure. However, if studies differ in terms of the number of answering possibilities, then the raw values are not really comparable.

Two ideas:

1) Divide by the SD. So you then compute d = (mean - neutral point) / SD. Then the (large-sample) sampling variance can be estimated with:

1/n + d^2/(2*n)

2) Divide by the possible range (not the observed one!). So you then compute d = (mean - neutral point) / range. Then the sampling variance can be estimated with:

SD^2 / (n * range^2)

Best,
Wolfgang