Specific Comparisons

Prerequisites
Difference Between Two Means (Independent Groups)

There are many occasions on which the comparisons among means are more complicated than simply comparing one mean with another. This section shows how to test these more complex comparisons. The methods in this section assume that the comparison among means was decided on before looking at the data. Therefore these comparisons are called planned comparisons. A different procedure is necessary for unplanned comparisons.

Let's begin with the made-up data from a hypothetical experiment shown in Table 1. Twelve subjects were selected from a population of high-self-esteem subjects (esteem = 1) and an additional 12 subjects were selected from a population of low-self-esteem subjects (esteem = 2). Subjects then performed on a task and (independent of how well they really did) half were told they succeeded (outcome = 1) and the other half were told they failed (outcome = 2). Therefore there were six subjects in each esteem/success combination and 24 subjects altogether.

After the task, subjects were asked to rate (on a 10-point scale) how much of their outcome (success or failure) they attributed to themselves as opposed to being due to the nature of the task.

Table 1. Distribution of colors.
outcome esteem attrib
1 1 7
1 1 8
1 1 7
1 1 8
1 1 9
1 1 5
1 2 6
1 2 5
1 2 7
1 2 4
1 2 5
1 2 6
2 1 4
2 1 6
2 1 5
2 1 4
2 1 7
2 1 3
2 2 9
2 2 8
2 2 9
2 2 8
2 2 7
2 2 6

 

The means of the four conditions are shown in Table 2.

Table 2. Mean ratings of attributions of
success or failure to oneself.
Success High Self Esteem
7.333
Low Self Esteem
5.500
Failure High Self Esteem
4.833
Low Self Esteem
7.833

There are several questions we can ask about the data. We begin by asking whether, on average, subjects who were told they succeeded differed significantly from subjects who were told they failed. The means for subjects in the success condition are 5.500 for the low-self-esteem subjects and 7.333 for the high-self-esteem subjects. Therefore, the mean of all subjects in the success condition is (5.500 + 7.333)/2 = 6.667. Similarly, the mean for all subjects in the failure condition is (7.833 + 4.833)/2 = 6.083. The question is, how do we do a significance test for this difference of 6.667 -6.083 = 0.084?

Table 3. Coefficients for comparing low and high self esteem.
Outcome Esteem Mean Coef-ficients Product
Success High Self Esteem
7.333
0.5 3.667
Low Self Esteem
5.500
0.5 2.750
Failure High Self Esteem
4.833
-0.5 -2.417
Low Self Esteem
7.833
-0.5 -3.917

 

The first step is to express this difference in terms of a linear combination of a set of coefficients and the means. This may sound complex, but it is really pretty easy. We can compute the mean of the success conditions by multiplying each success mean by 0.5 and then adding the result. This works because adding two numbers and then dividing by 2 is the same thing as multiplying each number by 0.5 and then adding the products. For example to get the mean of the numbers 1 and 3, we could add them together and divide by 2: (3+1)/2 = 2. Or, we could multiply each by 0.5 and add: (.5)(1) + (.5)(3) = .5 + 1.5 = 2.

We therefore can compute the difference between the "success" mean and the "failure" mean by multiplying each "success" mean by 0.5, each failure mean by -0.5 and adding the results. In Table 3, the coefficient column is the multiplier and the product column in the result of the multiplication. If we add up the four values in the product column we get

L = 2.750 + 3.667 - 3.917 - 2.417 = 0.083

This is the same value we got when we computed the difference between means previously (within rounding error). We call the value "L" for "linear combination." Now, the question is whether our value of L is significantly different from 0.

The formula for L is:

where ci is the ith coefficient and Mi is the ith mean. For this example,

L = (0.5)(5.50)+(0.5)(7.33)+(-0.5)(7.833)+(-0.5)(4.833) = 0.083.

The formula for testing L for ignorance is shown below

 

In this example,

 

MSE is the mean of the variances. The four variances are shown in Table 4. Their mean is 1.625. Therefore MSE = 1.625.

Table 4. Variances of attributions of
success or failure to oneself.
Success High Self Esteem
1.867
Low Self Esteem
1.100
Failure High Self Esteem
2.167
Low Self Esteem
1.367

 

The value of n is the number of subjects in each group. Here, n = 6.

Putting it all together,

We need to know the degrees for freedom in order to compute the probability value. The degrees of freedom is

df = N - k

where N is the total number of subjects (24) and k is the number of groups (4). Therefore, df = 20. Using the Online Calculator, we find that the two-tailed probability value is 0.874. Therefore, the difference between the "success" condition and the "failure" condition is not significant.

 

A more interesting question is whether the effect of outcome (success or failure) is different depending on the subject's self esteem. Perhaps, the effect of the outcome depends on the subject's self-esteem such that compared to high-self-esteem subjects, low-self-esteem subjects are less likely to attribute success to themselves and more likely to attribute failure to themselves.

To test this, we have to test a difference between differences. Specifically, is the difference between the two self-esteem groups different in the success condition than it is in the failure condition? The means shown in Table 5 show that this is the case. For the success condition, the low self-esteem subjects attributed the outcome to themselves less than did the high-self-esteem subjects. The opposite pattern occurred in the failure condition where the low-self-esteem subjects attributed the outcome to themselves more than did the high-self-esteem subjects. For the success condition, the difference between the high-self-esteem mean and the low-self-esteem mean is 7.333-5.500 = 1.833. For the failure condition, the difference is 4.833- 7.833=-3. The difference between differences is 1.833-(-3)=4.83.

The coefficients to test this difference between differences are shown in Table 5.

Table 5. Coefficients for testing differences between differences.
Outcome Esteem Mean Coef-ficients Product
Success High Self Esteem
7.333
1
7.333
Low Self Esteem
5.500
-1
-5.500
Failure High Self Esteem
4.833
-1
-4.833
Low Self Esteem
7.833
1
7.833

 

If it is hard to see where these coefficients came from, consider that our difference between differences were computed this way:

(7.33 - 5.5) - (4.83 - 7.83)

= 7.3 - 5.5 - 4.83 + 7.83

= (1)7.3 + (-1)5.5 + (-1)4.83 + (1)7.83

The values in parentheses are the coefficients.

To continue the calculations,


The two-tailed p value is 0.0002. Therefore, the difference between differences is highly significant.

Multiple Comparisons

The more comparisons you make, the greater your chance of a Type I error.