Balanced Two-Way ANOVA
The Model
In this section we will study the case of a balanced two-way model with interaction. Such a model can be written as
\[y_{ijk} = \mu +\alpha_i+\beta_j+\gamma_{ij}+\epsilon_{ijk}\]
where i=1,..,I, j=1,..,J and k=1,..,n.
Here the new term \(\gamma_{ij}\) is called the interaction term and is meant to encode a dependency between the factors A and B. If these are independent we would expect \(\gamma_{ij}=0\) for all i and j.
The assumptions are the same as in a one-way ANOVA, extended to this case.
Example (7.7.1)
The hearingaid data discussed in example (7.1.2) is a balanced two-way design, however by design one would not expect an interaction (dependency) between subjects and lists.
Example (7.7.2)
In an experiment to study gas mileage four different blends of gasoline are tested in each of three makes of automobiles. The cars are driven a fixed distance to determine the mpg (miles per gallon) The experiment is repeated three times for each blend-automobile combination. (Taken from Lyman Ott)
Note that the interest here is indifferent gasoline blends, automobile is a blocking variable, so this is a randomized block design.
head(gasoline)## MPG Gasoline Automobile
## 1 22.7 1 A
## 2 22.4 1 A
## 3 22.9 1 A
## 4 21.5 2 A
## 5 21.8 2 A
## 6 21.6 2 ANote that Gasoline is coded as 1, 2, 3 and 4, but these are simple codes without any meaning. So it is better to change them into factors.
gasoline$Gasoline = factor(gasoline$Gasoline)Now
table(gasoline$Gasoline, gasoline$Automobile)##
## A B C
## 1 3 3 3
## 2 3 3 3
## 3 3 3 3
## 4 3 3 3shows that each factor-level combination was tested three times, so here we have
I=4, J=3 and n=3
Interaction
Usually the first step in the analysis of a two-way design is to check for interaction. If we found \(\gamma_{ij}=0\) for all i and j, the model would simplify to an additive model of the form
\[y_{ijk} = \mu +\alpha_i+\beta_j+\epsilon_{ijk}\]
A graphical check is to draw the interaction plot: here we find the mean responses for each factor-level combination \(\bar{y}_{ij.}\), choose one of the factors to use on the x-axis, plot the means on the y-axis and connect the dots corresponding to the other factor.
Example (7.7.3)
For the gasoline data:
mns = tapply(gasoline$MPG, gasoline[, -1], mean)
df=data.frame(Gasoline=rep(1:4, 3),
Means=c(mns),
Automobile = rep(c("A", "B", "C"), each=4))
ggplot(data=df, aes(Gasoline, Means, color=Automobile)) +
geom_point() +
geom_line() 
Now in an additive model, going from level 1 of factor A to level 2, all responses get added the same (namely \(\beta_2-\beta_1\)), so the line segments should be parallel.
The problem with this approach is that each of the means is based on just three numbers, so the variance is quite high and the graphs can easily be far from parallel even if no interaction is present. It is therefore preferable to do a formal test.
Test for Interaction
It can be shown that estimable contrasts in \(\gamma_{ij}\)’s have the form
\[\gamma_{ij}-\gamma_{it}+\gamma_{sj}-\gamma_{st}\] We illustrate the ideas in the case of I=3, J=2. Then the cell means are
\[ \begin{pmatrix} \mu_{11} & \mu_{21} \\ \mu_{12} & \mu_{22} \\ \mu_{13} & \mu_{23} \\ \end{pmatrix} \] so if factor 1 is at level 1, the effect of factor 2 is \(\mu_{11}-\mu_{12}\). At level 2 it is \(\mu_{21}-\mu_{22}\) and at level 3 \(\mu_{31}-\mu_{32}\). If all of these effects were 0, there would be no interaction. Notice the similarity to the discussion above on the interaction plot.
So the hypothesis of no interaction can be written as
\[H_0:\mu_{11}-\mu_{12}=\mu_{21}-\mu_{22}=\mu_{31}-\mu_{32}\] Now
\[ \begin{aligned} &\mu_{11}-\mu_{12} = \mu+\alpha_1+\beta_1+\gamma_{11} - (\mu+\alpha_1+\beta_2+\gamma_{12})=\\ &\beta_1- \beta_2+\gamma_{11} -\gamma_{12} = \\ &\mu_{21}-\mu_{22} = \beta_1- \beta_2+\gamma_{21} -\gamma_{22}\\ \end{aligned} \]
and so \(\mu_{11}-\mu_{12}=\mu_{21}-\mu_{22}\) implies \(\beta_1- \beta_2+\gamma_{11} -\gamma_{12}=\beta_1- \beta_2+\gamma_{21} -\gamma_{22}\) or
\[\gamma_{11} -\gamma_{12}-\gamma_{21} +\gamma_{22}=0\] It can be shown that this is an estimable contrast. Similarly we find
\[\gamma_{21} -\gamma_{22}-\gamma_{31} +\gamma_{32}=0\]
and again this is an estimable contrast. Therefore the interaction hypothesis is testable.
To get a reduced model we reparametrize the model as follows: let
\[\gamma^*_{ij}=\mu_{ij}-\bar{\mu}_{.j}-\bar{\mu}_{i.}+\bar{\mu}_{..}\]
which is estimable.
Theorem (7.7.4)
Consider the model
\[y_{ijk} = \mu^* +\alpha^*_i+\beta^*_j+\gamma^*_{ij}+\epsilon_{ijk}\] where
\[ \begin{aligned} &\mu^*=\bar{y}_{..} \\ &\alpha^*_i = \bar{y}_{i.}-\bar{y}_{..}\\ &\beta^* = \bar{y}_{.j}-\bar{y}_{..} \end{aligned} \]
In this model the no interaction hypothesis \(H_0:\gamma^*_{ij}=0\) for all i,j is equivalent to
\[H_0:\mu_{11}-\mu_{12}=\mu_{21}-\mu_{22}=\mu_{31}-\mu_{32}\] which in turn is equivalent to
\[ H_0: \begin{pmatrix} \gamma_{11}-\gamma_{12}-\gamma_{21}+\gamma_{22} \\ \gamma_{21}-\gamma_{22}-\gamma_{31}+\gamma_{32} \\ \end{pmatrix} = \begin{pmatrix} 0 \\ 0 \end{pmatrix} \]
proof
We will show this in the case of I=3, J=2. Consider for example \(\gamma^*_{12}=\mu_{12}-\bar{\mu}_{.2}-\bar{\mu}_{1.}+\bar{\mu}_{..}\). Under \(H_0:\gamma^*_{12}=0\), and so
\[ \begin{aligned} &0 = \mu_{12}-\bar{\mu}_{.2}-\bar{\mu}_{1.}+\bar{\mu}_{..} =\\ &\text{ }\\ &\mu_{12} -\frac1I\sum_i \mu_{i2} - \frac1J\sum_j \mu_{1j} +\frac1{IJ}\sum_{i,j} \mu_{ij} = \\ &\mu+\alpha_1+\beta_2+\gamma_{12} -\\ &\frac1I\sum_i \left( \mu+\alpha_{i}+\beta_2 +\gamma_{ij} \right)-\\ &\frac1J\sum_j \left( \mu+\alpha_{1}+\beta_j +\gamma_{ij} \right)+\\ &\frac1{IJ}\sum_{ij} \left( \mu+\alpha_{i}+\beta_j +\gamma_{ij} \right) = \\ &\text{ }\\ &\mu+\alpha_1+\beta_2+\gamma_{12} -\\ &\frac1I \left( I\mu+\sum_i\alpha_{i}-I\beta_2 +\sum_i\gamma_{ij} \right)-\\ &\frac1J \left( J\mu+J\alpha_{1}+\sum_j\beta_j +\sum_j\gamma_{ij} \right)+\\ &\frac1{IJ} \left( IJ\mu+J\sum_{i}\alpha_{i}+I\sum_{j}\beta_j +\sum_{ij}\gamma_{ij} \right) = \\ &\frac1{IJ} \left( IJ\mu+IJ\alpha_1+IJ\beta_2+IJ\gamma_{12} \right. \\ &-IJ\mu-J\sum_i\alpha_{i}-IJ\beta_2 -J\sum_i\gamma_{ij} \\ &-IJ\mu-IJ\alpha_{1}-I\sum_j\beta_j -I\sum_j\gamma_{ij} +\\ &\left. IJ\mu+J\sum_{i}\alpha_{i}+I\sum_{j}\beta_j +\sum_{ij}\gamma_{ij} \right) =\\ &\text{ }\\ &\frac1{IJ} \left( IJ\alpha_1 - J\sum_i\alpha_{i}- IJ\alpha_{1} + J\sum_{i}\alpha_{i} \right. \\ &+IJ\beta_2 -IJ\beta_2 -I\sum_j\beta_j+ I\sum_{j}\beta_j\\ &\left. +IJ\gamma_{12}-J\sum_i\gamma_{i2} -I\sum_j\gamma_{1j} +\sum_{ij}\gamma_{ij} \right) = \\ &\text{ }\\ &\gamma_{12} - \frac1I \sum_i \gamma_{i2}- \frac1J \sum_j \gamma_{1j}+ \frac1{IJ}\sum_{ij} \gamma_{ij}=\\ &\gamma_{12} -\bar{\gamma}_{1.}-\bar{\gamma}_{.2}+\bar{\gamma}_{..} \end{aligned} \]
and so \(\gamma_{12} =\bar{\gamma}_{1.}+\bar{\gamma}_{.2}-\bar{\gamma}_{..}\)
In the same way we can show that
\[ \begin{aligned} &\gamma_{11} =\bar{\gamma}_{1.}+\bar{\gamma}_{.1}-\bar{\gamma}_{..}\\ &\gamma_{21} =\bar{\gamma}_{2.}+\bar{\gamma}_{.1}-\bar{\gamma}_{..} \\ &\gamma_{12} =\bar{\gamma}_{2.}+\bar{\gamma}_{.2}-\bar{\gamma}_{..} \\ \end{aligned} \]
and so
\[ \gamma_{11}-\gamma_{12}-\gamma_{21}+\gamma_{22} = \\ \bar{\gamma}_{1.}+\bar{\gamma}_{.1}-\bar{\gamma}_{..} \\ -\bar{\gamma}_{1.}-\bar{\gamma}_{.2}+\bar{\gamma}_{..}\\ -\bar{\gamma}_{2.}-\bar{\gamma}_{.1}+\bar{\gamma}_{..}\\ +\bar{\gamma}_{2.}+\bar{\gamma}_{.2}-\bar{\gamma}_{..} = 0 \]
The sum of squares for the test \(H_0:\gamma^*_{ij}=0\) is given by
\[\text{SS}(\gamma|\mu,\alpha,\beta)=\text{SS}(\mu,\alpha,\beta,\gamma)-\text{SS}(\mu,\alpha,\beta)\]
Using the usual side conditions \(\sum_i \hat{\alpha}_i=0\), \(\sum_j \hat{\beta}_j=0\), \(\sum_i \hat{\gamma}_{ij}=0\) and \(\sum_j \hat{\gamma}_{ij}=0\), a solution to the normal equation is given by
\[ \begin{aligned} &\hat{\mu} =\bar{y}_{...} \\ &\hat{\alpha}_i =\bar{y}_{i..}-\bar{y}_{...} \\ &\hat{\beta}_j =\bar{y}_{.j.}-\bar{y}_{...} \\ &\hat{\gamma}_{ij} =\bar{y}_{ij.}-\bar{y}_{i..}-\bar{y}_{.j.}+-\bar{y}_{...} \\ \end{aligned} \]
and so
\[ \begin{aligned} &\text{SS}(\mu,\alpha,\beta,\gamma) = \pmb{\hat{\beta}'X'y} = \\ &\hat{\mu}y_{...}+\sum_{i=1}^I \hat{\alpha}_iy_{i..}+\sum_{i=j}^J \hat{\beta}_jy_{.j.} +\sum_{i=1}^I\sum_{j=1}^J \hat{\gamma}_{ij}y_{ij.} = \\ &\bar{y}_{...}y_{...} + \sum_{i=1}^I \left(\bar{y}_{i..}-\bar{y}_{...}\right)y_{i..} +\\ &\sum_{j=1}^J \left(\bar{y}_{.j.}-\bar{y}_{...}\right)y_{.j.} + \sum_{i=1}^I\sum_{j=1}^J \left(\bar{y}_{ij.}-\bar{y}_{i..}-\bar{y}_{.j.}+-\bar{y}_{...}\right)y_{ij.}=\\ &\frac1n\sum_{i,j} y^2_{ij.} \end{aligned} \]
a similar calculation for the reduced model yields
\[\text{SS}(\mu,\alpha,\beta)=\frac{y^2_{...}}{nIJ}+\left(\sum_i\frac{y^2_{i..}}{nJ}-\frac{y^2_{...}}{nIJ}\right)+\left(\sum_j\frac{y^2_{.j.}}{nI}-\frac{y^2_{...}}{nIJ}\right)\]
and so
\[\text{SS}(\gamma|\mu,\alpha,\beta)=\sum_{ij} \frac{y^2_{ij.}}{n}-\sum_i\frac{y^2_{i..}}{nJ}-\sum_j\frac{y^2_{.j.}}{nI}+\frac{y^2_{...}}{nIJ}\]
As always
\[\text{SSE} = \pmb{y'y-\hat{\beta}X'y}= \sum_{ijk} y^2_{ijk}-\sum_{ij}\frac{y^2_{ij.}}{n}\]
There are IJ parameters involved in the null hypothesis. However, the side conditions impose I+J-1 restrictions. So the degrees of freedom for \(\text{SS}(\gamma|\mu,\alpha,\beta)\) are (I-1)(J-1).
Theorem (7.7.5)
To test \(H_0:\gamma^*_{ij}=0\) use
\[F=\frac{\text{SS}(\gamma|\mu,\alpha,\beta)/[(I-1)(J-1)]}{\text{SSE}/[IJ(n-1)]}\sim F((I-1)(J-1),IJ(n-1))\]
proof all of the above
Notice the degrees of freedom of SSE: IJ(n-1). This shows that the test is only possible if we have repeated measurements.
Example (7.7.6)
Let’s do the interaction test for the gasoline data:
y=gasoline$MPG
I=4;J=3;n=3
yij. = tapply(gasoline$MPG, gasoline[, -1], sum)
yi.. = tapply(gasoline$MPG, gasoline[, 2], sum)
y.j. = tapply(gasoline$MPG, gasoline[, 3], sum)
y...= sum(y)
sse=sum(y^2)-sum(yij.^2)/n
ssgamma=sum(yij.^2)/n-sum(yi..^2)/(n*J)-sum(y.j.^2)/(n*I)+y...^2/(n*I*J)
FTS=(ssgamma/((I-1)*(J-1)))/(sse/(I*J*(n-1)))
round(c(sse/c(1, I*J*(n-1)), ssgamma/c(1, (I-1)*(J-1)), FTS, 1-pf(FTS, (I-1)*(J-1), I*J*(n-1))), 3)## [1] 2.247 0.094 0.909 0.151 1.618 0.185or
gasoline$Gasoline=factor(gasoline$Gasoline)
fit=aov(MPG~.^2 , data=gasoline)
summary(fit)## Df Sum Sq Mean Sq F value Pr(>F)
## Gasoline 3 25.405 8.468 90.464 3.21e-13
## Automobile 2 0.527 0.263 2.813 0.0799
## Gasoline:Automobile 6 0.909 0.151 1.618 0.1854
## Residuals 24 2.247 0.094Tests for Main Effects
While it is possible to test for individual factors in the presence of interaction, it is usually better practice to first test for interaction and if such is found to be statistically significant, not to test for individual factors.
As before we consider the reparametrized model:
\[ \begin{aligned} &\alpha^*_i = \bar{\mu}_{i.}-\bar{\mu}_{..} = \\ &\frac1J\sum_j\mu_{ij} - \frac1{IJ}\sum_{ij} \mu_{ij} = \\ &\frac1J\sum_j \left( \mu_{ij} - \frac1{I}\sum_{i} \mu_{ij} \right)= \\ &\frac1J\sum_j \left( \mu_{ij} - \bar{\mu}_{.j} \right) \\ \end{aligned} \]
Now \(\mu_{ij} - \bar{\mu}_{.j}\) is the effect of the ith level of factor A at the jth level of factor B. So \(\alpha^*_i\) is the mean effect of the ith level of A. We therefore have the side condition \(\sum_i \alpha^*_i=0\). So now
\[H_0:\alpha_1=...=\alpha_I\]
is equivalent to
\[H_0:\alpha^*_1=...=\alpha^*_I=0\]
and this hypothesis is testable.
By their definition this is equal to
\[H_0: \bar{\mu}_{1.}-\bar{\mu}_{..}=...=\bar{\mu}_{I.}-\bar{\mu}_{..}\]
which is obviously equal to
\[H_0: \bar{\mu}_{1.}=...=\bar{\mu}_{I.}\]
Under the null hypothesis \(H_0:\alpha^*_1=...=\alpha^*_I=0\) the reduced model is
\[y_{ijk} = \mu^* +\beta^*_j+\gamma^*_{ij}+\epsilon_{ijk}\]
In the balanced case this model is orthogonal, so the estimators \(\hat{\mu}^*\), \(\hat{\beta}^*_j\) and \(\hat{\gamma}^*_{ij}\), and so we find
\[ \begin{aligned} &\text{SS}(\mu,\beta,\gamma) = \pmb{\hat{\beta}'X_1'y} = \\ &\hat{\mu}y_{...}+\sum_{i=j}^J \hat{\beta}_jy_{.j.} +\sum_{i=1}^I\sum_{j=1}^J \hat{\gamma}_{ij}y_{ij.} = \\ &\bar{y}_{...}y_{...} + \sum_{j=1}^J \left(\bar{y}_{.j.}-\bar{y}_{...}\right)y_{.j.} + \sum_{i=1}^I\sum_{j=1}^J \left(\bar{y}_{ij.}-\bar{y}_{i..}-\bar{y}_{.j.}+\bar{y}_{...}\right)y_{ij.}=\\ &y^2_{...}/nIJ + \sum_{j=1}^J \left(y^2_{.j.}/nI-y^2_{...}/nIJ\right) +\\ & \left(\sum_{ij}y^2_{ij.}/n-\sum_{i=1}y^2_{i..}/nJ-\sum_{j=1}y^2_{.j.}/nI+y^2_{...}/nIJ\right) \end{aligned} \]
and
\[\text{SS}(\alpha|\mu,\beta,\gamma)=\text{SS}(\mu,\alpha,\beta,\gamma)-\text{SS}(\mu,\beta, \gamma)=\sum_{i=1}y^2_{i..}/nJ-y^2_{...}/nIJ\]
Putting it all together we have the
ANOVA table for balanced two-way design
\[ \begin{array}{cccc} \text{Source} & \text{df} & \text{SS} \\ \hline \\ \text{Factor A} & I-1 & \sum_{i=1}y^2_{i..}/nJ-y^2_{...}/nIJ \\ \text{Factor B} & J-1 & \sum_{j=1}y^2_{.j.}/nI-y^2_{...}/nIJ \\ \text{Interaction} & (I-1)(J-1) & \sum_{ij} \frac{y^2_{ij.}}{n}-\sum_i\frac{y^2_{i..}}{nJ}-\sum_j\frac{y^2_{.j.}}{nI}+\frac{y^2_{...}}{nIJ} \\ \text{Error} & IJ(n-1) & \sum_{ijk}y^2_{ijk}-\sum_{ij}y^2_{ij.}/n \\ \text{Total} & nIJ-1 & \sum_{ijk}y^2_{ijk} - y^2_{...}/(nIJ) \\ \hline \\ \end{array} \]
Example (7.7.7)
Let’s do the test for the gasoline:
y=gasoline$MPG
I=4;J=3;n=3
yij. = tapply(gasoline$MPG, gasoline[, -1], sum)
yi.. = tapply(gasoline$MPG, gasoline[, 2], sum)
y.j. = tapply(gasoline$MPG, gasoline[, 3], sum)
y...= sum(y)
sse=sum(y^2)-sum(yij.^2)/n
ssalpha=sum(yi..^2)/(n*J)-y...^2/(n*I*J)
FTS=(ssalpha/(I-1))/(sse/(I*J*(n-1)))
round(c(sse/c(1, I*J*(n-1)), ssalpha/c(1, (I-1)), FTS, 1-pf(FTS, (I-1), I*J*(n-1))), 3)## [1] 2.247 0.094 25.405 8.468 90.464 0.000or again
gasoline$Gasoline=factor(gasoline$Gasoline)
fit=aov(MPG~.^2 , data=gasoline)
summary(fit)## Df Sum Sq Mean Sq F value Pr(>F)
## Gasoline 3 25.405 8.468 90.464 3.21e-13
## Automobile 2 0.527 0.263 2.813 0.0799
## Gasoline:Automobile 6 0.909 0.151 1.618 0.1854
## Residuals 24 2.247 0.094If we previously tested for interaction and failed to reject the null hypothesis of no interaction we can (and should) instead fit an additive model
\[y_{ijk} = \mu +\alpha_i+\beta_j\] and then we find
ANOVA table for balanced additive two-way design
\[ \begin{array}{ccc} \text{Source} & \text{df} & \text{SS} \\ \hline \\ \text{Factor A} & I-1 & \sum_{i=1} y^2_{i..}/nJ-y^2_{...}/nIJ \\ \text{Factor B} & J-1 & \sum_{i=1} y^2_{.j.}/nI-y^2_{...}/nIJ \\ \text{Error} & nIJ-I-J+1 & \sum_{ijk} y_{ijk}^2 - \sum_{i=1} y^2_{i..}/nJ -\\ & &\sum_{j=1} y^2_{.j.}/nI + y^2_{...}/nIJ \\ \text{Total} & nIJ-1 & \sum_{ijk}y^2_{ijk} - y^2_{...}/(nIJ) \\ \hline \\ \end{array} \]
Example (7.7.8)
Let’s do the test for the gasoline in the additive model:
ssalpha=sum(yi..^2)/(n*J) - y...^2/(n*I*J)
ssalpha/c(1, I-1)## [1] 25.405278 8.468426ssbeta=sum(y.j.^2)/(n*I) - y...^2/(n*I*J)
ssbeta/c(1, J-1)## [1] 0.5266667 0.2633333sse= sum(y^2)-sum(yi..^2)/(n*J)-sum(y.j.^2)/(n*I)+y...^2/(n*I*J)
FTS=(ssalpha/(I-1))/(sse/(n*I*J-I-J+1))
round(c(sse/c(1, n*I*J-I-J+1), ssalpha/c(1, (I-1)), FTS, 1-pf(FTS, (I-1), n*I*J-I-J+1)), 3)## [1] 3.156 0.105 25.405 8.468 80.510 0.000fit=aov(MPG~. , data=gasoline)
summary(fit)## Df Sum Sq Mean Sq F value Pr(>F)
## Gasoline 3 25.405 8.468 80.510 1.89e-14
## Automobile 2 0.527 0.263 2.504 0.0987
## Residuals 30 3.156 0.105