tapply on matrices of data and indices

I am calculating sums of matrix columns to each group, where the corresponding group values are contained in matrix columns as well. At the moment I am using a loop as follows:

index <- matrix(c("A","A","B","B","B","B","A","A"),4,2)
x <- matrix(1:8,4,2)

for (i in 1:2) {
  tapply(x[,i], index[,i], sum)
}

At the end of the day I need the following result:

   1  2
A  3  15
B  7  11

Is there a way to do this using matrix operations without a loop? On top, the real data is large (e.g. 500 x 10000), therefore it has to be fast.

Thanks in advance.