Collection was modified; enumeration may not execute error when removing a ListItem from a LIstBox

I have two ListBoxes, lstAvailableColors and lstSelectedColors. Between each listbox are two buttons, Add and Remove. When a color or colors is selected in lstAvailableColors and the Add button is clicked, I want to remove them from lstAvailableColors and display them in lstSelectedColors. Also, if colors are selected in lstSelectedColors and the Remove button is clicked, I want to remove the colors from lstSelectedColors and add them back to lstAvailableColors. When I do this, I get the following error when it removes the item:

Collection was modified; enumeration operation may not execute.

Here is the code for the Add Button and the Remove Button:

Add:

protected void btnAdd_Click(object sender, EventArgs e)
{
    foreach (ListItem item in lstAvailableColors.Items)
    {
        if (item.Selected)
        {
            lstSelectedColors.Items.Add(item);
            lstAvailableColors.Items.Remove(item);
        }
    }
}

Remove:

protected void btnRemove_Click(object sender, EventArgs e)
{
    foreach (ListItem item in lstSelectedColors.Items)
    {
        if (item.Selected)
        {
            lstAvailableColors.Items.Add(item);
            lstSelectedColors.Items.Remove(item);
        }
    }
}