Moving ListViewItems Up & Down

I have a ListView (WinForms) in which i want to move items up and down with the click of a button. The items to be moved are the ones who are checked. So if item 2, 6 and 9 are selected, they will become 1, 5 and 8 when I press the button for movement upwards and the items that were on those places are moved down a step.

I feel that I have made this unnecessarily complicated, as you can see below. The second SubItem of each ListViewItem is a number which represents its place in the list (starts on 1).

I blame lack of sleep and coffee for the following code, but if you could figure out a simpler way to complete this task I would be thankful.

private void sourceMoveUpButton_Click(object sender, EventArgs e)
    {
        List<Int32> affectedNumbers = new List<Int32>();
        bool foundNonChecked = false;

        List<KeyValuePair<int, ListViewItem>> newList = new List<KeyValuePair<int, ListViewItem>>();

        foreach (ListViewItem item in this.sourceListView.CheckedItems)
        {
            int newNum = int.Parse(item.SubItems[1].Text) - 1;

            if (newNum >= 1)
            {
                foreach (ListViewItem testItem in this.sourceListView.Items)
                {
                    if (int.Parse(testItem.SubItems[1].Text) == newNum && !testItem.Checked)
                    {
                        foundNonChecked = true;
                    }
                }

                if (foundNonChecked)
                {
                    item.SubItems[1].Text = newNum.ToString();
                    affectedNumbers.Add(newNum);
                }
            }
        }

        foreach (ListViewItem item in this.sourceListView.Items)
        {
            int num = int.Parse(item.SubItems[1].Text);

            if (affectedNumbers.Contains(num) && !item.Checked)
            {
                item.SubItems[1].Text = (num + affectedNumbers.Count).ToString();