Can't remove or stop AVPlayer

I have an AVPlayer that I load onto a new view whenever a link is clicked.

-(void)createAndConfigurePlayerWithURL:(NSURL *)movieURL sourceType:(MPMovieSourceType)sourceType {
self.playerItem = [AVPlayerItem playerItemWithURL:movieURL];
customControlOverlay = [[AFDetailViewController alloc] initWithNibName:@"AFMovieScrubControl" bundle:nil];
backgroundWindow = [[UIApplication sharedApplication] keyWindow];
[customControlOverlay.view setFrame:backgroundWindow.frame];
[backgroundWindow addSubview:customControlOverlay.view];

playerLayer = [AVPlayerLayer playerLayerWithPlayer:[AVPlayer playerWithPlayerItem:playerItem]];
[playerLayer.player play];
playerLayer.frame = customControlOverlay.view.frame;
[customControlOverlay.view.layer addSublayer:playerLayer]; 
}

The code above adds the AVPlayer to my app and works fine. I have a toggle in my customControlOverlay nib that should remove the view and stop the AVplayer from playing.

-(IBAction)toggleQuality:(id)sender {
if (qualityToggle.selectedSegmentIndex == 0) {
    NSLog(@"HD");
    [playerLayer.player pause];
    [self.view removeFromSuperview];

} else if (qualityToggle.selectedSegmentIndex == 1) {
    NSLog(@"SD");
}
}

The view is removed correctly but the player still plays in the background. After testing a bit the player wont respond to any code in the toggleQuality method but strings I have there as checks are getting logged.

Any thoughts on what I'm doing wrong?