Android Google Maps v2, MAP_TYPE_NONE and black squares

I've been struggeling with Google Maps v2 and custom tiles. It works fine, but we get black squares in Google Maps very often, and they are very distruptive.

I can reproduce the problem very simply by doing this:

mapFragment = new MapFragment() {
@Override
public void onActivityCreated(Bundle savedInstanceState) {
    super.onActivityCreated(savedInstanceState);
        mapFragment.getMap().setMapType(GoogleMap.MAP_TYPE_NONE);
    }
};

FragmentTransaction trans = getFragmentManager().beginTransaction();
trans.replace(R.id.map, mapFragment);
trans.commit();

I use this code and a standard activity with a framelayout. If I zoom far enough in the background on my Nexus4 it turns black. If I put tiles on top of that I will see black flashes as I zoom in.

I've seen a ton of workarounds on how to fix this, and none of them seem to work at all. It seems to only happen on certain zoom levels, but I (A) need those zoom levels, and (B) cannot seem to limit zooming (without building a custom pinch to zoom implementation anyways).

I've done some testing with custom tile providers. With MAP_TYPE_NONE, a tile provider which simply returns 1 blue pixel and my normal map tile provider I can visually see the map changing from black -> blue -> my map. Even though my custom blue tile provider returns immediately with a 1px blue tile.