ScrollViewer slow performance with DataGrid

I have the following scenario:

<ScrollViewer>
    <Grid>
         <!--many other controls-->
         <DataGrid />
    </Grid>
</ScrollViewer>

Now, when I bind DataGrid to large amount of data (around 10.000 rows) I am having very slow perfomance. In fact, i get OutOfmemory exception (and I have 8 GB memory)! I read somewhere that this is because ScrollViewer overrides DataGrid virtualisation (or something like that), but I don't know how to prevent that. If I remove the ScrollViewer, problem solved! The data loads in less than a second.

I want to keep the ScrollViewer (because of other controls) and have good performance. Is that possible? If not, is there any other solution-workaround?