Using QSortFilterProxyModel with a tree model

I have a QDirModel whose current directory is set. Then I have a QListView which is supposed to show the files in that directory. This works fine.

Now I want to limit the files shown, so it only shows png files (the filename ends with .png). The problem is that using a QSortFilterProxyModel and setting the filter regexp will try to match every parent of the files as well. According to the documentation:

For hierarchical models, the filter is applied recursively to all children. If a parent item doesn't match the filter, none of its children will be shown.

So, how do I get the QSortFilterProxyModel to only filter the files in the directory, and not the directories it resides in?