How to use AngularJS routes with Express (Node.js) when a new page is requested?

I'm using Express, which loads AngularJS from a static directory. Normally, I will request http://localhost/, in which Express serves me my index.html and all of the correct Angular files, etc. In my Angular app, I have these routes setup, which replace the content in an ng-view:

$routeProvider.when('/', {
    templateUrl: '/partials/main.html',
    controller: MainCtrl,
});

$routeProvider.when('/project/:projectId', {
    templateUrl: '/partials/project.html',
    controller: ProjectCtrl,
});

$locationProvider.html5Mode(true);

On my main page, I have a link to <a href="/project/{{project.id}}">, which will successfully load the template and direct me to http://localhost/project/3 or whatever ID I have specified. The problem is when I try to direct my browser to http://localhost/project/3 or refresh the page, the request is going to the Express/Node server, which returns Cannot GET /project/3.

How do I setup my Express routes to accommodate for this? I'm guessing it will require the use of $location in Angular (although I'd prefer to avoid the ugly ?searches and #hashes they use), but I'm clueless about how to go about setting up the Express routes to handle this.

Thanks.