How can I upgrade multiple projects (same solution) that's targeting 3.5 to 4.0 in one go?

The Target Framework Migrator extension works great.

After you install it simply go to Tools -> Target Framework Migrator

I'm not sure about moving from 3.5 to 4.0, but in case you're upgrading from 2.0 to 4.0, you'll see that VS2010 actually added a default app.config file (in case you didn't have your own), which means that you'll also have to copy this file to each of the projects. Also, in case you have some *.designer files, those may change as well, so you'll have to update accordingly...

To summarize, here's the procedure:

1. Upgrade one project and diff the changes on all files (I find that using source control is best applied for this)
2. write a script/tool that enumerates over the entire solution and... a. Copies app.config to each of the project folders b. Changes the *.*proj files to match the diff for p.1 c. Changes the rest of the files if needed