How do I debug an MPI program?

As someone else said, TotalView is the standard for this. But it will cost you an arm and a leg.

The OpenMPI site has a great FAQ on MPI debugging. Item #6 in the FAQ describes how to attach GDB to MPI processes. Read the whole thing, there are some great tips.

If you find that you have far too many processes to keep track of, though, check out Stack Trace Analysis Tool (STAT). We use this at Livermore to collect stack traces from potentially hundreds of thousands of running processes and to represent them intelligently to users. It's not a full-featured debugger (a full-featured debugger would never scale to 208k cores), but it will tell you which groups of processes are doing the same thing. You can then step through a representative from each group in a standard debugger.

As others have mentioned, if you're only working with a handful of MPI processes you can try to use multiple gdb sessions, the redoubtable valgrind or roll your own printf / logging solution.

If you're using more processes than that, you really start needing a proper debugger. The OpenMPI FAQ recommends both Allinea DDT and TotalView.

I work on Allinea DDT. It's a full-featured, graphical source-code debugger so yes, you can:

• Debug or attach to (over 200k) MPI processes
• Step and pause them in groups or individually
• Add breakpoints, watches and tracepoints
• Catch memory errors and leaks

...and so on. If you've used Eclipse or Visual Studio then you'll be right at home.

We added some interesting features specifically for debugging parallel code (be it MPI, multi-threaded or CUDA):

• Scalar variables are automatically compared across all processes:
  _{(source: allinea.co}