In linux, how can I test whether the output of a program is going to a live terminal or to a file?

isatty(int fd) will check whether the fd refers to a terminal or something else. It's part of unistd.h in the GNU C library.

Man page: http://linux.die.net/man/3/isatty

As an aside: if you want to read from a program using another program, but you want to fool isatty into thinking that your program is a human, there is a way to do that. You can use a pseudo-terminal (pty). This technique is used by expect, for example.

Can confirm that's what git relies on:

$ grep -ir "isatty" ./*
./builtin-commit.c:     if (isatty(0))
./builtin-config.c:         stdout_is_tty = isatty(1);
./builtin-pack-objects.c:   progress = isatty(2);
./builtin-prune-packed.c:   int opts = isatty(2) ? VERBOSE : 0;
./builtin-revert.c: if (isatty(0))
./builtin-shortlog.c:   if (!nongit && !rev.pending.nr && isatty(0))
./builtin-unpack-objects.c: quiet = !isatty(2);
./color.c:      stdout_is_tty = isatty(1);
./compat/winansi.c: if (!isatty(fileno(stream)))
./compat/winansi.c: if (!isatty(fileno(stream)))
./pack-redundant.c: if (!isatty(0)) {
./pager.c:  if (!isatty(1))
./pager.c:  if (isatty(2))
./remote-curl.c:    options.progress = !!isatty(2);
./transport.c:  args.no_progress = args.quiet || (!transport->progress && !isatty(1));
./transport-helper.c:   int no_progress = v < 0 || (!t->progress && !isatty(1));
./wt-status.c:   * will have checked isatty on stdout).

Run against the git source tree.

Note that fds 0=stdin, 1=stdout, 2=stderr by default, but these can of course be re-directed or closed (typically if you are a daemon you close your file descriptors and re-open the ones you want).