Alex Rivera | Logout

Random data in Unit Tests?

Asked 2008-08-28T14:45:26.807
191

I have a coworker who writes unit tests for objects which fill their fields with random data. His reason is that it gives a wider range of testing, since it will test a lot of different values, whereas a normal test only uses a single static value.

I've given him a number of different reasons against this, the main ones being:

  • random values means the test isn't truly repeatable (which also means that if the test can randomly fail, it can do so on the build server and break the build)
  • if it's a random value and the test fails, we need to a) fix the object and b) force ourselves to test for that value every time, so we know it works, but since it's random we don't know what the value was

Another coworker added:

  • If I am testing an exception, random values will not ensure that the test ends up in the expected state
  • random data is used for flushing out a system and load testing, not for unit tests

Can anyone else add additional reasons I can give him to get him to stop doing this?

(Or alternately, is this an acceptable method of writing unit tests, and I and my other coworker are wrong?)

unit-testing tdd mocking
Edit
Report

2 Answers

31

There is a half-way house here which has some use, which is to seed your PRNG with a constant. That allows you to generate 'random' data which is repeatable.

Personally I do think there are places where (constant) random data is useful in testing - after you think you've done all your carefully-thought-out corners, using stimuli from a PRNG can sometimes find other things.

answered 2008-08-28T15:09:04.613
0

I can envisage three solutions to the test data problem:

  • Test with fixed data
  • Test with random data
  • Generate random data once, then use it as your fixed data

I would recommend doing all of the above. That is, write repeatable unit tests with both some edge cases worked out using your brain, and some randomised data which you generate only once. Then write a set of randomised tests that you run as well.

The randomised tests should never be expected to catch something your repeatable tests miss. You should aim to cover everything with repeatable tests, and consider the randomised tests a bonus. If they find something, it should be something that you couldn't have reasonably predicted; a real oddball.

answered 2011-10-26T16:32:11.473

Your Answer