How to avoid the 80/20 rule in software development

How do you estimate the amount of work? You say that "the pesky 20% of work remaining seems to take 4 times as long as the previous 80%", but how did you get to the estimate that "20%" of the work is remaining and that "80%" is done? Obviously the estimates are wrong - in reality only 20% of the work is done and 80% is remaining.

In software development it's very hard to give accurate estimates long time in advance. The only way is to split the work into small manageable pieces (maybe less than 10 hours each). You can estimate accurately only the immediate next steps.

Some practices which help in estimating progress can be found in Scrum. The scope of what work will be done during the next sprint (one month or less) is fixed at the beginning of the sprint and rough estimates are given to each work. Then after the sprint the team can reflect on how much progress was done, how much is still missing, how accurate the estimates were, and what is slowing down the team. In Scrum and other agile methods an important point is getting fast feedback of what is done and how far we are in the project. I recommend reading more about them. The video about Scrum that Ólafur Waage linked in his message gives a good and quick introduction.

When it comes to time estimating this is my experience:

1. If you can't positively say that a task will take less than 4 hours you can't estimate it accurately. Break it down in smaller pieces and repeat recursively.

2. Making such a time estimate is no picnic, it will take time, you will basically have to iron out the complete project in manageable chunks meaning that any changes to the requirement will result in a changed time-plan (surprising, isn't it?)

3. The biggest problem is that we can't possibly foresee all the details (maybe, let's say 20% perhaps? Leaving you the rest 80% unestimated...) - see SCRUM as others already have pointed out.

4. Management will seldom "accept" such a detailed time estimate as it will "take too long" to implement.

However, as management is interested in making profit, they are also interested in cutting corners. So you should identify the corners possible to cut and make sophisticated compromises based on the real life scenarios involved. Backed by management you can accomplish a lot of these last 20% by doing nothing (sad in a way I guess, but still true).

Because the last 80% of the work which represents the last 20% of the final product is really polishing and ironing out bugs and adapting to changed requirements, etc. It might be possible to have some limited first version, etc, etc, be creative.

One of the root causes of the 80/20 phenomenon is that the unexpected always occurs for any difficult - and sometimes even trivial - tasks. For example: the documentation that your software design processes mandate suddenly get a new template format, thanks to some overzealous process managers. Suddenly, it isn't just a simple matter of updating the docs for your new release - you now have to restructure each of them, and all of them take significantly more time.

One of the best recommendations I've heard for handling this type of phenomenon is to always build buffer subtask(s) into the project schedule - recommended by Richard Whitehead. Every major task gets a 20% time increase (or somewhere thereabouts) noted as a subtask for that task. The purpose of each is to provide some measurement for what happens when "things go wrong" on that task. The author admits (and I've also found to be true) that often management will try to remove those buffer tasks - your only recourse is to either stand your ground, or pull a maneuver like Joel advocates (as @Casey already mentioned). In practice, I've found that a good number of buffer subtasks usually do stick around, and have helped out a few times in tight schedules.

Saying "yes things have gone OK so far, but there is still quite a bit left to do..." might not be the best way to get your point across. After hearing that a manager or client might think "yes, there is quite a bit left to do but he did this part so fast surely the rest is minimal".

Instead make sure to identify the remaining work and schedule them. This way you can show where you should be during th remaining 20% of the project. If it's taking too long your schedule will show a project behind schedule and that should raise some sense of urgency.

Keep your tasks updated weekly (or how ever regularly you have status reports). Identify areas that are in danger of falling behind especially if other areas depend on them.

I think it is best if you can to under-promise and over-deliver.

If your estimates are right, then you account for that pesky 20%. You obviously did not, and that is why it is an issue.

Maybe you are trying to give them everything they want which is unrealistic. Maybe you did not fully account for Murphy's Law, or did not give enough time for testing, finding bugs, then testing again, etc.

Looks like you should be doing a little more risk management...