Do I really need to encode '&' as '&'?

In HTML, a & marks the begin of a reference, either of a character reference or of an entity reference. From that point on, the parser expects either a # denoting a character reference, or an entity name denoting an entity reference, both followed by a ;. That’s the normal behavior.

But if the reference name or just the reference opening & is followed by a white space or other delimiters like ", ', <, >, &, the ending ; and even a reference to represent a plain, & can be omitted:

<p title="&amp;">foo &amp; bar</p>
<p title="&amp">foo &amp bar</p>
<p title="&">foo & bar</p>

Only in these cases can the ending ; or even the reference itself be omitted (at least in HTML 4). I think HTML 5 requires the ending ;.

But the specification recommends to always use a reference like the character reference & or the entity reference & to avoid confusion:

Authors should use "&" (ASCII decimal 38) instead of "&" to avoid confusion with the beginning of a character reference (entity reference open delimiter). Authors should also use "&" in attribute values since character references are allowed within CDATA attribute values.

A couple of years ago, we got a report that one of our web apps wasn't displaying correctly in Firefox. It turned out that the page contained a tag that looked like

<div style="..." ... style="...">

When faced with a repeated style attribute, Internet Explorer combines both of the styles, while Firefox only uses one of them, hence the different behavior. I changed the tag to

<div style="...; ..." ...>

and sure enough, it fixed the problem! The moral of the story is that browsers have more consistent handling of valid HTML than of invalid HTML. So, fix your damn markup already! (Or use HTML Tidy to fix it.)

If & is used in HTML then you should escape it.

If & is used in JavaScript strings, e.g., an alert('This & that'); or document.href, you don't need to use it.

If you're using document.write then you should use it, e.g. document.write(<p>this &amp; that</p>).