20
Java doesn't support variable length in look behind.
In this case, it seems you can easily ignore it (assuming your entire input is one word):
([a-z])(?!.*\1)([a-z])(?!.*\2)(.)(\3)(.)(\5)
Both lookbehinds do not add anything: the first asserts at least two characters where you only had one, and the second checks the second character is different from the first, which was already covered by (?!.*\1).
Working example: http://regexr.com?2up96