Automated x86 instruction obfuscation

I'm working on an x86 asm obfuscator that takes Intel-syntax code as a string and outputs an equivilent set of opcodes that are obfuscated.

Here's an example:

mov eax, 0x5523
or eax, [ebx]
push eax
call someAPI

Becomes something like:

mov eax, 0xFFFFFFFF ; mov eax, 0x5523
and eax, 0x5523     ;
push [ebx]          ; xor eax, [ebx]
or [esp], eax       ;
pop eax             ;
push 12345h         ; push eax
mov [esp], eax      ;
call getEIP         ; call someAPI
getEIP:             ;
add [esp], 9        ;
jmp someAPI         ;

This is just an example, I've not checked that this doesn't screw up flags (it probably does).

Right now I have an XML document that lists instruction templates (e.g. push e*x) and a list of replacement instructions that can be used.

What I'm looking for is a way to automatically generate opcode sequences that produce the same result as an input. I don't mind doing an educated bruteforce, but I'm not sure how I'd approach this.

Edit Report

What you need is an algebraic descripton of what the opcodes do, and a set of algebraic laws that allow you to determine equivalent operations.

Then for each instruction, you look up its algebraic description (for the sake of an example, an

 XOR  eax,mem[ecx]

whose algebraic equivalent is

 eax exclusive_or mem[ecx]

enumerate algebraic equivalences using those algebra equivalents, such as:

 a exclusive_or b ==> (a and not b) or (b and not a)

to generate equivalent algebraic statement for your XOR instruction

 eax exclusive_or mem[ecx] ==> (eax and not mem[ecx]) or (mem[ecx] and not eax)

You may apply more algebraic laws to this, for instance de morgans' theorem:

 a or b ==> not (not a and not b)

to get

(not (not (eax and not mem[ecx])) and (not (mem[ecx] and not eax)))

At this point you have a specification of an algebraic computation that will do the same thing as the original. There's your brute force.

Now you have to "compile" this to machine instructions by matching what instructions will do with what this says. Like any compiler, you likely want to optimize the generated code (no point in fetching mem[ecx] twice). (All of this hard... its a code generator!) The resulting code sequence would be something like:

mov ebx, mem[ecx]
mov edx, ebx
not edx
and edx, eax
not eax
and eax, ebx
not eax
or eax, edx

This is a lot of machinery to build manually.

Another way to do this is to take advantage of a program transformation system that allows you to apply source-to-source transformations to code. Then you can encode "equivalences" as rewrites directly on the code.

One of these tools is our DMS Software R

answered 2011-10-30T20:19:48.963

Automated x86 instruction obfuscation

1 Answer

Your Answer