Why INNER JOIN not equal (!=) hang forever

When I execute the following query:

SELECT * FROM `table1` 
 INNER JOIN table2 ON table2.number = table1.number

I get the result within 2 seconds. There are about 6 millions records in table2 and 1 million records in table1

table2.number and table1.number are indexed.

Now I want to get a list of numbers that not exist. Like this:

SELECT * FROM `table1` 
 INNER JOIN table2 ON table2.number != table1.number

It take forever and still hanging.. How to fix?