Lock only on an Id

I have a method which needs to run exclusivley run a block of code, but I want to add this restriction only if it is really required. Depending on an Id value (an Int32) I would be loading/modifying distinct objects, so it doesn't make sense to lock access for all threads. Here's a first attempt of doing this -

private static readonly ConcurrentDictionary<int, Object> LockObjects = new ConcurrentDictionary<int, Object>();
void Method(int Id)
{
    lock(LockObjects.GetOrAdd(Id,new Object())
    {
       //Do the long running task here - db fetches, changes etc
       Object Ref;
       LockObjects.TryRemove(Id,out Ref);
    }

}

I have my doubts if this would work - the TryRemove can fail (which will cause the ConcurrentDictionary to keep getting bigger).

A more obvious bug is that the TryRemove successfully removes the Object but if there are other threads (for the same Id) which are waiting (locked out) on this object, and then a new thread with the same Id comes in and adds a new Object and starts processing, since there is no one else waiting for the Object it just added.

Should I be using TPL or some sort of ConcurrentQueue to queue up my tasks instead ? What's the simplest solution ?