Cool the olives, toss them with the remaining oil, and serve.