In a skillet, heat the olive oil over low heat. When the oil is warmed up, add the garlic cloves and lemon rinds. When the garlic cloves are sizzling in the oil, add the oregano.
Dry the olives with a paper towel to remove excess moisture and add to the skillet. Season with black pepper and toss to combine.
Serve warm or at room temperature.