Heat the olive oil in a small saucepan over medium heat. Add the red pepper flakes thyme, garlic, lemon zest, orange zest, salt and pepper to taste and cook, stirring occasionally, until the garlic is pale golden, about 2 minutes. Stir in the olives and cook, stirring, until just warm, about 2 minutes. Turn off the heat. Stir in the lemon juice and orange juice and transfer to a decorative jar or serving bowl. Serve warm or at room temperature.