Drain the green and black olives from the brine or oil that they're packed in and place them in a medium bowl. Add the orange zest, garlic, fennel seeds, thyme leaves, red pepper flakes, 1/2 teaspoon salt, and 1/2 teaspoon black pepper. Pour the olive oil over the mixture, add the thyme sprigs, and toss to combine.
Transfer the mixture, including the olive oil, to a medium (10-inch) saute pan. Heat over medium heat until the oil begins to sizzle. Lower the heat and saute for 4 to 5 minutes, stirring occasionally, until the olives and garlic are heated through and fragrant. Serve warm right from the pan or transfer to a serving dish. Offer a small dish for the pits.