Grate the zest of one orange, remove the zest of the second orange with a vegetable peeler, and then juice both oranges. In a 9-by-13-inch glass or ceramic baking dish, toss together the olives, olive oil, 2 sprigs rosemary, the thyme, the zest removed with a peeler, orange juice, garlic, and 1 pinch crushed red pepper. Bake, tossing every 10 minutes, until the olives are slightly shriveled and the juices have reduced to a glaze, about 30 minutes.
While the olives are still hot, break the remaining sprig of rosemary over the top. Sprinkle with the finely grated orange zest, the fennel powder, and one more pinch crushed red pepper. Toss, and serve while still hot.