Put the olives, fennel, orange zest, orange segments, lavender, if using, and olive oil into a large bowl and toss well. Pour the olive mixture onto a baking sheet and place in the oven for 20 minutes. Turn oven to broil and broil olives for 1 to 2 minutes. Remove and transfer to a decorative bowl. Serve immediately.