For the rosemary syrup: Combine the sugar, rosemary sprigs and 1 cup water in a saucepan and bring to a boil, stirring until the sugar dissolves. Boil for 5 minutes. Let it cool, then strain out the rosemary. Stir in the lemon juice.
For the salad: To assemble, divide the yogurt among 8 glasses, top with a mix of the oranges, blood oranges, mandarin oranges and grapefruit and drizzle the rosemary syrup over the top.