Preheat the oven to 350 degrees F. Butter a 9-by-5-inch glass loaf pan or an 8-inch square glass baking pan.
Arrange the brioche on a wire rack to dry out. Crack the eggs into a mixing bowl and whisk until blended. Add the sugar and whisk until smooth. (Whisk really well or you'll get scrambled eggs when placed in the oven.) Add the milk, vanilla and salt and whisk until completely blended. Pour the custard through a fine-mesh sieve into another bowl. Whisk in the citrus zest.
Place the brioche in the prepared pan, cutting to fit as needed. Pour the custard evenly over the bread, filling the dish to the top. You may not be able to add all of the custard at this point. Let the mixture sit for 15 minutes, so that the bread can absorb the custard.
Just before baking, top off the dish with more of the remaining custard if the previous addition has been completely absorbed. Cover the dish with foil and bake until it has just set (no color is necessary), about 1 hour 15 minutes for a loaf pan and 35 to 45 minutes for an 8-inch square pan. To test for doneness, uncover the dish, slip a knife into the center and push the bread aside. If the custard is still very liquid, cover the dish and return the pudding to the oven for another 10 minutes. If only a little liquid remains, the pudding is ready to come out of the oven. The custard will continue to cook after it is removed from the oven and it will set up as it cools.
Let cool for about 10 minutes before serving. You can serve the bread pudding by slicing it and removing each slice with an offset spatula, or by scooping it out with a serving spoon. Divide the bread pudding among plates. Add a scoop of ice cream. Top with Burnt Oranges and drizzle with Caramel Sauce. Top with a dollop of whipped cream.
Preheat the grill to medium. Dust the orange wedges with sugar. Grill until char marks appear, 2 to 3 minutes on each side.
Yield: 3 cups.
In a medium, heavy saucepan, combine the sugar, 1/2 cup water, the corn syrup and salt. (Use a good-size pan because the caramel will boil vigorously and the volume will increase dramatically as soon as the hot cream is added.) Bring to a boil over medium heat, stirring to dissolve the sugar. Let the mixture boil, without stirring, until amber colored. Watch the sugar mixture carefully as it cooks to avoid burning the sugar. From the time the mixture started to boil, it takes over 15 minutes to reach an amber color. Remove from the heat. The mixture will continue to cook off the heat and become darker, so make sure to have the cream close by.
Carefully and slowly add the warm cream to the caramel. The mixture will boil vigorously at first. Let the mixture simmer down and then whisk until smooth. Add the lemon juice and vanilla.
Cut the butter into 1-inch chunks and add to the caramel sauce, one piece at a time, whisking constantly after each addition. Whisk the caramel periodically as it continues to cool. Pour the caramel sauce into an airtight container such as a glass jar and store in the refrigerator. This will keep for up to a month.