Pour the milk and heavy cream into a heavy bottomed saucepot and scrape the seeds from the vanilla bean into the pot (or add vanilla extract). (If using a vanilla bean, put the vanilla pod in the pot as well.) Add half the sugar to the pot and bring to a simmer (just below a boil). In a small bowl whisk together the egg yolks and the rest of the sugar and temper it by adding a small amount of the heated milk mixture to the bowl while whisking constantly (known as a liaison). Now pour the liaison (egg mixture) into the milk pot, stirring constantly. You are only heating it. Do not boil. Do not cook. The idea of a liaison is to incorporate the eggs and avoid making them into scrambled eggs! You want the mixture to begin to thicken so it will coat the back of a spoon. Remove from heat and allow to cool, then remove vanilla pod and discard, and refrigerate until chilled.
For the pineapple and cherries: In a skillet, bring 1 cup of water and the brandy to a boil, then reduce heat to a simmer. Add the pineapple to the simmering brandy mixture. (Add enough hot water to cover, if necessary.) Poach for about 10 minutes and using tongs, remove to a platter. In the same poaching liquid cook the cherries for about 10 minutes. Remove from heat and let cool to room temperature while you are preparing the batter.
For the cake: Preheat oven to 350 degrees F. Oil the bottom of a 9 by 2-inch deep round pan and lay parchment paper over the bottom. Then brush a thin layer of canola oil on top of the parchment paper. Sift together cake flour and just 1 tablespoon of the sugar several times to aerate. In a separate heatproof bowl combine the eggs and and the other 3 tablespoon sugar. Nest the bowl with the sugar and eggs in another larger bowl filled with about 2-inches hot water. Whisk together the eggs and sugar, being very careful NOT to splash any water into it. The idea here is to raise the temperature of the egg and 3/4 cup sugar mixture to between 100 and 120 degrees F. Use a candy thermometer to test the temperature. Once the mixture is warmed, remove the bowl from the water bowl and use an electric beater to beat it until it is about triple in volume and is the consistency of soft whipped cream. Sift the 1/3 of the flour and sugar mixture into the bowl of beaten eggs and sugar, and fold together. Repeat for each remaining 1/3 of the flour mixture, folding together after each addition. Then, gently fold in the melted butter and vanilla.
Arrange the pineapple circles with cherries placed in the center in the bottom of the cake pan. Start the pattern at the middle and work your way out. Pour batter over layer of pineapple, spreading evenly, but don't fill pan any more than within 1/2-inch of the top edge of the pan. Bang pan on countertop - 1 or 2 times only - to make sure the batter sinks down between the fruit, but don't overdo it or you will undo all your hard work in aerating the batter! Bake in oven until the surface springs back when touched and a toothpick inserted in the cake comes out clean, about 20 to 25 minutes. Let cake cool in pan 10 minutes to allow set up, then using oven mitts, press a serving plate firmly on the top edges of the pan, and invert the cake onto the plate. Replace any pineapple slices or cherries that become dislodged. This cake can be served warm or at room temperature, topped with creme anglaise.