For the caramel and fruit: Stir together the sugar and 1 1/2 tablespoons water in a 10-inch ovenproof skillet until the mixture looks like wet sand. Cook over medium-high heat, stirring occasionally, until melted and light amber, about 5 minutes. Stir in the salt if making a salted caramel. Arrange the plums, cut-side down, on top.
For the cake: Whisk the flour, almonds, baking powder, allspice, baking soda, cinnamon and salt together in a medium bowl.
Beat the sugar and butter together in a large bowl with an electric mixer on medium-high speed until light and fluffy, about 3 minutes. Add the eggs, 1 at a time, beating after each addition, then beat in the vanilla. Reduce the speed to low and add the flour mixture in 3 batches, alternating with the buttermilk, beating until just incorporated.
Pour the batter over the fruit and spread evenly. Bake until golden brown all over and a toothpick inserted into the center of the cake comes out clean, 35 to 50 minutes. Let stand 15 minutes and then invert onto a platter. Cool to room temperature before serving.
Copyright 2016 Television Food Network, G.P. All rights reserved.
When measuring flour, we spoon it into a dry measuring cup and level off the excess. (Scooping directly from the bag compacts the flour, resulting in dry baked goods.)