Position a rack in the center of the oven and preheat to 350 degrees F. Generously butter a 13 by 9-inch baking pan.
Heat 4 tablespoons of the butter in a large skillet over medium heat. Add the mango, pineapple, brown sugar, and rum. Cook, stirring often, until the sugar is melted and bubbling. Cool until tepid.
Sift together the flour, baking powder, ginger, and salt and set aside. Beat the remaining 8 tablespoons of butter and the sugar in a large bowl with a hand-held electric mixer at high speed until light and fluffy, about 3 minutes. Beat in the eggs, 1 at a time, then the vanilla. On low speed, add the flour in 3 additions, alternating with 2 additions of the milk, and beat until smooth, scraping down the sides of the bowl often with a rubber spatula.
Arrange the fruit in a single layer in the pan, and pour the syrup over all. Sprinkle with the pecans. Spread the batter evenly over the fruit.
Bake until a toothpick inserted in the center of the cake comes out clean, about 35 minutes. Cool for 10 minutes on a wire rack. Invert and unmold onto a large serving platter. (If any pieces of fruit stay in the pan, just remove them and arrange them in their place on the cake.) Pour any juices in the pan over the cake. Serve warm or completely cooled, with the ice cream.
Recipe courtesy Art Smith, Back to the Table, Hyperion, 2001