Position a rack in the center of the oven and preheat to 350 degrees F. Grease an 8-inch round cake pan with olive oil, and then line with parchment paper. Grease the parchment paper.
Add the milk, lime zest and lime juice to a small bowl and let stand for 5 minutes; the acid from the lime will curdle the milk and create "buttermilk."
Meanwhile, whisk the flour, baking powder and salt in a medium bowl. Beat 1 cup plus 2 tablespoons of the granulated sugar, the olive oil, eggs and vanilla in a large bowl with an electric hand mixer on medium-high speed until well combined and creamy, 2 to 3 minutes. Reduce the speed to low and beat in the flour mixture in three batches, alternating with the milk mixture. Use a rubber spatula to scrape down the sides of the bowl and give the batter a good stir to make sure everything is well-combined.
Transfer the batter to the prepared pan and arrange the mandarin oranges in a single layer on top in whatever pattern you like. Sprinkle the top evenly with the remaining 1 tablespoon granulated sugar. Bake until the cake is light golden brown and a toothpick inserted right off the center of the cake comes out clean, about 1 hour. (If you try testing with a toothpick in the very center of the cake, you might get a bit of moist batter on your toothpick, but the cake will continue to cook as it cools.) Transfer to a wire cooling rack and immediately run a small offset spatula or butter knife around the edges of the cake to release it from the pan. Let cool for 30 minutes in the pan, and then carefully invert directly onto the cooling rack. Don't worry if some of the crackly top comes off, any imperfections will be covered up with confectioners' sugar.
Once the cake is cooled completely, transfer to a serving plate and dust all over with confectioners' sugar.