Make the cake: Preheat the oven to 375 degrees F. Butter an 8-inch square baking dish. Whisk the flour, baking powder and salt in a medium bowl. Beat the butter, granulated sugar and orange zest in a large bowl with a mixer on medium-high speed until fluffy, about 3 minutes. Beat in the egg until combined. Reduce the mixer speed to low and gradually beat in the milk (it may look curdled). Add the flour mixture and beat, scraping down the bowl occasionally, until smooth. (The batter will be thick.) Stir in the blueberries and peach and spread in the prepared dish.
Make the topping: Mix the granulated sugar, brown sugar, flour, nutmeg and salt in a small bowl. Work in the butter with your fingers until clumpy; scatter over the batter.
Bake until the top is golden and set and a toothpick inserted into the center comes out clean, about 1 hour. Let cool at least 30 minutes. Serve with ice cream.