Make the filling: Preheat the oven to 375 degrees F. Butter a 9-by-13-inch baking dish. Toss the nectarines, plums, granulated sugar, all-purpose flour, lemon juice and salt in a large bowl.
Make the topping: Combine the all-purpose flour, whole-wheat flour, brown sugar, cinnamon and salt in a medium bowl, then stir in the hazelnuts. Work in the butter with your fingers until clumpy.
Spread the filling and any juices in the prepared dish. Pinch handfuls of the topping into clumps and scatter over the filling. Bake until the topping is browned and the filling is bubbling, 40 to 45 minutes. Let cool at least 15 minutes. Serve with ice cream or whipped cream.