For the crust: Pulse the flour, sugar and kosher salt in a food processor to combine. Scatter in the butter pieces and pulse until the butter is the size of small peas. Sprinkle with the vinegar and 5 tablespoons of the water. Pulse until the dough just begins to come together. (If the dough is still floury and crumbly, add the last tablespoon of water and pulse again.) Dump the dough on the counter and press into a disc with your hands. Wrap in plastic wrap and chill for 30 minutes.
Preheat the oven to 375 degrees F. Line a baking sheet with parchment paper.
For the filling: Melt the butter in a 10-inch cast-iron skillet over medium heat. Once the butter is melted, add the preserves and stir until smooth, then simmer until syrupy, about 1 minute. Remove from the heat and stir in the sugar, brandy, lemon juice, ginger and nutmeg. Add the peaches and raspberries, then sprinkle with the cornstarch and toss to coat the fruit in the cornstarch.
Roll the dough on a floured counter to a circle about an inch or so wider than the top of the skillet. Trim the dough to clean up the edges, then lay the dough over the top of the skillet, tucking the edges inside. Cut 3 small slits in the top of the dough to let steam escape. Brush the dough with the beaten egg and sprinkle with sugar.
Place the skillet on the lined baking sheet to catch any drips. Bake until the filling is bubbling and the crust is crisp and deep golden brown, 45 to 50 minutes. Let cool at least 30 minutes before serving.
For the sauce: Put the caramel sauce in a small saucepan and heat over medium-low heat until it starts to simmer, then whisk in the buttermilk and flaky salt. Return to a simmer and simmer until smooth, about 1 minute. Let cool before serving.
To serve, scoop the pie into serving dishes and drizzle with the sauce. Sprinkle with additional flaky salt, if desired.