Combine the wine, sugar, cinnamon, vanilla bean, cloves, orange zest, and lemon zest in a large (10 to 11-inch diameter) pot or Dutch oven, such as Le Creuset. Add 2 cups of water, bring to a boil, reduce the heat, and simmer uncovered for 15 minutes.
Add the figs, apricots, prunes, and peaches to the simmering liquid and bring to a boil. Lower the heat and simmer uncovered for 15 minutes, stirring occasionally, until all the fruit is tender and the liquid is reduced. (Not to worry if the figs are less tender; you don't want to overcook the rest of the fruit.) Remove and discard the cloves.
Off the heat, stir in the cherries and allow the fruit to cool to room temperature in the poaching liquid. Refrigerate for at least 4 hours. Reheat and serve warm or at room temperature with poaching liquid and a dollop of crème fraiche.