Put sugar and water in a 5 quart heavy pot over medium-high heat and stir until all of the sugar dissolves. Scrape the vanilla seeds from the pod and add it to the pot along with the vanilla bean and rosemary. Slowly simmer for an additional 5 minutes at very low heat.

Meanwhile, in a piece of cheesecloth place the cinnamon, cardamom and cloves in the center. Pull the four ends together and tie kitchen string around it to make a small pouch. Make sure to leave enough string to tie around the outside handle on the pot for easy removal. Add the spice pouch, wine, brandy and cut fruit. Let it simmer on low for about 1 hour. Never let it boil. Serve warm.