Place the potatoes, rosemary and garlic in a large saucepan and cover with cold water. Add a handful of salt and bring to a boil over medium-high heat. Boil until fork-tender, about 15 minutes. Drain well and remove the rosemary sprigs.
Working over the same saucepan, press the potatoes and garlic through a potato ricer. Add the warmed cream and olive oil. Place the pan back over low heat and stir everything together using a rubber spatula. Add the mozzarella, Parmesan and 1/2 teaspoon salt and continue to stir until the mixture is hot, the cheese is melted and the potatoes are almost stringy in texture--this will take 5 to 10 minutes. Serve hot.