Preheat the oven to 375˚. Slice off about 1/4 inch from the top of the garlic head. Place cut-side up on a piece of foil. Top with the olive oil and a pinch of salt; snap a rosemary sprig in half and place on top of the garlic. Wrap in the foil. Roast until the garlic is soft and golden, about 1 hour. Carefully open the foil and let the garlic cool slightly, then squeeze the cloves out of the skin; discard the rosemary.
Meanwhile, butter a 2-quart baking dish. Smash the remaining 2 garlic cloves; add to a large pot along with the potatoes and the remaining 3 rosemary sprigs. Cover with cold water by 1 inch and season generously with salt. Bring to a boil, then reduce the heat to a simmer and cook until the potatoes are tender, 15 to 18 minutes. Drain well; discard the rosemary sprigs.
Increase the oven temperature to 450˚. Press the potatoes, boiled garlic and roasted garlic through a ricer or food mill back into the pot. (Alternatively, return the potatoes and boiled garlic to the pot along with the roasted garlic and mash with a potato masher.)
Fold in the butter until melted. Gradually stir in 3/4 cup milk until smooth. If the potatoes still seem thick, gradually add the remaining 1/4 cup milk as needed. Season with salt.
Transfer the mashed potatoes to the prepared baking dish. Bake until browned on top, 30 to 40 minutes.