Put the potatoes in a large saucepan and cover with cold water by 1 inch. Season generously with salt and bring to a boil. Reduce the heat to medium low and simmer until the potatoes are very tender but not falling apart, about 15 minutes.
Meanwhile, melt 4 tablespoons butter in a large skillet over medium heat. Add the leeks, thyme and rosemary. Cook, stirring, until the leeks soften, 12 to 15 minutes; season with salt and pepper and set aside.
Drain the potatoes and return to the pot over low heat; cook until any excess water evaporates, about 2 minutes. Add the remaining 1½ sticks butter, reserving 1 tablespoon for topping, and mash with a potato masher until smooth and the butter has melted. Gradually mash in ½ cup milk and ½ teaspoon salt until creamy, adding up to ¼ cup more milk as needed. Season with salt and pepper and stir in the cooked leeks.
Transfer the potatoes to a bowl; top with the remaining butter before serving.
For plain mashed potatoes, follow the recipe for Mashed Potatoes with Leeks, but skip Step 2.