Add cut potatoes into a large pot and cover by about 1-inch with cold water. Place over medium heat and simmer (do not boil) potatoes for 15 minutes or until fork tender.
Strain potatoes from pot, and set aside. In same pot add oil, garlic and rosemary. Lightly saute over medium heat. When slightly brown add potatoes back to the pot, and mix with garlic and rosemary.
Remove from heat, add milk, salt and pepper, and mash with potato masher. Garnish with chives.