Place the potatoes and yams in a large saucepan, cover with cold water and season with salt. Bring to a boil and cook until fork-tender, 15 to 20 minutes.
Drain the potatoes and yams and place in a stand mixer fitter with a whisk attachment. Add the goat cheese, butter and garlic and mix on low just until smooth, being careful not to over-mix or the potatoes will become gummy. Season with salt and pepper. If necessary, add the milk to loosen.