Put all the potatoes in a large saucepan and cover with cold water by 1 inch. Season generously with salt and bring to a boil. Reduce the heat to medium and simmer until very tender, about 15 minutes.
Drain the potatoes; return to the saucepan over low heat. Cook until any remaining water evaporates, about 2 minutes. Add the cayenne, 1/2 teaspoon salt and all but 2 tablespoons butter; mash with a potato masher until smooth. Gradually mash in the milk until creamy. Season with black pepper; transfer to a serving dish.
Melt the remaining 2 tablespoons butter in a skillet over medium heat until just beginning to brown, 3 to 5 minutes. Stir in the pecans; cook until lightly toasted, 1 to 2 minutes, then remove from the heat. Stir in the brown sugar; pour over the mashed potatoes.