Add the potatoes and oil to a resealable bag and season with salt and pepper; shake the bag to thoroughly coat the potatoes. Transfer to a baking sheet and bake until easily pierced with a paring knife, 15 to 20 minutes; cool.
Cut the potatoes in half so the biggest surface area is exposed. Remove all of the flesh from 4 of the potato halves with a 1/4-teaspoon measuring spoon or melon baller. Put the potato flesh in a medium bowl and discard the skins. Use the spoon to scoop out most of the potato flesh from the remaining 12 potato halves, leaving about 1/4 inch of flesh on the skins so the "shells" firmly hold their shape. Transfer the potato flesh to the medium bowl.
Add the mayonnaise, relish, mustard, dill and cayenne to the bowl and mash with a potato masher until well mixed and fairly smooth. Season with salt and pepper.
Spoon the mixture back into the potato shells. Dust lightly with smoked paprika and garnish with dill. Serve at room temperature.