Place the potatoes on a baking sheet and rub with the olive oil. Bake until soft enough to yield to gentle pressure, about 1 hour. Let cool for 10 minutes.
Heat the milk and butter in a small pot over low heat. Slice the potatoes in half lengthwise and scoop out the flesh into a bowl, leaving just a little bit behind to give the skins some structure. Place the scooped-out potato skins back on the baking sheet and set aside.
Mash the potatoes with a masher until there are no large clumps. Add the heated milk and butter as well as the Gruyere, sour cream, half of the chives, and salt and pepper to taste. Mix with a wooden spoon until well blended and smooth. Scoop the mashed potatoes back into the potato skins and top each with some of the Cheddar.
Return the potatoes to the oven; bake for 20 minutes. Transfer to a serving platter, garnish with the remaining chives, and serve.