Position a rack in the upper third of the oven and preheat to 450 degrees F. Peel the sweet potatoes and slice into 1/8-inch-thick rounds (use a mandoline if you have one). Peel half of the russet potatoes and slice into 1/8-inch-thick rounds. (Do not rinse-you'll need the starch for the dish to set.)
Coat the bottom of a 10-inch nonstick ovenproof skillet with 4 tablespoons melted butter. Starting in the center of the skillet, arrange a layer of russet potato slices in concentric circles, slightly overlapping them. Sprinkle with a pinch of salt and some of the rosemary leaves.
Set the skillet over medium heat. Working quickly, add the remaining sliced russets in 2 more layers, then the sliced sweet potatoes in 2 layers, sprinkling lightly with more salt and rosemary between layers. (Check to make sure the bottom is browning evenly; adjust the heat as needed.) Drizzle with 4 tablespoons melted butter. Peel and thinly slice the remaining russet potatoes and add them to the skillet in 2 or 3 more layers, lightly seasoning each layer with salt and rosemary. Drizzle with the remaining 4 tablespoons melted butter.
When the potatoes are golden brown on the bottom, transfer the skillet to the oven and bake until the potatoes soften slightly, about 30 minutes. Remove from the oven; center the bottom of a 9-inch cake pan on the surface of the potatoes and press firmly a few times to help the layers stick together. Remove the cake pan and return the skillet to the oven; bake until the potatoes are tender and the top is golden brown, about 20 more minutes.
Let the potatoes cool 5 minutes in the skillet, then run a small knife around the edge to loosen. Set an inverted serving plate on top of the skillet and carefully flip the potatoes onto the plate. Season with salt and pepper.