Preheat the oven to 400 degrees F.

Cut the potatoes crosswise into 1/8-inch thick slices and place them in a large bowl. Heat the oil in a 10-inch nonstick ovenproof skillet over high heat. When the oil is hot, add the rosemary and olives. Cook for 10 seconds. Immediately remove the skillet from the heat and pour the mixture over the potatoes. Add the salt and pepper. Gently stir with a spoon to evenly coat the potatoes.

In the same 10-inch nonstick skillet, starting in the center of the skillet, arrange about 1/3 of the potatoes slices in a single layer so they form concentric circles that slightly overlap. Pick out about half the olives and sprinkle them and half of the lemon zest over the first layer of potatoes. Create a second layer of overlapping potato slices. Sprinkle the remaining olives and lemon zest over the second layer. Then create a third layer of potatoes. Finally, pour the oil from the bowl over the potatoes. Put the pan over high heat and cook until the oil sizzles and bubbles, about 1 minute. Place the pan in the oven and bake for 20 minutes, use a spatula to press down the top layer of potato slices. Continue to bake until the tip of a knife slides easily through the potatoes, about 20 minutes more. Remove the pan from the oven and pour off the excess oil. Allow the tart to sit at room temperature for at least 10 minutes.

Run a spatula along the outer edge of the tart to release any potatoes stuck to the side of the skillet. Place a dinner plate larger than the skillet on top of the skillet, with the serving side facing down. Invert the skillet and plate to transfer the tart to the plate. Cut the tart into wedges. Serve warm.