Combine the lemon zest, oil, rosemary and garlic on a large, shallow platter or a tray large enough to hold the swordfish steaks in a single layer. Add the swordfish, sprinkle with salt and pepper and turn to coat with the marinade. Cover and marinate in the refrigerator for 1 to 2 hours.
Brush the grill or broiler pan with a thin film of oil. Just before serving time, preheat the grill or broiler. Transfer the swordfish to grill or broiler pan, reserve the marinade left on the platter. If broiling, adjust the oven rack so that the tops of the fish are about 3 inches from the heat source. Grill or broil until lightly browned, 5 to 8 minutes, depending on the intensity of the heat. Using a spatula, carefully turn each steak. Spoon any marinade and pieces of lemon zest and rosemary onto the tops of the steaks. Broil until lightly browned and the center of the fish is opaque when tested with the tip of a knife, about 5 to 8 minutes. Serve garnished with capers and lemon wedges.