Pour 1/4 cup oil into a wide, shallow, nonreactive dish. Stir in the lemon and orange zests, 1/2 tablespoon of the thyme and all but 1 tablespoon of the chopped rosemary. Add the swordfish to the marinade and turn to coat the fish completely. Refrigerate 2 to 24 hours. Preheat gas grill to high or prepare a charcoal grill, letting the coals burn until white ash is visible. Remove the swordfish from the marinade, brushing off any solids. Skewer the fish pieces with the rosemary branches (about 4 pieces per branch). Season each kebab with salt and pepper. Place the skewers on the hot grill and cook, turning as needed, until cooked through, approximately 4 minutes total.
If you can't find long, sturdy rosemary branches, use bamboo skewers. Soak them in warm water for 15 minutes before using them to keep them from catching fire on the grill.
Recipe courtesy of Jimmy Bradley