Prepare an outdoor barbecue grill with good charcoal briquets or wood chips stacked against one side of the grill and let them burn until ash white. Brush the grill lightly with olive oil, then brush the eggplant and peppers lightly with the oil. Place them skin-side down on the opposite side of the grill -- not over direct heat. Roast 4 to 5 minutes on each side, until well grilled and cooked. Remove to a warm platter and reserve while grilling the fish.
Brush the outside of each fish with olive oil. Fill the cavity of each fish with 2 sprigs of rosemary. Rub the garlic on the outside of the fish. Season with salt and pepper, and grill the fish in the same indirect manner 8 to 10 minutes on each side.
Meanwhile, remove the leaves from the remaining 4 sprigs of rosemary. Discard the stems, coarsely chop the leaves and reserve.
Remove the fish to second platter and cool 5 or 6 minutes. Because the fish has already been cooked, it can be filleted easily by running a sharp knife along the dorsal fin and lifting the fillet away from the bones using two forks to tease it free gently. Place the first fillet on a serving plate, turn over the fish, and repeat the filleting process for the second side.
Serve 1 fish fillet to each person, accompanied with the grilled vegetables. Drizzle each serving with some of the remaining olive oil and sprinkle with the chopped rosemary. *If you substitute fillets for whole fish, keep in mind that they can be tricky to grill and that they cook in half the time.