Preheat a grill to high. Combine the oil, garlic and rosemary in a shallow dish. Add the lamb, turn to coat and season with salt and pepper. Mix the parsley and olives in a bowl.
Squeeze the juice of 1 lemon into a bowl. Put the peppers skin-side down on the grill; cook until blistered, brushing with the lemon juice and seasoning with salt, about 4 minutes per side. Transfer to a platter.
Meanwhile, grill the remaining 2 lemons cut-side down until charred, about 5 minutes; set aside. Grill the scallions, turning, until tender, about 4 minutes. Transfer to a cutting board and coarsely chop, then add to the olive mixture and season with salt and pepper.
Grill the lamb until marked, about 4 minutes per side for medium rare. (A thermometer inserted into the thickest part should register 140 degrees F.) Transfer to a cutting board and let rest 5 minutes. Slice against the grain, add to the platter and top with the scallion-olive mixture. Squeeze the grilled lemons on top.