Mix together the olive oil, oregano, salt, pepper and garlic in a small mason jar. Set aside.
Place the cherry tomatoes, bell peppers and feta on a sheet pan. Add 2 tablespoons of the olive oil mixture and toss. Add the lemon halves to the pan and roast until the vegetables and feta have browned, 15 to 17 minutes. Allow them to cool, about 20 minutes.
To assemble, toss the roasted tomatoes, bell pepper and feta together with the cucumbers, olives and red onion, then place on a platter.
Add the juice from the roasted lemons to the remaining olive oil mixture in the mason jar. Secure the lid and shake until combined. Drizzle the salad with half of the dressing. Arrange dill leaves over the top and serve with the remaining dressing on the side.