Brush both sides of all bread slices with 1 tablespoon of the oil. Toast on a grill pan over medium-high heat for 1 minute. Turn each slice 90 degrees on the same side and cook for another minute so your bread chars with beautiful crisscross grill marks. Flip over and repeat. Rub each slice of hot bread with the cut garlic clove as it comes off the grill.
Meanwhile, add the remaining 3 tablespoons oil to a medium skillet over medium heat. Add the red grapes and saute until warm and softened, about 3 minutes. Add the green grapes and toss just to warm. Remove to a bowl.
Add the sage leaves to the hot oil and fry until crisp, 15 to 20 seconds. Remove to a paper towel-lined plate.
Spread 2 tablespoons of the chevre over one side of each piece of toast, covering completely. Top with a mound of warm grapes on top. Sprinkle a bit of sea salt and a few grinds of pepper over the grapes. Top with a sage leaf. Zest the lemon generously over the top of each slice.