Preheat the oven to 350 degrees F. Butter and flour a 9-inch springform pan.
Whisk together the all-purpose flour, almond flour, baking powder and salt in a bowl.
Beat the butter and sugar in a large bowl with an electric mixer until light and fluffy, about 2 minutes. Add the lemon zest and eggs one at a time, beating well after each addition. Beat in the olive oil and vanilla. Beat in half of the flour mixture, followed by the milk, and then the remaining flour, beating until just incorporated. Add about three-quarters of the grapes and gently fold them into the batter with a rubber spatula.
Transfer the batter to the prepared pan and smooth the top. Gently press the remaining grapes just barely into the surface of the cake. Bake until the cake is golden brown, pulling away from the sides and a wooden pick inserted in the center comes out clean, 40 to 45 minutes. Transfer to a rack to cool 10 minutes, then remove the springform rim and continue to cool completely.
Serve the cake with whipped cream and drizzled with Honey Grape Glaze.
Honey Grape Glaze:
Put the grapes, honey and lemon juice in a medium saucepan with 2 tablespoons water. Bring to a simmer and cook, stirring and smashing the grapes, until the grapes have broken down and the syrup is slightly thickened, 10 to 12 minutes. Strain the mixture into a bowl, pressing hard on the solids before discarding. Let cool before drizzling on the cake.