Preheat the oven to 375 degrees F. Toss the berries with 1 teaspoon vanilla and 3 tablespoons flour; set aside. Combine the brown sugar and 3 tablespoons water in a medium ovenproof skillet. Cook over medium heat, stirring, until thick and caramelized, about 7 minutes. Remove from the heat; stir in the lemon juice then the berry mixture and set aside.
Whisk the remaining 1 cup flour, the granulated sugar, baking powder and salt in a large bowl. Add the butter and work it into the flour mixture with your fingers until evenly incorporated and the mixture resembles fine meal. Whisk the egg yolks with the buttermilk and the remaining 1 teaspoon vanilla in a small bowl, then gradually whisk into the flour mixture until the batter is smooth. Spread the batter evenly in the skillet so the berries are almost covered.
Bake until the top is golden, about 35 minutes. Let stand at least 10 minutes before serving.