For the streusel: Whisk the flour, almond flour, granulated sugar, brown sugar and salt together in a medium bowl. Use your fingertips to work the softened butter into the dry ingredients until sandy. Set aside
For the cake: Line a 9-by-13-inch baking pan with parchment paper and spray the paper with baking spray.
Mix the butter, oil, granulated sugar, brown sugar, lemon zest, baking powder, salt and vanilla on low speed with an electric mixer fitted with a paddle attachment until combined. Raise the speed to medium and mix until fluffy, about 2 minutes. Scrape the bowl and paddle. Add 1 egg and mix on medium speed for 30 seconds. Add the second egg and mix for another 30 seconds. Add a third of the flour to the bowl and blend on low. Add half the buttermilk, blending on low. Add another third of the flour, then the remaining buttermilk. Scrape the bowl. Add the remaining third flour and mix just until combined.
Spread the batter into the prepared pan. Scatter half of the blackberries on top and push them into the batter. Sprinkle the streusel over the batter. Scatter the remaining blackberries on top and push them into the crumble just a bit. Bake until a toothpick inserted in the center comes out with moist crumbs, 45-48 minutes. Cool the cake completely on a wire rack. Remove it from the pan before slicing and serving.
To toast almond flour, put the flour in a dry skillet and place over medium low heat. Use a spatula or wooden spoon to stir the flour continuously until it begins to smell nutty and turns light golden, 3 to 4 minutes.